This study formulates an elastoplastic-damage constitutive model derived from a potential energy function. Drawing upon the potential-based approach of Watanabe et al. (2008), we derived the constitutive equations for elastic, plastic, and continuum damage and obtained were derived and the evolution equations for internal variables by maximizing dissipation. Furthermore, an implicit stress update algorithm was developed to facilitate implementation within finite element analysis. We applied this constitutive modeling approach to both isotropic constitutive model and anisotropic constitutive model characterized by a crystal structure. Finally, finite element analyses of periodic microstructures were performed using the constitutive models, showcasing its ability to capture ductile fracture originating from microscopic heterogeneity.