Under mechanical loading, metallic materials can fail in various ways, including yielding, fracture, buckling, wear, fatigue, and so on, with fracture being particularly destructive. Ductile fracture, characterized by dimples on the fracture surface, is commonly observed in pure metals and alloys. From the microscopic point of view, the ductile fracture of metals and alloys is closely associated with the nucleation, propagation, and coalescence of voids, influenced by factors such as stress state, void size, void volume fraction, void shape, and temperature. Micromechanics-based models developed for ductile damage considering the void evolution, such as the Gurson model and its extensions, usually presume spherical voids, but creating models that consider realistic void shapes and their evolution presents significant challenges. Moreover, conducting mechanical analyses of ductile failure across specimen and component scales requires addressing cross-scale issues. This study first constructed representative volume element models incorporating isolated voids of different initial shapes. Finite element simulations were carried out based on the representative volume elements by adopting a J2 plasticity model for the matrix, systematically analyzing how initial void shape affects stress-strain responses and ductile damage under triaxial tensile and shear loading conditions. A neural network-based surrogate model was trained with the numerical data generated by the simulations to approximate stress-strain responses and damage evolution. This model effectively predicted how initial void shape influences ductile damage. Subsequently, a user-defined material subroutine was developed and integrated into a commercial finite element code to simulate the impact of initial void shapes on the ductile failure process in notched specimens. Results indicated that a reduced aspect ratio for the voids decreased the damage rate, leading to delayed softening at the specimen level. This work demonstrates the potential of using surrogate models to predict ductile damage involving complex microstructural features.