The duration of ground motion can significantly influence the cumulative damage and failure levels of structures, but its impact on structural response is inadequately considered in current seismic design. To explore the effect of ground motion duration on structural damage, this study employs a spectral matching method combined with wavelet transform to select 115 ground motion records of varying durations from the Japan strong-motion database (K-NET). Earthquake damage spectra under two different restoring force models were calculated, and a comprehensive analysis was conducted on the effects of ground motion duration, yield strength reduction factor, structural natural period and other factors on the structural damage spectrum. Additionally, 585 natural earthquake records of varying durations and different site conditions were selected from the Pacific Earthquake Engineering Research Center (PEER) database. Taking into account the combined influences of ground motion duration, yield strength reduction factor, structural restoring force model, and site conditions, a damage spectrum prediction model was developed using a differential evolution algorithm. Compared with existing prediction models, the proposed model reduces relative errors by more than 40%, which offers improved accuracy for predicting damage spectra that consider the effect of duration. The results of this study provide valuable insights for structural seismic design and damage assessment when considering the impact of ground motion duration.