To enhance the protection of masonry pagodas and improve the damping performance of suspension pendulum damper (SPD), a novel SMA composite pendulum damping system is proposed, integrating shape memory alloy (SMA) into SPD. This system enhances the inertia force generated by the pendulum within SPD by leveraging the super elasticity and high damping characteristics of SMA, thereby improving the overall energy dissipation capability of the damping system. Initially, the study outlines the process of obtaining the equivalent restoring force of SMA through stochastic equivalent linearization. Subsequently, it establishes the SDOF computational model of the SMA composite pendulum damping system by drawing parallels with the SDOF computational model of SPD. The paper elucidates the structure and operational principles of the damping system, followed by an exploration of the effects of two control parameters on its performance. Further, the study applies both SPD and the new composite pendulum damping system to a masonry pagoda. The structural vibration response is analyzed under three distinct seismic excitations following the implementation of the damping system using ABAQUS software. The findings reveal that the SMA composite pendulum damping system outperforms SPD in vibration control under various seismic excitations, demonstrating its superior applicability.