Structural seismic response monitoring plays a crucial role in the earthquake damage assessment and evaluation of seismic resilience for urban building clusters. Addressing the issues of high cost and low prevalence of existing seismic sensors, this paper proposes a structural seismic response monitoring system based on surveillance cameras. The system develops a hierarchical line segment descriptor matching algorithm for building structures and introduces a time-history data extraction technique for structural seismic displacement responses based on line matching. This effectively resolves the challenge of targetless structural surfaces and indistinct natural feature points in real earthquake scenarios. Thereby, enabling real-time monitoring of inter-story drift at a sub-pixel level. The system has been successfully demonstrated in the world's first practical earthquake visual monitoring application at a middle school in Sichuan Province. The results show that even under complex and varying lighting conditions at night, the monitoring system maintains high accuracy, achieving sub-pixel-level inter-story drift monitoring with peak inter-story drift errors within 35% and structural natural frequency errors within 5%. Furthermore, the monitoring system adopts a lightweight design, with resource utilization rates of both the central processing unit (CPU) and graphics processing unit (GPU) below 15% when processing a single surveillance video, meeting the requirements for real-time multi-node processing and ensuring efficient system operation. Compared to traditional accelerometer solutions, the visual monitoring system eliminates the need for additional dedicated sensors, leveraging existing security surveillance equipment to construct a building cluster monitoring network. This approach not only enables dual-purpose use of existing surveillance cameras for both routine and emergency scenarios but also provides a new technological approach for seismic monitoring of urban building clusters.