Real-time hybrid testing is an important test method for exploring the seismic performance of structures incorporating velocity-dependent components. However, current real-time hybrid tests encounter the challenge that the numerical substructure calculation efficiency fails to meet the real-time requirements, thereby restricting the application of this method in seismic tests of large-scale engineering structures. In order to improve the computational efficiency of the numerical substructures, a physical information neural network suitable for real-time hybrid testing is proposed, and a real-time hybrid testing method for neural network surrogate models is implemented. First, a neural network model was constructed based on different physical constraint equations. Then the seismic response of a two-story frame structure with a damper was numerically simulated by finite element software, and these simulation data were employed to train the network model. Finally, the trained physical information neural network was used to carry out real-time hybrid test simulation. The simulation results show that the physical information neural network has high prediction accuracy, among which the physical information neural network using resilience as the loss function has the highest accuracy. The real-time hybrid test method based on the physical information neural network agent model is feasible.