A semi-active impact damper (SAID) with controllable timing is presented based on vibro-impact mechanism, aiming to control the multi-modal vibration of civil structures under earthquake excitations. The proper impact actuating force can interfere with the accumulation of structural vibration amplitude without additional controllable elements. In this paper, a device construction with adjustable collision clearance was proposed based on the SAID mechanical model and corresponding semi-active control strategy. The damping performance of the SAID system in a five-story frame structure is studied through shaking table tests. The results showed that the SAID system could effectively reduce the acceleration at the top floor of the structure and the inter-story drift. Time-frequency energy analysis revealed that the SAID system can transfer the structural vibration energy from low-order modes to high-order modes to accelerate the energy dissipation during the controlled impact process and suppress the structural response caused by dominant modes. An enhanced semi-active impact damper (ESAID) is further proposed to reduce the adverse effect of the acceleration sudden change caused by the impacts while improving the damping performance of the SAID system.