Planetary gear transmission systems are extensively used in industrial applications. Due to their compact and complex configurations, mechanical components are prone to failure during long-term operation. Compared to fixed-axis gear systems, planetary gear systems exhibit multiple excitation sources and time-varying signal transmission paths stemming from their intricate structural and kinematic characteristics. Consequently, condition monitoring techniques based on fixed vibration measurement points face significant challenges in compound fault diagnosis, especially when a planet gear fault is coupled with a bearing fault. To address these issues, this study proposes a non-contact torsional vibration monitoring and residual vibration analysis method utilizing laser doppler vibrometry (LDV). The laser beam is positioned on the low-speed shaft surface to directly acquire torsional vibration information from the gear system. To mitigate the impact of measurement noise on fault feature extraction, a hybrid denoising strategy combining cepstrum-based soft-threshold editing and median filtering is developed to suppress pseudo-vibration artifacts and random impulse noise, respectively. For different types of compound faults, a progressive residual vibration decomposition framework is established. This framework systematically peels off residual broadband response and residual meshing sideband components from the torsional vibration signal. Specifically, optimized filtering is applied to the broadband response obtained via cepstrum short-pass to extract the second-order cyclostationary features of bearing faults. Concurrently, a phase self-demodulation-based order domain resampling method is proposed to highlight gear fault features by reconstructing meshing sideband residual signals of different orders. Experiments involving tooth spalling on planetary gears and raceway spalling on the sun gear bearing demonstrate that the proposed method can effectively achieve non-contact compound fault diagnosis for planetary gear systems. Compared to conventional flexible synchronous averaging and accelerometer-based methods, the proposed approach exhibits superior performance in early-stage planet gear fault detection under varying speeds.