To address the problems of low detection accuracy and severe missed detections of small targets for hydraulic support protection plates caused by the complex underground environments of coal mines, an improved YOLOv8n detection algorithm (CMH-YOLO) was proposed. In the CMH-YOLO algorithm, a SPDConv space-to-depth convolution module was first introduced into the shallow network to encode spatial information into the channel dimension, thereby preserving the detailed features of small targets. Subsequently, a lightweight C2f_MSA multi-scale attention module was designed, which integrated the long-range dependency modeling of large-kernel separable attention with the adaptive feature selection of efficient channel attention. This enabled the shallow network to simultaneously capture protection plate features at various distances and scales. To tackle the blurred surface features and partial occlusion of protection plates caused by the highly dusty underground environment, a C2f_DRFE dust-robust feature enhancement module was constructed. This module expanded the receptive field through multi-scale dilated convolutions to acquire information from surrounding unconcluded regions, and worked in tandem with deformable convolutions to adaptively avoid dense dust areas. Meanwhile, a BiFPN bidirectional feature pyramid network was introduced to replace the traditional PAN-FPN, which further improved the detection performance of the model. Finally, the Inner-IoU loss function was adopted, utilizing an auxiliary bounding box mechanism to enhance bounding box regression accuracy. Experimental validation was conducted on a coal mine hydraulic support dataset containing 20 045 annotated images. The results show that the mAP@0.5, mAP@0.5:0.95, precision, and recall of CMH-YOLO reach 97.9%, 81.4%, 93.8% and 94.5%, respectively. Compared with the baseline model YOLOv8n, these metrics are improved by 1.7, 1.8, 2.7, 1.6 percentage points, respectively, successfully maintaining the lightweight nature of the model while enhancing detection accuracy. The proposed algorithm effectively resolves issues such as small target recognition, multi-scale adaptation, and dust interference in extreme environments. It provides a reliable real-time monitoring solution for the safety monitoring of coal mine hydraulic supports, which holds significant engineering value for preventing roof accidents and ensuring miner safety.