To improve the accuracy and real-time performance of identifying unsafe behaviors of miners in the mine belt transportation area, and to address the problems of poor real-time performance and high false detection rate in existing manual monitoring methods, a dual-stream spatiotemporal fusion network (DS-SFNet) that integrated image features and human skeleton features was proposed. First, challenges such as low illumination and dust interference in underground environments were addressed by designing a sub-pixel convolutional block attention module (SPCBAM), which combined with sub-pixel convolution and depth wise separable convolution to optimize feature representation. Second, to mitigate the high computational resource consumption of the OpenPose model, its backbone feature extraction network was reconstructed using MobileNet v3 by incorporating dilated convolutions and cross-layer connections. Finally, a hierarchical feature fusion module was constructed to deeply integrate image features and skeletal trajectory features through spatiotemporal alignment and complementary modeling. The results demonstrate a recognition accuracy of 76.4% on HMDB51 (Human Motion Database 51) and 97.9% on UCF101 (University of Central Florida 101), outperforming the SlowFast model by 1.5% and 1.1%, respectively. On a self-built coal mine dataset containing four unsafe behaviors (climbing, crossing, leaning, and hand-leaning), the average recognition accuracy reaches 92.3%. The MobileNet v3-reconstructed OpenPose model reduces parameters to 11.5% of the original Visual Geometry Group 19 (VGG19) network while increasing inference speed by over 3 times. The complete framework achieves a single-frame processing time of 38.7 ms and a parameter count of 57.3 M.