To overcome the challenges of slow speed, prolonged time consumption, and high costs in traditional seed vigor detection methods, we propose a non-destructive seed vigor detection approach that integrates laser speckle technology, a STCD algorithm, and a deep learning model, which facilitate the construction of a comprehensive seed vigor detection system encompassing “image acquisition, feature processing, model training, and vigor determination.” An imaging system was developed by utilizing a 632.8 nm He-Ne laser and a CCD camera to capture laser speckle images of pea seeds. After image preprocessing, the individual pea seed area was extracted. Then, two-dimensional wavelet transform was applied to denoise the speckle images. Subsequently, the STCD method was utilized for image data processing. The STCD method is deeply integrated with the image difference improvement strategy to highlight the feature information of the pea seed laser spot image. The weighted average of the first 4 frames is taken as the reference frame. The 3×3 neighborhood local variance weighting mechanism and the forgetting factor α=0.7 are introduced to construct the cumulative difference map. Finally, 1 600 difference images are obtained, with 800 viable seeds and 800 non-viable seeds. Divide the dataset in a ratio of 7∶2∶1. In terms of the model, MobileNetV3 was adopted as the backbone network to construct the lightweight Faster R-CNN model. The parameters were adjusted and the training strategy was optimized until the Accuracy of the validation set was stable. Subsequently, the optimally trained model was employed for target detection, feature extraction, and deep learning on the speckle images of maize seeds. Ultimately, seed classification and regression were accomplished, enabling efficient and accurate detection of seed viability. The results indicate that the laser speckle images of seeds processed by STCD exhibit distinct feature information, with a maximum signal-to-noise ratio of 35.8 dB for the differential images. After training with the lightweight Faster R-CNN model, the classification accuracy for viable seeds is 92.25% and the accuracy for non-viable seeds is 93.25%. The model achieves an AUC value of 98% in the ROC curve, an F1-score of 92.63%, a false positive rate (FPR) of 4.75%. The recognition time for a single image is 0.031 5 seconds, which is 0.049 3 seconds shorter than that of the original model. The detection speed is 2.5 times faster than the original model, and the convergence epoch is 25, reduced by more than half compared to the original model, demonstrating high training efficiency. These findings confirm that the proposed method enables rapid and non-destructive detection of seed viability.