Recognition accuracy of small targets is often required for dense fruit distribution in natural orchard environments. Therefore, a flat peach detection model based on an improved YOLOv8 architecture, termed CCGs-YOLO, was proposed in this study. The proposed model integrates the MetaFormer framework with a convolutional gated linear unit module. A hybrid module combining convolution and attention mechanisms was introduced to enhance spatial feature extraction and improve feature representation under complex background conditions. Meanwhile, a channel-aware module was incorporated to simulate inter-channel dependencies, thereby improving the discrimination capability between fruit targets and cluttered backgrounds. Specifically, the C2f_ConvFormer module was employed to simultaneously capture local and global contextual information, while the C2f_CaFormer module was introduced to enhance channel interaction and feature aggregation. In addition, a convolutional gated linear unit mechanism was embedded into the network to improve feature selection capability and robustness against background noise. To address the small object detection problem in densely distributed fruit scenarios, localization accuracy was further improved. An optimized regression loss function based on an inner-overlap constraint, named Inner-CIoU, was adopted to achieve more accurate bounding box regression and reduce localization errors caused by overlapping targets. Experimental results demonstrated that the ConvFormer module improved the F1-score to 90.44% and the mAP50 to 96.40%, indicating enhanced feature extraction capability. The CaFormer module increased the F1-score from 89.96% to 90.63%, while the mAP50 further improved to 96.12%, demonstrating effective channel modeling capability under relatively high inference efficiency. When only the convolutional gated linear unit mechanism was applied, the F1-score reached 90.20% and the mAP50 achieved 96.12%, verifying its effectiveness in enhancing feature representation. Furthermore, the combination of CaFormer and convolutional gated linear unit improved the Precision to 91.47%, the F1-score to 90.77%, and the mAP50 to 96.24%, demonstrating the complementary advantages of channel modeling and gated feature selection. In terms of localization performance, Inner-CIoU improved both mAP and model convergence stability compared with the conventional CIoU loss function. After integrating all improved components, the model achieved relatively better overall performance. Precision reached 93.07%, representing an increase of 3.31 percentage points compared with the baseline model. The F1-score reached 90.87%, while the mAP50 achieved 96.24%. Meanwhile, the model size was reduced from 5.97 MB to 4.88 MB, and the number of parameters decreased from 3.01 million to 2.42 million, indicating that the proposed model possesses favorable lightweight characteristics while maintaining relatively high inference speed of 362.07 FPS. In addition, comparative experiments were conducted with several mainstream models, including different versions of the YOLO series. Under challenging scenarios such as occlusion, small targets, dense distribution, and edge targets, the proposed model achieved relatively superior performance in terms of Precision, F1-score, and mAP, demonstrating improved detection stability and feature perception capability. Visualization analysis further indicated that the improved model could focus more accurately on fruit regions and suppress background interference to a certain extent. Furthermore, deployment experiments on edge computing devices demonstrated that the proposed model could still maintain relatively high detection accuracy and stable performance under practical application conditions, with the mAP50-95 reaching 88.82%, indicating potential for real-world applications. Overall, the proposed model effectively balanced detection accuracy, model lightweight characteristics, and computational efficiency, demonstrating good robustness and adaptability in complex orchard environments. The proposed approach can provide a feasible technical solution for rapid and accurate fruit recognition in flat peach harvesting.