Deep neural networks have demonstrated exceptional performance in the extraction of image information, with CNNs commonly employed for the extraction and compression of feature dimensions [21]. Convolution, the process by which a kernel slides across the input plane to perform computations, is significantly influenced by the size of the kernel and the stride setting, both of which affect the efficiency of target feature extraction. The computational process of a two-dimensional convolutional layer is as follows:where $ {C}_{{\mathrm{in}}} $ and $ {C}_\text{out} $ correspond to the quantities of input and output channels, respectively. $ {\mathrm{out}}(\cdot ) $, which can be derived by the convolution ($ * $) of weights $ w(\cdot ) $ and input image $ {\mathrm{input}}(\cdot ) $ and then adding the bias term $ b(\cdot ) $, signifies the output features. The process essentially constitutes a cross-correlation operation. When the bias term is nullified and the convolution is executed optically, (1) is transformed to be

The input to the simplified convolutional layer can be considered as a monochrome image with a single channel, and the computation is equivalent to the inner product of the illumination and detection process, which means that the convolution process can be implemented by the optical manner to extract target feature. By comparing (1) and (2), optical convolution can be perceived as an unbiased, single-channel convolutional layer. In this process, the kernel is distributed over the input optical image to execute a template matching operation, of which the result is collected by a detector and can be represented aswhere $ F $ denotes the collected feature intensity values, $ {O}_{(x,y)} $ represents the object’s grayscale distribution at the pixel location $ (x,y) $, and $ K $ refers to the kernel distribution that is projected.

Leveraging the principles of optical convolution, a hybrid optoelectronic neural network framework can be implemented as depicted in Fig. 1. The network architecture comprises an input layer, convolutional layers, and a decoding classification network. During the training process, the input layer receives the target images, while the output layer provides the predicted class labels. The convolutional and decoding networks are jointly optimized to extract target features and classify the targets accurately. The kernels of the convolutional layer, as the components of network optimization, can be mapped into the optical inference process. The inference process of the network involves a programmable SLM (in Fig. 1), that modulates the light field according to the kernel distribution, and a detector completing the feature collection of the actual target. Finally, the decision network is deployed on a platform with limited computing power to implement feature inference.

The process of implementing optical convolution through the encoding of light fields is equivalent to single-kernel full convolution, which does not facilitate the sliding of the kernel to capture the coupled feature information between the pixels at different spatial locations. The optical convolution process necessitates an incoherent optical system for the physical mapping. The compressive nature of convolution precludes the ability to achieve deep convolution through stacking processes, leading to weaker feature extraction capability of optical convolution compared to fully optical diffraction neural networks. Consequently, optical convolution has inherent limitations in feature extraction, impeding the extraction of more information from complex scenes.

The importance of analyzing targets at multiple scales stems from the nature of the targets themselves. Scenes in the world contain objects of various sizes, and objects themselves have features at different scales. Applying any analytical process at a single scale may result in the loss of information at other scales. A better solution is to perform analysis simultaneously at multiple scales [22,23]. Modern image classification networks aggregate MS features using various methods, such as integrating information through consecutive convolutional and pooling layers or performing convolutions with kernels of different sizes. The specificity of optical convolution makes it challenging to achieve the former, as it often requires the additional creation of non-programmable diffractive media to construct the optical network. For the latter, the physical mapping method of the projected coded light field allows optical convolution to freely adjust kernel sizes and achieve light field illumination to extract features at different scales without facing computational issues associated with MS convolution. This inspired us to construct the MS optical convolution module shown in Fig. 2.

By progressively downsampling to generate images of different resolutions, an image pyramid is formed [24]. Optical convolution is then performed with kernels of the same size as each respective image, outputting features at different scales. Smaller convolutional kernels, corresponding to lower resolution, are capable of capturing detailed features, whereas larger convolutional kernels can capture the global features of the target. The integration of information from different scales enhances the recognition capability of the backend network. The principle of extracting MS features of the target using this method can be expressed aswhere $ {F}_{m\times m} $ ($ {F}_{n\times n} $) denotes the image features collected by optical convolution at a scale of $ m\times m $ ($ n\times n $),$ {O}_{(i,j)} $ represents the object’s grayscale distribution at the pixel location $ (i,j) $, and $ {K}^{m\times m} $ ($ {K}^{n\times n} $) refers to the kernel distribution that is projected at an $ m\times m $ ($ n\times n $) resolution.

Furthermore, unlike traditional convolutional layers, optical convolution kernels of different sizes possess the same semantic level. This means that during inference, all convolution kernels can directly illuminate the target objects based on the trained distribution without the need for additional serial arrangement of more optical systems, facilitating the simplification and integration of hardware systems.

Traditional target recognition neural networks rely on sensor-captured target images for inference and classification. In contrast, OCNNs based on spatial light modulation achieve target feature perception through light field propagation and photoelectric detection, which are then used for decision-making on target categories. Unlike traditional image-based neural networks, OCNNs do not generate image matrices. Instead, they directly utilize the sequence of light intensity values captured by detectors as target features for numerical computation. This approach significantly reduces the complexity and computational load of the network, increases computation speed, and lowers the deployment difficulty of neural networks on resource-constrained devices.

Therefore, introducing MS optical convolution in non-imaging target recognition networks can significantly enhance the model’s recognition capabilities and contribute to the advancement of non-imaging target recognition applications.

Fig. 3 illustrates the schematic of the designed MS optical convolution non-imaging target recognition training framework. To simulate the MS characteristics of the target, we use the process of hierarchical downsampling of the original image to multiple resolutions as the generative model to generate MS dataset from the original images in the dataset. The training batch consisting of the generated images is fed into the MS-OCNN object classification model. We build parallel multi-resolution convolution layers to optimize the distribution of optical convolution kernels. What is more, we combine channel attention mechanisms and residual blocks to improve the performance of the model.

The parallel gradient-dimension convolutional structure ensures that the convolutional kernel in each channel operates at an equivalent depth, facilitating the real-time optimization of optical convolution kernels. In consistency with the execution modality of optical convolution, we implement the extraction of target features through full convolution. For the inputs with varying resolutions, the convolutional kernels matching the respective sizes are endowed with their channel resources, achieving the extraction of MS features. In this setup, the number of channels within the convolutional kernels corresponds to the number of modulations performed on the light field. Furthermore, specialized fusion blocks are engineered to cater to feature information at different hierarchical levels, maintaining the distinctiveness among the features of various scales.

In the feature inference module, residual blocks and attention mechanisms are utilized for feature identification. As shown in Fig. 4(a), features from various semantic levels are reshaped to form feature maps. Once multiple-channel feature maps are established, the channel attention module (CAM) is integrated to derive attention vectors for the respective channels. These vectors are combined with the original channel distributions through weighting, resulting in weighted feature maps [25]. The introduction of the attention mechanism also facilitates the prioritization of modulation patterns based on the participatory weights of the convolutional kernels in inference, thereby identifying the more beneficial kernels for inference and discarding those with minimal influence, which can further reduce the sampling rate by retaining these kernels [26]. Furthermore, for computations involving deep features, residual blocks are implemented for the processing and transmission of deep features [27]. Fig. 4(b) illustrates the residual architecture, which consists of convolutional layers (conv in Fig. 4) and rectified linear unit (ReLU). This architecture is introduced to mitigate the excessive consumption of computational resources and the problem of vanishing gradients, as well as for effective feature learning and signal propagation, while significantly reducing the network’s parameter count. Focusing on the lightness and efficiency throughout the target inference process, we have crafted multi-resolution convolutional layers and embedded channel attention mechanisms. The complete network architecture is delineated in Fig. 4(c) with fully connected layers (fc in Fig. 4) inserted. Within the MS-OCNN that accommodates the inputs of various resolutions, we have realized the extraction, integration, and inference of MS features from the target, culminating in the use of fully connected layers to yield the prediction outcomes.

We train and test the model on the PyTorch platform. To assess the model’s generalization capabilities, we select the modified national institute of standards and technology (MNIST) handwritten digit dataset and the Fashion-MNIST apparel dataset, which feature grayscale details, as our training data inputs. The MNIST dataset encompasses digits ‘0’ to ‘9’ and consists of simple binary images, whereas the Fashion-MNIST dataset comprises images of 10 popular types of clothing, presenting sparser grayscale images [28]. We generate multi-resolution datasets from both datasets through hierarchical down-sampling. Each dataset is composed of a training set, a validation set and a test set, while maintaining the original class configurations. The higher-resolution images contain more detailed information among images of varying resolutions, while the lower-resolution images exhibit more pronounced contour features. For full convolution, lower-resolution input images provide intuitive contour information, which is less susceptible to the interference from the distribution of details within the image, such as texture and noise. High-resolution images, on the other hand, are employed to supplement fundamental detail information. Compared to the MNIST dataset, the Fashion-MNIST dataset is more complex, with a certain degree of grayscale distribution, making it more suitable for testing the robustness and generalization capabilities of the proposed model, especially when subjected to added noise.

To independently verify the effectiveness of the designed MS optical convolutional layer, we employ a fully connected neural network (FCNN) as the baseline and construct fully connected networks with both SS and MS optical convolutional structures. We align the number of convolutional kernel channels for the SS recognition model with that of the MS recognition model, using the number of convolutional kernel channels as the variable for recognition simulation. The results are displayed in Fig. 5. As the number of convolutional kernel channels increases, the recognition accuracy for targets is improved within a certain range and then becomes nearly stable. Compared to the SS FCNN model, the introduction of the MS optical convolutional structure endows the network with a higher ceiling of recognition accuracy. With direct decision-making on input features, the accuracy is increased by nearly 1%. This indicates that the MS optical convolutional structure can effectively augment the quantity of features fed into the network which are derived from various scales of the images. The enhancing of the volume of valid information drives correct model decisions, which is consistent with our theoretical analysis.

To substantiate the efficacy of the introduced residual structures and attention mechanisms in enhancing model performance, we compare four models: (i) a multi-scale fully convolutional network MS-FCN classification model constructed with MS optical convolutional layers and fully connected layers; (ii) a MS-FCN classification model incorporating an attention mechanism; (iii) a MS-FCN classification model integrating a residual structure; (iv) the proposed MS-OCNN, a multi-resolution CNN classification model employing both attention mechanisms and residual structures. The experimental platform is configured with an Ubuntu 22.04 operating system, 16 GB random-access-memory (RAM), and an Nvidia MX450 graphics card. All four models are trained under identical training parameters. Training is completed over 100 epochs with a batch size of 256. We employ a stochastic gradient descent (SGD) optimizer with the initial learning rate of 0.002, the momentum set to 0.9, and the weight decay at 1.2×10⁻⁴. A learning rate scheduler is also utilized, which dynamically reduces the learning rate based on the loss changes of the model on the validation set, combined with an early-stopping mechanism to ensure better convergence and improved training efficiency. Table 1 presents the accuracy metrics of the aforementioned four models on the two datasets. The results indicate that the integration of the attention modules and the residual structures significantly enhances the recognition capabilities of the model. Our proposed MS-OCNN achieves 3.8% and 5.9% higher recognition accuracies on MNIST and Fashion-MNIST datasets compared to the basic multi-resolution CNN classification model, as indicated by the bold values in Table 1.

To validate the performance of the proposed MS-OCNN, we compare it with several reported SS non-imaging recognition models. These selected SS models utilize neural network-optimized optical convolutional distributions and single-pixel detection recognition methods. They are trained and fine-tuned on the same dataset to achieve efficient non-imaging classification of specific targets. The classification simulation results with different sampling times (ST) are shown in Tables 2 and 3. In the tables, sample rates (SR) represents the ratio of the actual number of samples to the number of samples required to reconstruct an image using a single-pixel detector, the latter typically being the total number of pixels in the image. With the same number of convolution kernel channels, which corresponds to the same sample times in the optical convolution process, the model with the specifically selected combination of resolutions demonstrated higher recognition accuracy compared to SS models, thus exhibiting superior feature extraction and recognition efficiency. This is validated across recognition tasks for both datasets. However, in the more complex task of recognizing the Fashion-MNIST dataset, hyper-parameters including optical convolution resolution need to be fine-tuned. This discrepancy is primarily attributed to the distinct characteristics of objects within different datasets, necessitating the model to adjust the resolution as a hyperparameter through training to achieve optimal performance. In our simulations, we incorporate Gaussian noise of varying intensities into the test dataset to simulate pixel-level fluctuations characteristic of light source noise. This approach is utilized to evaluate the effect of noise on the accuracy of the recognition process. Fig. 6(a) illustrates the images with superimposed Gaussian noise of different magnitudes. The recognition outcomes are depicted in Fig. 6(b), where the proposed model can resist a certain level of noise interference. However, as the noise intensity increases (when the standard deviation exceeds 0.1), there is a significant drop in recognition accuracy.

To validate the practical effectiveness of the proposed method, we construct an experimental setup illustrated in Fig. 7 to realize the optical CNN for target recognition and classification. The emitted continuous-wave laser (Rayziss, 532 nm) beam is expanded by a beam expander to produce a uniform light spot. The projected light is modulated by the pre-imported convolution kernel distribution on a digital micromirror device (DMD, Texas Instruments Discovery 7001, resolution of 1 024×768, refresh rate of 22 kHz), and the exiting modulated light field is reflected off the target surface. The reflected light, after passing through the collection lens, is gathered by a quadrant detector (First sensor) and converted into point signals. These signals are then amplified by a signal conditioning circuit and acquired by a microprocessor chip with a 12-bit analog-to-digital converter (ADC), converting them into digital signals. The processor performs real-time calculations to produce recognition results. The ADC used in the experiment is configured with four channels to detect the light intensity signals of the four pixels of the quadrant detector. Each channel’s conversion time is set to more than 1 μs to ensure sufficient sampling accuracy. Given the SLM’s modulation rate of 20 kHz, the detection time required for each recognition is approximately 1.2 ms, providing the system with good real-time performance.

In the experiments, to approximate the recognition of real-world objects, we print randomly selected images from the test set on A4 papers. Through printed grayscale calibration, we correct the reflectance of the images. After training the proposed model, we extract the convolutional kernels used for feature extraction. Utilizing a grayscale dithering algorithm, we transform the continuously distributed convolutional kernels into binary discrete modulation patterns for spatial light modulation. Furthermore, we increas the ADC sampling period to 10 ms. After five repeated samplings, we take the mean value as the effective signal for processing, thereby reducing the impact of random fluctuations in the light source on recognition accuracy. Leveraging the effectiveness of the proposed network model, we decouple the decision-making part of the model and deploy it into the microcontroller unit (MCU) after quantization and compression, achieving online target recognition.

To evaluate the performance of our trained target classification model, we conduct recognition experiments with images from different datasets. Fig. 8 illustrates the recognition results for 100 randomly selected printed images from each dataset in actual experiments. The results indicate that the proposed method achieves the accuracy of 97% in the recognition task of printed MNIST images (Fig. 8(b)) and the accuracy of 87% on the Fashion-MNIST dataset (Fig. 8(d)). Numerically, the MS-OCNN attains classification accuracies of 98.6% over 10 000 handwritten digits in the blind testing set (Fig. 8(a)) and of 90.6% for Fashion-MNIST dataset in the blind testing set (Fig. 8(c)). The experimental classification results demonstrate consistency with the simulation results, despite an approximate 2% loss in accuracy. This reduction is primarily attributed to some interference factors such as the quantization of network parameters in deployment, the deviations in light spot illumination, and the circuit sampling errors. The experiment demonstrates that the proposed model and recognition method can effectively enhance the sampling efficiency of target features and improve the recognition accuracy of targets.