Latest ArticlesLow-light image enhancement (LLIE) is a key challenge in computational photography and computer vision,with critical applications ranging from autonomous navigation and night surveillance to computational imaging. Although existing methods in sRGB,HSV,and YCbCr color spaces have shown effectiveness,they face an irreconcilable trade-off between preserving color fidelity and suppressing luminance artifacts,often introducing chromatic shifts or halo distortions during the enhancement process. Recent studies focus on color space optimization. YCbCr-based methods perform shallow luminance processing (e.g.,linear filtering),causing an imbalance between chrominance denoising and detail preservation. Accordingly,the performance of these methods is capped. HSV space strategies decouple the hue and saturation components but cause color discontinuities due to polar coordinate quantization. This study proposes a novel lightweight LLIE network to address these limitations. This network fully leverages the channel decoupling properties of the HVI color space,minimizing the mutual interference between luminance information and color structural information during enhancement.
We propose GCDINet,an end-to-end framework leveraging HVI color space’s physical decoupling. The architecture comprises: 1) a dual-branch structure that processes value-intensity and hue components independently via the GCFE module,which utilizes dual GELU activations and element-wise multiplication to generate spatial attention masks; 2) a dual-path adaptive feature calibration (DAFC) structure that nonlinearly modulates illumination and chrominance features before reconstruction; 3) a composite loss combining HVI-space and sRGB-space constraints. The novel HVI color space decouples images into luminance and color structure maps at the original resolution,allowing the dual-branch network to process luminance and color structure information independently. This mechanism avoids interference from processing in the same feature stream and provides independent branch information for the GCFE module. A DAFC structure nonlinearly modulates the illumination features from the luminance branch and the chrominance vectors from the color branch before feature reconstruction. This approach guides information reconstruction during decoding,suppressing color fidelity loss and luminance artifact amplification caused by information loss. Although the dual-branch structure and gated convolutions decouple luminance and color structure information for a refined feature enhancement,uneven channel weight distribution during feature fusion can result in the loss of key luminance and chrominance information. During training,we optimize using a composite loss function that combines L1 intensity loss,structural similarity index measure (SSIM) loss,and edge-aware regularization. The model is implemented in PyTorch with the Adam optimizer(
Comprehensive evaluations on the LOLv1,LOLv2-Real,and LOLv2-Synthetic datasets demonstrate the state-of-the-art performance of GCDINet. On the LOLv1 dataset,our method achieves the best results in terms of SSIM and learned perceptual image patch similarity (LPIPS) and is only 0.76 dB short of the best method in peak signal-to-noise ratio (PSNR). On the LOLv1-Real dataset,our method attains optimal results in PSNR and SSIM and is only 0.006 short of the best method in LPIPS. Experimental results on the LOLv2-Synthetic dataset demonstrate that the proposed method achieves a PSNR of 25.698 dB,an SSIM of 0.935,and an LPIPS score of 0.047. These metrics show improvements of 0.028 dB,5.3%,and 20.3% compared with the state-of-the-art LLIE method Retinexformer. Furthermore,the computational complexity is 7.94 GFLOPs,which is 50% lower than that of Retinexformer,achieving an effective balance between speed and performance compared to the existing methods. Ablation studies show that the PSNR,SSIM,and LPIPS can be improved by 1.9%,0.2%,and 9.4%,respectively,using only the dual-path adaptive feature calibration structure. When integrated with the GCM module,these improvements increase to 2.8%,0.64%,and 11.3%. The utilization of HVI color space decomposition yields in enhanced PSNR by 18.9% over the HSV baseline,SSIM by 6.4%,and perceptual quality by 64.3%. These three evaluation metrics improve by 4.9%,2.1%,and 36.4% compared with YCbCr. The use of only the HVI loss lacks pixel-space consistency constraints,resulting in large pixel errors and decreased performance across all three metrics,especially PSNR. By contrast,the use of only sRGB loss focuses on pixel-space enhancement but ignores the low-light probability distribution in the HVI color space,resulting in color imbalance. However,from a visualization perspective,the pixel-level errors and color distribution of using only HVI/sRGB loss functions are not intuitively observable.
The proposed LLIE method in this study effectively combines the advantages of a dual-branch structure and the HVI color space,achieving a balance between color fidelity and luminance artifact suppression in low-light image enhancement. GCDINet resolves the fundamental trade-off between color and artifacts through deterministic component decoupling and adaptive feature calibration,with ablation studies confirming the synergistic benefits of the GCFE and DAFC modules.
Image super-resolution (SR) is a fundamental low-level vision task in computer vision that aims to reconstruct high-resolution images from low-resolution inputs,thereby enhancing image detail quality and sharpness. High-quality reconstruction not only improves human visual perception but also provides clear and highly structured input features for high-level vision tasks,such as object detection,semantic segmentation,and face recognition. Therefore,SR technology has broad application prospects in intelligent surveillance,medical imaging,autonomous driving,satellite remote sensing,and related fields. Convolutional neural networks (CNNs) have been widely used for SR because of their ability to capture local textures and edges. However,their limited receptive fields hinder the modeling of long-range dependencies,which are essential for preserving global structural consistency. Transformer-based methods,by contrast,leverage self-attention mechanisms to capture the global context,achieving superior reconstruction performance. Despite this advantage,conventional Transformers suffer from high computational costs and redundant attention weight,limiting their applicability in lightweight or real-time scenarios. To address these challenges,we propose a lightweight SR network that integrates the advantages of CNN and Transformer architectures. The network integrates efficient local feature extraction with global context modeling,achieving balance between reconstruction quality and computational efficiency.
We propose SPSANet,a lightweight image SR network that is based on sparse and permuted self-attention (SPSA) and composed of shallow feature extraction,deep feature extraction,and high-resolution reconstruction. At the shallow feature extraction stage,a standard 3 × 3 convolution captures basic texture information,and shallow features are forwarded to the reconstruction stage via a long skip connection to provide residual guidance. The deep feature extraction stage consists of four SPSA groups,each containing six SPSA blocks and a convolutional layer. Each block integrates an SPSA module (SPSAM) that expands the receptive field while reducing attention redundancy,a high-frequency enhancement module (HFEM) for refining texture and edge details,and a dual-branch feature fusion module (DBFFM) that effectively fuses global and local features through spatial–channel interactions. Then,shallow and deep features are fused via a global residual connection,and a pixel shuffle operation generates the final high-resolution output. Our method is implemented on the PyTorch framework and trained using an NVIDIA RTX 3090 GPU. The training dataset is DIV2K,which contains 800 images. For the ×2 SR task,the model is trained for a total of 500 K iterations by using the Adam optimizer with an initial learning rate of 2 × 10⁻⁴. The input image patch size is fixed at 64 × 64,the batch size is set to 16,and the MultiStepLR scheduler is applied to halve the learning rate at iterations [250 K,400 K,450 K,475 K]. For the ×3 and ×4 SR tasks,the model is initialized with the pretrained weights from the ×2 model,and the total number of training iterations is reduced by half.
The proposed method is evaluated on five public benchmark datasets and compared with 11 state-of-the-art (SOTA) SR approaches. Experimental results demonstrate that under the constraint of lightweight design,SPSANet achieves either the best or second-best performance across different upscale factors and datasets,showing strong generalization ability and stability. Specifically,when the upscale factor is ×3,SPSANet surpasses the latest SOTA methods by 0.15 dB and 0.25 dB in peak signal-to-noise ratio (PSNR) on Urban100 and Manga109 datasets,respectively,and achieves corresponding improvements in the structural similarity index measure (SSIM). Furthermore,when the self-ensemble strategy is applied during the testing phase,the model’s performance is further enhanced. Under the same scaling factor,PSNR increases by an additional 0.11 dB and 0.23 dB on Urban100 and Manga109,respectively,indicating that the proposed method maintains strong robustness under multiview inference. Ablation studies are conducted on SPSAM,HFEM,and DBFFM to further investigate the contribution of each component. The results show that removing any of these modules leads to a noticeable degradation in performance. SPSAM plays a key role in global dependency modeling and attention sparsification,HFEM effectively enhances texture restoration quality,and DBFFM substantially improves the fusion of global and local features. In terms of visual quality,SPSANet can reconstruct clear,natural,and sharp image textures,particularly in scenes with complex structures and abundant details. It preserves the structural consistency and perceptual realism of the original images more effectively than existing approaches do.
In this work,we propose SPSANet,a lightweight image SR network that integrates an SPSA mechanism to efficiently capture long-range dependencies while minimizing redundant computations. By establishing interlayer attention connections,SPSANet enables focused feature learning,ensuring that the network prioritizes tokens that are the most critical to the reconstruction process. Moreover,the proposed HFEM enhances the recovery of fine textures and structural details,and DBFFM effectively aligns and integrates global and local representations. Through extensive experiments,SPSANet demonstrates superior reconstruction performance and visual fidelity compared with existing lightweight SR approaches,especially in challenging scenarios with intricate patterns and rich textures. Despite these promising results,opportunities for further improvement remain. Future work could enhance model compactness and generalization through knowledge distillation,hybrid convolution-Transformer integration,and adaptation to broad low-level vision tasks (e.g.,image denoising and deblurring). Overall,SPSANet provides a practical,interpretable solution for lightweight Transformer-based image restoration,offering insights for the design of efficient and high-performing SR networks.
Intelligent analysis and processing of UAV-captured images by using computer vision has become the mainstream approach for power transmission line inspection. As the most widely used connection components in transmission lines,bolts play a critical role in structural integrity. However,under complex environmental conditions,bolts are prone to defects,such as lost nuts and pins,which may pose serious threats to the safe and stable operation of transmission systems. Therefore,bolt defect classification via computer vision techniques is essential to ensure transmission reliability. Vision-language models (VLMs) have high potential in bolt defect classification tasks where visual information is limited. However,vision-language pretraining approaches that are based on image-text contrastive learning typically require large-scale datasets and diverse textual descriptions. For instance,general-domain multimodal datasets often comprise millions of image-text pairs,and bolt defect datasets are characterized by small-scale data and limited textual descriptions. As a result,contrastive learning approaches cannot be directly applied to such tasks. To expand the applicability of VLMs in bolt defect classification,this study proposes a transmission line bolt defect classification method that integrates vision-language feature alignment and textual decoupling (VL-Bolt).
We analyze the alignment behavior of image-text pairs in a training batch and identify a key limitation of standard image-text contrastive learning: intraclass semantic overlap. When multiple samples from the same class are present in a batch,image-text contrastive learning incorrectly treats identical class descriptions as negative pairs,hindering effective alignment between images and their textual counterparts. This issue is particularly pronounced in small-scale bolt defect datasets with limited textual diversity. Then,we adopt the CLIP ViT-B/32 model as the backbone. However,because of the lack of power domain knowledge in CLIP’s pretraining and the high similarity between different bolt defect features,its performance in fine-grained classification is constrained. To address this issue,we introduce a text feature decoupling (TFD) loss,which explicitly disentangles textual features of different defect categories in the feature space. Building upon the decoupled textual features,we further design a text-anchor-guided visual feature alignment (TA-VFA) strategy tailored to bolt defect classification. In TA-VFA,decoupled text features serve as anchors to guide the alignment of their corresponding image features,facilitating the effective separation of image representations across different defect categories. This architecture inherently resolves the intraclass semantic overlap issue,making it suitable for the bolt defect classification task. Moreover,considering the limited scale of the bolt defect dataset,we propose a progressive fine-tuning strategy for the image encoder to mitigate overfitting and improve generalization. Specifically,only Transformer layer 11 is unfrozen during early training,and additional layers are gradually unfrozen at the late stages. This scheme enables the model to prioritize learning high-level semantic features relevant to bolt defects while retaining low-level structural information and the generalization ability gained during pretraining.
In comparative experiments,VL-Bolt achieves a classification accuracy of 92.2% on the bolt defect dataset,surpassing the baseline by 3.9% and outperforming multiple ImageNet-pretrained vision models and four recent bolt defect classification models fine-tuned on bolt datasets. Compared with the multimodal MUCO-BD (multi-modal contrastive learning for bolt defect classification) model,the proposed VL-Bolt model has a slightly lower classification accuracy; however,it has approximately 29.9% fewer trainable parameters. Moreover,when the VL-Bolt model is pretrained with domain-specific knowledge with the same configuration as MUCO-BD,its performance surpasses that of MUCO-BD,demonstrating the superior adaptability of the proposed approach. We also compare our transfer strategy with four recently proposed transfer learning methods. Results demonstrate that VL-Bolt achieves superior accuracy and generalization performance. From the perspectives of algorithm principle and structure,the main reason for the poor classification performance of CLIP-Adapter,CoOp (context optimization),CoCoOp (conditional context optimization),and Tip-Adapter models is the excessively high interclass similarity of bolt defects. Compared with WiSE-FT (ensembling the weights of the zero-shot and fine-tuned models),a transfer method designed to enhance model generalization,VL-Bolt outperforms it by 2.2%,demonstrating the generalization advantage of the proposed transfer method. To verify the robustness of the proposed method,this study applies VL-Bolt to three publicly available datasets and compares its classification performance with that of end-to-end fine-tuning model. On the fine-grained classification datasets Flowers102 and StanfordCars,VL-Bolt achieves substantial improvements,particularly on StanfordCars,where it surpasses the end-to-end fine-tuning model by 10.7%. On the Caltech101 dataset,which is characterized by large interclass variations,VL-Bolt still demonstrates strong task adaptability,further validating the robustness of the proposed approach. Ablation studies confirm the contribution of each component of our method and its bolt classification task compatibility. The TA-VFA architecture avoids the intraclass semantic overlap issue,and its integration leads to a notable improvement in accuracy. The text feature decoupling mechanism can effectively mitigate interclass confusion to a certain extent. Notably,the progressive image encoder fine-tuning strategy guides the model to focus on target semantics,substantially improving accuracy. To identify the optimal layer depth for progressive fine-tuning,we analyze training loss and validation accuracy curves and find that unfreezing only Transformer layers 5 to 11 achieves optimal performance. Compared with end-to-end fine-tuning,the progressive image encoder fine-tuning strategy effectively stabilizes the training process and mitigates overfitting. In visualization experiments,the near-orthogonality of interclass text feature similarities validates the effectiveness of the proposed text feature decoupling loss. Moreover,using t-distributed stochastic neighbor embedding,we visualize the feature spaces learned by image-text contrastive learning,CLIP ViT-B/32,and VL-Bolt. Compared with the other methods,VL-Bolt shows better interclass separability with more natural boundary distributions,achieving reasonable balance in feature representation. Gradient-weighted class activation map visualizations further reveal that VL-Bolt focuses on bolt targets and defect regions while reducing attention to irrelevant background areas,thereby providing a solid foundation for downstream classification tasks.
The proposed VL-Bolt substantially improves the classification performance of VLMs in bolt defect classification tasks. Specifically,the TFD loss and TA-VFA architecture demonstrate strong task adaptability and effectively mitigate the challenge of intraclass semantic overlap while reducing the burden of data preparation. Moreover,the progressive fine-tuning strategy for the image encoder suppresses overfitting and directs the model’s attention toward defect-relevant regions. Overall,this work provides new insights into the practical application of VLMs in power transmission line inspection.
In autonomous driving scenarios,pedestrian detection performance can degrade substantially because of dramatic variations in pedestrian scale,frequent occlusions,and interference caused by complex backgrounds. To overcome these difficulties,we introduce a novel real-time pedestrian detection algorithm called frequency-enhanced and boundary-aware detection Transformer (FEBA-DETR),which integrates multiple frequency-domain enhancement strategies with a boundary-aware mechanism. By leveraging these complementary components,the proposed method achieves reliable pedestrian detection under the aforementioned challenging conditions.
Real-time detection Transformer (RT-DETR) has demonstrated promising object detection performance,but it still faces challenges in capturing fine-grained details of objects at multiple scales,particularly for small objects,and in effectively leveraging frequency-domain information. To address these limitations,we propose an enhanced RT-DETR architecture that integrates several innovative modules and techniques. For improved backbone efficiency and feature propagation,our model incorporates cross-stage partial connections,which strengthen gradient flow across network stages and reduce redundant computations without sacrificing representation power. Building on this foundation,we introduce a frequency-aware module that explicitly harnesses high- and low-frequency visual cues to enrich the feature representations,ensuring that important textural and edge details are preserved and emphasized throughout the network. At the feature interaction level,we design a frequency-aware attention-based intra-scale feature interaction (FAIFI) mechanism. By enabling features at the same scale to communicate through attention operations modulated by frequency-domain signals,FAIFI allows the detector to capture subtle object characteristics that might otherwise be missed in single-scale processing. Moreover,we employ a boundary-aware feature aggregation (BAFA) strategy to improve multiscale feature fusion. BAFA focuses on preserving object boundary information when combining features from different resolution levels,thus helping maintain precise localization cues and contributing to accurate detection of objects with distinct edges. In addition to these architectural improvements,we refine the training process with an Inner-GIoU loss,a modified bounding box regression loss that places high emphasis on the overlapping area between predicted and ground-truth boxes. This tailored loss function drives the model to achieve tight and highly reliable localization. Furthermore,to bolster the model’s robustness and expand the diversity of training data,we introduce a novel frequency-based data augmentation approach that utilizes 2D discrete wavelet transform. By decomposing and altering the frequency components of training images,this augmentation method exposes the network to a broad variety of spatial-frequency patterns and helps prevent overfitting to common textures. Together,these enhancements considerably boost the detection accuracy and reliability of RT-DETR. Experiments on challenging object detection benchmarks indicate that the improved RT-DETR achieves higher average precision than its baseline counterpart,especially on small or texture-rich objects,and maintains real-time inference speed. Our results underscore the benefit of incorporating frequency-domain insights and boundary-aware feature processing into the RT-DETR framework for high-performance object detection.
We rigorously evaluate our approach on three widely used person-detection benchmarks: CityPersons,WiderPersons,and ACDC. On CityPersons,our FEBA-DETR model achieves notable improvements over the baseline RT-DETR,with AP50 increasing by 2.3% and AP50:95 increasing by 2.0%; incorporating our data augmentation technique boosts performance,raising the aforementioned gains to 3% and 2.4%,respectively. On the occlusion-specific subsets of CityPersons,the miss rate decreases by 4.92% for the partially occluded “Reasonable” category and by 2.82% for the heavily occluded “Heavy” category. Similarly,on the challenging WiderPersons dataset,FEBA-DETR again surpasses the baseline,with AP50 and AP50:95 improvements of 1.1% and 0.7%,respectively. These consistent gains underscore the robust generalization of our method across diverse datasets. Moreover,on the ACDC dataset,which specifically assesses robustness in complex environmental scenarios,FEBA-DETR boosts AP50 and AP50:95 by 1.9% and 1.3%,respectively,with even greater improvements of 3.1% and 2.4%,respectively,after applying our data augmentation strategy. These extensive experimental results convincingly affirm the effectiveness of our multilevel frequency enhancement and boundary-awareness techniques,substantially advancing pedestrian detection in challenging autonomous driving contexts.
Experimental evaluations show that FEBA-DETR delivers notable advancements in detecting pedestrians,particularly those who are either small in scale or partially occluded. The model also maintains consistent performance under highly challenging and dynamic conditions. These findings underscore the practical advantages of incorporating frequency-domain enhancements and boundary-awareness strategies. Together,these design choices contribute to a dependable,accurate pedestrian detection system,which is especially valuable for autonomous driving applications where safety and precision are critical.
Infrared small-target detection plays a critical role in military surveillance,security monitoring,remote sensing,and unmanned aerial vehicle (UAV) operations. Accurate detection of such targets is essential for threat assessment,object tracking,and decision-making in complex operational scenarios. In real-world environments,infrared targets are often extremely small in size,have low contrast against complex backgrounds,and are easily affected by noise. These characteristics make the extraction of discriminative features that can effectively separate targets from surrounding clutter challenging. Moreover,differences in semantic content across feature layers impede the effective fusion of shallow detailed features with deep semantic features,further limiting the accuracy and robustness of detection. Designing efficient,accurate,and robust methods for infrared small-target detection under complex backgrounds remains a critical research problem in the field of computer vision and defense-related applications.
To address these challenges,we propose a novel multibranch perception and cross-layer semantic fusion network (MPCF-Net) for infrared small-target detection. The network is built on an encoder-decoder architecture designed to capture both local and global contextual information while mitigating the computational cost. In the encoder,we introduce a multibranch perception fusion attention module (MPFM) that simultaneously processes local,global,and serial convolutional branches to extract multiscale features. The MPFM module is combined with local-global guided attention (LGGA),which integrates local spatial features and global semantic information to enhance the target response,and global channel spatial attention (GCSA),which models channel dependencies and spatial relationships to improve feature representation quality. A spatial-channel cross transformer block (SCTB) replaces conventional skip connections to effectively bridge semantic gaps between shallow and deep layers. SCTB models cross-layer context dependencies,allowing the network to capture hierarchical feature correlations while alleviating semantic differences between layers. This module ensures that shallow detailed and deep semantic features are adequately fused,improving the overall discriminative power of the network. In the decoder,depthwise separable convolution is employed to reduce computational complexity without sacrificing representational capability. Moreover,a light gradient gate (LGG) guided by Sobel gradients is introduced at the output stage to enhance small-target edge details and reinforce feature localization precision. By combining attention mechanisms with multibranch and cross-layer strategies,MPCF-Net can robustly detect infrared small targets in challenging and cluttered backgrounds.
The effectiveness of MPCF-Net is evaluated on three public infrared small-target datasets,namely,SIRST,IRSTD,and NUDT-SIRST,which contain diverse scenes with varying target sizes,shapes,and noise levels. Quantitative results demonstrate that MPCF-Net achieves intersection over union (IoU) scores of 80.12%,66.28%,and 84.26% and normalized intersection over union (nIoU) scores of 78.23%,64.58%,and 86.48% on SIRST,IRSTD,and NUDT-SIRST datasets,respectively. In terms of detection performance,MPCF-Net attains a high probability of detection of 99.88%,94.23%,and 98.21% while maintaining extremely low false alarm rates of 1.12 × 10-6,4.39 × 10-6,and 14.57 × 10-6 on SIRST,IRSTD,and NUDT-SIRST datasets,respectively. We also analyze the effectiveness of individual components. Ablation studies reveal that the MPFM module substantially enhances multiscale feature extraction,LGGA improves the network’s sensitivity to target regions,and GCSA strengthens feature representation by emphasizing spatial-channel relationships. The SCTB module demonstrates clear advantages in reducing semantic conflicts between layers,leading to coherent and accurate feature fusion. LGG ensures that edge details are preserved and small targets are precisely localized,which is crucial for applications where missed detection could result in operational failure.
MPCF-Net effectively combines multibranch perception,attention mechanisms,and cross-layer semantic fusion to enhance discriminative feature extraction and hierarchical context modeling for infrared small targets. The network demonstrates high accuracy,robustness,and low false alarm rates in complex scenes. Its design provides a practical solution for real-world surveillance,military monitoring,and UAV-based remote sensing applications. Future work may explore adaptive integration with other sensor modalities and real-time deployment strategies to further enhance operational effectiveness.
Dougong is a distinctive structural and decorative component of traditional Chinese wooden architecture,playing a critical role in load transfer,seismic resistance,and architectural aesthetics. Moreover,Dougong,a key element connecting columns,beams,and roof structures,embodies sophisticated construction wisdom and regional architectural characteristics accumulated over centuries. The structural configuration of Dougong is composed of multiple interlocking wooden elements arranged in a highly ordered and hierarchical manner,exhibiting strong geometric regularity and pronounced directional dependency. These characteristics make Dougong not only architecturally significant but also visually complex. In real-world heritage environments,the automated recognition of Dougong faces substantial challenges. Variations in spatial orientation,component location (corner and non-corner Dougong),viewing angle,and scale significantly affect visual appearance. Moreover,complex backgrounds,uneven illumination,occlusion by surrounding architectural elements,and long-term material degradation further complicate accurate recognition. In particular,damaged Dougong components often exhibit incomplete structures and blurred boundaries,which severely reduce the effectiveness of conventional image recognition methods. Although deep learning techniques have been increasingly applied to cultural heritage documentation and architectural analysis,numerous existing approaches primarily rely on convolutional neural networks. Such methods often struggle to capture long-range dependencies and direction-sensitive structural features,limiting their adaptability to ancient architectural components with strong directional organization. Accordingly,a recognition framework that can explicitly model directional structural information while maintaining robustness under complex environmental conditions must be developed. The objective of this study is to develop a robust and accurate Dougong recognition model by integrating Transformer-based feature extraction with a novel direction-aware attention mechanism. This study aims to advance the intelligent recognition of intact and damaged Dougong components by enhancing directional feature representation and channel-wise feature interaction,thereby supporting the digital preservation and systematic documentation of ancient wooden architectural heritage.
This study proposes an improved Dougong recognition framework based on the Mask R-CNN architecture enhanced with a Swin Transformer backbone. Unlike the traditional convolutional backbones,the Swin Transformer enables hierarchical feature extraction and efficient modeling of global contextual information,which is particularly beneficial for complex architectural scenes. A dedicated Dougong dataset is constructed,consisting of 3 014 carefully annotated images that include corner and non-corner Dougong instances. The dataset is collected from real architectural environments at historically significant sites,such as Lingyin Temple and Faxisi Temple,complemented by high-quality images from publicly accessible online repositories. The collected samples cover diverse viewpoints,spatial scales,lighting conditions,and background environments,ensuring strong representativeness and practical applicability. A directional coordinate attention (DIRCA) mechanism is introduced to address the limitations of conventional attention mechanisms in capturing the directional structural characteristics of Dougong. Unlike traditional spatial or channel attention methods that treat spatial dimensions uniformly,DIRCA decomposes spatial attention into horizontal and vertical directions. This design enables the network to independently encode direction-sensitive information along orthogonal axes,which closely corresponds to the intrinsic structural logic of Dougong,including component stacking order,load transmission paths,and repetitive decorative patterns. DIRCA improves the model’s ability to distinguish subtle structural variations under complex background interference by explicitly strengthening the representation of direction-dependent features. In addition,a 1D squeeze-and-excitation (1D-SE) module is incorporated to achieve lightweight channel recalibration. The proposed 1D-SE mechanism models inter-channel dependencies using a simplified 1D operation,effectively minimizing computational cost while preserving channel discrimination capability. This lightweight design facilitates seamless integration with the Transformer-based backbone and enhances the interaction between global contextual features and local structural details. The DIRCA and 1D-SE modules are jointly embedded within the Swin Transformer-enhanced Mask R-CNN framework,forming the proposed SwinE_AM_DIRCA model. Model training and evaluation are conducted under consistent experimental settings. Comparative experiments are performed against representative baseline models,including Mask R-CNN_ResNet50 and Swin_AM,to verify the effectiveness of the proposed architectural improvements. Multiple evaluation metrics are used to assess recognition performance,stability,and robustness,with particular emphasis on corner Dougong and damaged Dougong scenarios.
Experimental results demonstrate that the proposed SwinE_AM_DIRCA model achieves substantial and consistent performance improvements over baseline methods across multiple recognition tasks. The recognition performance for non-corner Dougong reaches 94.4%,while the performance for corner Dougong attains 89.2%,outperforming the Swin_AM model by 6.7% and 8.9%,respectively. The proposed framework achieves even more pronounced improvements of 10.8% for non-corner Dougong and 20.4% for corner Dougong recognition compared with the conventional Mask R-CNN_ResNet50 model,highlighting the effectiveness of Transformer-based feature extraction combined with direction-aware attention. In damaged Dougong recognition tasks,the proposed model exhibits strong robustness and stability. Despite challenges,such as partial occlusion,surface erosion,structural incompleteness,and background clutter commonly observed in ancient architectural heritage sites,SwinE_AM_DIRCA maintains accurate localization and recognition of the key structural components. Qualitative visualization results further indicate that the model effectively suppresses irrelevant background features while preserving the integrity of direction-sensitive structural information,which is crucial for a reliable heritage documentation. Ablation experiments further confirm the contribution of each proposed module. The DIRCA mechanism significantly enhances the model’s sensitivity to directional features,while the 1D-SE module improves channel-wise feature discrimination with minimal computational overhead. When these two modules are combined,they produce complementary effects,resulting in consistent performance gains and improved model robustness across different Dougong types and damage conditions.
This study presents a direction-aware and lightweight attention-enhanced deep learning framework for Dougong recognition in ancient Chinese wooden architecture. The proposed SwinE_AM_DIRCA model effectively addresses the challenges posed by complex morphology,strong directional characteristics,and real-world damage conditions of Dougong structures by integrating the DIRCA mechanism and 1D-SE module into a Swin Transformer-based Mask R-CNN architecture. The primary innovation of this study lies in the unified modeling of direction-sensitive structural features and efficient channel recalibration,enabling improved recognition performance and robustness without excessive computational complexity. Experimental results demonstrate that the proposed method significantly outperforms existing baseline models in standard and damaged Dougong recognition tasks. Overall,this study provides an effective and practical technical solution for the intelligent recognition and digital preservation of ancient architectural components. The proposed framework contributes to the advancement of deep learning applications in cultural heritage analysis and offers valuable methodological support for the systematic documentation and sustainable protection of ancient buildings.
Face recognition (FR) has achieved remarkable progress in recent years mainly because of deep learning techniques and the availability of massive,annotated face datasets. Large-scale models,such as ResNet,demonstrate excellent discriminative capability,but their deployment on resource-limited devices,such as mobile and embedded systems,remains highly challenging because of prohibitive computational and storage demands. Lightweight networks,including MobileFaceNet,can reduce complexity and enable real-time applications but often suffer from a noticeable accuracy drop,especially under unconstrained and cross-domain conditions. To address this dilemma,researchers have introduced knowledge distillation (KD) as an effective strategy for compressing large teacher models into small student models while retaining accuracy. However,conventional logit-based distillation is not very effective in FR because this task emphasizes discriminative feature embeddings rather than direct classification logits and faces additional challenges,including extremely large class space and flat soft targets. Recent studies have explored feature-based or relational distillation,but these approaches often couple student and teacher objectives too tightly,leading to gradient interference and unstable convergence. Furthermore,many methods rely solely on hard labels and overlook the rich relational information embedded in the training data. Motivated by these limitations,this study proposes a novel two-stage contrastive KD framework called TC-Face,which is designed to simultaneously enhance stability,efficiency,and recognition accuracy in lightweight FR models.
The proposed TC-Face framework consists of two decoupled yet complementary stages. At stage one,we introduce a self-supervised contrastive distillation strategy. Instead of directly relying on logit outputs,the student network learns from the relational structure of teacher embeddings through a momentum-updated dynamic feature memory bank. This design allows the student to mimic fine-grained intersample relations derived from the teacher without being overwhelmed by noisy gradients. A difficulty-aware weighting mechanism is employed to further improve robustness. Each training sample is adaptively assigned a weight in accordance with its alignment difficulty: simple and overly hard samples are down-weighted,and moderately challenging samples contribute the most to optimization. This approach balances the knowledge transfer process and prevents overfitting to ambiguous or mislabeled identities. Moreover,momentum updating ensures that the teacher feature bank evolves smoothly,stabilizing supervision signals across iterations. The overall loss integrates embedding alignment,contrastive distribution matching,and adaptive weighting. At stage two,the student transitions from imitation to independent optimization. Specifically,the classifier parameters pretrained by the teacher are reused to initialize the student classifier,thereby reducing redundant training cost and accelerating convergence. With ArcFace-based angular margin loss,the student network now learns to refine its feature space independently; it explores discriminative embeddings while avoiding collapse into mere replicas of the teacher. The classifier is trained with frozen student parameters during the first epochs,followed by joint fine-tuning with shared learning rate schedules,to accelerate early-stage convergence. This two-stage decoupling ensures that the student absorbs structured relational knowledge from the teacher then refines its representation capacity through direct discriminative optimization. Compared with single-stage or tightly coupled KD,TC-Face balances imitation and independence,thus achieving training stability and strong recognition accuracy.
We conduct extensive experiments on multiple public datasets to verify the effectiveness of TC-Face. Training is performed on MS1MV2 and MS1MV3 datasets (5.8 M and 5.1 M images,respectively),and evaluation covers widely used benchmarks,including labeled faces in the wild(LFW),AgeDB,cross-age labeled faces in the wild(CALFW),cross-pose labeled faces in the wild(CPLFW),celebrities in frontal-profile(CFP-FP),MegaFace,IJB-B,and IARPA janus benchmark(IJB-C). On the lightweight backbone MobileFaceNet (2.06 M parameters,0.45 giga floating point operations per second),our method consistently outperforms prior distillation strategies. For example,the silhouette coefficient of embeddings trained with vanilla ArcFace is 0.200,whereas our method improves it to 0.236,surpassing even large models,such as ResNet18 (0.224). This result demonstrates superior intraclass compactness and interclass separability in the feature space. Ablation studies on the IJB-C dataset reveal the contribution of each component. Using vanilla KD with mean squared error loss yields true acceptance rate at false acceptance rate (TAR@FAR) = 1E-4 of 91.29% and TAR@FAR = 1E-5 of 79.79%. Introducing contrastive KD (CKD) improves performance to 92.81% and 84.24%. Our full method with adaptive weighting (CKD+) further increases accuracy to 93.07% and 88.33%. Varying the momentum update rate for the teacher feature bank shows that
This work proposes TC-Face,a novel two-stage contrastive KD framework for lightweight face recognition. By decoupling imitation and discriminative optimization,TC-Face successfully mitigates the limitations of conventional KD methods. The first stage leverages a momentum-based feature bank and adaptive weighting to stabilize relational knowledge transfer,and the second stage enables independent embedding optimization with classifier parameter sharing. Extensive experiments across various benchmarks demonstrate that TC-Face not only improves accuracy by up to 5% TAR on challenging datasets,such as IJB-C,but also accelerates training and maintains efficiency suitable for mobile deployment. Compared with prior state-of-the-art KD methods,our approach achieves superior stability,faster convergence,and more discriminative embeddings,thus setting a new standard for lightweight FR training. In summary,TC-Face bridges the gap between large,high-accuracy teacher models and lightweight,deployable student models,providing a practical solution for real-world face recognition systems constrained by computational resources. Future work may extend this framework to other metric learning tasks,such as person re-identification or fine-grained visual categorization,where structured relational knowledge and decoupled training could similarly enhance lightweight model performance.
Point cloud data has become a fundamental modality for human body modeling and pose estimation because of their explicit and direct representation of 3D geometric structure. In contrast to image-based or volumetric methods,point clouds preserve raw geometric information without discretization or projective distortion,making them especially suitable for capturing detailed surface characteristics. However,in practical acquisition scenarios,such as those using consumer-grade LiDAR sensors or depth cameras,point clouds often suffer from substantial local missing regions caused by sensor viewpoint limitations,object self-occlusions,and environmental obstructions. These incompleteness issues severely degrade the completeness and accuracy of reconstruction,posing substantial challenges for downstream tasks such as animation,virtual try-on,or motion analysis. Moreover,as unstructured data with irregular sampling,point clouds lack explicit connectivity information and exhibit permutation invariance. This inherent nature complicates the learning of continuous surface geometry and topological connectivity of the human body,which is essential for recovering plausible shapes and joint kinematics. Consequently,many existing approaches produce results with structural distortions,loss of fine-grained details,or inaccurate pose predictions,especially around articulated regions such as elbows,knees,and shoulders. Although recent deep learning models have made progress in processing point sets,they still struggle to maintain structural coherence and geometric fidelity when handling highly incomplete inputs under complex articulations. Therefore,methods that can infer missing geometry with strong contextual awareness and robust priors on human body shape and mobility are still needed.
To address these issues,we propose a novel method for human body reconstruction and point cloud completion by leveraging the topological prior of the skinned multiperson linear (SMPL) model. Our approach begins with a multiscale feature extraction module that incorporates per-point semantic awareness,enabling the network to capture fine-grained details and broader contextual information. This module employs a combination of local geometric feature learners and hierarchical aggregation layers to enhance the understanding of structural patterns at various scales. By integrating point-wise semantics during the feature learning process,it effectively encodes biological constraints into the representation,thereby generating more discriminative and context-aware point cloud features that better preserve the natural articulation and proportionality of human body shapes. Based on this enriched feature representation,the parametric mesh topology of the SMPL model guides the prediction of an initial deformable human skeleton. This step effectively regularizes the structural reasoning in missing or incomplete regions by introducing strong anatomical constraints,considerably alleviating topological ambiguities caused by severe data incompleteness,such as occlusions or large-scale scanning artifacts. The parametric model acts as a structural template that ensures biomechanically consistent predictions,even when large portions of the input are absent. As a result,the proposed framework maintains robust performance under challenging conditions,producing plausible human body proportions and pose-dependent shape variations that align with realistic anatomical properties. Furthermore,we design a vertex offset-aware refinement module that learns spatial residuals between the input sparse point cloud and the vertices of the predicted mesh through fine-grained alignment. This module operates in a hierarchical manner,progressively correcting geometric discrepancies across multiple resolution levels. It utilizes a cascaded residual learner that first handles large-scale deformations and then gradually focuses on recovering subtle surface details,such as muscle curves,soft-tissue contours,and joint depressions,which are often lost in coarse reconstruction stages. By iteratively aligning the predicted mesh with the observed points,our method achieves precise reconstruction of detailed surface geometry with remarkably improved geometric fidelity. The final output is a high-fidelity and anatomically plausible human body model,even when reconstructed from highly incomplete or noisy input data.
Experimental results demonstrate that the proposed method consistently outperforms existing approaches across the SURREAL and AMASS datasets,highlighting its effectiveness and broad applicability. On the SURREAL dataset,the method reduces the vertex-to-vertex reconstruction error by 4.6%,setting a new state-of-the-art with a remarkable pose estimation accuracy of 16.2 mm. This improvement not only reflects enhanced precision in capturing complex human motions but also indicates stronger alignment with ground-truth data. Similarly,on the AMASS dataset,the approach achieves a considerably lower Chamfer distance compared with current point cloud completion techniques,underscoring its ability to generate more geometrically consistent and structurally accurate reconstructions. These outcomes across diverse datasets further validate the strong generalization capacity and robustness of the proposed framework,suggesting its potential for real-world applications where reliability under varying conditions is critical.
In conclusion,this work demonstrates that the effective integration of SMPL’s topological prior into the point cloud reconstruction pipeline leads to remarkable improvements in reconstruction quality. This approach proves especially advantageous when dealing with unstructured and highly incomplete input data,where traditional methods often struggle to produce plausible results. By leveraging the strong semantic and structural constraints provided by the SMPL model,our method enhances detail recovery,ensures improved shape consistency,and reduces artifacts in challenging scenarios. These findings highlight the value of incorporating semantic priors in geometric reconstruction tasks and suggest promising directions for future research in robust 3D understanding and generation.
Reconstructing clothed human bodies from a single perspective-distorted image is a fundamental yet challenging problem in computer vision,with broad applications in virtual try-on,digital human creation,AR/VR content generation,and human-computer interaction. Although recent advances in monocular human reconstruction have achieved impressive progress,most existing approaches implicitly assume orthographic or weak-perspective projection models. These simplified assumptions neglect the actual imaging geometry of real cameras,reducing depth to a single global scaling factor and failing to capture the inherently nonlinear spatial variations caused by strong perspective distortion. Consequently,the reconstructed geometry often suffers from inaccurate limb proportions,inconsistent body scale,and distortion-dependent artifacts,particularly when the subject is close to the camera or occupies a non-central region of the image. Such limitations significantly undermine the applicability of current methods to real-world scenarios where perspective effects are inevitable.To address these shortcomings,This study proposes PerRec,a novel perspective-aware reconstruction framework specifically designed for recovering high-fidelity clothed human geometry from a single perspective-distorted RGB image.
The key idea of PerRec is to explicitly account for perspective-induced depth ambiguity and projection nonlinearity through a combination of distortion decoupling,geometric field representation,and pseudo-multiple-view supervision. Unlike existing models that attempt to implicitly learn perspective from data,PerRec introduces structural priors and tailored network designs that directly target the characteristics of perspective distortion,thereby enabling a more accurate and physically plausible 3D inference.The proposed framework contains four major components. First,a distortion decoupling module is introduced to separate image position from the underlying perspective distortion pattern. The method effectively normalizes spatial variance while preserving layout-dependent cues by uniformly partitioning the input image and generating virtual viewpoint transformations for each region. This mechanism allows the model to effectively capture how projection distortion changes across the field of view,mitigating the tendency of conventional models to collapse depth variations into oversimplified estimates. Second,we design a multi-scale attention hourglass network (MA-HGNet) to robustly extract local and global features under varying degrees of distortion. The network integrates a multi-level hourglass structure with channel attention mechanisms to refine fine-grained patterns,such as clothing folds,body boundaries,and high-curvature regions that are particularly susceptible to distortion. MA-HGNet demonstrates enhanced sensitivity to spatially localized distortions while maintaining stable global shape perception compared with standard convolutional backbones. Third,we convert the SMPL mesh into a Fourier occupancy field (FOF) to effectively capture continuous geometry and enhance high-frequency surface details. This representation encodes occupancy values using Fourier basis functions,enabling the network to model sharp transitions,intricate clothing topology,and subtle geometric variations that are difficult to represent with parametric models alone. The proposed model unifies the advantages of parametric priors and implicit fields,resulting in substantially detailed and expressive reconstructions. Finally,considering that a single image inherently lacks sufficient viewpoint diversity,we introduce a pseudo-multi-view module that synthesizes multiple virtual observations during training. These synthesized views act as implicit geometric constraints,guiding the network to learn distortion-invariant features and improving its ability to reason about occluded or heavily distorted regions. This strategy significantly boosts reconstruction robustness without requiring additional input views or multi-camera setups.
Extensive experiments are conducted on the Thuman2.0 and CustomHumans datasets,covering controlled and in-the-wild settings with diverse degrees of perspective distortion. Quantitative evaluations demonstrate that PerRec substantially outperforms state-of-the-art monocular reconstruction methods across multiple metrics,including chamfer distance,normal consistency,and point-to-surface error. Specifically,improvements of approximately 60%–80% are observed on average,highlighting the effectiveness of the proposed design in addressing global and local reconstruction quality. Qualitative comparisons further show that PerRec produces a stable body scale,highly realistic limb proportions,and detailed clothing geometry,particularly in strongly distorted images where competing methods typically fail.
The contributions of this study are threefold. First,this study provides a comprehensive analysis of the limitations of existing projection assumptions in monocular reconstruction and establishes perspective distortion as a core challenge in real-world 3D human modeling. Second,this study proposes a unified reconstruction framework that integrates distortion decoupling,implicit field representation,and pseudo-multi-view learning,offering a principled and effective solution for handling perspective effects. Third,this study demonstrates that accurate and high-fidelity clothed human reconstruction from a single perspective-distorted image is achievable without relying on multi-view capture systems or camera calibration. Overall,this study offers a practical and extensible perspective-aware solution for real-world human reconstruction,bridging the gap between theoretical projection modeling and data-driven 3D inference. The proposed approach not only enhances geometric accuracy under challenging perspective conditions but also expands the applicability of monocular reconstruction methods to everyday images captured by mobile devices and consumer cameras. Results indicate that perspective distortion,long regarded as a major obstacle for monocular reconstruction,can be effectively resolved by combining structural modeling cues with carefully designed learning-based strategies.
Currently,the speech-driven 3D talking face generation technology has considerably matured,particularly with recent models based on vector quantized variational autoencoders (VQ-VAE),which achieve highly vivid facial expression generation by quantizing facial motion features. However,existing methods still suffer from limitations in facial reconstruction accuracy and emotional control stability. On the one hand,the facial detail reconstruction capability of VQ-VAE models requires further improvement. On the other hand,emotion guidance strategies in existing models remain relatively simplistic,failing to fully utilize multimodal information beyond speech,resulting in generated facial expressions that lack realism.
Inspired by the structures of conditional variational autoencoders and VQ-VAE-2,this study proposes the DecTalk3D model,a hierarchical decoupling method within the VQ-VAE framework. Facial features are decoupled into high-level and low-level features. Identity vectors and descriptive text are introduced as external conditions to decouple the high-level features. Afterward,the decoupled high-level features are used as internal conditions,working together with the two external conditions (identity vector and text) to further decouple the low-level features. This hierarchical decoupling mechanism aims to improve the facial reconstruction quality and the stability of emotional expression. During generation,speech and text information jointly guide facial generation in different ways: the text is encoded by the CLIP text encoder to leverage its rich semantic information for holistic expression control,while the speech is encoded by Wav2Vec 2.0 to precisely control lip movements and other subtle expressions. The model generates accurate facial motions by integrating multimodal information at different levels.
Experiments were conducted on the multi-view emotional audio-visual dataset(MEAD),3DMEAD and TA-MEAD datasets and included quantitative evaluation,visual analysis,and ablation studies. The proposed model was compared with FaceFormer,CodeTalker,FaceDiffuser,and ProbTalk3D. In terms of overall facial reconstruction accuracy,the proposed model achieved a 5.9% lower mean vertex error (MVE) than ProbTalk3D—which also incorporates emotional input—demonstrating that explicit emotion modeling effectively enhances geometric consistency and preserves facial structural accuracy,particularly for complex emotions. In terms of lip motion accuracy (LVE),the proposed model ranked second,with a 16.7% gap compared with the top-performing FaceDiffuser,while still outperforming FaceFormer and ProbTalk3D. The limited improvement may stem from the hierarchical decoupling approach,which,while decoupling emotional conditions,offers restricted decoupling for the mouth region,affecting detailed performance. In upper-face dynamic deviation (FDD),the proposed model significantly surpassed traditional methods even though it is inferior to ProbTalk3D,reflecting its ability to express emotional intensity. Moreover,the model showed notable advantages in distribution-based quality metrics: it achieved optimal mean estimation error (MEE) and coverage error (CE),with MEE 4.1% lower than ProbTalk3D and 58.4% lower than CodeTalker and CE 8.2% lower than ProbTalk3D and 61.6% lower than CodeTalker. This result indicated that hierarchical decoupling enhances distribution concentration and coverage of real samples,making generated facial motions close to real faces. However,the model had the lowest diversity score,highlighting a trade-off where hierarchical decoupling ensures semantic consistency but constrains variation—beneficial for emotion accuracy yet potentially limiting naturalness. Although traditional methods showed orders-of-magnitude differences in MVE and FDD due to lacking emotional input,and ProbTalk3D excelled in FDD and Diversity but lagged in accuracy,the proposed model balances high-precision reconstruction with superior distribution estimation through hierarchical decoupling and multimodal modeling.
Experimental results demonstrate that the proposed hierarchical decoupling-based method produces accurate and stable facial reconstructions,excelling in maintaining facial geometric consistency and expressing complex emotions. The model’s key innovation lies in its layered feature decoupling within the VQ-VAE framework,using identity vectors and text as external conditions and high-level features as an internal condition,which significantly enhances reconstruction quality and emotional controllability. This framework offers practical value for HCI,digital humans,and animation. Quantitative evaluation shows superior performance in MVE,LVE,and FDD,alongside stable MEE and CE scores,confirming the model’s ability to effectively integrate multimodal information for generating realistic expressions. Ablation studies validate the critical roles of the hierarchical feature decoupling and the synergy between internal and external conditions. However,limitations remain: generated lip movements are less pronounced compared with real speech,the current two-level feature decoupling is relatively simplistic,and the diversity and subtlety of emotional expressions require further improvement. Future work will focus on improving lip-sync quality,exploring finer-grained decoupling dimensions for differentiated guidance,and enhancing expression diversity while preserving accuracy. The source code is available on GitHub (https://github.com/chen114514sheng/DecTalk3D) and archived on ScienceDB (https://doi.org/10.57760/sciencedb.j00240.00095).