Home Latest Articles
Latest Articles
  • Qi He, Zhengyuan Zang, Zengzhou Hao
    Journal of Image and Graphics. 2026, 31(9): 3412-3426.
    Objective

    Accurate short-term prediction of satellite cloud imagery is crucial for modern meteorological operations,including severe weather nowcasting,aviation safety,and disaster preparedness. However,this task remains profoundly challenging due to the inherently nonlinear and chaotic dynamics of atmospheric motion. Cloud systems undergo complex spatiotemporal transformations,such as genesis,dissipation,deformation,rotation,merging,and splitting,which are difficult to capture using conventional forecasting techniques. Traditional approaches,including optical flow and block-matching methods,primarily estimate displacement vectors based on local brightness constancy or regional correlation. While computationally manageable,these methods often fail to model the nonstationary and nonrigid transformations characteristic of real cloud evolution,leading to considerable error accumulation over extended prediction horizons.

    Method

    To overcome these challenges,this study proposes CloudPredUNet,a novel and lightweight neural network architecture for satellite cloud image sequence prediction. The model is built upon a U-Net skeleton,renowned for its effective encoder-decoder structure with skip connections,which is well-suited for dense prediction tasks. CloudPredUNet introduces three core innovative modules designed to enhance global feature aggregation,detail reconstruction,and long-range spatiotemporal dependency modeling,respectively. First,in the encoder pathway,we design a frequency domain self-attention (FDSA) module. Instead of applying global Fourier transforms,which can be sensitive to noise and computationally heavy for high-resolution images,our module operates on localized image patches. The input features are projected and split into query (Q),key (K),and value (V) components. The Q and K tensors are partitioned into nonoverlapping patches. A 2D fast Fourier transform (FFT) is applied to each patch,transitioning the representation into the frequency domain. A Hadamard (element-wise) product between the spectra of Q and K patches is then performed,effectively computing cross-correlation in the frequency domain,a process equivalent to spatial convolution but with considerably lower computational complexity (O(HW log (HW))). The result is transformed back to the spatial domain via inverse FFT,and then gated with the V component through element-wise multiplication. This design enables efficient aggregation of global contextual information across the entire image while preserving local texture details,addressing the limited receptive field of standard convolutions. Second,the decoder pathway incorporates a multiscale spatial attention (MSA) module to recover high-resolution details and suppress irrelevant background regions during upsampling. This module employs a series of cascaded depthwise separable convolutions with progressively increasing dilation rates (e.g.,d = 1,3,5,7). This multibranch structure captures features at multiple scales,mitigating the gridding artifacts common in single-dilation dilated convolutions. The outputs from all branches are concatenated and processed through a spatial attention mechanism. This mechanism generates an attention map by applying max-pooling and average-pooling across the channel dimension,followed by a small multilayer perceptron (MLP) and a sigmoid activation. This map highlights structurally important regions (e.g.,cloud edges,cores) and attenuates uniform areas,thereby refining the reconstruction quality. Third,a dedicated temporal mapper is introduced between the encoder and decoder to model the evolution of features over time explicitly. Its core component is a spatiotemporal feature extraction (SFE) module. This module first merges the time and channel dimensions of the encoded feature sequence. The merged features are then split into four subsets. One subset is preserved,while the other three are processed by distinct convolutional branches: a standard 3 × 3 depthwise convolution,a 1 × 11 horizontal strip convolution,and an 11 × 1 vertical strip convolution. This heterogeneous receptive field design allows the model to capture isotropic cloud expansion as well as anisotropic motion patterns (e.g.,shear,directional advection). The outputs are fused and further refined by a channel recalibration block,which employs channel-wise max and average pooling followed by an MLP to weight the importance of different temporal channels adaptively,enhancing the modeling of long-range dependencies. The complete CloudPredUNet model is trained end-to-end using the mean squared error (MSE) loss function,optimizing it to minimize pixel-wise prediction error directly.

    Result

    Extensive experiments were conducted on a dataset constructed from the FY-2D geostationary meteorological satellite,covering the region 26° E—146° E,60° S—60° N. The dataset was temporally and spatially partitioned to ensure no overlap between training,validation,and independent test sets. CloudPredUNet was evaluated against a comprehensive suite of state-of-the-art models,including traditional optical flow,autoregressive models (ConvLSTM,PredRNN,MIM),and nonautoregressive models (MMVP,SimVP,TAU). Results demonstrate the superior performance and remarkable efficiency of CloudPredUNet. In terms of model efficiency,CloudPredUNet is exceptionally lightweight,containing only 0.101 million parameters and requiring approximately five GFLOPs per forward pass. This represents a drastic reduction compared with competitors; for instance,its parameter count is less than one-fifth of the next most efficient model (MMVP),and its computational cost is nearly two orders of magnitude lower than ConvLSTM. Regarding prediction accuracy,CloudPredUNet achieved the best scores across all four primary evaluation metrics on the test set: MAE of 8.393,MSE of 188.922,SSIM of 0.825,and PSNR of 25.367. This consistent outperformance indicates its exceptional capability in minimizing prediction error while maintaining high structural fidelity and image quality relative to the ground truth. A critical test for any sequence prediction model is its performance in multistep forecasting. CloudPredUNet demonstrated outstanding robustness against error accumulation. In a five-frame-ahead prediction task,it secured the lowest MAE for every single predicted frame. Notably,by the fifth frame,its MAE (10.589) was approximately 2.4% lower than that of the second-best model,MIM (10.852),underscoring its superior stability for longer-term predictions. Ablation studies systematically validated the contribution of each proposed module. Removing any of the three core modules (FDSA,MSA,SFE) led to a measurable drop in performance. The complete model provided an 11.63% improvement in MAE over a baseline U-Net without these modules,confirming their effectiveness and complementary nature. Qualitative visual analysis further reinforced these quantitative findings. In complex meteorological scenarios such as cloud growth,cloud merging,and the preservation of a typhoon eye’s characteristic structure,CloudPredUNet generated predictions that were visually more coherent,detailed,and consistent with the actual evolution compared with other models. It successfully maintained sharp boundaries and realistic textures where other models produced blurring,fragmentation,or loss of critical features.

    Conclusion

    This study addresses key limitations in satellite cloud image prediction,namely,the modeling of nonlinear cloud dynamics,the efficient capture of global spatial dependencies,and the mitigation of multistep error accumulation,by introducing CloudPredUNet. The core innovations lie in its integrative design: The frequency domain self-attention module enables efficient global context modeling; the multiscale spatial attention module enhances detail reconstruction; and the temporal mapper with its spatiotemporal feature extraction module strengthens long-range evolutionary pattern learning. The experimental outcomes affirm that CloudPredUNet successfully bridges the gap between high predictive accuracy and operational efficiency. It establishes a new state-of-the-art on the FY-2D benchmark while being orders of magnitude more parameter- and compute-efficient than prevailing methods. This combination makes it a highly promising candidate for real-time,operational short-term cloud forecasting systems where computational resources may be constrained. Future research will focus on several avenues to advance this work toward greater practical utility and robustness,including the following: 1) expanding the spatiotemporal diversity of training and testing datasets to evaluate and improve the model’s generalizability across different seasons,climatic zones,and synoptic conditions; 2) investigating multimodal fusion frameworks that incorporate ancillary meteorological data fields (e.g.,wind vectors,humidity,atmospheric pressure) to inject more physical constraints into the learning process,potentially improving the prediction of physically coherent evolutions; and 3) further refining the frequency-domain processing mechanism,potentially by integrating adaptive filters or multiresolution analysis,to better handle high-frequency noise and further enhance the reconstruction of subtle cloud microstructures. Through these efforts,we aim to evolve CloudPredUNet from a high-performing research prototype into a reliable and versatile tool for operational meteorology.

  • Zhaoyang Wang, Yun Ge, Mengqi Wu, Jinliang Chen
    Journal of Image and Graphics. 2026, 31(9): 3427-3440.
    Objective

    Remote sensing image object detection is fundamental to the automatic interpretation of aerial and satellite imagery. It aims to identify objects of predefined categories from given images and regress the localization information for each object. Remote sensing image object detection technology has broad applications in practical scenarios,such as intelligent surveillance,urban planning,port management,and military surveys. In contrast to natural images,where objects are typically horizontally aligned and backgrounds are relatively simple,remote sensing images are captured from a top-down perspective using spaceborne or airborne sensors,offering a much broader field of view. Consequently,objects in remote sensing images are often distributed in arbitrary orientations,and the background information is more complex and diverse. Although some horizontal object detectors perform well on natural images,horizontal bounding boxes (HBB) cannot describe the orientation of objects and often introduce unnecessary background redundancy. Moreover,in densely arranged scenes,the overlap between HBBs and neighboring boxes can be substantial,which may lead to missed or false detections. By contrast,oriented bounding boxes (OBB) can represent objects more accurately and are better suited for object detection in dense scenes. However,during regression,the skew intersection over union (SkewIoU) is difficult to compute and nondifferentiable,preventing the model from optimizing its predicted OBB parameters via backpropagation. To address this,current mainstream methods convert oriented bounding boxes into 2D Gaussian distribution and compute the distance between these distributions as the regression loss. This approach approximates the calculation of SkewIoU while remaining differentiable,and such methods are collectively referred to as Gaussian distance loss (GDL). However,given the periodic nature of angles,GDL-based methods exhibit discontinuities in angle regression. Furthermore,they suffer from imbalanced angle perception across targets with varying aspect ratios. The interplay between these issues considerably degrades the detection performance of the model. In this study,a detection algorithm named joint angle continuous representation and perceptual balance (JCPB) is proposed to mitigate the mentioned challenges.

    Method

    To address the angle boundary discontinuity problem in oriented object detection,this study proposes an angular vector decomposition method (AVDM),which decomposes angle parameters into an angle vector representation. To eliminate discontinuous regions inherent in traditional angular representations,this vector representation is further extended to encompass the entire unit circle plane,ensuring continuous variation of the angular vector during OBB regression. Aiming at the periodic discrepancy issue in angle loss calculation between rectangular and square targets,this study introduces an innovative shape aware coefficient method. Without increasing the encoding length of the angular vector,a shape-aware coefficient dynamically adjusts its value on the basis of the target’s aspect ratio,thereby modulating the change period of the angular loss function. To address the problem of imbalanced angle perception,we proposed the angle perceptual balance strategy,which adaptively calculates loss weighting coefficients based on the aspect ratio of targets and adjusts their application timing according to the SkewIoU value. This enhances the model’s angle perception capability for low-aspect-ratio targets during the later stages of training. Finally,on the basis of a 2D Gaussian distribution representation method,we further enhance the model’s detection performance by joint angle continuous representation and perceptual balance.

    Result

    This study conducted comparative experiments using FCOS and RetinaNet detectors on the DOTA,DIOR-R,and HRSC2016 datasets to validate the effectiveness of the proposed methods. Furthermore,comparisons with current mainstream models on the DOTA dataset demonstrate the superiority of the proposed model. Evaluation metrics include average precision at 75% (AP₇₅) intersection over union threshold,mean average precision (mAP),and parameters. On the DOTA dataset,incorporating AVDM improves AP₇₅ and mAP by 2.47% and 1.03% on average compared with the baseline model. Further incorporating APBS resulted in additional average improvements of 0.84% and 0.43% for AP75 and mAP,respectively. On the DIOR dataset,integrating AVDM achieved average gains of 1.37% and 0.95% for AP75 and mAP. Further incorporating APBS resulted in additional improvements of 0.82% and 0.36% for AP75 and mAP,respectively. On the HRSC dataset,the AVDM method achieved average gains of 5.63% and 3.86% for AP75 and mAP,respectively. After incorporating APBS,AP75 and mAP improved by 0.44% and 0.49% on average. Experimental results across multiple datasets demonstrate that APBS effectively enhances the model’s angular learning capability for small aspect ratio targets,while AVDM achieves continuous angular representation,effectively resolving boundary discontinuity issues. Compared with other encoding methods,AVDM introduces only 0.04 M additional parameters,achieving continuous angular representation and periodic adaptation to angular losses for rectangular and square targets at minimal parametric cost. Experiments on the HRSC dataset show that detection accuracy for slender objects is highly susceptible to boundary discontinuity issues,while angle perceptual imbalance has a relatively weaker impact. Finally,comprehensive comparisons with current mainstream remote sensing image orientation object detection methods on the DOTA dataset demonstrate that our approach achieves an mAP of 73.78%,outperforming other methods. Moreover,it achieves optimal accuracy on small vehicles,ships,tennis courts,storage tanks,and roundabouts categories. Additionally,when using Swin-Transformer as the backbone network,the model’s performance is further improved,reaching an mAP of 74.71%.

    Conclusion

    Extensive experimental results demonstrate that the proposed AVDM achieves continuous angular representation,while APBS effectively enhances the model’s perceptual capability for small aspect ratio targets. Building upon this foundation,JCPB remarkably improves the overall performance of the detection model in complex remote sensing scenarios by integrating continuous angular representation with perceptual balance.

  • Muwei Jian, Yukun Ling, Haoran Zhang, Linsong Zhang, Jiajun Ma
    Journal of Image and Graphics. 2026, 31(8): 2689-2704.

    The National Highway Traffic Safety Administration asserted that distracted driving has become one of the leading causes of motor vehicle accidents,resulting in severe economic losses and casualties worldwide. Distracted driving refers to behaviors in which a driver’s attention shifts from the primary driving task to unrelated activities,impairing visual perception,cognitive awareness,decision-making,and operational control. Common distractions,such as calling,texting,eating,and conversing,are often covert,dynamic,and difficult to detect using a single sensing modality. Therefore,the development of a framework capable of robust perception,adaptive cognition,and real-time reasoning remains a major challenge in intelligent driving research. Embodied intelligence,a core concept of intelligent manufacturing,emphasizes deep interaction and tight coordination between intelligent agents and the physical environment. Moreover,embodied intelligence enables the emergence and evolution of cognition through continuous perception-action coupling,endowing intelligent systems with adaptive behavior generation,self-optimization,and environmental awareness. In recent years,embodied intelligence has been widely applied in bionic robotics,unmanned systems,and intelligent vehicles,driving innovation and theoretical progress in human-machine collaboration and cognitive autonomy. Building upon this theoretical foundation,this study systematically reviews the key technological pathways for distracted driving detection from the perspective of embodied cognition,focusing on frontier research trends,such as large-scale multimodal models,domain adaptation,and continual learning. This study introduces a hierarchical theoretical framework consisting of five interrelated layers that together form a closed-loop structure of perception,cognition,and execution by integrating the cognitive logic of embodied intelligence with the dynamic characteristics of complex traffic environments. The physical layer serves as the foundation,responsible for multimodal data acquisition through cameras,infrared sensors,microphones,and physiological sensors,ensuring a comprehensive coverage of driver behaviors and environmental conditions. The communication and fusion layer handles data transmission,synchronization,and multimodal fusion across heterogeneous devices,enabling reliable multi-source collaboration and adaptive information exchange under varying network and vehicular conditions. The data and knowledge layer manages and semantically interprets the collected information,constructs knowledge graphs of driver states and contextual events,and supports the integration of prior driving knowledge for enhanced reasoning. The cognition and reasoning layer applies advanced learning and inference mechanisms to interpret multimodal inputs,recognize driver states,predict behavioral intentions,and formulate context-aware strategies for distraction mitigation. Finally,the capability and application layer functions as the executive component,connecting cognitive outputs to real-world interaction,and supports human-machine communication through digital human interfaces,semantic prompts,and feedback control mechanisms for adaptive intervention and behavioral regulation. Together,these five layers establish a theoretical structure that transitions from passive perception to active cognition and embodied execution,forming an integrated mechanism for closed-loop understanding and intervention. Unlike existing detection systems that primarily focus on recognition and alerting,the proposed embodied-intelligence-based framework highlights a conceptual transformation toward human-centered adaptability,self-evolving learning,and knowledge-driven decision-making. This study provides a unified and extensible foundation that bridges multimodal perception,cognitive reasoning,and control feedback under the embodied intelligence paradigm as a theoretical contribution. The proposed framework is conceptual rather than engineering-oriented,serving as a blueprint for future empirical validation,experimental simulation,and real-world deployment. Meanwhile,the implementation of the proposed framework still faces considerable challenges. From the technical perspective,the constrained computational resources of in-vehicle terminals limit the deployment of complex deep learning models. Insufficient multimodal sensing and synchronization hinder the scalability of perception systems. Meanwhile,inference hallucination in large-scale reasoning models introduces uncertainty that threatens decision reliability and system safety. From the application perspective,ethical and legal controversies related to driver monitoring,data privacy,and algorithmic transparency remain unresolved,while high economic costs continue to impede large-scale deployment and commercialization. Future research on lightweight optimization,trustworthy and interpretable inference,distributed learning,and regulatory standardization is necessary to address these challenges. Such efforts are expected to ensure that embodied-intelligence-based distracted driving detection can evolve toward safe,efficient,transparent,and socially sustainable intelligent driving ecosystems. Overall,this study presents a systematic,forward-looking,and theoretically grounded exploration of embodied-intelligence-driven distracted driving detection. This study redefines the principles of human-machine collaboration,establishes a pathway for embodied cognitive understanding,and provides theoretical guidance for the evolution of next-generation intelligent and safe driving systems by consolidating fragmented advances in multimodal perception,driver behavior modeling,and adaptive decision-making into a unified theoretical architecture.

  • Chunyi Li, Jianbo Zhang, Jiahao Xiao, Bowen Yan, Shengyu Guo, Tongrui Ye, Weisi Lin, Guangtao Zhai
    Journal of Image and Graphics. 2026, 31(8): 2705-2730.

    Embodied intelligence refers to agents that sense,reason,act,and learn through a tight “body-environment-task” feedback loop. It is widely regarded as the most promising route to general-purpose artificial intelligence (AI). Paradoxically,although model parameters,training data,and compute budgets have grown by four orders of magnitude in only five years,the evaluation ecosystem remains fragmented and largely irreproducible. The same functional capability (e.g.,“pick-and-place”) is formalized as object-detection mean average precision in one paper,as 3D intersection over union in a second,as task success rate in a third,and as human preference score in a fourth,with absolute gaps that exceed 10%—15% among “state-of-the-art” results. Environments,random seeds,physics parameters,sensor noise models,and success criteria are rarely open-sourced,and thus,the community cannot tell whether an apparent improvement originates from algorithmic innovation,data scale,evaluation cherry picking,or simple benchmark over-fitting. This uncertainty has become a critical bottleneck for scientific progress and industrial adoption. To address the gap,we conduct a systematic survey of 100+ papers published in IEEE/CVF Conference on Computer Vision and Pattern Recognition,IEEE International Conference on Robotics and Automation,Conference on Neural Information Processing Systems,Robotics: Science and Systems,International Conference on Learning Representations,International Journal of Robotic Research,and Journal of Machine Learning Research between 2020 and 2025. Then,we present the first holistic analysis of embodied AI evaluation. We organize the landscape into a three-tier pyramid: static datasets,simulation benchmarks,and real-robot trials. Each tier trades off cost,controllability,and ecological validity. Across all tiers,we decompose evaluation space into four tightly coupled stages: perception,cognition,decision,and execution. Then,we identify fine-grained capability dimensions along with 20+ quantitative metrics. For every dimension,we analyze the following: 1) the underlying methodological assumptions,2) the boundary conditions under which the metric is valid,3) the intrinsic limitations (e.g.,sim-to-real physics mismatch,dataset bias,human-label variance),and 4) the cost-credibility curve that determines when a researcher should ascend from static to simulation,and finally to the real world.All evaluations in this study are based on a theoretical “perception-cognition-decision-execution loop” framework. In embodied AI,perception is the process by which an agent converts raw,asynchronous,and often noisy signals from heterogeneous physical sensors,such as RGB-D cameras,LiDAR,tactile skins,joint encoders,force/torque,and inertial measurement unit units,into a registered,time-aligned digital twin that supplies metric 3D geometry and semantic labels. Cognition is then built on this torrent to distill structured,memory-enabled,and generalizable world models that encode object affordances,physical common sense,and relational and causal links,grounding free-form human language into the resulting latent space. Decision-making combines these world models with an explicit task goal and searches across symbolic,kinematic,and dynamic levels to output a coherent,risk-aware,and resource-efficient plan that is expressed as a hierarchy of subgoals,trajectories,and low-level set points. Finally,execution transforms this plan into real-world mechanical energy through high-rate feedback controllers that track desired motions or forces while compensating for actuator nonlinearities,shock,thermal limits,and human safety constraints,closing the perception-cognition-decision-execution loop that defines embodied AI. In contrast with humans,whose perception-cognition-decision-execution loop is seamlessly integrated,currently available embodied agents suffer structural flaws in every stage. Consequently,evaluation must decouple the four blocks and score accuracy,error,and latency separately to pinpoint bottlenecks. Static tests,i.e.,“written exams”,run on a GPU with a few megabytes of images or point clouds in minutes,but erase dynamics and physics. Simulation,i.e.,the “job interview”,adds interactive physics in tens of seconds but still drifts away from reality. Real robot trials,i.e.,“practical tests”,cost thousands,take days to deploy and minutes per run,but expose every sensing drift,prediction error,or control delay,offering the ultimate ground truth before field deployment. Therefore,by integrating the three-tier pyramid of static test-simulation-real robot evaluation into the closed perception-cognition-decision-execution loop,this work delivers the first systematic quantification of coverage,cost,and accuracy trade-offs across more than 100 embodied AI benchmarks,establishing a reproducible,traceable,and extensible evaluation baseline that not only exposes the exact stage at which state-of-the-art systems fail,but also provides a principled road map for allocating research budgets,standardizing protocols,and certifying safety before real-world deployment,marking a pivotal transition from empirical competition to scientific measurement,and accelerating the field’s development toward trustworthy,deployable,and general-purpose embodied intelligence. The detailed static-dynamic-real embodied intelligence evaluation can be accessed at https://opencompass.org.cn/embodied-intelligence.

  • Rui Liu, Wenguan Wang, Jun Wang, Yi Yang
    Journal of Image and Graphics. 2026, 31(8): 2731-2773.

    Embodied artificial intelligence aims to endow agents with the ability to perceive,reason,and act through continuous interaction with the physical world. In contrast with disembodied intelligence,which mostly operates on static data and symbolic tasks,embodied intelligence must handle sensorimotor coupling,environmental uncertainty,long-horizon decision-making,and real-time control in open-world settings. These requirements expose the limitations of traditional perception-planning-control pipelines. Conventional robotic systems are often built from cascaded modules for visual perception,language understanding,state estimation,task planning,and low-level control. Although such pipelines benefit from modular engineering and interpretability,they frequently suffer from error accumulation across stages,brittle generalization to novel tasks and scenes,and heavy dependence on manually designed interfaces,reward functions,and domain-specific annotations. Recent advances in large-scale foundation models have opened a new path for embodied intelligence by enabling the integration of visual grounding,language reasoning,and action generation within a shared model space. This trend has led to the rapid emergence of vision-language-action (VLA) models,which aim to translate multimodal observations and task instructions directly into executable actions or action-conditioned policies,narrowing the gap between semantic understanding and physical execution.This survey presents a systematic review of VLA models in embodied intelligence from the perspectives of historical development,technical architecture,system design,training strategies,evaluation protocols,and future trends. We first revisit the technological trajectory that made VLA possible. The evolution of large vision models and large language models has substantially improved semantic representation,multimodal alignment,and knowledge transfer,while progress in robot learning has highlighted the importance of scalable policy learning from heterogeneous data sources. Within this context,VLA models can be considered the convergence of three lines of research: multimodal foundation models for perception and reasoning,imitation and reinforcement learning for embodied decision-making,and robot action modeling for continuous control. Their significance lies not merely in adding language prompts to robot policies,but also in establishing a unified interface through which high-level task intent,environmental perception,and low-level action generation can be jointly optimized.A central objective of this survey is to clarify the internal structure of VLA systems. We analyze the key technical modules that constitute current VLA pipelines. On the perception side,VLA models rely on visual encoders to transform raw observations into compact and task-relevant representations. These observations may include RGB images,depth maps,multi-view input,temporal image sequences,or structured spatial representations,such as 3D or bird’s-eye-view features. The choice of visual representation directly influences the model’s capability to capture object semantics,spatial relations,affordances,and scene dynamics. On the language side,textual encoders provide instruction understanding,goal specification,contextual reasoning,and cross-modal alignment. Language not only describes tasks but also offers a flexible interface for compositional generalization,abstract planning,and human-robot interaction. The crucial problem of action representation lies between perception and control. Compared with visual and textual tokens,action signals are continuous,temporally dense,and strongly constrained by embodiment. Therefore,we pay special attention to action tokenization and decoding,reviewing how actions can be represented as discrete tokens,continuous vectors,chunked trajectories,latent action codes,or generative distributions. We further compare the major decoding paradigms,including direct regression,classification after discretization,autoregressive sequence modeling,diffusion-based policy generation,and flow-based generation,and then discuss their respective trade-offs in precision,expressivity,stability,and inference speed.Beyond module-level analysis,this survey proposes a system-level taxonomy that categorizes existing VLA models into three broad classes: single systems,dual-system architectures,and hierarchical systems. Single systems integrate perception,reasoning,and action generation within a unified model,and they are attractive for end-to-end deployment,parameter sharing,and simplified optimization. However,they may experience difficulties in balancing high-level deliberation with fast control. Dual-system architectures address this issue by explicitly separating slow reasoning from fast acting,frequently coupling a deliberative module for semantic planning with a lightweight controller for reactive execution. This design is particularly relevant to robotics because reasoning and motor control operate under different temporal constraints. Hierarchical systems introduce structured intermediate abstractions,such as subgoals,skills,waypoints,object-centric plans,or symbolic task graphs,to improve modularity,interpretability,and reuse across tasks. We argue that this taxonomy provides a useful perspective for understanding the design space of VLA models and analyzing how different systems trade off generalization,efficiency,and robustness.The survey also reviews the training and evaluation ecosystem that supports VLA research. Current VLA models are typically trained through multistage pipelines that involve internet-scale visual-language corpora,simulated interactive data,teleoperated demonstrations,and real robot trajectories. Pretraining equips the model with broad perceptual and linguistic priors,while post-training and task-specific adaptation align those priors with embodiment,action semantics,and control requirements. We summarize commonly used strategies,such as supervised imitation learning,behavior cloning,instruction tuning,parameter-efficient fine-tuning,model distillation,reinforcement learning augmentation,and hybrid training paradigms that combine offline and online data. We further discuss the role of simulator-based benchmarks and real-world evaluations. Simulation enables scalable and reproducible testing under controlled settings,while real robot experiments reveal embodiment gaps,latency constraints,and deployment failures that are not fully captured by simulation alone. In reviewing the literature,we emphasize that cross-paper comparisons must be interpreted with caution because reported results are often influenced by differences in robot morphology,sensor configurations,action horizons,training data quality,evaluation metrics,hardware platforms,and implementation details.Despite their remarkable progress,VLA models still face several fundamental challenges. First,real-time inference remains a major bottleneck. Many powerful multimodal models rely on autoregressive decoding or iterative generative sampling,which may be too slow for high-frequency control. Second,action learning suffers from limited robotic data,inconsistent annotation quality,and poor cross-platform transfer,especially when embodiments differ substantially. Third,environmental generalization is far from solved. Models trained on curated datasets may degrade sharply in cluttered scenes,long-horizon tasks,dynamic environments,or safety-critical settings where perception is ambiguous and recovery from failure is required. Fourth,current VLA systems still exhibit limited causal reasoning and world modeling capabilities. Many methods align observations and instructions effectively but lack deep predictive understanding of object dynamics,interaction consequences,and long-term task structure. Fifth,evaluation remains fragmented. Existing studies often emphasize task success rate while underreporting execution frequency,safety violations,intervention cost,recovery capability,and robustness under distribution shift. Finally,broader issues of trustworthiness,including interpretability,failure diagnosis,ethical risk,and human-centered deployment,have become increasingly important as VLA systems move from laboratory demonstrations toward real applications.In the future,we argue that the next stage of VLA research will be defined by several converging directions. One is the shift from passive perception-conditioned action generation to active reasoning-driven embodied intelligence,in which agents can query,explore,verify,and adapt during execution. Another is the integration of richer world models that support prediction,imagination,and planning across time and modality. Lifelong and continual learning will also be essential for agents that are operating in evolving environments. In addition,future VLA systems are likely to expand beyond fixed single-robot manipulation toward integrated mobile manipulation,long-horizon household assistance,multi-agent collaboration,and broader open-world autonomy. Achieving these objectives will require not only stronger models,but also better data curation,more principled action representations,more realistic evaluation standards,and more reliable safety mechanisms. By synthesizing the development of VLA models from architecture to deployment,this survey aims to provide a structured reference for researchers and practitioners,and to support the advancement of embodied intelligence toward more general,robust,and trustworthy interactions in the physical world. Furthermore,a comprehensively curated list of open-source VLA algorithms,datasets,and simulation benchmarks discussed in this survey is updated at https://github.com/DefaultRui/vision-language-action-models-for-embodied-AI.

  • Xinjing Wang, Ying Gao, Yaqi Zou, Zhengyu Zhu, Chunxue Xu, Qi Zhao
    Journal of Image and Graphics. 2026, 31(8): 2774-2784.
    Objective

    Accurate and fine-grained scene understanding remains a key challenge for embodied intelligence and humanoid robotic systems,which require real-time and robust perception. In particular,when applying vision-language models to pixel-level segmentation,issues,such as localization blur and sensitivity to scale and viewpoint,often impede reliable spatial reasoning. Open-vocabulary semantic segmentation (OVSS) offers a promising solution by enabling pixel-level recognition of arbitrary textual categories without relying on exhaustive pixel-wise annotations. However,large-scale vision-language models (VLMs),such as CLIP,are primarily optimized for image- or region-level tasks,resulting in limited spatial granularity and degraded dense prediction performance. This study proposes a training-free inference paradigm that enhances the pixel-wise discriminability and engineering readiness of VLM-based OVSS frameworks without modifying pretrained model weights,thereby improving their applicability to real-time robotic perception and decision-making and addressing these limitations.

    Method

    We introduce TG-CLIP,a practical framework built on a frozen CLIP (ViT-B/16) backbone,incorporating two lightweight and complementary inference-stage operators. These operators inject text-conditioned semantic bias into local visual representations and amplify prediction robustness across scales and viewpoints. The first operator,text-guided recalibration (TGR),treats text queries as conditional semantic probes: given a set of class queries encoded into normalized query vectors,TGR computes per-location query-visual affinities and back-projects the resulting semantic signal into the local feature space. The injected signal is blended with the original visual features under a tunable injection strength and renormalized to ensure that the final logits retain a cosine-similarity interpretation. Moreover,TGR is a parameter-free forward operator that requires no pixel-level supervision or fine-tuning,solely relying on the pretrained visual and textual encoders to perform a semantic recalibration of local responses. The second operator,multi-view consistency reasoning (MCR),aggregates evidence across a compact set of resampling scales and mirrored views. MCR first resamples the input at multiple scales that are rounded to integer multiples of the model’s patch grid to avoid misalignment artifacts. Thereafter,MCR performs forward inference (optionally via an overlap-aware sliding window) and up-samples logits back to the original image resolution. In each scale,MCR computes a horizontally mirrored view,maps its outputs back to the canonical coordinate frame,and aggregates the mirrored and canonical predictions. A final cross-scale averaging yields the comprehensive dense prediction. The overall design explicitly preserves the zero-shot characteristics of the underlying VLM while using only forward-time operations to improve spatial precision.

    Result

    We evaluate TG-CLIP on eight public segmentation benchmarks that represent diverse settings with and without explicit background classes. These benchmarks include PASCAL VOC,PASCAL-Context,COCO-Object,ADE20K,COCO-Stuff,Cityscapes,and commonly used VOC subsets. In feature extraction,we use the CLIP ViT-B/16 encoder in a frozen configuration. All experiments are performed without any additional training or pixel-level fine-tuning. The hyperparameters are kept consistent across datasets. Across the evaluated benchmarks,TG-CLIP achieves an average mIoU of 45.5%,outperforming several recent baselines. In particular,TG-CLIP exceeds the second-ranked ProxyCLIP (40.1%) by 4.4 percentage points in averaged mIoU. Qualitative inspections indicate that TG-CLIP yields crisp boundaries,optimal preservation of thin structures,and reduced spurious activations in complex scenes. Detailed ablation studies confirm that both operators contribute to performance gains: TGR primarily improves local category separability and boundary fidelity,while MCR increases robustness to scale and viewpoint variability. Sensitivity analyses show that TGR is robust to a wide range of temperature and injection-strength settings,and MCR exhibits a measurable accuracy-efficiency tradeoff dictated by the number of scales and whether mirrored views are included. We report and recommend practical operating points that balance latency and accuracy for real-world deployment. TG-CLIP demonstrates that carefully designed inference-stage interventions can substantially close the gap between region-optimized VLM representations and the needs of dense,pixel-level segmentation without retraining. Given that all modifications occur at inference time,our framework is particularly attractive for engineering-constrained scenarios,such as embedded perception or rapid prototyping,where re-training is costly or infeasible. Nevertheless,several limitations remain: the multi-scale and mirrored inference steps introduce additional computation proportional to the number of scales and views; performance on extremely small or heavily occluded instances is still constrained by the spatial granularity of the frozen encoder; and our method does not eliminate the potential gains from task-specific supervised fine-tuning when labels are available. Finally,although our experiments include multiple standard benchmarks,further validation on real-world robotic perception tasks and diverse domain shifts can effectively elucidate generalization properties.

    Conclusion

    We propose TG-CLIP,a training-free inference methodology that combines text-guided recalibration with multi-scale flipped fusion to enhance pixel-level semantic discrimination and robustness of VLM-based open-vocabulary segmentation. TG-CLIP is simple to implement,preserves the zero-shot strengths of pretrained models,and yields consistent empirical gains across diverse benchmarks. We release implementation details and deployment recommendations to facilitate adoption and further research on inference-centric approaches for dense vision-language tasks. The codes are available at:https://www.scidb.cn/s/7baeYf.

  • Qiangqiang Zhou, Jiacong Yu, Jiawei Xu, Yong Chen, Xin Huang, Yanjiao Shi, Qing Zhang
    Journal of Image and Graphics. 2026, 31(8): 2785-2813.

    With the rapid development and widespread adoption of virtual reality (VR),360° omnidirectional image (ODI) and omnidirectional video (ODV) have become important media for entertainment,education,healthcare,gaming,and immersive training. By capturing a full 360° field of view,they enable users to navigate and interact with content freely,offering a sense of presence that conventional 2D media cannot provide. However,ODI and ODV data also introduce distinctive challenges,including spherical distortion caused by projection methods,uneven viewpoint distributions arising from user-controlled perspectives,and strict real-time processing requirements needed to preserve immersion. These characteristics make saliency detection—the task of identifying regions that attract human attention—particularly important in VR environments,while also rendering traditional 2D saliency models inadequate for full-sphere attention prediction. This survey provides a comprehensive review of recent advances in saliency detection for VR 360° content,spanning traditional feature-based methods and modern deep-learning approaches,including convolutional neural network (CNN),Transformers for long-range dependency modeling,and long short-term memory (LSTM) networks for temporal modeling in video. The survey begins by outlining the fundamental principles of VR 360° content,including imaging mechanisms,geometric properties of spherical representations,and key characteristics such as boundary discontinuity and projection-induced artifacts that distinguish it from conventional 2D imagery and necessitate specialized saliency models. Existing methods are then organized into several major categories: traditional handcrafted approaches based on low-level cues such as color,contrast,and texture; advanced deep architectures designed to learn robust features under spherical constraints; multi-projection modeling strategies for equirectangular,cubemap,and spherical domains; and multimodal fusion methods that integrate visual,auditory,and depth information. Traditional methods are computationally efficient and conceptually simple,but they often struggle with spherical distortion and spatial discontinuity. For example,early superpixel-based segmentation methods are sensitive to boundary effects,and color-dictionary sparse-representation approaches do not adapt well to dynamic VR content. In contrast,deep-learning methods have transformed the field by enabling end-to-end optimization and better handling of dynamic viewpoints and multimodal information. Representative examples include CNN-based models such as SalGCN,which use spherical graph convolution to mitigate projection distortion; Transformer-based models such as SalViT360,which capture global dependencies in 360° video effectively; and LSTM-based models such as HiBayes-LSTM,which improve temporal modeling for scanpath prediction by incorporating user-behavior cues. A major part of this survey is devoted to datasets and evaluation metrics. We review prominent ODI and ODV saliency datasets,including Salient360!,HTRO,and PAVS10K,and summarize their scale,annotation types (such as fixation maps and scanpaths),and applicability to tasks including saliency prediction and salient-object detection. We also analyze commonly used metrics,including Pearson correlation coefficient (CC),normalized scanpath saliency (NSS),Kullback–Leibler divergence (KLD),F-measure,mean absolute error (MAE),S-measure,and E-measure,discussing both their usefulness and their limitations in the presence of 360°-specific distortions,especially near the poles in equirectangular projection. Comparative studies indicate that deep-learning methods,particularly those using multimodal fusion and advanced projection strategies,consistently outperform traditional approaches on key indicators such as AUC and CC. The survey further examines practical applications of saliency detection in VR,including image and video quality assessment,adaptive compression for bandwidth optimization,and virtual cinematography. In quality assessment,saliency-guided models such as SG360BIQA improve prediction accuracy by emphasizing user-attended regions and thereby aligning objective metrics more closely with subjective experience. In compression,approaches such as RoSal360 allocate more bitrate to salient regions to improve transmission efficiency in resource-constrained VR systems. In virtual cinematography,attention-driven deep reinforcement learning models simulate human gaze patterns to support smoother and more intuitive camera control. These examples highlight the broader importance of saliency detection in optimizing VR systems and improving user immersion across applications from immersive storytelling to teleconferencing. Despite this progress,substantial challenges remain,including geometric distortion in spherical representations,the need for real-time processing under strict latency constraints,and the scarcity of unified multimodal datasets that combine visual,auditory,and interaction signals. Future work should therefore explore richer multimodal integration,lightweight and unsupervised learning frameworks,and large-scale standardized benchmarks. Emerging directions such as transfer learning for cross-domain adaptation,contrastive learning for more discriminative representations,and spherical-geometry-aware data augmentation may also help improve generalization and alleviate data scarcity. In summary,this survey offers a detailed analysis of saliency detection for VR 360° content,bridging traditional heuristic methods and modern data-driven approaches. By reviewing algorithms,datasets,evaluation metrics,applications,and open challenges,it aims to support future advances toward more efficient,immersive,and user-centered VR systems. The algorithms and datasets discussed in this article have been summarized at https://github.com/jiacongyu/PVSD.

  • Pin Lyu, Yiquan Wu
    Journal of Image and Graphics. 2026, 31(8): 2814-2843.

    In the era of big data and artificial intelligence,image processing has emerged as a cornerstone technology that drives advancements across diverse fields,including medical diagnostics,autonomous systems,and remote sensing. The exponential growth of high-resolution image data has created unprecedented computational demands that classical computing architectures increasingly struggle to meet. As the progress under Moore’s law slows,quantum computing has emerged as a transformative paradigm; it leverages fundamental quantum properties,including superposition,entanglement,and quantum parallelism,to overcome classical limitations. This convergence has given rise to the rapidly evolving field of quantum image processing (QIP),which promises to redefine the boundaries of computational imaging capabilities. This comprehensive survey provides a systematic analysis of QIP’s remarkable evolution over the past five years,documenting its transition from a theoretical concept to an emerging practical technology. We introduce an integrated classification framework that mirrors the complete image processing pipeline——from fundamental representation to advanced applications——thus creating a coherent perspective that effectively connects quantum mechanical principles with practical implementation requirements. The foundation of this framework lies in quantum image representation models,where innovative schemes,such as novel enhanced quantum repre-sentation,flexible representation of quantum images,and generalized quantum image representation,have advanced to support arbitrary image dimensions and complex color spaces while simultaneously optimizing critical qubit utilization efficiency. Building upon this foundation,QIP techniques demonstrate increasingly sophisticated capabilities. Advanced filtering algorithms achieve exponential speedups in noise reduction tasks,and quantum Fourier and wavelet transforms enable unprecedented efficiency in image compression. Quantum parallelism facilitates rapid morphological analysis and feature extraction,substantially accelerating fundamental image processing operations. Aside from basic processing,quantum-enhanced methods for edge detection,segmentation,and feature matching also demonstrate remarkable efficiency improvements in extracting meaningful information from complex visual data. The security domain benefits substantially from quantum approaches,with innovative encryption protocols based on quantum walks and chaos theory providing theoretically unbreakable protection for sensitive visual data. Quantum watermarking techniques that utilize entanglement phenomena offer fundamentally undetectable data-hiding capabilities,creating new paradigms for digital rights management and secure communication. In machine learning applications,quantum support vector machines enhance classification accuracy for high-dimensional image data,and quantum convolutional neural networks revolutionize feature extraction through inherent quantum parallelism. Quantum generative adversarial networks enable sophisticated synthetic data generation and have notable applications in data augmentation and privacy preservation. This review emphasizes an integrated perspective that connects theoretical advances with practical implementations across multiple application domains. Our structured analytical framework clearly reveals how abstract quantum principles translate into operational capabilities while simultaneously addressing the critical challenges currently hindering widespread adoption. The noisy intermediate-scale quantum era presents substantial hardware limitations,where qubit decoherence and elevated error rates remain persistent barriers to reliable operation. The considerable theory-practice implementation gap,where algorithm performance on actual quantum devices often falls short of theoretical predictions,requires careful consideration. Moreover,scalability issues in real-world applications,where the substantial overhead of quantum data encoding and measurement can diminish computational advantages,present additional hurdles that must be overcome. Resource management considerations examine how different representation models directly affect quantum circuit complexity and execution time,providing crucial insights for practical algorithm design and implementation. Looking forward,several promising pathways can be taken. Examples include hybrid quantum-classical architectures that leverage the complementary strengths of both computational paradigms,advanced error mitigation techniques that compensate for current hardware imperfections,and quantum-inspired classical algorithms that can deliver enhanced performance on conventional hardware while preserving certain quantum advantages. The discussion extends to current hardware platforms,including superconducting qubits,trapped ions,and photonic quantum systems,to provide practical guidance for experimental research and implementation strategies. Through systematic analysis and critical evaluation,this review demonstrates quantum computing’s massive potential to achieve exponential acceleration in computational tasks,enhance security beyond classical limitations,and enable unprecedented capabilities in image analysis and understanding. Emerging applications in medical imaging,remote sensing,and security systems highlight the transformative potential of these technologies. As quantum hardware continues its steady evolution,the strategic integration of quantum approaches into conventional image processing workflows promises to redefine computational boundaries across numerous disciplines,including healthcare,environmental monitoring,industrial inspection,and artificial intelligence. The field currently stands at the threshold of a new computational paradigm,requiring continued technical innovation and strategic vision to successfully navigate the coming transition. By synthesizing recent advances,clearly identifying current limitations,and outlining promising research directions,this survey aims to accelerate the development and adoption of QIP technologies. The ultimate integration of quantum approaches with classical image processing methodologies promises to create powerful hybrid frameworks that leverage the distinctive strengths of both computational paradigms,fundamentally transforming how we extract knowledge and value from visual data in the emerging quantum era and enabling capabilities beyond the reach of either approach independently. The mentioned algorithms are summarized at https://github.com/cococococo7/QIIP.

  • Jinghong Zhang, Pan Su, Qingyuan Zhu, Chaogang Zhang, Bing Li, Wangbin Cao
    Journal of Image and Graphics. 2026, 31(8): 2844-2867.

    With the accelerating transition of the global energy landscape toward renewables,wind power has been widely promoted as a vital component of clean energy. Alongside the rapid expansion of wind turbine installed capacity,the structural health of wind turbine blades (WTBs),which are among the critical power-generating components,directly influences the safety and efficiency of wind power systems. In recent years,unmanned aerial vehicles (UAVs) have been equipped with high-resolution imaging devices for blade image acquisition. By combining advanced image processing and object detection techniques,they have achieved automated,noncontact defect detection,which not only enhances detection efficiency and coverage but also reduces the risks and costs of traditional manual inspections. Therefore,this approach has become a mainstream methodology in the industry and a focal point of research. However,images captured by UAVs often suffer from poor quality (primarily manifested as noise,motion blur,and low resolution),posing critical challenges to defect detection. Moreover,defect features,such as erosion marks,stains,or manufacturing traces,frequently exhibit high visual similarity to blade surface textures and present multiscale and diverse morphological characteristics. This visual ambiguity further complicates accurate localization and classification. In addition,the lack of standardized annotation protocols and the heavy reliance on manual labeling hinder the construction of high-quality datasets. These issues collectively constrain the detection accuracy and generalization performance of deep learning-based object detection models. Existing survey literature on WTB defect detection has primarily emphasized nondestructive testing methods,with a noticeable scarcity of comprehensive reviews dedicated exclusively to image-based detection approaches. Current surveys in this field focus `on architectural advancements within deep learning models,including the integration of lightweight modules,attention mechanisms,and multiscale feature extraction techniques. Although these architectural improvements have enhanced detection accuracy,they often overlook essential aspects,including image preprocessing methods and learning strategies,both of which play a crucial role in determining overall model performance. In this review,we systematically incorporate perspectives on image preprocessing and learning strategies to analyze defect detection methods for WTBs. The main contributions of this review are as follows. 1) From the perspective of defect manifestation in WTBs,defects can be broadly categorized into internal and surface types. We systematically present and analyze the causes,formation mechanisms,and visual characteristics of these defect categories with respect to their structural properties. We also summarize the most frequently observed defect types in datasets. These datasets provide valuable data support for the design and optimization of subsequent defect detection models. 2) This review summarizes the characteristics of defects in detection tasks across various power system components,such as foreign object recognition on transmission lines and insulator defect detection. Through a comparative analysis,this review demonstrates that WTB defect detection uniquely encompasses all the aforementioned challenges and variabilities. Consequently,WTB defect detection necessitates simultaneously addressing a spectrum of challenges,including poor image quality,multiscale defect manifestations,class imbalance,lack of standardized annotation protocols,and the need for lightweight deployable models. Such a problem-driven perspective provides a foundation for inspiring future methodological advancements. 3) We conduct an in-depth investigation into the key components of deep learning-based image-driven WTB defect detection. Three critical aspects,namely,image preprocessing,model architecture,and dominant learning strategies,are presented. With respect to image preprocessing,task-driven methodologies are categorized into image quality enhancement,geometric correction,and structural feature extraction. Each of them is examined for its critical role in improving the saliency of defects and bolstering the robustness of detection models. At the architectural level,the enhancement strategies across mainstream detection frameworks,including two-stage,one-stage,and Transformer-based models,are presented. In terms of learning strategies,structured analyses of the practical effectiveness and potential of approaches,including transfer learning,ensemble learning,cascade strategies,and unsupervised/semisupervised learning,are presented. These strategies are evaluated in terms of their capability to address limited data availability,class imbalance,and label uncertainty in defect detections. 4) This review also provides an overview of publicly available datasets released in recent years,examining their data types,annotation protocols,and defect categories,thereby offering guidance for researchers in selecting appropriate research sources. A summary of commonly used evaluation metrics is provided,underscoring the necessity of image quality assessment in diverse application scenarios. 5) This review outlines critical challenges in image-based WTB defect detection,including imbalanced defect distributions,heavy reliance on manual annotations,limited model generalization under environmental variations,insufficient data modality diversity,and lack of innovation in model architectures. In light of these challenges,several promising research directions are proposed; these include the effective integration of multimodal data,creative design of model structures,and in-depth adoption of weakly or self-supervised learning strategies,with the aim of inspiring and guiding future advancements in WTB defect detection.

  • Jiaxin Liu, Sheng Li, Zhenxing Qian, Xinpeng Zhang, Qichao Ying
    Journal of Image and Graphics. 2026, 31(8): 2868-2881.
    Objective

    The widespread use of face-retouching techniques across social media platforms has generated significant societal concerns that extend beyond aesthetics,giving rise to psychological consequences,such as body dysmorphic disorder,undermining trust in digital imagery and reinforcing pervasive cultural norms surrounding beauty standards. Face-retouching is widely perceived as more convenient than face swapping and has been integrated into numerous social network applications. Although existing research has made substantial progress in detecting retouched faces using a range of computational approaches,the critical task of reversing these modifications to reconstruct original facial features and restore authenticity in digital images remains largely unexplored. This disparity highlights a significant gap between detection capabilities and the development of practical restoration solutions. This study formally introduces face retouching restoration (FRR) as a novel computer vision task,distinguished from related problems,such as makeup restoration and general image restoration,by its explicit focus on subtle textural modifications and extreme structural alterations characteristic of retouched facial images. The primary objective is to accurately recover authentic facial characteristics from heavily manipulated images by addressing two key technical challenges: restoring fine-grained textures,such as skin smoothing,and reconstructing facial geometry,including eye enlargement and jaw reshaping. This dual focus on texture and structure advances beyond existing methods that typically address only a single aspect of facial modification. Furthermore,this study aims to establish standardized evaluation protocols and datasets for this emerging research direction,recognizing that existing benchmarks designed for retouching detection are inadequate for assessing restoration quality. The proposed approach advances digital media forensics by providing practical tools for content authentication while simultaneously contributing to the broad discussions on image manipulation in the digital age. This study aims to shift from passive detection to active restoration,with potential applications in forensic analysis,psychological counseling,and digital media literacy initiatives.

    Method

    The proposed retouching-guided diffusion model for FRR (RT-FRR) introduces a novel dual-component architecture that synergistically integrates diffusion-based generation modeling with targeted retouching artifact analysis. The framework incorporates a face retouching estimation module (FREM) that utilizes constrained BayarConv layers with 5 × 5 × 3 kernels,specifically designed to amplify manipulation artifacts while suppressing authentic facial features. This preprocessing stage extracts multi-scale retouching traces through four successive ResNet blocks,capturing local pixel-level inconsistencies and global structural alterations. These hierarchical features serve as conditional guidance for the second component—a denoising diffusion probabilistic model configured with 1 000 timesteps that learns to predict the precise residual between retouched and original facial images. The model’s U-Net backbone incorporates innovative cross-attention mechanisms at four distinct network depths,allowing progressive integration of retouching traces with diffusion priors throughout the denoising process. This multi-scale attention strategy facilitates simultaneous recovery of high-frequency textural details,such as skin pores and fine wrinkles,and macroscopic structural features,including facial contours and organ geometry. Training optimization utilizes the Adam algorithm across 100 000 iterations with a fixed learning rate of 10-4 using a composite loss function that balances pixel-wise reconstruction accuracy with perceptual quality metrics. The authors created the extreme retouched face dataset (ExRF) comprising 1 000 high-resolution image pairs with severe yet plausible retouching operations,manually generated using professional editing software with consistent extreme parameter settings,for a comprehensive evaluation. This dataset complements the existing RetouchingFFHQ benchmark,ensuring robust validation across the full spectrum of retouching intensities from subtle enhancements to dramatic transformations. The framework operates on 512 × 512 resolution images,maintaining architectural consistency with state-of-the-art diffusion models while being specifically optimized for facial feature recovery tasks.

    Result

    Experimental evaluation demonstrates that RT-FRR achieves state-of-the-art performance in face retouching restoration,significantly outperforming existing methods across multiple quantitative and qualitative metrics. On the ExRF dataset,which features significantly retouched faces,RT-FRR achieves a PSNR of 25.23 dB,surpassing the second-best method ResDiff by 1.9 dB,along with superior scores in SSIM at 0.806,FID at 48.802,and LPIPS at 0.241. These results highlight the model’s effectiveness in recovering subtle textural details and major structural modifications,such as eye enlargement,nose reshaping,and facial contour adjustments. Comparative analysis against nine baseline methods,including makeup removal techniques,such as CSD-MIT and BeautyGAN,as well as general image restoration approaches,such as Restormer and SR3,demonstrates that RT-FRR consistently delivers high-fidelity reconstructions,particularly in cases of significant retouching where prior methods frequently fail. On the RetouchingFFHQ dataset,which features mild retouching,RT-FRR maintains strong performance with a PSNR of 25.39 dB,demonstrating robustness across different retouching intensities. Qualitative assessments reinforce these findings,with feature similarity heatmaps and t-SNE visualizations confirming that RT-FRR more accurately recovers facial identity characteristics than competing approaches. The model successfully reconstructs fine-grained skin textures while correcting exaggerated structural changes,such as unnaturally thinned jawlines or enlarged eyes,without introducing artifacts common in GAN-based or transformer-based methods. Ablation studies underscore the importance of key design choices,where the face residual generation strategy alone contributes a 4 dB PSNR improvement,while the FREM module adds an additional 2 dB by localizing retouching traces. Visual comparisons further demonstrate RT-FRR’s ability to restore natural facial proportions and skin details,even in cases where retouching has heavily distorted the original appearance. These results establish RT-FRR as a comprehensive solution for face retouching restoration,effectively addressing textural and structural recovery with unprecedented accuracy.

    Conclusion

    In this study,we focus on FRR,a novel task focused on recovering original facial features from retouched ones,particularly those that are significantly retouched. We propose RT-FRR to recover the textural and structural face features from a retouched face,where a face retouching estimation module is designed and incorporated to extract the retouching trace to guide the FRR. The retouching traces are integrated with the denoising U-Net of the DDPM through an attention mechanism for an accurate face residual generation. We evaluate the performance of RT-FRR on an existing face retouching dataset and the newly constructed ExRF. On both datasets,our RF-FRR outperforms existing makeup removal and image restoration schemes in the task of FRR.