Latest ArticlesHan portrait stones are decorative stone carvings on funerary architectural structures,such as tombs and ancestral halls,dating back to the Han Dynasty. The contents of these stones depict mythological legends,institutional systems,and everyday life scenes,including chariot processions,kitchen banquets,textile farming,and other scenes that directly reflect the social practices of the Han period. Iconic figures,such as the Queen Mother of the West (Xiwangmu),Fuxi and Nüwa,and feathered immortals (Yuren),represent the Han people’s cosmology of “immortal souls”. In particular,Xiwangmu images are prevalent in the tombs of the middle and lower classes but seldom appear in noble tombs,reflecting hierarchical differences in beliefs. Themes,such as ancient sages and war scenes,which combine architectural practicality and artistic decoration,serve educational and commemorative functions,embodying realism and romanticism. This study proposes an image-text retrieval method that leverages semantic information from mask images of Han portrait stones to enhance images and texts,addressing the domain shift problem present in existing image-text retrieval methods. The method introduces a semantic enhancement module and integrates semantic information with randomly generated prompt templates for text reorganization. This design enables the model to learn additional discriminative features,understand different objects in images,align text with images,and improve the accuracy and comprehensiveness of image-text retrieval.
This study utilizes the architecture of Chinese contrastive language-image pre-training (Chinese CLIP) using a vision transformer (ViT) as the image encoder and robustly optimized bidirectional encoder representations from Transformers (RoBERTa) model as the text encoder to extract image and text features. Semantic information from mask images and original text is used to calculate object-specific weights. Attention weights are derived by combining these weights with features from mask and original images. After a residual connection,these attention weights are used to generate attention features,while the weights themselves serve as semantic enhancement features. The original image features,attention features,and enhancement features are combined through a multi-stage feature fusion strategy at a ratio of 0.4:0.5:0.1 to form composite features. This design enables the model to focus on the primary subjects and important objects in images while enriching feature representation without information loss. Effective semantic objects are extracted from the mask images,and the original texts are embedded into randomly generated prompt templates incorporating these objects to produce enhanced texts. This process structures text descriptions,strengthens semantic expression,and improves matching with image features. Furthermore,dynamic temperature adjustment and asymmetric similarity calculation are introduced: the contrastive learning temperature is adaptively adjusted based on feature similarity (low temperatures applied to high-similarity samples to enhance discriminative learning and high temperatures applied to low-similarity samples to increase tolerance). The different temperature coefficients (0.9 for image-to-text and 1.1 for text-to-image) balance the matching difficulty in both directions,accounting for the distribution differences between image and text features to improve the accuracy and stability of cross-modal matching.
In this study,a series of comparative experiments,ablation experiments,and visual comparison experiments is conducted on the proposed model to evaluate the model’s performance and the contributions of each module. Results show that,on the Han portrait stone test set,the proposed method significantly improves the text-to-image retrieval metrics Mean_Recall (%),Recall@1 (%),Recall@5 (%),and Recall@10 (%) by 15.20%,21.84%,14.29%,and 9.48%,respectively,compared with the second-ranked bootstrapping language-image pre-training model in terms of precision and comprehensiveness. During the image-to-text retrieval,the metrics Mean_Recall (%),Recall@1 (%),Recall@5 (%),and Recall@10 (%) are improved by 17.47%,23.22%,18.45%,and 10.72%,respectively,compared with the second-ranked cross-modal vision-language model (X2-VLM),demonstrating significant improvements in image-text retrieval performance. Visualization experiments can effectively assist us in observing the improvement of the model’s performance and the enhanced effects brought about by each added module,making the experimental results highly intuitive. Additionally,the proposed method maintains good retrieval performance in data robustness experiments. The average recall rate of the model consistently exceeds 70%,and it maintains excellent retrieval performance. This outcome indicates that the proposed model exhibits remarkable data robustness and is applicable to various scenarios,such as fuzzy retrieval.
The proposed method enhances the model’s ability to align Han portrait stone images with their descriptive texts. The model can learn effective and crucial features by simultaneously enhancing text and images. The proposed dynamic temperature mechanism adapts to the structural characteristics of the Han portrait stone dataset and enhances the model’s ability to learn from difficult samples. effectively improving the performance of image-text retrieval in terms of accuracy and comprehensiveness. However,when objects in images are similar but differ in positional information,the image-text retrieval results may suffer from confusion and false positives. Given the limited scale and diversity of the dataset for Han portrait stone image-text retrieval,the model’s cross-modal retrieval performance may be constrained. In the future,we will optimize the model’s capability to semantically analyze the different positions and actions of objects in images and expand the Han portrait stone dataset.
Synthetic aperture radar (SAR) and optical remote sensing represent two complementary Earth observation modalities. SAR imagery enables all-weather,all-day observation due to its active imaging mechanism,while optical imagery provides higher spatial resolution and more intuitive visual details. Cross-modal translation between SAR and optical images not only compensates for missing data in either modality but also enhances downstream applications,such as image matching and semantic segmentation,through multi-modal information fusion. However,current methods encounter notable challenges. Approaches based on cycle-consistent generative adversarial networks (CycleGAN) predominantly emphasize macroscopic structural reconstruction and fail to exploit deep semantic correlations across source and target domains. In particular,traditional contrastive learning techniques are limited in remote sensing due to the high spatial autocorrelation of similar land-cover features,which blurs the distinction between positive and negative samples and causes the contrastive mechanism to underperform. This study addresses these challenges by proposing a semantic-guided contrastive learning method for SAR and optical image translation,which effectively mitigates feature homogenization in remote sensing image translation and significantly improves semantic fidelity and downstream task performance.
The proposed framework comprises three core components: 1) a semantic feature extraction module,2) a semantic-guided contrastive learning module,and 3) a joint optimization scheme combining cyclic generation and contrastive learning. Initially,this approach utilizes pre-trained SAR and optical semantic segmentation models (DeepLabV3) to extract pixel-level semantic features from real and reconstructed images. Subsequently,the semantic-guided contrastive learning module implements a class-consistency-based positive-negative sample selection strategy: In each query patch in the generated image,patches sharing the same semantic class in the real image serve as positive samples,while those from differing classes are treated as negatives. This strategy effectively mitigates the feature homogenization problem that challenges traditional contrastive learning in remote sensing contexts. Contrastive loss is computed within a shared feature space projected by a lightweight perceptron. Finally,the joint optimization framework integrates cyclic consistency losses at pixel and semantic levels with adversarial losses. The cyclic semantic segmentation loss enforces consistency in structure,texture,and semantics between generated and real images,while adversarial losses enhance image realism. The overall loss function balances these components via weighted hyperparameters.
Comprehensive experiments on two public datasets—WHU-OPT-SAR and DDHRNet—evaluate the proposed method against state-of-the-art approaches,including CycleGAN,contrastive unpaired translation,query-selected attention,and conditional diffusion,across image translation quality and downstream tasks. Regarding image translation,our method consistently achieves superior performance. On WHU-OPT-SAR,SAR-to-optical translation yielded a peak signal-to-noise ratio (PSNR) improvement of 11.9% and a mean absolute error (MAE) reduction of 31.1% over the second-best method; optical-to-SAR translation achieved gains of 3.8% in PSNR,5.6% in structural similarity index measure (SSIM),and a 5.2% reduction in MAE. On DDHRNet,our method sustained leading performance across diverse geographical contexts. In downstream tasks,the semantic segmentation and feature matching results confirm marked gains. The semantic segmentation pixel accuracy improved by 49.26% for optical image generation and 10.60% for SAR image generation on WHU-OPT-SAR,outperforming all baselines. In feature matching tasks,the inlier ratio (IR) and Euclidean distance-based inlier ratio improved by up to 1% and 0.49%,respectively. Qualitative comparisons reveal superior visual quality with enhanced detail preservation and grayscale consistency. Ablation studies demonstrate the necessity of each component: removal of the semantic-guided contrastive learning module caused a 10.75% drop in pixel accuracy for optical images,while omitting the cyclic semantic segmentation loss reduced PSNR by 7.0% in optical-to-SAR translation. These findings validate the critical role of our innovations in overcoming remote sensing translation challenges.
The proposed semantic-guided contrastive learning method for SAR and optical image translation effectively addresses the prevalent issue of feature homogenization in remote sensing image translation. Our method substantially enhances the semantic fidelity and downstream utility of generated images by integrating semantic segmentation guidance with contrastive learning within a cyclic generative framework,offering a novel solution for unsupervised SAR and optical image translation. Extensive experiments demonstrate that this approach provides a robust and effective solution for unpaired SAR-optical image translation with notable advantages in semantic consistency and cross-modal feature alignment. Our main contributions include: 1) identifying the feature homogenization problem inherent in traditional contrastive learning for remote sensing translation; 2) proposing a novel semantic-guided contrastive learning framework with category-consistent sample selection; and 3) developing a unified architecture combining cyclic generation and contrastive learning. Future work will explore self-supervised semantic guidance to improve applicability in scenarios lacking semantic annotations,thereby enhancing generalizability and robustness across diverse remote sensing applications. Our codes are available at https://www.scidb.cn/s/VVVBnu.
Accurate segmentation of colorectal polyps in colonoscopy images is essential for the early diagnosis and prevention of colorectal cancer. However,colonoscopy images often exhibit blurred boundaries,irregular shapes,and large-scale variations,making automated segmentation highly challenging. Conventional convolutional neural network (CNN)-based methods struggle to capture long-range dependencies,often leading to inaccurate boundary localization and missed detection of small lesions. Although Transformer-based models improve global context modeling,they present quadratic computational complexity and high memory consumption when processing high-resolution medical images. The Mamba architecture,built upon selective state space models,has recently shown strong potential for efficient sequence modeling with linear computational complexity. Despite its advantages,directly applying Mamba-based models to polyp segmentation remains challenging due to its limited capability to extract local features and insufficient multiscale feature interaction. Aiming to address these issues,this paper proposes a multilevel and multiscale feature fusion enhanced network based on the Mamba architecture (MMF-MambaNet),which intends to improve segmentation accuracy while maintaining computational efficiency.
MMF-MambaNet adopts an encoder-decoder architecture and employs the visual state space model,a variant of Mamba,as the backbone to efficiently capture long-range contextual dependencies. Aiming to enhance feature diversity and robustness,a local sequence-wise style transformation augmentation module is integrated into the backbone. This module performs stochastic re-parameterization by perturbing the mean and variance of feature distributions,thereby improving model generalization under varying imaging conditions. Three specialized modules are also incorporated to strengthen multi-scale representation and boundary perception. First,a cross-scale aware self-attention (CASA) module is introduced to facilitate hierarchical feature interaction by using deep semantic features to refine shallow features. By combining channel and spatial attention mechanisms,CASA effectively fuses multilevel features while preserving fine boundary details. Second,an adaptive detail enhancing module (ADEM) is designed to improve structural representation across different scales. Using an anchor-scale fusion strategy,this module adaptively integrates information from different scales to construct highly comprehensive detail representations of polyps,thereby enhancing boundary structures and improving the segmentation of small lesions. Third,an adaptive global-local integration (AGLI) gating module is introduced in the decoder stage. This lightweight mechanism adaptively combines global semantic features with local detailed features from skip connections,enabling the model to emphasize reliable regions while suppressing background interference. Furthermore,MMF-MambaNet introduces a reverse feature fusion pathway that progressively combines deep semantic features with shallow spatial information through upsampling and feature aggregation. A pyramid pooling module is also employed to capture multiscale global contextual information,thereby improving semantic consistency in the segmentation results. The model is optimized using a hybrid binary cross-entropy and dice loss,which balances pixel-level classification accuracy with region-level overlap while alleviating class imbalance in medical image segmentation.
Extensive experiments were conducted on five publicly available colorectal polyp segmentation datasets: CVC-300,CVC-ClinicDB,Kvasir-SEG,CVC-ColonDB,and ETIS-LaribPolypDB. The proposed MMF-MambaNet was compared with eight state-of-the-art segmentation methods,including PraNet,VM-UNetV2,and Polyp-Mamba. Experimental results demonstrate that MMF-MambaNet achieves competitive and stable performance across multiple datasets. Compared with the VM-UNet baseline,MMF-MambaNet improves the mean Dice coefficient (mDice) by 1.1%,5.0%,4.2%,9.1%,and 2.6% on the CVC-300,CVC-ClinicDB,Kvasir-SEG,CVC-ColonDB,and ETIS-LaribPolypDB datasets,respectively. On the challenging CVC-ColonDB dataset,the proposed model further outperforms VM-UNetV2,achieving improvements of 4.6% in mDice and 6.2% in mean intersection over union (mIoU). Visual comparisons show that MMF-MambaNet can accurately delineate polyp boundaries even under challenging cases,such as low contrast,irregular shapes,and small lesion sizes. In addition to segmentation accuracy,the model also demonstrates favorable computational efficiency. With 18.84 M parameters and 5.74 GFLOPs,MMF-MambaNet maintains a relatively lightweight architecture while achieving an inference speed of 57.58 frames per second. Ablation studies further confirm the effectiveness of the proposed modules,revealing that CASA,ADEM,and AGLI each contribute to enhanced segmentation accuracy and improved boundary preservation.
This study presents MMF-MambaNet,a Mamba-based network that integrates multilevel and multiscale feature fusion mechanisms designed for colorectal polyp segmentation. By combining efficient sequence modeling with adaptive feature interaction strategies,the proposed method effectively addresses the challenges such as blurred boundaries and scale variations in colonoscopy images. Experimental results demonstrate that MMF-MambaNet exhibits strong segmentation performance while maintaining lightweight computational complexity,making it a promising approach for computer-aided colonoscopy diagnosis and assisting the early detection of colorectal cancer.
As an interdisciplinary field that spans computer vision,graphics,artificial intelligence (AI),and optical imaging,3D vision serves as the cornerstone for constructing embodied general intelligence and the metaverse. The “scaling law” paradigm,upon which AI development relies,faces significantly diminishing marginal returns and encounters bottlenecks. Thus,the focus of academia and industry is pivoting ever more clearly toward foundational subjects that are closely related to 3D vision,such as “world models”,“spatial intelligence”,and “embodied intelligence”,granting 3D vision with unprecedented strategic attention and developmental opportunities. In 2025,the primary frontier trends in the field of 3D vision can be summarized as follows. 1) Feed-forward 3D reconstruction that supports spatiotemporal multi-image input: with breakthroughs in feed-forward 3D reconstruction technologies,such as visual geometry grounded Transformer(VGGT),obtaining scene structure and motion information through spatiotemporal multi-image feed-forward methods has become increasingly simple,exerting two profound effects. First,it provides a solid foundation to 3D scene understanding for spatial intelligence,allowing many traditional 2D vision problems to be solved more fundamentally in 3D space. Second,combined with efficient rendering technologies,such as 3D Gaussian splatting (3DGS),the threshold for high-quality 3D content production has been significantly lowered,paving the way for large-scale applications,such as digital twins and the metaverse. 2) Gradual fusion of 3D generation and 3D reconstruction: 3D AI generated content technologies,such as SAM3D support compositional and instance-level object generation under single-image input,with generation quality gradually reaching industrial-grade scanning standards while simultaneously integrating feed-forward reconstruction methods to gradually achieve the generation of authentic 3D structures and textures that are consistent with the input images. This condition will support the feed-forward multi-instance reconstruction of dynamic complex scenes,significantly improving real-time,multimodal perception and understanding capabilities in complex scenarios. 3) Integration of video generation and world models into embodied intelligence: Video generation technology is rapidly incorporating explicit or implicit 3D representations and evolving toward multi-view consistency,long sequences,and physical plausibility,directly driving the development of integrated “perception-generation-interaction” world model technologies. These types of world models,combined with feed-forward 3D reconstruction technology,will form a complete “multimodal perception-3D modeling-4D generation-real-time interaction” 4D world model. Simultaneously,world model methods have begun to serve embodied intelligence,and a unified framework of “understanding-generation-execution” has begun to emerge. World models are widely regarded by the academic community as the key path to achieving generalizable embodied intelligence and ultimately leading to artificial general intelligence. 4) Human behavior and video data becoming the core fuel that drives breakthroughs: Human operational spaces and interaction videos constitute a “data goldmine” for training embodied intelligence. The vast amount of human behavior videos on the internet,along with first-person perspective data collected through simple devices,contain physical common sense,causal reasoning,and interaction preferences that serve as the natural fuel for breaking through the current data bottlenecks of embodied intelligence. By performing explicit 3D perceptual reconstruction or latent space action alignment and learning on these data,a “data pyramid” base can be constructed to drive the scaling of embodied intelligence. 5) Evolution of the embodied training paradigm from imitation learning to interaction-driven reinforcement learning: The technical evolution of embodied intelligence vision-language-action (VLA) models is shifting from a supervised fine-tuning paradigm that relies on expert demonstrations to a composite training architecture that integrates online reinforcement learning. This shift effectively breaks the dependence on scarce high-quality data,enabling policies driven by sparse rewards to obtain generalization and exploration capabilities that surpass those of imitation learning,and solving the challenges of exploration and stable updates in continuous action spaces. Simultaneously,the development of high-performance training systems and action-conditioned world models provides infrastructure support for large-scale interaction data generation and efficient policy evolution,marking a new “post-training” stage for embodied intelligence that is centered on “interaction-driven” approaches. The top 10 selected research advancements of 2025 in the field of 3D vision include the following: 1) feed-forward 3D reconstruction that builds the foundation models for 3D vision (spatial intelligence); 2) the convergence of the reconstruction and generation technical routes (video generation/3D generation),moving away from mutual assistance toward preliminary integration; 3) 3DGS/4DGS continuously improving representation efficiency,triggering a surge in scene modeling and volumetric video applications; 4) 3D generation: a shift from single-object visual realism to structuralized components/scenes and physical interactivity; 5) from video generation to world models: oriented toward spatiotemporal consistency,physical plausibility,and interactivity; 6) unified multimodal large models for understanding and generation,serving spatial intelligence perception; 7) frontier shifts in digital humans: from appearance modeling to multimodal interaction; 8) human data becoming the essential fuel for breaking through the scaling law of embodied intelligence; 9) embodied intelligence foundation models that are evolving toward unified models of integrated “understanding-imagination-execution”; and 10) “post-training” moment of embodied intelligence: the paradigm shift of VLA models from imitation learning to online reinforcement learning. Collectively,these breakthroughs have established the prototype of an integrated intelligent architecture characterized by “multimodal perception-3D modeling-4D generation-real-time interaction”,providing critical technical support for the substantive advancement of spatial and embodied intelligence. To promote academic discourse,this study extensively analyzes frontier trends in 3D vision and curates the top 10 annual research advances,offering valuable reference perspectives for academia and industry.
With the exponential growth of the metaverse,immersive virtual reality,and advanced human-computer interaction paradigms,audio-driven digital human generation has rapidly emerged as a paramount research frontier in computer vision and computer graphics. This transformative technology aims to establish a robust and seamless cross-modal mapping from 1D speech signals to 3D visual video streams. In contrast with traditional motion capture-based approaches,which are constrained by expensive hardware and rigid physical setups,audio-driven methodologies offer a highly scalable,cost-effective,and interactive solution. From a cognitive psychology perspective,human speech perception is inherently a multimodal process. The classic McGurk effect demonstrates that visual cues,such as precise lip movements,play a decisive role in deciphering auditory information,especially in noisy environments. Consequently,synthesizing a lifelike digital human requires not only ultrahigh-definition visual rendering but also millimeter-level,time-synchronized alignment between auditory signals and visual lip dynamics to prevent the notorious “uncanny valley” effect,where subtle unnaturalness provokes discomfort in human observers. Despite the proliferation of extensive research and rapid algorithmic advancements,generating photorealistic and real-time interactive digital avatars continues to encounter three formidable challenges. First,bridging the immense semantic gap between distinct modalities—specifically,translating semantic and acoustic features into complex spatial facial muscle deformations—remains fundamentally difficult. Second,the trade-off between high-fidelity rendering and real-time inference latency poses a critical bottleneck. Achieving pore-level skin details and realistic hair rendering typically demands immense computational resources,conflicting with the strict low-latency requirements (often under 100 ms) necessary for seamless live streaming or real-time conversational agents. Third,zero-shot cross-identity generalization and the expansion of driving scope—from isolated talking heads to full-body expressive agents that encompass co-speech gestures—demand robust structural priors that current isolated models struggle to provide. To deconstruct this complex domain systematically,this comprehensive review first proposes a unified “encoding-mapping-rendering” technical framework that encapsulates the majority of audio-driven digital human architectures. The framework initiates with an audio feature extractor,utilizing state-of-the-art self-supervised foundational models (e.g.,WavLM and Whisper) to capture rich phonetic,semantic,and emotional representations from raw audio waveforms. These acoustic features are then fed into visual geometric representations,transitioning from sparse 2D landmarks into sophisticated 3D morphable models,which provide explicit facial motion space. Finally,a cross-modal mapping and rendering network translates these spatial configurations into final visual video frames. Following this structural foundation,this study meticulously reviews the evolutionary trajectory of audio-driven rendering technologies,categorizing them into four dominant paradigms. First,early 2D image-based generation methods,predominantly driven by generative adversarial networks,are examined. Although these models excel in achieving rapid inference speed and fundamental lip synchronization,they inherently lack topological constraints,suffering from severe structural degradation and texture stretching when handling large-pose head rotations. Second,the integration of neural radiance fields (NeRFs) introduces a paradigm shift by leveraging implicit volumetric representations. NeRF-based approaches successfully resolve multi-view consistency issues; however,the heavy computational burden of volumetric ray marching restricts their real-time application,while their subject-specific optimization nature limits broad generalization. Third,the recent breakthrough of 3D Gaussian splatting (3DGS) has revolutionized the field by combining explicit geometric primitives with highly optimized rasterization pipelines. 3DGS-based models break real-time rendering constraints,easily surpassing 100 frames per second,although they still struggle with dynamic consistency during extreme facial deformations due to their discrete point cloud nature. Lastly,diffusion models represent the current pinnacle of visual fidelity and generative diversity. By iteratively denoising latent spaces,these models synthesize unprecedented skin textures,hair dynamics,and micro-expressions,although their inherent Markov chain sampling mechanisms introduce significant inference latency and temporal jitter challenges into long video generation. To facilitate an objective assessment of these diverse methodologies,this study synthesizes mainstream datasets and rigorous evaluation metrics. We detail the utilization of large-scale in-the-wild datasets,such as VoxCeleb,alongside high-resolution benchmarks,like high-definition talking face,and emotional datasets,such as the multi-view emotional audio-visual dataset. The evaluation criteria are multidimensional,encompassing lip-sync accuracy through SyncNet metrics (lip sync error distance and lip sync error confidence),visual quality via Fréchet inception distance and learned perceptual image patch similarity,temporal video consistency that utilizes Fréchet video distance,and identity preservation measured using cosine similarity. Finally,this study provides an in-depth analysis of open challenges and outlines prospective future research trajectories. We hypothesize that the integration of multimodal large language models will fundamentally redefine cross-identity generalization,serving as powerful world models for decoding audio into structured motion commands. Furthermore,transitioning from purely correlation-based learning to causal modeling will be essential for decoupling authentic emotional expressions from entangled acoustic noise. To address the theoretical bottlenecks of long-sequence temporal consistency,we propose exploring state space models,such as Mamba architectures,and flow matching trajectories. Concurrently,given that these generative capabilities approach indistinguishable realism,ethical implications,particularly those that concern deepfake misuse,necessitate the parallel development of robust forensic detection and defense mechanisms. Ultimately,this review aims to serve as a definitive guide for researchers,navigating the transition of digital humans from mere visual replicas into fully embodied,emotionally intelligent interactive agents in the digital era.
For the 21st century,this study is dedicated to exploring the fundamental theories,models,and architectures of next-generation artificial neural networks (ANNs),with the objective of constructing high-performance ANNs that featuring adaptive topology,interpretability,strong generalization capability,and high energy efficiency. Since the initial proposition of ANNs in the 1940s,the field has witnessed over 80 years of development. Extending to the mid-21st century,ANNs can be categorized into five generations based on five core dimensions,and such generational evolution constitutes the core trajectory of neural network research and advancement. This study defines five generations of ANNs (abbreviated as xG-ANNs) from five perspectives: neuronal unit,information coding,network structure,learning mechanism,and Turing test. 1G-ANNs: Threshold logic networks,represented by the McCulloch-Pitts model and the perceptron; 2G-ANNs: Continuous activation networks (e.g.,sigmoid,tanh),typified by the classical back-propagation network; 3G-ANNs: Represented by spiking neural networks 4G-ANNs: Deep neural networks,represented by AlexNet,ResNet,Transformer architectures,and the attention mechanism,which have passed the conversational Turing test for disembodied intelligence; 5G-ANNs: Cognitive neural networks,with five defining characteristics: 1) cognitive units that integrate memory,reasoning,and human-like attention; 2) hybrid coding of semantic symbols and distributed representations; 3) modular cognitive architecture with dynamic reconfigurable topology; 4) meta-learning,causal reasoning,and lifelong learning; and 5) the embodied Turing test that remains as a breakthrough. A core academic consensus has been formed regarding 5G-ANNs. That is,such networks integrate neural computing,symbolic reasoning,and cognitive architecture with low-power consumption,support dynamic topology,memory cognition,neuro-symbolic fusion,and embodied/world models. They exhibit intrinsic advantages,including adaptive structure,few-shot generalization,interpretability,low energy consumption,and embodied cognition. At present,the field is in the era of 4G-ANNs,which are characterized by data-driven fitting,deep learning,the attention mechanism,and Transformer frameworks. Represented by large language model-based ChatGPT,4G-ANNs have passed the conversational Turing test,but such validation is a “black box” assessment that is restricted to the emergence of disembodied intelligence. The root causes lie in the inherent asymmetry of large language models built on the scaling law of parameter expansion,their lack of comprehension of physical laws that govern the real world,and critical drawbacks,including jagged multimodal and multiform intelligent output and inferior energy efficiency. By contrast,neural networks that are deployed in embodied intelligent robots lack autonomous intelligence and can only execute predefined actions per programmed instructions,leading to a huge gap between disembodied intelligence and embodied intelligence in 4G-ANN systems. Addressing these critical limitations of 4G-ANNs calls for the support of novel theories,models,and architectural designs. At present,extensive debates and divergence persist over the developmental orientation and technical routes of next-generation ANNs. This study analyzes and summarizes the developmental progress of mainstream theories,models,and architectures of the first four generations of ANNs. It focuses on the characteristics of several typical 4G-ANN models and their enhanced variants,and reviews representative architectures for 5G-ANNs,including world models,joint embedding predictive architecture,cognitive spiral models,and intelligence-trace cellular network frameworks. Finally,grounded in theory of cognitive physics and the practical technologies of driving brain cognition,this work proposes a lightweight architecture,called embodied cognitive physics neural network (E-CoPNN),which incorporates the core characteristics of 5G-ANNs. In particular,E-CoPNN features dynamic topology (three-layer nested structure and dual systems),memory cognition (three categories of memory),neuro-symbolic fusion (integration of Euclidean and non-Euclidean data,combination of lexemes and concepts),and embodied cognition/world models (fusion of disembodied and embodied intelligence,integration of cognitive space and physical space). The proposed architecture embodies the typical properties of 5G-ANNs: an adaptive structure,few-shot generalization,interpretability,low power consumption,and embodied cognition. Targeting brain-like general intelligence,5G-ANNs regard dynamic topology,autonomous memory,cognitive reasoning,and structural evolution as core attributes,shifting the ANN research paradigm from data fitting to structural reconstruction,and thus,representing a pivotal direction for achieving machine autonomous intelligence and continuous learning.Conclusion and Significance: At present,the development of next-generation ANNs for the 21st century will drive a series of transformative revolutions: philosophically,a paradigm shift from mind-body dualism to embodied cognitive science based on embodied perception monism; theoretically,a disciplinary expansion from 20th century biophysics to 21st century cognitive physics; model-wise,a fundamental transition of ANN research from data fitting into structural reconstruction; in application,bridging the long-standing gap between disembodied intelligence and embodied intelligence in ANN development; generationally,an evolutionary leap from 4G-ANNs to 5G-ANNs characterized by brain-like cognitive capabilities and adaptive topological structures. This work lays a solid foundation for the widespread application of embodied cognitive robots with learning,self-development,self-correction,and human-robot interaction capabilities. Furthermore,it supports the convergent development of nano-bio-info-cogno,or the four cutting-edge technologies led by cognitive integration. Finally,this work enhances human intellectual competence and ushers in a new round of cognitive revolution.
Driven by the rapid advancement of deep learning and large-scale computing,visual perception systems have achieved remarkable progress in a wide range of domains,including autonomous driving,intelligent transportation,security surveillance,medical diagnostics,industrial inspection,and human-robot interaction. Modern vision algorithms,empowered by massive datasets and increasingly sophisticated neural architectures,are now capable of performing object detection,semantic segmentation,scene understanding,and even causal reasoning with unprecedented accuracy. Despite this rapid growth,the development of visual intelligence still faces several critical bottlenecks. Chief among these challenges are the highly imbalanced data distributions that characterize real-world environments,the scarcity of long-tail or rare-event samples essential for robustness,and the substantial human and financial cost associated with large-scale manual annotation. These factors significantly hinder the performance,safety,and generalizability of deep perception systems,especially in complex,dynamic,or safety-critical scenarios. Parallel images technology,emerging as a novel image generation and modeling methodology grounded in parallel systems theory and the artificial systems,computational experiments,and parallel execution (ACP) framework,offers a promising pathway to address these limitations. The core idea behind parallel images is to construct controllable,high-fidelity artificial scene systems that reflect the structure,behavior,physics,and semantics of their real-world counterparts. Within these artificial systems,computational experiments can be conducted at scale,allowing for the controlled generation of diverse visual data that capture variations in illumination,geometry,environmental conditions,sensor characteristics,and task-specific factors. Through the interaction and iterative feedback between virtual and real environments,parallel images establish a closed-loop mechanism of “modeling-training-feedback-optimization”,enabling perception models to evolve continuously,validate hypotheses,and improve performance under systematically generated variations. This closed-loop mechanism differentiates parallel images from traditional synthetic data generation in several important aspects. First,instead of passively producing static rendered images,parallel images emphasize dynamic parallelism,where virtual agents,environments,and tasks evolve in sync with real-world processes. Second,the approach integrates multimodal feedback,bridging visual,geometric,physical,and semantic modalities to ensure consistency and translatability across domains. Third,the framework supports scalable modeling of rare,dangerous,or expensive scenarios that are difficult or impossible to capture in real life,such as near-crash events in autonomous driving,rare diseases in medical imaging,or hazardous industrial operations. These capabilities make parallel images a powerful tool for enhancing the robustness,safety,and domain generalization of modern perception systems. This paper provides a comprehensive,systematic review of the theoretical foundations,methodological innovations,and developmental trajectory of parallel images technology. This paper begins by revisiting its roots in parallel intelligence and the ACP paradigm and detailing how artificial systems serve as controlled experimental platforms that complement real-world data collection. Then,this paper examines recent technical advances encompassing three major research directions aligned with the ACP framework: 1) multimodal data-driven virtual scene generation,which employs generative adversarial networks,diffusion models,neural radiance fields,and 3D Gaussian splatting to overcome data scarcity and annotation bottlenecks,enabling the creation of controllable,editable,and semantically consistent synthetic environments; 2) multiview feature fusion and virtual-real model transfer,aimed at addressing feature discrepancies and semantic misalignment across heterogeneous visual modalities through cross-modal alignment,multigranularity adaptive transfer,and domain-bridging strategies that enhance generalization and adaptability in hybrid virtual-real environments; 3) parallel reasoning through heterogeneous data and knowledge fusion,which integrates structured information extraction,external knowledge guidance,scene graphs,temporal logic,and large language models to advance perceptual understanding toward semantic-level reasoning and decision-making,thereby supporting continuous optimization and closed-loop evolution in complex scenes. Beyond summarizing technological developments,this paper also situates parallel images within the broader context of emerging trends in generative artificial intelligence and foundation models. With the rise of diffusion models,neural radiance fields,and large-scale multimodal models,parallel images are poised to integrate more deeply with generative simulation pipelines. This paper discusses how these innovations can strengthen the fidelity,controllability,and adaptability of artificial visual data and potentially enable new capabilities such as task-conditioned scene synthesis,human-AI cosimulation,interactive data generation,and closed-loop autonomous scenario exploration,and provides key capabilities for building general visual systems with continuous learning and feedback optimization. This paper provides crucial support for building general visual systems with continuous learning and feedback-driven optimization. Finally,this paper identifies several open challenges and future research directions that are essential for advancing the development of parallel images systems. These challenges include achieving high-quality expansion of virtual data,bridging the semantic gap between virtual and real domains,and enabling real-time,tightly coupled virtual-real interaction. Addressing these issues will require advances in intelligent generation models,self-supervised quality evaluation,unified data standards,causality-aware cross-domain alignment,and low-latency virtual-real collaboration supported by next-generation communication and sensing technologies. Solving these challenges will be critical for pushing forward the frontier of synthetic visual intelligence and unlocking the full potential of parallel images in real-world applications.
Remote sensing technology serves as the core mechanism for the observation of the Earth and the understanding of surface environments. It plays an irreplaceable role in critical fields,such as natural disaster monitoring,urban planning,resource exploration,and ecological protection. Driven by the rapid advancement of deep learning over the past decade,the intelligent interpretation of remote sensing images has achieved breakthrough progress in fundamental vision tasks. However,the traditional deep learning paradigm is intrinsically built upon a close-set assumption,i.e.,that models can only recognize a predefined and human-annotated set of fixed categories during the inference stage. When confronted with highly complex surface environments in real-world Earth observation scenarios,dynamic object morphology,and rare ground objects with long-tail distributions,this traditional paradigm not only incurs prohibitive costs for the construction of massive pixel-level annotated datasets but also easily falls into the trap of domain-specific overfitting. Consequently,the generalization and response capabilities of this paradigm are severely challenged by unseen categories or sudden events,making this paradigm inadequate for meeting the highly dynamic interpretation demands of the open world. In recent years,the rapid development of vision-language models has catalyzed a paradigm shift in artificial intelligence from task-specific models into general-purpose perception models. By mapping visual representations and natural language into a unified feature space through contrastive learning on massive image-text pairs,these models have broken the constraints of discrete labels,enabling a direct response to arbitrary natural language prompts. This capability is known as open vocabulary perception. Although this technology has demonstrated remarkable zero-shot generalization and cross-modal reasoning capabilities in the natural image domain,the direct application of these general vision-language models to the remote sensing domain encounters a severe domain gap. The uniqueness of remote sensing data poses multiple challenges to the adaptability of existing models. First,the distinct overhead imaging perspective causes drastic variations in object scale and complex background textures. Second,Earth observation tasks rely on multisource heterogeneous data from synthetic-aperture radar,multispectral or hyperspectral imaging,and thermal infrared sensors. The underlying physical mechanisms of these sensors exceed the inherent inductive biases of models that are pretrained solely on natural RGB images. Third,remote sensing objects often exhibit strong geospatial attributes and complex topological associations. To address these critical challenges,this study provides a comprehensive and systematic review of recent advancements in open vocabulary perception for remote sensing images. We first delve into the foundational aspect of this field: vision-language pretraining for remote sensing. We extensively review the evolution of construction strategies for large-scale datasets. We highlight the transition from limited,human-annotated image-text pairs into massive datasets generated via heuristic rules,the integration of geographic metadata,and advanced multimodal large language models,including innovative approaches that leverage OpenStreetMap and geographical coordinates to produce fine-grained,physics-aware descriptions across multiple modalities. Concurrently,we systematically summarize the progression of pretraining methodologies. Although early approaches have primarily focused on simple domain adaptation through continuous pretraining,recent state-of-the-art frameworks emphasize physics-aware encoding,fine-grained multilevel consistency learning,and geography-enhanced architectures. These frameworks better capture the intricate spatial relationships and modality diversities that are inherent in Earth observation data. Subsequently,this review conducts an in-depth analysis of the adaptation and optimization of open vocabulary perception techniques across a wide spectrum of crucial downstream tasks. For zero-shot scene classification and cross-modal retrieval,we discuss advanced strategies that are designed to mitigate the high intra-class similarity and complex interclass variances typical in remote sensing. We emphasize the shift toward fine-grained local-global alignment,hard negative mining,dynamic soft labeling,and prompt engineering. In the realm of open vocabulary image segmentation,we categorize the existing literature into training-based methods and training-free or annotation-free paradigms. Training-based methods leverage base categories to adapt models while preventing catastrophic forgetting through pseudo-label distillation and knowledge retention mechanisms. Training-free paradigms synergize foundational models,such as CLIP and the segment anything model,to extract structural masks and align semantics without the updating of network weights. For open vocabulary object detection and remote sensing visual grounding,we explore the approaches of researchers to deal with extreme scale variations,arbitrary orientations,and dense object distributions. These approaches include innovative frameworks for pseudo-label generation,multi-scale feature alignment,cross-modality context modeling,and interactive grounding mechanisms. Furthermore,we examine open vocabulary change detection,wherein recent studies employ either combinations of pretrained vision-language models or generative models to generate large-scale data. These approaches aim to identify arbitrary,text-specified surface transitions and simulate complex spatiotemporal changes without reliance on massive and costly bitemporal pixel-level annotations. We also briefly discuss emerging open vocabulary applications in 3D urban point clouds and cross-domain archeological remote sensing,illustrating the expanding horizon of this technology. Despite remarkable progress,the field of open vocabulary perception for remote sensing remains in a crucial developmental stage and faces several critical bottlenecks. This study critically identifies the limitations of current research,including the severe scarcity of high-quality and geographically balanced training data. This scarcity leads to geographic biases and performance degradation in data-poor regions. In addition,genuinely fine-grained and long-tailed open vocabulary evaluation benchmarks that can accurately reflect the performance of a model are prominently absent in extreme or unknown real-world scenarios. The inadequate physical understanding of heterogeneous modalities and the inherent black box unreliability of current large models in high-stake decision-making scenarios further constrain practical deployments. To chart the course for future research,we outline several promising and essential trajectories. First,we anticipate a paradigm shift toward generative perception that is driven by multimodal large language models. This shift unifies various spatial localization tasks into the direct generation of coordinate sequences or geometric property tokens to utilize fully the logical reasoning capabilities of foundational models. Second,we strongly advocate for the construction of rigorous,real-world,and fine-grained evaluation systems that incorporate complex spatiotemporal logic,diverse geographic conditions,and comprehensive evaluation metrics. Third,the development of omni-modal foundation models that explicitly integrate physical priors and deep learning is considered crucial for the achievement of all-weather and all-spectrum Earth observations,moving beyond pure data-driven approaches. Furthermore,we highlight the necessity to extend perception from static spatial analysis to dynamic spatiotemporal causal reasoning to decode the evolutionary processes of the Earth. Finally,addressing the severe conflict between the massive parameter scale of foundation models and the limited computing power of aerospace edge devices requires focused research into efficient,trustworthy,and safe edge-cloud collaborative computing architectures. By systematically synthesizing these advancements and challenges,this comprehensive review aims to serve as a foundational road map for researchers and practitioners. It accelerates the transition of the intelligent interpretation of remote sensing from isolated,close-set recognition toward artificial general intelligence that is capable of highly reliable,dynamic,and open world perception.
With the rapid emergence of large multimodal models (LMMs),massive heterogeneous data are proliferating across various industrial and scientific domains. In this context,multi-view clustering (MVC) serves as a cornerstone technology for unsupervised knowledge discovery and latent correlation mining. At present,MVC is undergoing a profound and historical paradigm shift. Traditional surveys predominantly focus on the horizontal categorization of algorithmic network structures. However,this approach often fails to reveal the intrinsic evolutionary logic across different technological eras. Departing from these conventions,this study proposes a pioneering,prior-driven theoretical perspective to reconstruct systematically the developmental trajectory of MVC over the past two decades. This goal is achieved through a trans-paradigm analytical framework of geometry-semantics-cognition. During the initial stage of shallow structural mining,the research paradigm focused on explicit mathematical constraints within original or kernel-induced feature spaces. Euclidean space methods,such as multi-view K-Means and nonnegative matrix factorization,identify global prototypes by minimizing squared error or Frobenius norm reconstruction loss. By contrast,affine space methods leverage self-representation properties to model data as a union of low-dimensional subspaces,while manifold space techniques utilize spectral graph theory to transform clustering into optimal graph cut problems by preserving local topological correlations. As the field transitioned into deep spatial modeling based on semantic collaborative priors,researchers utilized the powerful nonlinear mapping capabilities of deep neural networks to project heterogeneous data into high-order semantic spaces. This evolution encompasses several distinct research paradigms. 1) Embedding space research focuses on deep subspace clustering,employing autoencoders to learn discriminative features while maintaining cross-view consistency. 2) Latent space methods utilize probabilistic generative models,such as variational autoencoders and generative adversarial networks,to align latent distributions and infer missing view information through adversarial games. 3) Augmented space paradigms introduce contrastive learning to maximize mutual information between views via InfoNCE-like losses,enhancing representation robustness. 4) Topological space studies leverage graph neural networks to mine intra-view geometric structures and inter-view complementary semantics synergistically. Moving into the current era of LMMs,this study progressively explores deep alignment based on cognitive priors,where LMMs are viewed not merely as data sources but as knowledge bases that contain human-level common sense and logical reasoning capability. We systematically elucidate the infrastructural role of MVC in empowering massive data governance. Specifically,MVC facilitates semantic deduplication to enhance data quality and employs token-level clustering to optimize mixture-of-experts(MoE) routing for expert specialization. Furthermore,MVC enables hierarchical semantic chunking,which is critical for precise document retrieval within retrieval-augmented generation(RAG) frameworks. Beyond these applications,we analyze the potential for LMM logical reasoning to back-propagate into clustering tasks. This synergy elevates MVC from pure statistical feature alignment to a new dimension of knowledge-driven cognitive logic consistency. Beyond theoretical frameworks,the review highlights the transformative effect of MVC in diverse real-world scenarios,ranging from multi-omics fusion in smart healthcare for cancer subtyping and spatial-spectral fusion in remote sensing for urban functional zone identification to cross-perspective pedestrian recognition in public security and unsupervised anomaly detection in industrial internet of things networks. Despite these advancements,achieving a transition from laboratory benchmarks to robust industrial infrastructure requires addressing several core challenges. First,the scaling bottleneck where in the O(N2) or O(N3) computational complexity of traditional methods must be reduced to linear levels to handle million-scale web data. Second,the extreme robustness required in open environments to handle not only severely incomplete views but also the pervasive long-tailed category imbalance where minority abnormal samples are frequently overwhelmed by dominant patterns. Third,the urgent need for evaluation system reconstruction to move away from legacy small-scale datasets with shallow features toward native heterogeneous multimodal benchmarks that reflect real-world weak alignment and complex noise distributions. Ultimately,this survey aims to provide a novel research road map for theoretical innovation and engineering practice,advocating for a transition toward intelligent decision systems characterized by high-level semantic decoupling and cognitive logic alignment in the era of LMMs.
Nighttime perception remains a critical bottleneck for the autonomous operation of unmanned aerial vehicles (UAVs). Under low-light conditions,weak and uneven illumination amplifies noise and glare,background interference becomes increasingly complex,and targets often appear small,low in contrast,or partially occluded. Meanwhile,UAV platforms impose strict constraints on onboard computation and power consumption,requiring detection and tracking methods to achieve balance between robustness and efficiency. This paper presents a comprehensive review and empirical study of the application of deep learning in UAV nighttime target detection and tracking by synthesizing methodological trends,experimental findings,and future research directions. We characterize the unique challenges of UAV nighttime vision in contrast to daytime scenarios and highlight four interrelated issues: 1) limited perceptual signals and variable illumination,leading to degraded pixel-level cues; 2) feature degradation and background interference caused by artificial light sources and shadows; 3) restricted onboard resources that make the deployment of large-scale models difficult; and 4) complex imaging conditions,where target scale and appearance fluctuate frequently. These constraints underscore the need for specialized algorithmic paradigms rather than direct transfer of daytime methods. From the perspective of UAV nighttime target detection,this study systematically reviews four major research directions. First,nighttime image enhancement methods employ dedicated restoration networks to improve brightness and contrast before detection. Although these approaches can effectively enhance visual quality,the two-stage pipeline may introduce artifacts and does not always optimally align with detector objectives. Second,domain adaptation methods leverage adversarial learning,style transfer,or feature alignment to bridge the day-night distribution gap. Such methods reduce the dependence on annotated nighttime data but remain sensitive to interdomain diversity and often require carefully designed pseudo-labeling or regularization. Third,multimodal perception fusion,especially RGB-infrared (IR) fusion,utilizes complementary thermal cues to mitigate illumination deficiencies. State-of-the-art fusion networks (e.g.,M2D-LIF) have demonstrated substantial mean average precision (mAP) improvements on UAV nighttime benchmarks,with some models achieving over 80% mAP@0.5 on the DroneVehicle-Night subset. Fourth,lightweight models (e.g.,CSPDarkNet-based backbones) focus on balancing accuracy and efficiency under limited computational budgets. Experiments indicate that backbone choice is critical for enabling real-time nighttime detection without a considerable accuracy loss. For nighttime target tracking,this study categorizes four representative paradigms. 1) Enhancement-before-tracking: low-light enhancers are used as a preprocessing stage (sometimes co-optimized) to improve input quality and thereby downstream tracker performance. 2) Domain adaptation: extending static feature alignment to temporal modeling,sometimes combined with synthetic nighttime sequences or foundation models,to enhance robustness. 3) Prompt learning: reframing day-night transfer as a fast adaptation task and leveraging small-scale prompts for efficient transfer with minimal parameter updates. 4) Curriculum learning: ordering training samples from easy to difficult (e.g.,progressively increasing darkness or motion blur) to stabilize training and improve performance under extreme degradation. 5) Multimodal fusion: we further analyze the interplay among these paradigms and the tension between restoration- and task-oriented objectives. Subsequently,commonly used evaluation metrics and benchmark datasets for nighttime and all-weather UAV detection and tracking are introduced,and a nighttime UAV vehicle detection dataset called DroneVehicle-Night is constructed. Two complementary empirical studies are conducted to support this survey. The first study,which is based on DroneVehicle-Night (derived from DroneVehicle),contains tens of thousands of paired RGB-IR annotated images covering urban roads,residential areas,and parking lots,with five classes of vehicles annotated. Experiments reveal that under identical training conditions,IR consistently outperforms RGB (e.g.,CSPDarkNet-based detectors achieve high mAP@0.5 in IR),and multimodal fusion methods,such as M2D-LIF,achieve over 83.9% mAP@0.5,underscoring the value of modality complementarity in nighttime detection. The second study evaluates the transferability of generic UAV detectors to the nighttime scenario. From VisDrone2019,we construct a paired subset (521 nighttime/513 daytime images). Although generic detectors achieve 30%-45% mAP@0.5 on daytime data,their performance declines by 8%-12% at night,with small and low-contrast objects exhibiting the most severe degradation. These findings confirm that specialized nighttime and modality-aware methods are effective,and simple day-night transfer is insufficient. To further reflect the state of literature on nighttime tracking,this study summarizes the performance of 20 representative trackers on UAVDark135 and NAT2021 (results are cited from the original publications). Among them,Mamba-based methods (e.g.,MambaNUT-Small and MambaTrack) and several domain adaptation methods (e.g.,DARTer and DCPT) consistently rank among the top,achieving robust AUC-accuracy trade-offs while maintaining reasonable frame rates. These results highlight the importance of efficient sequence modeling and domain-aware adaptation in nighttime tracking. On the basis of the survey and empirical evidence,several key conclusions are derived. First,sensor modality selection is critical. IR and RGB-IR fusion substantially enhance robustness to low-light and complex backgrounds,whereas RGB-only pipelines remain fragile. Second,backbone design strongly influences the accuracy-efficiency balance. Compared with heavy backbones,such as ResNet50,CSPDarkNet variants offer more attractive real-time trade-offs on embedded platforms. Third,aside from frame-by-frame strategies,explicit temporal modeling and sequence-oriented architectures are indispensable for robust nighttime tracking. Fourth,day-night transfer remains a major challenge,as evidenced by the VisDrone experiments that highlight the urgent need for specialized nighttime designs and datasets. Finally,four promising research directions are identified: 1) cross-modal vision-language fusion that leverages large-scale language/vision models to enhance multimodal alignment and interactive tracking; 2) construction of rich multimodal nighttime datasets that extend beyond urban scenes to cover wild,mountainous,and aquatic environments; 3) unsupervised and weakly supervised learning to reduce annotation costs and improve cross-domain robustness; and 4) lightweight architecture design. Aside from accuracy,UAV tasks demand real-time and efficient models. Future work may explore long-sequence modeling with Mamba in combination with neural architecture search,dynamic inference,and efficient operators to balance accuracy,latency,and energy. In conclusion,this survey and empirical study provide a clear roadmap toward robust UAV nighttime perception and practical deployment. All algorithms and datasets used in this work are publicly available at https://github.com/bsfsf/DroneVehicle-Night and https://doi.org/10.57760/sciencedb.32435, to facilitate future research.