The rational and effective use of multisource information tend to expand the spatial and temporal coverage of measurements,fully excavate the target feature information contained in various sensors,greatly reduce the ambiguity of information,and improve detection performance. Faced with multimodal target data from different types,characteristics,and perception methods in some applications,such as in airborne multisensor target-tracking systems,various types of sensors with different working attributes (e.g.,visible light images,infrared images,optical sensors,microwave radar,and LiDAR) are often involved,presenting measurement information of the target from different perspectives. The point cloud information obtained by LiDAR not only achieve high-precision ranging but also provide more accurate spatial information due to the varying reflectivities of different objects. However,this is susceptible to noise generated by complex environments and lacks semantic information. Compared to point cloud information,target image information provides rich surface textures and contextual semantic information that can more accurately restore the appearance and structure of objects; however,it often lacks in-depth information. The current landscape is marked by the rapid development of sensor networks,information perception methods,and big data processing technologies. Faced with information perception from different types,characteristics,and methods,multimodal information fusion technology has received increasing attention. This is due to its advantages of human-like automatic perception and powerful comprehensive reasoning capabilities; its multidomain and multimodal nature,as well as its ambiguity,and incomplete data association. The paper first introduces the significance and necessity of multimodal information fusion. Furthermore,at the fusion architecture level,it outlines the evolution from early fusion,late fusion,to hybrid fusion,revealing the trade-off relationship between information retention and computational efficiency in each architecture. At the same time,this paper elaborates on three cutting-edge multimodal fusion methods: attention-based interaction modeling,semantic alignment based on contrastive learning,and generative fusion with LLMs as the carrier. Then,it introduces typical multimodal datasets,such as the COCO dataset,LAION-400M dataset,Visual Genome dataset,MS COCO Captions dataset,Conceptual Captions dataset,Flickr30k dataset,and aviation-related datasets,along with their corresponding application fields. Moreover,the paper further explores the typical military applications of multimodal fusion,including battlefield situation awareness,multisource intelligence analysis,and unmanned system collaboration. This work also emphasizes that high-quality single-modal representations are no longer a bottleneck with the maturity of contrastive learning and multimodal pre-training. To date,the research focus has shifted from how to map heterogeneous modalities to a unified space in the early stage to how to design interaction mechanisms that can capture complex,dynamic,and even contradictory relationships between modalities based on representation alignment. On the basis of fusion architecture,fusion models,and computational costs,we propose three development directions for multimodal information fusion. We argue that the choice of multimodal fusion architecture directly impacts the degree of information retention,computational efficiency,and model interpretability. Thus,existing methods can be categorized into early fusion,late fusion,hybrid fusion based on the stage at which information fusion occurs. Early fusion,also known as data-level fusion,involves the integration of multisource information prior to or at the shallowest level of modality-specific feature extraction. This process preserves the richest original information and multimodal interaction details. Typical implementations include multimodal concatenation and multi-view encoding. Taking the visual-language model as an example,early fusion concatenates image patch embeddings and text word embeddings into a unified sequence,which is then fed into a joint encoder for processing. The advantage of this architecture lies in its ability to establish multimodal associations at a shallow level,thus facilitating the capture of fine-grained modality interactions. However,early fusion faces a severe dimensionality catastrophe problem. When the number of modalities increases or the feature dimensions of each modality become extremely high,the joint representation space grows exponentially. At the same time,during early fusion,the noise of some modalities may be amplified due to the varying noise characteristics of different modalities,resulting in poor representation quality. Meanwhile,late fusion adopts a diametrically opposite strategy in which each modality independently performs feature extraction and task prediction,with integration,weighted average,and meta-learner only at the final decision-making layer. In this “divide and conquer” design,the modal branches can be trained in parallel,achieving high computational efficiency; meanwhile,overfitting or noise in one modality is less likely to affect other modalities. Additionally,in scenarios wherein some modalities are missing,the remaining branches can still work normally,demonstrating strong system robustness. However,late fusion may lead to a higher consumption of computational resources as it requires training independent models for each modality. At the same time,independent models for each modality struggle to capture low-level interactions between them,making it difficult to model simple fusion at the decision-making level. Furthermore,data from different modalities may face alignment issues in time or space,such as the synchronization of video frames and audio signals. Hybrid fusion introduces cross-modal interaction in the middle layers of the network while maintaining modality-specific processing paths,aiming to combine the strengths of both. Its typical application is to use modality-independent encoders at the bottom layer,introduce a cross-attention module in the middle layer to achieve feature-level interaction,and separate again at the top layer to preserve modality-specific information. A deeper evolutionary direction is dynamic fusion,which allows the network to autonomously decide where and with what intensity information from various modalities can be fused. For example,a gated mechanism-based visual image and LiDAR multimodal fusion network dynamically adjusts the weights of information from each modality based on input data. Specifically,it places more trust in visual images under strong lighting conditions,while giving higher weight to LiDAR point cloud information under low-light conditions. The choice of fusion architecture does not have an “optimal solution”; rather,it depends on the function of task characteristics and resource constraints. Generally,when there are fine-grained,location-related interactions between modalities (e.g.,image-text alignment),early fusion is better. When each modality can complete predictions independently and some modalities are prone to missing,late fusion is more robust. Meanwhile,dynamic fusion strikes a good balance between performance and robustness. Looking back at the development of multimodal fusion,the bottleneck of fusion has shifted from “representation” to “interaction”. With the maturity of contrastive learning and multimodal pre-training,high-quality single-modality representation is no longer a bottleneck. In particular,the research focus has shifted from how to map heterogeneous modalities into a unified space in the early stage to how to design interaction mechanisms that can capture complex,dynamic,and even contradictory relationships between modalities based on representation alignment. Furthermore,fusion architecture has evolved from “static design” to “dynamic adaptation”. In the real world,the correlation and reliability of modalities change dynamically with the environment,and the limitations of fixed fusion strategies are becoming increasingly evident. In military confrontations,electronic jamming may render specific sensors inoperative. Therefore,to enhance the robustness of the fusion system the activation state of modalities and fusion weights must be dynamically adjusted on the basis of input content and task context. The causal fusion model holds promise for breaking through the current limitations of relational learning. At present,most existing methods focus on learning statistical correlations between modalities,rather than causal relationships. This leads to two issues. First,the model is prone to learning spurious correlations; second,it struggles to generalize to environments beyond the training distribution. Therefore,introducing causal inference tools can enhance the fusion model’s adversarial robustness and environmental transferability. Furthermore,a new multimodal fusion mechanism is designed in this paper. To date,redundant information from each modality may cause computational waste,and the corresponding noise information may affect fusion performance. Thus,how to utilize information theory techniques to identify and retain complementary information while suppressing redundancy and noise is a new development direction for multimodal fusion technology.
| 科 Family | 属数 Number of genus | 种数 Number of species | 占总种数比例 Percentage of total species (%) | 属 Genus | 种数 Number of species | 占总种数比例 Percentage of total species (%) |
|---|---|---|---|---|---|---|
| 鹅膏菌科Amanitaceae | 2 | 11 | 5.26 | 鹅膏菌属 Amanita | 10 | 4.78 |
| 小菇科 Mycenaceae | 2 | 12 | 5.74 | 丝盖伞属 Inocybe | 5 | 2.39 |
| 多孔菌科 Polyporaceae | 8 | 14 | 6.70 | 蜡蘑属 Laccaria | 5 | 2.39 |
| 红菇科 Russulaceae | 3 | 23 | 11.00 | 小皮伞属 Marasmius | 6 | 2.87 |
| 小菇属 Mycena | 11 | 5.26 | ||||
| 光柄菇属 Pluteus | 5 | 2.39 | ||||
| 红菇属 Russula | 17 | 8.13 | ||||
| 栓菌属 Trametes | 5 | 2.39 |