收藏切换
Multimodal Learning Method Based on Intra- and Inter-Sample Cooperative Representation and Adaptive Fusion
收藏切换
Xuejian Huang1, 2, Tinghuai Ma2, *, Gensheng Wang3
Journal of Computer Research and Development | 2024, 61(5) : 1310 - 1324
Less
收藏切换
Journal of Computer Research and Development | 2024, 61(5): 1310-1324
• Artificial Intelligence •
Multimodal Learning Method Based on Intra- and Inter-Sample Cooperative Representation and Adaptive Fusion
Full
Xuejian Huang1, 2, Tinghuai Ma2, *, Gensheng Wang3
Affiliations
  • 1College of VR Modern Industry, Jiangxi University of Finance and Economics, Nanchang 330013
  • 2College of Computer, Nanjing University of Information Science and Technology, Nanjing 210044
  • 3College of Information Management, Jiangxi University of Finance and Economics, Nanchang 330013
  • born in 1990. PhD candidate, lecturer. Member of CCF. His main research interests include multimodal machine learning and social network analysis

    born in 1974. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include social network privacy protection, big data mining, and text emotion computing

    born in 1974. PhD, professor, PhD supervisor. Member of CCF. His main research interests include data mining and social network

About Author:

born in 1990. PhD candidate, lecturer. Member of CCF. His main research interests include multimodal machine learning and social network analysis

born in 1974. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include social network privacy protection, big data mining, and text emotion computing

born in 1974. PhD, professor, PhD supervisor. Member of CCF. His main research interests include data mining and social network

Published: 2024-05-01 doi: 10.7544/issn1000-1239.202330722
Outline
收藏切换

Multimodal machine learning represents a novel paradigm in artificial intelligence, leveraging various modalities and intelligent processing algorithms to achieve enhanced performance. Multimodal representation and fusion are two pivotal tasks in multimodal machine learning. Currently, most multimodal representation methods pay little attention to inter-sample collaboration, leading to a lack of robustness in feature representation. Additionally, most multimodal feature fusion methods exhibit sensitivity to noisy data. Therefore, in the realm of multimodal representation, an approach based on both intra-sample and inter-sample multimodal collaboration is proposed to facilitate a comprehensive understanding of interactions within and between modalities, ultimately enhancing the robustness of feature representation. Firstly, text, speech, and visual features are individually extracted based on pre-trained models such as BERT, Wav2vec 2.0, and Faster R-CNN. Subsequently, considering the complementarity and consistency of multimodal data, two categories of encoders, namely modality-specific and modality-shared, are constructed to learn both modality-specific and shared feature representations. Furthermore, intra-sample collaboration loss functions are formulated using central moment differences and orthogonality, while inter-sample collaboration loss functions are established using contrastive learning. Lastly, a representation learning function is designed based on intra-sample collaboration, inter-sample collaboration, and sample reconstruction errors. Regarding multimodal fusion, an adaptive multimodal feature fusion method is designed, accounting for the possibility that each modality may exhibit varying types of effects and levels of noise at different times, using attention mechanisms and gated neural networks. Experimental results on the multimodal intent recognition dataset MIntRec and emotion datasets CMU-MOSI and CMU-MOSEI demonstrate that this multimodal learning approach outperforms baseline methods across multiple evaluation metrics.

multimodal representation  /  multimodal fusion  /  multimodal learning  /  collaborative representation  /  adaptive fusion
黄学坚, 马廷淮, 王根生. 基于样本内外协同表示和自适应融合的多模态学习方法. 计算机研究与发展, 2024 , 61 (5) : 1310 -1324 . DOI: 10.7544/issn1000-1239.202330722
Xuejian Huang, Tinghuai Ma, Gensheng Wang. Multimodal Learning Method Based on Intra- and Inter-Sample Cooperative Representation and Adaptive Fusion[J]. Journal of Computer Research and Development, 2024 , 61 (5) : 1310 -1324 . DOI: 10.7544/issn1000-1239.202330722
Year 2024 volume 61 Issue 5
82
40
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202330722
  • Receive Date:2023-09-06
  • Online Date:2026-11-17
  • Published:2024-05-01
Article Data
Affiliations
History
  • Received:2023-09-06
  • Revised:2024-02-20
Affiliations
    1College of VR Modern Industry, Jiangxi University of Finance and Economics, Nanchang 330013
    2College of Computer, Nanjing University of Information Science and Technology, Nanjing 210044
    3College of Information Management, Jiangxi University of Finance and Economics, Nanchang 330013
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202330722
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT