收藏切换
Multimodal Emotion Recognition Method Based on Mixture of Fusion Experts
收藏切换
Haifeng Chen1, Xi Li1, Yan Li2, Jian Li3, *, Lang He4, Dongmei Jiang5
Journal of Computer Research and Development | 2026, 63(6) : 1478 - 1491
Less
收藏切换
Journal of Computer Research and Development | 2026, 63(6): 1478-1491
• Artificial Intelligence •
Multimodal Emotion Recognition Method Based on Mixture of Fusion Experts
Full
Haifeng Chen1, Xi Li1, Yan Li2, Jian Li3, *, Lang He4, Dongmei Jiang5
Affiliations
  • 1School of Electronic Information and Artificial Intelligence, Shaanxi University of Science & Technology, Xi’an 710021
  • 2School of Computer Science, Northwestern Polytechnical University, Xi’an 710072
  • 3School of Electrical and Control Engineering, Shaanxi University of Science & Technology, Xi’an 710021
  • 4School of Computer Science & Technology, Xi’an University of Posts and Telecommunications, Xi’an 710061
  • 5Pengcheng Laboratory, Shenzhen, Guangdong 518055
  • born in 1988. PhD, lecturer. Member of CCF. His main research interests include affective computing, multimodal emotion recognition, emotion recognition in conversations, multimodal depression analysis, and facial action unit recognition

    born in 2001. Master candidate. Student member of CCF. Her main research interest includes multimodal emotion recognition

    born in 1998. Master candidate. His main research interests include machine learning and multimodal understanding

    born in 1975. PhD, professor. His main research interests include computer vision and digital twin

    born in 1983. PhD, associate professor. His main research interests include multimodal depression recognition and analysis, discrete emotion analysis, dimensional emotion recognition, and image processing

    born in 1973. PhD, professor. Her main research interests include audio-visual fusion based speech processing, audio-visual emotion analysis and recognition, and facial animation synthesis of speakers with expressions

About Author:

born in 1988. PhD, lecturer. Member of CCF. His main research interests include affective computing, multimodal emotion recognition, emotion recognition in conversations, multimodal depression analysis, and facial action unit recognition

born in 2001. Master candidate. Student member of CCF. Her main research interest includes multimodal emotion recognition

born in 1998. Master candidate. His main research interests include machine learning and multimodal understanding

born in 1975. PhD, professor. His main research interests include computer vision and digital twin

born in 1983. PhD, associate professor. His main research interests include multimodal depression recognition and analysis, discrete emotion analysis, dimensional emotion recognition, and image processing

born in 1973. PhD, professor. Her main research interests include audio-visual fusion based speech processing, audio-visual emotion analysis and recognition, and facial animation synthesis of speakers with expressions

Published: 2026-06-01 doi: 10.7544/issn1000-1239.202550544
Outline
收藏切换

Multimodal emotion recognition aims to integrate data from multiple modalities for accurate inference of emotional states. Existing research either treats all modalities equally or adopts fixed fusion strategies based on a single modality, failing to adequately address the imbalance of modality contributions. To tackle this, we propose a dynamic multimodal fusion method MoMFE based on the mixture of experts (MoE) framework, incorporating an adaptive router module and a multimodal fusion expert module. This method dynamically evaluates the contribution of each modality and selects appropriate fusion strategies accordingly. The adaptive router mechanism dynamically assesses modality contributions through inter-modal correlation analysis and dynamic weighting to guide the selection of fusion experts, while an expert-guided loss function is integrated to further optimize the expert selection process. The multimodal fusion experts perform complementary fusion for different modality contribution combinations and introduce a shared expert to mitigate the loss of global information and reduce parameter redundancy. Comparative experiments on three benchmark datasets (MER2024, CMU-MOSEI, and CH-SIMS) for multimodal emotion recognition demonstrate that MoMFE method outperforms state-of-the-art (SOTA) multimodal fusion methods in core metrics, including binary emotion recognition accuracy (Acc-2) and F1 score. Notably, the method achieves an average improvement of approximately 2% on CH-SIMS dataset.

emotion recognition  /  multimodal fusion  /  mixture of experts  /  adaptive router  /  cross-modal attention
陈海丰, 李溪, 李岩, 李健, 何浪, 蒋冬梅. 基于混合融合专家的多模态情感识别方法. 计算机研究与发展, 2026 , 63 (6) : 1478 -1491 . DOI: 10.7544/issn1000-1239.202550544
Haifeng Chen, Xi Li, Yan Li, Jian Li, Lang He, Dongmei Jiang. Multimodal Emotion Recognition Method Based on Mixture of Fusion Experts[J]. Journal of Computer Research and Development, 2026 , 63 (6) : 1478 -1491 . DOI: 10.7544/issn1000-1239.202550544
Year 2026 volume 63 Issue 6
83
40
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202550544
  • Receive Date:2025-07-20
  • Online Date:2026-11-17
  • Published:2026-06-01
Article Data
Affiliations
History
  • Received:2025-07-20
  • Revised:2025-12-15
  • Accepted:2026-01-07
Affiliations
    1School of Electronic Information and Artificial Intelligence, Shaanxi University of Science & Technology, Xi’an 710021
    2School of Computer Science, Northwestern Polytechnical University, Xi’an 710072
    3School of Electrical and Control Engineering, Shaanxi University of Science & Technology, Xi’an 710021
    4School of Computer Science & Technology, Xi’an University of Posts and Telecommunications, Xi’an 710061
    5Pengcheng Laboratory, Shenzhen, Guangdong 518055
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202550544
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT