Article(id=1329419929369030861, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1329419793389695520, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202550544, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=null, receivedDate=1752940800000, receivedDateStr=2025-07-20, revisedDate=1765728000000, revisedDateStr=2025-12-15, acceptedDate=1767715200000, acceptedDateStr=2026-01-07, onlineDate=1794888010589, onlineDateStr=2026-11-17, pubDate=1780243200000, pubDateStr=2026-06-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1794888010589, onlineIssueDateStr=2026-11-17, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1794888010589, creator=13701087609, updateTime=1794888010589, updator=13701087609, issue=Issue{id=1329419793389695520, tenantId=1146029695717560320, journalId=1324759914146529478, year='2026', volume='63', issue='6', pageStart='1377', pageEnd='1638', issueExtLink='null', onlineDate='null', pubDate='1780243200000', pubDateStr='2026-06-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1794887978168, creator='13701087609', updateTime=1794892139295, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext={EN=IssueExt(id=1329437246815580885, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1329419793389695520, language=EN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=), CN=IssueExt(id=1329437246815580886, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1329419793389695520, language=CN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=)}, issueFiles=null, downloadFileDto=null}, startPage=1478, endPage=1491, ext={EN=ArticleExt(id=1329419929566163151, articleId=1329419929369030861, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=Multimodal Emotion Recognition Method Based on Mixture of Fusion Experts, columnId=1325489891789910943, journalTitle=Journal of Computer Research and Development, columnName=Artificial Intelligence, runingTitle=null, highlight=null, articleAbstract=

Multimodal emotion recognition aims to integrate data from multiple modalities for accurate inference of emotional states. Existing research either treats all modalities equally or adopts fixed fusion strategies based on a single modality, failing to adequately address the imbalance of modality contributions. To tackle this, we propose a dynamic multimodal fusion method MoMFE based on the mixture of experts (MoE) framework, incorporating an adaptive router module and a multimodal fusion expert module. This method dynamically evaluates the contribution of each modality and selects appropriate fusion strategies accordingly. The adaptive router mechanism dynamically assesses modality contributions through inter-modal correlation analysis and dynamic weighting to guide the selection of fusion experts, while an expert-guided loss function is integrated to further optimize the expert selection process. The multimodal fusion experts perform complementary fusion for different modality contribution combinations and introduce a shared expert to mitigate the loss of global information and reduce parameter redundancy. Comparative experiments on three benchmark datasets (MER2024, CMU-MOSEI, and CH-SIMS) for multimodal emotion recognition demonstrate that MoMFE method outperforms state-of-the-art (SOTA) multimodal fusion methods in core metrics, including binary emotion recognition accuracy (Acc-2) and F1 score. Notably, the method achieves an average improvement of approximately 2% on CH-SIMS dataset.

, authors=Haifeng Chen1, Xi Li1, Yan Li2, Jian Li3, *, Lang He4, Dongmei Jiang5, authorsList=Haifeng Chen, Xi Li, Yan Li, Jian Li, Lang He, Dongmei Jiang, authorCompany=null, correspAuthors=Jian Li, authorNote=

born in 1988. PhD, lecturer. Member of CCF. His main research interests include affective computing, multimodal emotion recognition, emotion recognition in conversations, multimodal depression analysis, and facial action unit recognition

born in 2001. Master candidate. Student member of CCF. Her main research interest includes multimodal emotion recognition

born in 1998. Master candidate. His main research interests include machine learning and multimodal understanding

born in 1975. PhD, professor. His main research interests include computer vision and digital twin

born in 1983. PhD, associate professor. His main research interests include multimodal depression recognition and analysis, discrete emotion analysis, dimensional emotion recognition, and image processing

born in 1973. PhD, professor. Her main research interests include audio-visual fusion based speech processing, audio-visual emotion analysis and recognition, and facial animation synthesis of speakers with expressions

, correspAuthorsNote=null, copyrightStatement=Copyright © 2026 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1329419931608789222, articleId=1329419929369030861, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=基于混合融合专家的多模态情感识别方法, columnId=1325489892104483745, journalTitle=计算机研究与发展, columnName=人工智能, runingTitle=null, highlight=null, articleAbstract=

多模态情感识别旨在整合不同模态数据以推断情感状态。现有研究通常平等对待各模态或依赖单一模态设计固定策略的融合方法,难以应对模态数据贡献度不均衡的问题。为此,提出一种动态的多模态混合融合方法(MoMFE),并基于混合专家模型框架设计了自适应路由模块和多模态融合专家模块,动态评估各模态的贡献度并选择相应的融合策略。其中,自适应路由机制通过模态间相关性分析动态评估模态贡献度,并为融合专家的选择提供指导,同时引入专家引导损失,进一步优化融合专家选择;多模态融合专家则针对不同模态贡献度组合进行互补融合,并通过引入共享专家以避免全局信息丢失和参数冗余。基于MER2024,CMU-MOSEI,CH-SIMS数据集上的多模态情感识别任务的对比实验表明,相比其他先进的多模态融合方法,MoMFE方法在情感识别准确率(Acc-2)、F1分数等核心评估指标上均取得优异的性能。

, authors=陈海丰1, 李溪1, 李岩2, 李健3, *, 何浪4, 蒋冬梅5, authorsList=陈海丰, 李溪, 李岩, 李健, 何浪, 蒋冬梅, authorCompany=null, correspAuthors=李健, authorNote=

作者贡献声明

陈海丰提出方法思路以及修改意见;李溪实现方法设计并撰写论文;李岩负责部分实验以及论文修改;李健负责实验监督以及论文修改;何浪负责项目管理以及论文修改;蒋冬梅负责项目监督以及论文修改。

1988年生。博士,讲师。CCF会员。主要研究方向为情感计算、多模态情感识别、对话中的情感识别、多模态抑郁症分析、面部动作单元识别

2001年生。硕士研究生。CCF学生会员。主要研究方向为多模态情感识别

1998年生。硕士研究生。主要研究方向为机器学习、多模态理解

1975年生。博士,教授。主要研究方向为计算机视觉、数字孪生

1983年生。博士,副教授。主要研究方向为多模态抑郁症识别和分析、离散情感分析、维度情感识别、图像处理

1973年生。博士,教授。主要研究方向为听视觉融合的语音处理、听视觉情感分析与识别、带表情的说话人面部动画合成

, correspAuthorsNote=
李健()
, copyrightStatement=版权所有 © 2026 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=4J0wv8sD9ne0T9Y0hbQDPg==, magXml=vlu7352g9oll082ZhvDdiQ==, pdfUrl=null, pdf=30ZAYQc1pf3aOAl5e9Luyw==, pdfFileSize=1319565, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=rbp7quKlsEnVafS+IPjGFw==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=J3ZmzYeQiovZm72t8BUCZw==, mapNumber=null, fund=null)}, authors=[Author(id=1329419932334403832, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=chenhaifeng@sust.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329419932418289914, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419932334403832, language=EN, stringName=Haifeng Chen, firstName=Haifeng, middleName=null, lastName=Chen, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Electronic Information and Artificial Intelligence, Shaanxi University of Science & Technology, Xi’an 710021, bio={"content":"

born in 1988. PhD, lecturer. Member of CCF. His main research interests include affective computing, multimodal emotion recognition, emotion recognition in conversations, multimodal depression analysis, and facial action unit recognition

"}, bioImg=null, bioContent=

born in 1988. PhD, lecturer. Member of CCF. His main research interests include affective computing, multimodal emotion recognition, emotion recognition in conversations, multimodal depression analysis, and facial action unit recognition

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329419932489593083, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419932334403832, language=CN, stringName=陈海丰, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1陕西科技大学电子信息与人工智能学院 西安 710021, bio={"img":"O6v6TIhveb2oZj/xzAMCyA==","content":"

1988年生。博士,讲师。CCF会员。主要研究方向为情感计算、多模态情感识别、对话中的情感识别、多模态抑郁症分析、面部动作单元识别

"}, bioImg=O6v6TIhveb2oZj/xzAMCyA==, bioContent=

1988年生。博士,讲师。CCF会员。主要研究方向为情感计算、多模态情感识别、对话中的情感识别、多模态抑郁症分析、面部动作单元识别

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329419931822698727, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, xref=1, ext=[AuthorCompanyExt(id=1329419931831087336, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419931822698727, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Electronic Information and Artificial Intelligence, Shaanxi University of Science & Technology, Xi’an 710021), AuthorCompanyExt(id=1329419931839475945, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419931822698727, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1陕西科技大学电子信息与人工智能学院 西安 710021)])]), Author(id=1329419932548313341, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329419932619616511, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419932548313341, language=EN, stringName=Xi Li, firstName=Xi, middleName=null, lastName=Li, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Electronic Information and Artificial Intelligence, Shaanxi University of Science & Technology, Xi’an 710021, bio={"content":"

born in 2001. Master candidate. Student member of CCF. Her main research interest includes multimodal emotion recognition

"}, bioImg=null, bioContent=

born in 2001. Master candidate. Student member of CCF. Her main research interest includes multimodal emotion recognition

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329419932686725376, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419932548313341, language=CN, stringName=李溪, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1陕西科技大学电子信息与人工智能学院 西安 710021, bio={"img":"m4VIMPi/oNkdn9KNShSmzA==","content":"

2001年生。硕士研究生。CCF学生会员。主要研究方向为多模态情感识别

"}, bioImg=m4VIMPi/oNkdn9KNShSmzA==, bioContent=

2001年生。硕士研究生。CCF学生会员。主要研究方向为多模态情感识别

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329419931822698727, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, xref=1, ext=[AuthorCompanyExt(id=1329419931831087336, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419931822698727, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Electronic Information and Artificial Intelligence, Shaanxi University of Science & Technology, Xi’an 710021), AuthorCompanyExt(id=1329419931839475945, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419931822698727, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1陕西科技大学电子信息与人工智能学院 西安 710021)])]), Author(id=1329419932774805762, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329419932854497540, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419932774805762, language=EN, stringName=Yan Li, firstName=Yan, middleName=null, lastName=Li, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2School of Computer Science, Northwestern Polytechnical University, Xi’an 710072, bio={"content":"

born in 1998. Master candidate. His main research interests include machine learning and multimodal understanding

"}, bioImg=null, bioContent=

born in 1998. Master candidate. His main research interests include machine learning and multimodal understanding

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329419932925800709, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419932774805762, language=CN, stringName=李岩, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2西北工业大学计算机学院 西安 710072, bio={"img":"Bwmsy3UZLO9e+qARuRfiwQ==","content":"

1998年生。硕士研究生。主要研究方向为机器学习、多模态理解

"}, bioImg=Bwmsy3UZLO9e+qARuRfiwQ==, bioContent=

1998年生。硕士研究生。主要研究方向为机器学习、多模态理解

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329419931923362026, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, xref=2, ext=[AuthorCompanyExt(id=1329419931935944939, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419931923362026, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Computer Science, Northwestern Polytechnical University, Xi’an 710072), AuthorCompanyExt(id=1329419931940139244, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419931923362026, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2西北工业大学计算机学院 西安 710072)])]), Author(id=1329419932997103879, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=lijianjsj@sust.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329419933089378569, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419932997103879, language=EN, stringName=Jian Li, firstName=Jian, middleName=null, lastName=Li, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, *, address=3School of Electrical and Control Engineering, Shaanxi University of Science & Technology, Xi’an 710021, bio={"content":"

born in 1975. PhD, professor. His main research interests include computer vision and digital twin

"}, bioImg=null, bioContent=

born in 1975. PhD, professor. His main research interests include computer vision and digital twin

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329419933152293130, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419932997103879, language=CN, stringName=李健, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, *, address=3陕西科技大学电气与控制工程学院 西安 710021, bio={"img":"oxZUsu9gMuKzRosUQspqeg==","content":"

1975年生。博士,教授。主要研究方向为计算机视觉、数字孪生

"}, bioImg=oxZUsu9gMuKzRosUQspqeg==, bioContent=

1975年生。博士,教授。主要研究方向为计算机视觉、数字孪生

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329419932091134190, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, xref=3, ext=[AuthorCompanyExt(id=1329419932099522799, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419932091134190, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3School of Electrical and Control Engineering, Shaanxi University of Science & Technology, Xi’an 710021), AuthorCompanyExt(id=1329419932112105712, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419932091134190, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3陕西科技大学电气与控制工程学院 西安 710021)])]), Author(id=1329419933227790604, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329419933328453902, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419933227790604, language=EN, stringName=Lang He, firstName=Lang, middleName=null, lastName=He, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=4, address=4School of Computer Science & Technology, Xi’an University of Posts and Telecommunications, Xi’an 710061, bio={"content":"

born in 1983. PhD, associate professor. His main research interests include multimodal depression recognition and analysis, discrete emotion analysis, dimensional emotion recognition, and image processing

"}, bioImg=null, bioContent=

born in 1983. PhD, associate professor. His main research interests include multimodal depression recognition and analysis, discrete emotion analysis, dimensional emotion recognition, and image processing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329419933399757071, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419933227790604, language=CN, stringName=何浪, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=4, address=4西安邮电大学计算机学院 西安 710061, bio={"img":"tYH1eHu03Lbb9ZCtJxiZCw==","content":"

1983年生。博士,副教授。主要研究方向为多模态抑郁症识别和分析、离散情感分析、维度情感识别、图像处理

"}, bioImg=tYH1eHu03Lbb9ZCtJxiZCw==, bioContent=

1983年生。博士,副教授。主要研究方向为多模态抑郁症识别和分析、离散情感分析、维度情感识别、图像处理

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329419932175020273, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, xref=4, ext=[AuthorCompanyExt(id=1329419932191797490, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419932175020273, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=4School of Computer Science & Technology, Xi’an University of Posts and Telecommunications, Xi’an 710061), AuthorCompanyExt(id=1329419932200186099, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419932175020273, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=4西安邮电大学计算机学院 西安 710061)])]), Author(id=1329419933471060241, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, orderNo=5, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329419933596889363, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419933471060241, language=EN, stringName=Dongmei Jiang, firstName=Dongmei, middleName=null, lastName=Jiang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=5, address=5Pengcheng Laboratory, Shenzhen, Guangdong 518055, bio={"content":"

born in 1973. PhD, professor. Her main research interests include audio-visual fusion based speech processing, audio-visual emotion analysis and recognition, and facial animation synthesis of speakers with expressions

"}, bioImg=null, bioContent=

born in 1973. PhD, professor. Her main research interests include audio-visual fusion based speech processing, audio-visual emotion analysis and recognition, and facial animation synthesis of speakers with expressions

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329419933668192532, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419933471060241, language=CN, stringName=蒋冬梅, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=5, address=5鹏城实验室 广东深圳 518055, bio={"img":"EREeNgT5sQem+X5oaK3f2A==","content":"

1973年生。博士,教授。主要研究方向为听视觉融合的语音处理、听视觉情感分析与识别、带表情的说话人面部动画合成

"}, bioImg=EREeNgT5sQem+X5oaK3f2A==, bioContent=

1973年生。博士,教授。主要研究方向为听视觉融合的语音处理、听视觉情感分析与识别、带表情的说话人面部动画合成

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329419932263100660, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, xref=5, ext=[AuthorCompanyExt(id=1329419932271489269, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419932263100660, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=5Pengcheng Laboratory, Shenzhen, Guangdong 518055), AuthorCompanyExt(id=1329419932279877878, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419932263100660, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=5鹏城实验室 广东深圳 518055)])])], keywords=[Keyword(id=1329419933764661525, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=EN, orderNo=1, keyword=emotion recognition), Keyword(id=1329419933840158998, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=EN, orderNo=2, keyword=multimodal fusion), Keyword(id=1329419933903073559, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=EN, orderNo=3, keyword=mixture of experts), Keyword(id=1329419933970182424, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=EN, orderNo=4, keyword=adaptive router), Keyword(id=1329419934058262809, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=EN, orderNo=5, keyword=cross-modal attention), Keyword(id=1329419934146343194, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=CN, orderNo=1, keyword=情感识别), Keyword(id=1329419934272172315, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=CN, orderNo=2, keyword=多模态融合), Keyword(id=1329419934360252700, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=CN, orderNo=3, keyword=混合专家), Keyword(id=1329419934423167261, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=CN, orderNo=4, keyword=自适应路由), Keyword(id=1329419934536413470, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=CN, orderNo=5, keyword=跨模态注意力)], refs=[Reference(id=1329419936662925623, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=Das R, Singh T D. Multimodal sentiment analysis: A survey of methods, trends, and challenges[J]. ACM Computing Surveys, 2023, 55(13): 1−38, articleTitle=null, refAbstract=null), Reference(id=1329419936734228792, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Poria S, Hazarika D, Majumder N, et al. Beneath the tip of the iceberg: Current challenges and new directions in sentiment analysis research[J]. IEEE Transactions on Affective Computing, 2020, 14(1): 108−132, articleTitle=null, refAbstract=null), Reference(id=1329419936792949049, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Islam M S, Kabir M N, Ghani N A, et al. Challenges and future in deep learning for sentiment analysis: A comprehensive review and a proposed novel hybrid approach[J]. Artificial Intelligence Review, 2024, 57(62): 1−79, articleTitle=null, refAbstract=null), Reference(id=1329419936864252218, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Jian, Ji Yanli, Qin Zhen, et al. Dominant single-modal supplementary fusion (SIMSUF) for multimodal sentiment analysis[J]. IEEE Transactions on Multimedia, 2023, 26: 8383−8394, articleTitle=null, refAbstract=null), Reference(id=1329419936927166779, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Lian Zhen, Sun Haiyang, Sun Licai, et al. MER2024: Semi-supervised learning, noise robustness, and open-vocabulary multimodal emotion recognition[C]//Proc of the 2nd Int Workshop on Multimodal and Responsible Affective Computing. New York: ACM, 2024: 41−48, articleTitle=null, refAbstract=null), Reference(id=1329419936985887036, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=罗渊贻, 吴锐, 刘家锋, 等. 面向情感语义不一致的多模态情感分析方法[J]. 计算机研究与发展, 2025, 62(2): 374−382, articleTitle=null, refAbstract=null), Reference(id=1329419937052995901, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Luo Yuanyi, Wu Rui, Liu Jiafeng, et al. Multimodal sentiment analysis method for sentimental semantic inconsistency[J]. Journal of Computer Research and Development, 2025, 62(2): 374−382 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1329419937128493374, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Liang Yang. A dynamic weighted fusion model for multimodal sentiment analysis[J]. Signal, Image and Video Processing, 2025, 19(8): 609−623, articleTitle=null, refAbstract=null), Reference(id=1329419937191407935, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Feng Xinyu, Lin Yuming, He Lihua, et al. Knowledge-Guided dynamic modality attention fusion framework for multimodal sentiment analysis[J]. arXiv preprint, arXiv: 2410.04491, 2024, articleTitle=null, refAbstract=null), Reference(id=1329419937275294016, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=申锦涛. 基于模态交互和动态融合的多模态情感识别研究[D]. 沈阳: 辽宁大学, 2024, articleTitle=null, refAbstract=null), Reference(id=1329419937354985793, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Shen Jintao. Research on multimodal emotion recognition based on modal interaction and dynamic fusion[D]. Shenyang: Liaoning University, 2024 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1329419937434677570, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhuang Yan, Zhang Yanru, Hu Zheng, et al. GLoMo: Global-local modal fusion for multimodal sentiment analysis[C]//Proc of the 32nd ACM Int Conf on Multimedia. New York: ACM, 2024: 1800−1809, articleTitle=null, refAbstract=null), Reference(id=1329419937501786435, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Xueliang, Wang Mingyang, Tan Yingchun, et al. TGMoE: A text guided mixture-of-experts model for multimodal sentiment analysis[J]. International Journal of Advanced Computer Science & Applications, 2024, 15(8): 1227−1234, articleTitle=null, refAbstract=null), Reference(id=1329419937585672516, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=贾熹滨, 曾檬, 米庆, 等. 领域对齐对抗的无监督跨领域文本情感分析算法[J]. 计算机研究与发展, 2022, 59(6): 1225−1270, articleTitle=null, refAbstract=null), Reference(id=1329419937652781381, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Jia Xibin, Zeng Meng, Mi Qing, et al. Domain alignment adversarial unsupervised cross-domain text sentiment analysis algorithm[J]. Journal of Computer Research and Development, 2022, 59(6): 1225−1270 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1329419937753444678, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Haifeng, Zhang Peng, Guo Chujia, et al. Facial action unit representation based on self-supervised learning with ensembled priori constraints[J]. IEEE Transactions on Image Processing, 2024, 33: 5045−5069, articleTitle=null, refAbstract=null), Reference(id=1329419937828942151, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Yuhua, Shen Guang, Xu Yuezhu, et al. Learning mutual correlation in multimodal transformer for speech emotion recognition[C/OL]//Proc of the 2021 Interspeech. 2021: 4518−4522.[2025-04-16]. https://www.isca-archive.org/interspeech_2021/wang21ga_interspeech.html, articleTitle=null, refAbstract=null), Reference(id=1329419937891856712, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Pennington J, Socher R, Manning C, et al. GloVe: Global vectors for word representation[C]//Proc of the 2014 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2014: 1532−1543, articleTitle=null, refAbstract=null), Reference(id=1329419938055434569, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Devlin J, Chang Mingwei, Lee K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]//Proc of the 2019 Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, PA: ACL, 2019: 4171−4186, articleTitle=null, refAbstract=null), Reference(id=1329419938130932042, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv preprint, arXiv: 2010.11929, 2020, articleTitle=null, refAbstract=null), Reference(id=1329419938210623819, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Hsu W N, Bolte B, Tsai Y H, et al. Hubert: Self-supervised speech representation learning by masked prediction of hidden units[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 3451−3460, articleTitle=null, refAbstract=null), Reference(id=1329419938286121292, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Radford A, Kim J W, Xu Tao, et al. Robust speech recognition via large-scale weak supervision[C]//Proc of the 38th Int Conf on Machine Learning. New York: ACM, 2023: 28492−28518, articleTitle=null, refAbstract=null), Reference(id=1329419938340647245, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Wei Han, Chen Hui, Poria S. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis[C]//Proc of the 2021 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2021: 9180–9192, articleTitle=null, refAbstract=null), Reference(id=1329419938403561806, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=Cai Yunrui, Wu Zhiyong, Jia Jia, et al. Lora-Mer: Low-rank adaptation of pre-trained speech models for multimodal emotion recognition using mutual information[C/OL]//Proc of the 2024 Interspeech. 2024: 4658−4662.[2025-04-15]. https://www.isca-archive.org/interspeech_2024/cai24b_interspeech.html, articleTitle=null, refAbstract=null), Reference(id=1329419938462282063, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=Kumar A, Vepa J. Gated mechanism for attention based multi modal sentiment analysis[C]//Proc of the 2020 IEEE Int Conf on Acoustics, Speech and Signal Processing. Piscataway, NJ: IEEE, 2020: 4477−4481, articleTitle=null, refAbstract=null), Reference(id=1329419938546168144, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=Majumder N, Hazarika D, Gelbukh A, et al. Multimodal sentiment analysis using hierarchical fusion with context modeling[J]. Knowledge-Based Systems, 2018, 161: 124−133, articleTitle=null, refAbstract=null), Reference(id=1329419938613277009, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=26, authorNames=null, journalName=null, refType=null, unstructuredReference=Jacobs R A, Jordan M I, Nowlan S J, et al. Adaptive mixtures of local experts[J]. Neural Computation, 1991, 3(1): 79−87, articleTitle=null, refAbstract=null), Reference(id=1329419938734911826, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=27, authorNames=null, journalName=null, refType=null, unstructuredReference=Fedus W, Zoph B, Shazeer N. Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity[J]. Journal of Machine Learning Research, 2022, 23(120): 1−39, articleTitle=null, refAbstract=null), Reference(id=1329419938810409299, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=28, authorNames=null, journalName=null, refType=null, unstructuredReference=Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proc of the 31st Int Conf on Neural Information Processing Systems. New York: ACM, 2017: 6000−6010, articleTitle=null, refAbstract=null), Reference(id=1329419938911072596, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=29, authorNames=null, journalName=null, refType=null, unstructuredReference=Lepikhin D, Lee H J, Xu Yuanzhong, et al. Gshard: Scaling giant models with conditional computation and automatic sharding[C/OL]//Proc of the 9th Int Conf on Learning Representations. 2021: 1−23.[2025-04-23]. https://openreview.net/forum?id=qrwe7XHTmYb, articleTitle=null, refAbstract=null), Reference(id=1329419938982375765, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=28, rfOrder=30, authorNames=null, journalName=null, refType=null, unstructuredReference=Du Nan, Huang Yanping, Dai A M, et al. GlaM: Efficient scaling of language models with mixture-of-experts[C]//Proc of the 39th Int Conf on Machine Learning. New York: ACM, 2022: 5547−5569, articleTitle=null, refAbstract=null), Reference(id=1329419939045290326, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=29, rfOrder=31, authorNames=null, journalName=null, refType=null, unstructuredReference=Cai Weilin, Jiang Juyong, Wang Fan, et al. A survey on mixture of experts in large language models[J]. IEEE Transactions on Knowledge and Data Engineering, 2025, 37(7): 3896−3915, articleTitle=null, refAbstract=null), Reference(id=1329419939095621975, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=30, rfOrder=32, authorNames=null, journalName=null, refType=null, unstructuredReference=Sarkar R, Liang Hanxue, Fan Zhiwen, et al. Edge-MoE: Memory-efficient multi-task vision transformer architecture with task-level sparsity via mixture-of-experts[C]//Proc of the 2023 IEEE/ACM Int Conf on Computer Aided Design. Piscataway, NJ: IEEE, 2023: 1−9, articleTitle=null, refAbstract=null), Reference(id=1329419939166925144, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=31, rfOrder=33, authorNames=null, journalName=null, refType=null, unstructuredReference=Lin X V, Shrivastava A, Liang Luo, et al. Moma: Efficient early-fusion pre-training with mixture of modality-aware experts[J]. arXiv preprint, arXiv: 2407.21770, 2024, articleTitle=null, refAbstract=null), Reference(id=1329419939234034009, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=32, rfOrder=34, authorNames=null, journalName=null, refType=null, unstructuredReference=Han Xing, Nguyen H, Harris C, et al. FuseMoE: Mixture-of-experts transformers for fleximodal fusion[C]//Proc of the 38th Int Conf on Neural Information Processing Systems, New York: ACM, 2025: 67850–67900, articleTitle=null, refAbstract=null), Reference(id=1329419939301142874, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=33, rfOrder=35, authorNames=null, journalName=null, refType=null, unstructuredReference=Xiong Congxiong, Chen Hao, Zheng Hao, et al. Mome: Mixture of multimodal experts for cancer survival prediction[C]//Proc of the 27th Int Conf on Medical Image Computing and Computer-Assisted Intervention. Springer: Berlin, 2024: 318−328, articleTitle=null, refAbstract=null), Reference(id=1329419939364057435, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=34, rfOrder=36, authorNames=null, journalName=null, refType=null, unstructuredReference=Shazeer N, Mirhoseini A, Maziarz K, et al. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer[J]. arXiv preprint, arXiv: 1701.06538, 2017, articleTitle=null, refAbstract=null), Reference(id=1329419939422777692, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=35, rfOrder=37, authorNames=null, journalName=null, refType=null, unstructuredReference=Dai Damai, Deng Chengqi, Zhao Chenggang, et al. Deepseek-Moe: Towards ultimate expert specialization in mixture-of-experts language models[C]//Proc of the 62nd Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2024: 1290−1297, articleTitle=null, refAbstract=null), Reference(id=1329419939481497949, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=36, rfOrder=38, authorNames=null, journalName=null, refType=null, unstructuredReference=Zadeh A B, Liang P P, Poria S, et al. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph[C]//Proc of the 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2018: 2236−2246, articleTitle=null, refAbstract=null), Reference(id=1329419939544412510, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=37, rfOrder=39, authorNames=null, journalName=null, refType=null, unstructuredReference=Yu Wenmeng, Xu Hua, Meng Fanyang, et al. CH-SIMS: A Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality[C]//Proc of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 3718−3727, articleTitle=null, refAbstract=null), Reference(id=1329419939607327071, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=38, rfOrder=40, authorNames=null, journalName=null, refType=null, unstructuredReference=Baltrusaitis T, Zadeh A, Lim Y C, et al. OpenFace 2.0: Facial behavior analysis toolkit[C]//Proc of the 13th IEEE Int Conf on Automatic Face & Gesture Recognition. Piscataway, NJ: IEEE, 2018: 59−66, articleTitle=null, refAbstract=null), Reference(id=1329419939674435936, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=39, rfOrder=41, authorNames=null, journalName=null, refType=null, unstructuredReference=Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//Proc of the 38th Int Conf on Machine Learning. New York: ACM, 2021: 8748−8763, articleTitle=null, refAbstract=null), Reference(id=1329419939728961889, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=40, rfOrder=42, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Zhixian, Chen Haifeng, Li Xi, et al. Improving multimodal emotion recognition by leveraging acoustic adaptation and visual alignment[C]//Proc of the 2nd Int Workshop on Multimodal and Responsible Affective Computing. New York: ACM, 2024: 67−71, articleTitle=null, refAbstract=null), Reference(id=1329419939787682146, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=41, rfOrder=43, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Aiyuan, Xiao Bin, Wang Bingning, et al. Baichuan 2: Open large-scale language models[J]. arXiv preprint, arXiv: 2309.10305, 2023, articleTitle=null, refAbstract=null), Reference(id=1329419939867373923, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=42, rfOrder=44, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Tianyi, Liu Daming. MPID: A modality-preserving and interaction-driven fusion network for multimodal sentiment analysis[C]//Proc of the 31st Int Conf on Computational Linguistics. Stroudsburg, PA: ACL, 2025: 4313−4322, articleTitle=null, refAbstract=null), Reference(id=1329419939926094180, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=43, rfOrder=45, authorNames=null, journalName=null, refType=null, unstructuredReference=Xin Feng, Gong Wenjuan, Shi Tingbo, et al. Gated fusion network with progressive modality enhancement for sentiment analysis[J]. IEEE Access, 2024, 12: 165810−165821, articleTitle=null, refAbstract=null), Reference(id=1329419939984814437, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=44, rfOrder=46, authorNames=null, journalName=null, refType=null, unstructuredReference=Fan Qi, Li Yong, Xin Yi, et al. Leveraging contrastive learning and self-training for multimodal emotion recognition with limited labeled samples[C]//Proc of the 2nd Int Workshop on Multimodal and Responsible Affective Computing. New York: ACM, 2024: 72−77, articleTitle=null, refAbstract=null), Reference(id=1329419940110643558, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=45, rfOrder=47, authorNames=null, journalName=null, refType=null, unstructuredReference=Gao Fei, Shi Pujin. Audio-Guided fusion techniques for multimodal emotion analysis[C]//Proc of the 2nd Int Workshop on Multimodal and Responsible Affective Computing. New York: ACM, 2024: 62−66, articleTitle=null, refAbstract=null), Reference(id=1329419940194529639, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=46, rfOrder=48, authorNames=null, journalName=null, refType=null, unstructuredReference=Lei Ling, He Wangjun, Zheng Qiuyan, et al. Multimodal sentiment analysis based on expert mixing of subtask representations[J]. IEEE Access, 2025, 13: 44278−44287, articleTitle=null, refAbstract=null), Reference(id=1329419940295192936, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=47, rfOrder=49, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Yihe, Yuan Ziqi, Mao Huisheng, et al. Make acoustic and visual cues matter: CH-SIMS v2. 0 dataset and av-mixup consistent module[C]//Proc of the 24th Int Conf on Multimodal Interaction. New York: ACM, 2022: 247−258, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1329419931822698727, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, xref=1, ext=[AuthorCompanyExt(id=1329419931831087336, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419931822698727, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Electronic Information and Artificial Intelligence, Shaanxi University of Science & Technology, Xi’an 710021), AuthorCompanyExt(id=1329419931839475945, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419931822698727, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1陕西科技大学电子信息与人工智能学院 西安 710021)]), AuthorCompany(id=1329419931923362026, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, xref=2, ext=[AuthorCompanyExt(id=1329419931935944939, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419931923362026, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Computer Science, Northwestern Polytechnical University, Xi’an 710072), AuthorCompanyExt(id=1329419931940139244, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419931923362026, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2西北工业大学计算机学院 西安 710072)]), AuthorCompany(id=1329419932091134190, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, xref=3, ext=[AuthorCompanyExt(id=1329419932099522799, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419932091134190, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3School of Electrical and Control Engineering, Shaanxi University of Science & Technology, Xi’an 710021), AuthorCompanyExt(id=1329419932112105712, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419932091134190, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3陕西科技大学电气与控制工程学院 西安 710021)]), AuthorCompany(id=1329419932175020273, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, xref=4, ext=[AuthorCompanyExt(id=1329419932191797490, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419932175020273, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=4School of Computer Science & Technology, Xi’an University of Posts and Telecommunications, Xi’an 710061), AuthorCompanyExt(id=1329419932200186099, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419932175020273, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=4西安邮电大学计算机学院 西安 710061)]), AuthorCompany(id=1329419932263100660, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, xref=5, ext=[AuthorCompanyExt(id=1329419932271489269, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419932263100660, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=5Pengcheng Laboratory, Shenzhen, Guangdong 518055), AuthorCompanyExt(id=1329419932279877878, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419932263100660, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=5鹏城实验室 广东深圳 518055)])], figs=[ArticleFig(id=1329419934632882463, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=EN, label=Figure 1, caption=Classification statistics of CH-SIMS dataset samples and classification examples, figureFileSmall=CiT4ecHDdGb36rKzQJ1sPg==, figureFileBig=rbp7quKlsEnVafS+IPjGFw==, tableContent=null), ArticleFig(id=1329419934720962848, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=CN, label=图1, caption=CH-SIMS数据集样本分类统计以及分类示例, figureFileSmall=CiT4ecHDdGb36rKzQJ1sPg==, figureFileBig=rbp7quKlsEnVafS+IPjGFw==, tableContent=null), ArticleFig(id=1329419934934872353, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=EN, label=Figure 2, caption=The method model diagram, figureFileSmall=svCd1Is6a88YAAzR9NdfWw==, figureFileBig=EfgMy270oRC1s008SZintQ==, tableContent=null), ArticleFig(id=1329419935010369826, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=CN, label=图2, caption=方法模型图, figureFileSmall=svCd1Is6a88YAAzR9NdfWw==, figureFileBig=EfgMy270oRC1s008SZintQ==, tableContent=null), ArticleFig(id=1329419935090061603, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=EN, label=Figure 3, caption=Expert activation rates under different router settings on CH-SIMS dataset, figureFileSmall=KID1z6hqes97q531fTGWrw==, figureFileBig=pzB8RagRTl/bLQbBj/V3Mw==, tableContent=null), ArticleFig(id=1329419935178141988, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=CN, label=图3, caption=CH-SIMS数据集上不同路由设置下的专家激活率, figureFileSmall=KID1z6hqes97q531fTGWrw==, figureFileBig=pzB8RagRTl/bLQbBj/V3Mw==, tableContent=null), ArticleFig(id=1329419935262028069, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=EN, label=Figure 4, caption=Performance of each fusion expert on CH-SIMS classification dataset, figureFileSmall=wrd8ElTWRVhOoWOFGI3EFQ==, figureFileBig=YZzyaFDkZhOTA/47rwzC/A==, tableContent=null), ArticleFig(id=1329419935350108454, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=CN, label=图4, caption=CH-SIMS分类数据集上各融合专家的表现, figureFileSmall=wrd8ElTWRVhOoWOFGI3EFQ==, figureFileBig=YZzyaFDkZhOTA/47rwzC/A==, tableContent=null), ArticleFig(id=1329419935429800231, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=EN, label=Figure 5, caption=Assessment results of d and α for MoMFE on CH-SIMS dataset, figureFileSmall=tDq6GT2UvFPuK6LltEif4g==, figureFileBig=gPY1b1P/+QZCICHVAiMWTA==, tableContent=null), ArticleFig(id=1329419935505297704, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=CN, label=图5, caption=CH-SIMS数据集上d与α对MoMFE的评估结果, figureFileSmall=tDq6GT2UvFPuK6LltEif4g==, figureFileBig=gPY1b1P/+QZCICHVAiMWTA==, tableContent=null), ArticleFig(id=1329419935580795177, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=EN, label=Table 1, caption=

Introduction of Datasets

, figureFileSmall=null, figureFileBig=null, tableContent=
数据集训练集验证集测试集语言
MER202450301169中文
CMU-MOSEI1632618714659英文
CH-SIMS1368456457中文
), ArticleFig(id=1329419935635321130, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=CN, label=表1, caption=

数据集介绍

, figureFileSmall=null, figureFileBig=null, tableContent=
数据集训练集验证集测试集语言
MER202450301169中文
CMU-MOSEI1632618714659英文
CH-SIMS1368456457中文
), ArticleFig(id=1329419935702429995, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=EN, label=Table 2, caption=

Results of Different Methods on MER2024 Dataset %

, figureFileSmall=null, figureFileBig=null, tableContent=
方法训练及验证 F1测试 F1Acc-6
MR-CCL86.5388.25
LAA83.8588.90
AGT83.2289.83
MoMFE(本文)83.2889.8890.01
), ArticleFig(id=1329419935790510380, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=CN, label=表2, caption=

不同方法在MER2024数据集上的结果

, figureFileSmall=null, figureFileBig=null, tableContent=
方法训练及验证 F1测试 F1Acc-6
MR-CCL86.5388.25
LAA83.8588.90
AGT83.2289.83
MoMFE(本文)83.2889.8890.01
), ArticleFig(id=1329419935882785069, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=EN, label=Table 3, caption=

Results of Different Methods on CMU-MOSEI and CH-SIMS Datasets %

, figureFileSmall=null, figureFileBig=null, tableContent=
方法CMU-MOSEI数据集CH-SIMS数据集
MaeCorrAcc-7Acc-2F1MaeCorrAcc-5Acc-3Acc-2F1
MPID53.377.653.6485.1085.3042.161.844.2181.5681.91
GFNPME52.877.153.9086.5286.2640.261.683.1583.10
SIMSUF52.977.253.6886.2386.12
KuDA52.977.652.8986.4686.5940.861.343.5466.5280.7480.71
DWE52.676.854.8085.9785.9841.659.342.9080.7480.54
AC-MoE31.573.282.0582.04
TGMoE53.575.753.7085.5185.86
FuseMoE54.081.086.4185.43
GLoMo53.977.155.0086.5086.40
MoMFE(本文)53.478.854.7286.8386.6937.775.141.3871.9984.3384.51
), ArticleFig(id=1329419935945699630, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=CN, label=表3, caption=

不同方法在CMU-MOSEI和CH-SIMS数据集上的结果

, figureFileSmall=null, figureFileBig=null, tableContent=
方法CMU-MOSEI数据集CH-SIMS数据集
MaeCorrAcc-7Acc-2F1MaeCorrAcc-5Acc-3Acc-2F1
MPID53.377.653.6485.1085.3042.161.844.2181.5681.91
GFNPME52.877.153.9086.5286.2640.261.683.1583.10
SIMSUF52.977.253.6886.2386.12
KuDA52.977.652.8986.4686.5940.861.343.5466.5280.7480.71
DWE52.676.854.8085.9785.9841.659.342.9080.7480.54
AC-MoE31.573.282.0582.04
TGMoE53.575.753.7085.5185.86
FuseMoE54.081.086.4185.43
GLoMo53.977.155.0086.5086.40
MoMFE(本文)53.478.854.7286.8386.6937.775.141.3871.9984.3384.51
), ArticleFig(id=1329419936037974319, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=EN, label=Table 4, caption=

Experimental Results Under Different Router Settings %

, figureFileSmall=null, figureFileBig=null, tableContent=
路由机制MER2024数据集CH-SIMS数据集
Acc-6F1Acc-2F1
Softmax路由87.6887.4581.1481.19
ARM路由89.3988.8183.3683.11
ARM路由+Lna90.0189.8884.3384.51
), ArticleFig(id=1329419936096694576, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=CN, label=表4, caption=

不同路由设置下的实验结果

, figureFileSmall=null, figureFileBig=null, tableContent=
路由机制MER2024数据集CH-SIMS数据集
Acc-6F1Acc-2F1
Softmax路由87.6887.4581.1481.19
ARM路由89.3988.8183.3683.11
ARM路由+Lna90.0189.8884.3384.51
), ArticleFig(id=1329419936151220529, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=EN, label=Table 5, caption=

Experimental Results of Different Expert Combinations Setting %

, figureFileSmall=null, figureFileBig=null, tableContent=
多模态融合专家CH-SIMS数据集
DFLFEFSFAcc-3Acc-2F1
√√68.1980.3980.37
√√√69.3781.3381.53
√√69.1082.3582.39
√√√70.4882.3982.77
√√70.0782.1382.55
√√√70.3982.5382.68
√√√71.2583.4283.71
√√√√71.9984.3384.51
), ArticleFig(id=1329419936209940786, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=CN, label=表5, caption=

不同专家组合设置的实验结果

, figureFileSmall=null, figureFileBig=null, tableContent=
多模态融合专家CH-SIMS数据集
DFLFEFSFAcc-3Acc-2F1
√√68.1980.3980.37
√√√69.3781.3381.53
√√69.1082.3582.39
√√√70.4882.3982.77
√√70.0782.1382.55
√√√70.3982.5382.68
√√√71.2583.4283.71
√√√√71.9984.3384.51
), ArticleFig(id=1329419936285438259, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=EN, label=Table 6, caption=

Experimental Results of Model Performance with Missing Modalities %

, figureFileSmall=null, figureFileBig=null, tableContent=
模态缺失
类型
缺失样本
比例
MER2024数据集CMU-MOSEI数据集CH-SIMS数据集
AccF1Acc-2F1Acc-2F1
单模态
缺失
1089.27±0.0788.83±0.1286.12±0.1286.25±0.0883.81±0.1184.03±0.14
3088.01±0.1287.55±0.1784.89±0.2485.02±0.2783.11±0.1783.37±0.21
5086.43±0.1086.13±0.0584.14±0.2084.37±0.1682.69±0.2682.84±0.19
双模态
缺失
1087.32±0.0387.10±0.1084.26±0.1384.53±0.1283.28±0.1283.81±0.15
3086.52±0.1186.31±0.1883.52±0.2483.88±0.1782.03±0.0682.44±0.09
5084.35±0.0983.87±0.2383.04±0.1583.29±0.1881.42±0.2581.75±0.14
), ArticleFig(id=1329419936373518644, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=CN, label=表6, caption=

缺失模态下模型性能的实验结果

, figureFileSmall=null, figureFileBig=null, tableContent=
模态缺失
类型
缺失样本
比例
MER2024数据集CMU-MOSEI数据集CH-SIMS数据集
AccF1Acc-2F1Acc-2F1
单模态
缺失
1089.27±0.0788.83±0.1286.12±0.1286.25±0.0883.81±0.1184.03±0.14
3088.01±0.1287.55±0.1784.89±0.2485.02±0.2783.11±0.1783.37±0.21
5086.43±0.1086.13±0.0584.14±0.2084.37±0.1682.69±0.2682.84±0.19
双模态
缺失
1087.32±0.0387.10±0.1084.26±0.1384.53±0.1283.28±0.1283.81±0.15
3086.52±0.1186.31±0.1883.52±0.2483.88±0.1782.03±0.0682.44±0.09
5084.35±0.0983.87±0.2383.04±0.1583.29±0.1881.42±0.2581.75±0.14
), ArticleFig(id=1329419936453210421, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=EN, label=Table 7, caption=

Experimental Results of Noise Robustness %

, figureFileSmall=null, figureFileBig=null, tableContent=
模态加噪
类型
加噪样本
比例
MER2024数据集
AccF1
语音模态
加噪
1088.57±0.1588.21±0.13
3087.44±0.0587.10±0.08
5086.23±0.1485.95±0.15
视觉模态
加噪
1089.11±0.0888.85±0.10
3088.64±0.0688.26±0.05
5087.34±0.1187.05±0.13
语音+视觉
加噪
1088.23±0.0787.94±0.09
3087.29±0.1587.12±0.11
5085.86±0.0885.62±0.05
), ArticleFig(id=1329419936541290806, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, language=CN, label=表7, caption=

噪声鲁棒性实验结果

, figureFileSmall=null, figureFileBig=null, tableContent=
模态加噪
类型
加噪样本
比例
MER2024数据集
AccF1
语音模态
加噪
1088.57±0.1588.21±0.13
3087.44±0.0587.10±0.08
5086.23±0.1485.95±0.15
视觉模态
加噪
1089.11±0.0888.85±0.10
3088.64±0.0688.26±0.05
5087.34±0.1187.05±0.13
语音+视觉
加噪
1088.23±0.0787.94±0.09
3087.29±0.1587.12±0.11
5085.86±0.0885.62±0.05
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202550544, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202550544, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202550544, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202550544, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1780243200000, fullTextJson=null, articleText=null, reference=null)
收藏切换
基于混合融合专家的多模态情感识别方法
陈海丰 1 , 李溪 1 , 李岩 2 , 李健 3, * , 何浪 4 , 蒋冬梅 5
计算机研究与发展 | 人工智能 2026,63(6): 1478-1491
收起
收藏切换
计算机研究与发展 |人工智能 2026 , 63 (6) : 1478 -1491
基于混合融合专家的多模态情感识别方法
全屏
[Author(id=1329419932334403832, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=chenhaifeng@sust.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329419932418289914, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419932334403832, language=EN, stringName=Haifeng Chen, firstName=Haifeng, middleName=null, lastName=Chen, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Electronic Information and Artificial Intelligence, Shaanxi University of Science & Technology, Xi’an 710021, bio={"content":"

born in 1988. PhD, lecturer. Member of CCF. His main research interests include affective computing, multimodal emotion recognition, emotion recognition in conversations, multimodal depression analysis, and facial action unit recognition

"}, bioImg=null, bioContent=

born in 1988. PhD, lecturer. Member of CCF. His main research interests include affective computing, multimodal emotion recognition, emotion recognition in conversations, multimodal depression analysis, and facial action unit recognition

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329419932489593083, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419932334403832, language=CN, stringName=陈海丰, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1陕西科技大学电子信息与人工智能学院 西安 710021, bio={"img":"O6v6TIhveb2oZj/xzAMCyA==","content":"

1988年生。博士,讲师。CCF会员。主要研究方向为情感计算、多模态情感识别、对话中的情感识别、多模态抑郁症分析、面部动作单元识别

"}, bioImg=O6v6TIhveb2oZj/xzAMCyA==, bioContent=

1988年生。博士,讲师。CCF会员。主要研究方向为情感计算、多模态情感识别、对话中的情感识别、多模态抑郁症分析、面部动作单元识别

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329419931822698727, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, xref=1, ext=[AuthorCompanyExt(id=1329419931831087336, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419931822698727, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Electronic Information and Artificial Intelligence, Shaanxi University of Science & Technology, Xi’an 710021), AuthorCompanyExt(id=1329419931839475945, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419931822698727, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1陕西科技大学电子信息与人工智能学院 西安 710021)])]), Author(id=1329419932548313341, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329419932619616511, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419932548313341, language=EN, stringName=Xi Li, firstName=Xi, middleName=null, lastName=Li, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Electronic Information and Artificial Intelligence, Shaanxi University of Science & Technology, Xi’an 710021, bio={"content":"

born in 2001. Master candidate. Student member of CCF. Her main research interest includes multimodal emotion recognition

"}, bioImg=null, bioContent=

born in 2001. Master candidate. Student member of CCF. Her main research interest includes multimodal emotion recognition

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329419932686725376, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419932548313341, language=CN, stringName=李溪, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1陕西科技大学电子信息与人工智能学院 西安 710021, bio={"img":"m4VIMPi/oNkdn9KNShSmzA==","content":"

2001年生。硕士研究生。CCF学生会员。主要研究方向为多模态情感识别

"}, bioImg=m4VIMPi/oNkdn9KNShSmzA==, bioContent=

2001年生。硕士研究生。CCF学生会员。主要研究方向为多模态情感识别

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329419931822698727, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, xref=1, ext=[AuthorCompanyExt(id=1329419931831087336, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419931822698727, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Electronic Information and Artificial Intelligence, Shaanxi University of Science & Technology, Xi’an 710021), AuthorCompanyExt(id=1329419931839475945, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419931822698727, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1陕西科技大学电子信息与人工智能学院 西安 710021)])]), Author(id=1329419932774805762, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329419932854497540, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419932774805762, language=EN, stringName=Yan Li, firstName=Yan, middleName=null, lastName=Li, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2School of Computer Science, Northwestern Polytechnical University, Xi’an 710072, bio={"content":"

born in 1998. Master candidate. His main research interests include machine learning and multimodal understanding

"}, bioImg=null, bioContent=

born in 1998. Master candidate. His main research interests include machine learning and multimodal understanding

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329419932925800709, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419932774805762, language=CN, stringName=李岩, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2西北工业大学计算机学院 西安 710072, bio={"img":"Bwmsy3UZLO9e+qARuRfiwQ==","content":"

1998年生。硕士研究生。主要研究方向为机器学习、多模态理解

"}, bioImg=Bwmsy3UZLO9e+qARuRfiwQ==, bioContent=

1998年生。硕士研究生。主要研究方向为机器学习、多模态理解

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329419931923362026, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, xref=2, ext=[AuthorCompanyExt(id=1329419931935944939, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419931923362026, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Computer Science, Northwestern Polytechnical University, Xi’an 710072), AuthorCompanyExt(id=1329419931940139244, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419931923362026, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2西北工业大学计算机学院 西安 710072)])]), Author(id=1329419932997103879, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=lijianjsj@sust.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329419933089378569, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419932997103879, language=EN, stringName=Jian Li, firstName=Jian, middleName=null, lastName=Li, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, *, address=3School of Electrical and Control Engineering, Shaanxi University of Science & Technology, Xi’an 710021, bio={"content":"

born in 1975. PhD, professor. His main research interests include computer vision and digital twin

"}, bioImg=null, bioContent=

born in 1975. PhD, professor. His main research interests include computer vision and digital twin

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329419933152293130, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419932997103879, language=CN, stringName=李健, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, *, address=3陕西科技大学电气与控制工程学院 西安 710021, bio={"img":"oxZUsu9gMuKzRosUQspqeg==","content":"

1975年生。博士,教授。主要研究方向为计算机视觉、数字孪生

"}, bioImg=oxZUsu9gMuKzRosUQspqeg==, bioContent=

1975年生。博士,教授。主要研究方向为计算机视觉、数字孪生

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329419932091134190, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, xref=3, ext=[AuthorCompanyExt(id=1329419932099522799, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419932091134190, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3School of Electrical and Control Engineering, Shaanxi University of Science & Technology, Xi’an 710021), AuthorCompanyExt(id=1329419932112105712, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419932091134190, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3陕西科技大学电气与控制工程学院 西安 710021)])]), Author(id=1329419933227790604, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329419933328453902, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419933227790604, language=EN, stringName=Lang He, firstName=Lang, middleName=null, lastName=He, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=4, address=4School of Computer Science & Technology, Xi’an University of Posts and Telecommunications, Xi’an 710061, bio={"content":"

born in 1983. PhD, associate professor. His main research interests include multimodal depression recognition and analysis, discrete emotion analysis, dimensional emotion recognition, and image processing

"}, bioImg=null, bioContent=

born in 1983. PhD, associate professor. His main research interests include multimodal depression recognition and analysis, discrete emotion analysis, dimensional emotion recognition, and image processing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329419933399757071, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419933227790604, language=CN, stringName=何浪, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=4, address=4西安邮电大学计算机学院 西安 710061, bio={"img":"tYH1eHu03Lbb9ZCtJxiZCw==","content":"

1983年生。博士,副教授。主要研究方向为多模态抑郁症识别和分析、离散情感分析、维度情感识别、图像处理

"}, bioImg=tYH1eHu03Lbb9ZCtJxiZCw==, bioContent=

1983年生。博士,副教授。主要研究方向为多模态抑郁症识别和分析、离散情感分析、维度情感识别、图像处理

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329419932175020273, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, xref=4, ext=[AuthorCompanyExt(id=1329419932191797490, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419932175020273, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=4School of Computer Science & Technology, Xi’an University of Posts and Telecommunications, Xi’an 710061), AuthorCompanyExt(id=1329419932200186099, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419932175020273, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=4西安邮电大学计算机学院 西安 710061)])]), Author(id=1329419933471060241, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, orderNo=5, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329419933596889363, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419933471060241, language=EN, stringName=Dongmei Jiang, firstName=Dongmei, middleName=null, lastName=Jiang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=5, address=5Pengcheng Laboratory, Shenzhen, Guangdong 518055, bio={"content":"

born in 1973. PhD, professor. Her main research interests include audio-visual fusion based speech processing, audio-visual emotion analysis and recognition, and facial animation synthesis of speakers with expressions

"}, bioImg=null, bioContent=

born in 1973. PhD, professor. Her main research interests include audio-visual fusion based speech processing, audio-visual emotion analysis and recognition, and facial animation synthesis of speakers with expressions

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329419933668192532, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, authorId=1329419933471060241, language=CN, stringName=蒋冬梅, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=5, address=5鹏城实验室 广东深圳 518055, bio={"img":"EREeNgT5sQem+X5oaK3f2A==","content":"

1973年生。博士,教授。主要研究方向为听视觉融合的语音处理、听视觉情感分析与识别、带表情的说话人面部动画合成

"}, bioImg=EREeNgT5sQem+X5oaK3f2A==, bioContent=

1973年生。博士,教授。主要研究方向为听视觉融合的语音处理、听视觉情感分析与识别、带表情的说话人面部动画合成

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329419932263100660, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, xref=5, ext=[AuthorCompanyExt(id=1329419932271489269, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419932263100660, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=5Pengcheng Laboratory, Shenzhen, Guangdong 518055), AuthorCompanyExt(id=1329419932279877878, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329419929369030861, companyId=1329419932263100660, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=5鹏城实验室 广东深圳 518055)])])]
陈海丰1 , 李溪1, 李岩2, 李健3, * , 何浪4, 蒋冬梅5
作者信息
  • 1陕西科技大学电子信息与人工智能学院 西安 710021
  • 2西北工业大学计算机学院 西安 710072
  • 3陕西科技大学电气与控制工程学院 西安 710021
  • 4西安邮电大学计算机学院 西安 710061
  • 5鹏城实验室 广东深圳 518055
通讯作者:
作者简介:

作者贡献声明

陈海丰提出方法思路以及修改意见;李溪实现方法设计并撰写论文;李岩负责部分实验以及论文修改;李健负责实验监督以及论文修改;何浪负责项目管理以及论文修改;蒋冬梅负责项目监督以及论文修改。

1988年生。博士,讲师。CCF会员。主要研究方向为情感计算、多模态情感识别、对话中的情感识别、多模态抑郁症分析、面部动作单元识别

2001年生。硕士研究生。CCF学生会员。主要研究方向为多模态情感识别

1998年生。硕士研究生。主要研究方向为机器学习、多模态理解

1975年生。博士,教授。主要研究方向为计算机视觉、数字孪生

1983年生。博士,副教授。主要研究方向为多模态抑郁症识别和分析、离散情感分析、维度情感识别、图像处理

1973年生。博士,教授。主要研究方向为听视觉融合的语音处理、听视觉情感分析与识别、带表情的说话人面部动画合成

Multimodal Emotion Recognition Method Based on Mixture of Fusion Experts
Haifeng Chen1 , Xi Li1, Yan Li2, Jian Li3, * , Lang He4, Dongmei Jiang5
Affiliations
  • 1School of Electronic Information and Artificial Intelligence, Shaanxi University of Science & Technology, Xi’an 710021
  • 2School of Computer Science, Northwestern Polytechnical University, Xi’an 710072
  • 3School of Electrical and Control Engineering, Shaanxi University of Science & Technology, Xi’an 710021
  • 4School of Computer Science & Technology, Xi’an University of Posts and Telecommunications, Xi’an 710061
  • 5Pengcheng Laboratory, Shenzhen, Guangdong 518055
  • born in 1988. PhD, lecturer. Member of CCF. His main research interests include affective computing, multimodal emotion recognition, emotion recognition in conversations, multimodal depression analysis, and facial action unit recognition

    born in 2001. Master candidate. Student member of CCF. Her main research interest includes multimodal emotion recognition

    born in 1998. Master candidate. His main research interests include machine learning and multimodal understanding

    born in 1975. PhD, professor. His main research interests include computer vision and digital twin

    born in 1983. PhD, associate professor. His main research interests include multimodal depression recognition and analysis, discrete emotion analysis, dimensional emotion recognition, and image processing

    born in 1973. PhD, professor. Her main research interests include audio-visual fusion based speech processing, audio-visual emotion analysis and recognition, and facial animation synthesis of speakers with expressions

出版时间: 2026-06-01 doi: 10.7544/issn1000-1239.202550544
文章导航
收藏切换

多模态情感识别旨在整合不同模态数据以推断情感状态。现有研究通常平等对待各模态或依赖单一模态设计固定策略的融合方法,难以应对模态数据贡献度不均衡的问题。为此,提出一种动态的多模态混合融合方法(MoMFE),并基于混合专家模型框架设计了自适应路由模块和多模态融合专家模块,动态评估各模态的贡献度并选择相应的融合策略。其中,自适应路由机制通过模态间相关性分析动态评估模态贡献度,并为融合专家的选择提供指导,同时引入专家引导损失,进一步优化融合专家选择;多模态融合专家则针对不同模态贡献度组合进行互补融合,并通过引入共享专家以避免全局信息丢失和参数冗余。基于MER2024,CMU-MOSEI,CH-SIMS数据集上的多模态情感识别任务的对比实验表明,相比其他先进的多模态融合方法,MoMFE方法在情感识别准确率(Acc-2)、F1分数等核心评估指标上均取得优异的性能。

情感识别  /  多模态融合  /  混合专家  /  自适应路由  /  跨模态注意力

Multimodal emotion recognition aims to integrate data from multiple modalities for accurate inference of emotional states. Existing research either treats all modalities equally or adopts fixed fusion strategies based on a single modality, failing to adequately address the imbalance of modality contributions. To tackle this, we propose a dynamic multimodal fusion method MoMFE based on the mixture of experts (MoE) framework, incorporating an adaptive router module and a multimodal fusion expert module. This method dynamically evaluates the contribution of each modality and selects appropriate fusion strategies accordingly. The adaptive router mechanism dynamically assesses modality contributions through inter-modal correlation analysis and dynamic weighting to guide the selection of fusion experts, while an expert-guided loss function is integrated to further optimize the expert selection process. The multimodal fusion experts perform complementary fusion for different modality contribution combinations and introduce a shared expert to mitigate the loss of global information and reduce parameter redundancy. Comparative experiments on three benchmark datasets (MER2024, CMU-MOSEI, and CH-SIMS) for multimodal emotion recognition demonstrate that MoMFE method outperforms state-of-the-art (SOTA) multimodal fusion methods in core metrics, including binary emotion recognition accuracy (Acc-2) and F1 score. Notably, the method achieves an average improvement of approximately 2% on CH-SIMS dataset.

emotion recognition  /  multimodal fusion  /  mixture of experts  /  adaptive router  /  cross-modal attention
陈海丰, 李溪, 李岩, 李健, 何浪, 蒋冬梅. 基于混合融合专家的多模态情感识别方法. 计算机研究与发展, 2026 , 63 (6) : 1478 -1491 . DOI: 10.7544/issn1000-1239.202550544
Haifeng Chen, Xi Li, Yan Li, Jian Li, Lang He, Dongmei Jiang. Multimodal Emotion Recognition Method Based on Mixture of Fusion Experts[J]. Journal of Computer Research and Development, 2026 , 63 (6) : 1478 -1491 . DOI: 10.7544/issn1000-1239.202550544
情感识别通过文本、语音、图像等数据识别情感状态,在人机交互、舆情分析等领域广泛应用[1]。多模态情感识别凭借整合多种模态数据的优势日益受到重视[2-3]。而多模态情感识别的一大挑战是如何有效地融合不同模态传达的情感信息[4],这主要源于人类复杂的情感表达方式,导致不同模态包含的情感信息量及其对情感识别任务的重要性不同。
现有研究主要通过单一固定的融合方式实现多模态融合。例如,Lian等人[5]通过计算模态间注意力分数作为权重进行加权融合;Luo等人[6]提出对情感语义表达一致的各模态特征表示进行加权连接;Liang[7]和Feng等人[8]分别利用遗传算法和情感知识指导衡量每个模态的贡献,从而选出主导模态,构建单一模态为主导的多模态融合。然而面对不平衡的模态贡献分布,单一的融合范式缺乏对模态间主导关系的进一步研究,动态选择多种融合策略的方法被引入到多模态融合中。申锦涛[9]提出一种基于GRU门的多模态动态融合方法,选择是否以文本模态为主而进行加权融合,利用门控实现对文本主导和非文本模态主导的2种融合方法的选择。然而,在实际场景中,主导模态往往不是单一的,模态贡献度组合多样化。如图1呈现了CH-SIMS数据集的样本分类统计,该数据集同时包含单模态和多模态标签。经分析发现,数据集中的模态贡献度呈现出3种典型分布模式:单模态主导型(单一模态贡献显著)、双模态协同型(2种模态共同主导)以及三模态均衡型(各模态贡献相当)。因此,需要依据不同模态贡献度分布动态选择对应的互补融合方法。
近年来,混合专家(mixture of experts,MoE)模型凭借其动态选择专家和数据处理的灵活性,在情感识别领域广泛应用。例如:Zhuang等人[10]利用MoE层的每个专家专注于单模态表示的不同方面,自主融合针对每个模态独特的多个局部表示;Zhao等人[11]则是将MoE作为多模态融合后的增强模块,通过加权专家组合来增强多模态表示中的情感信息细节。以上方法存在2个关键局限:一是多数工作仅将MoE用于单模态表示增强或后处理阶段,缺乏面向多模态融合策略的动态选择机制;二是所设计的专家结构多采用标准前馈网络,难以针对不同模态间的贡献分布进行专业化建模,从而限制了多样化融合需求的适应能力。为此,本文提出构建具备不同专业化属性的互补融合专家。这一设计不仅使MoE框架有望成为实现动态融合的有效工具,更能通过组建多样化且专业化的互补融合专家、灵活选择适配的融合策略,进一步提升多模态特征的融合效果。
本文提出一种多模态混合融合专家(mixture of multimodal fusion experts,MoMFE)方法来实现自适应的多模态融合。MoMFE基于混合专家模型框架,设计了自适应路由模块(adaptive router module,ARM)和多模态融合专家模块(multimodal fusion expert module,MFEM),实现依据模态贡献度分布动态选择融合方法。具体而言,在ARM中,首先计算模态间相互依赖系数,以此量化模态间相关性和信息共享程度,并为路由选择融合专家提供指导,从而增强路由对模态间主导关系的感知能力。接着引入专家引导损失来优化专家选择,确保选择的融合专家更符合模态主导关系,减少因信息不对称导致的路由决策偏差。在MFEM中,基于跨模态注意力机制计算模态间的加权特征进行互补融合,构建了多种适应不同模态主导关系的融合方法,为多模态特征融合提供了更灵活且适配的解决方案,以解决单一固定融合方式所造成的重要信息丢失问题。本文的主要贡献总结有3点:
1)提出一种基于混合专家模型的多模态融合方法,可依据各模态的贡献度动态选择适配的融合策略;
2)在路由机制中引入模态间相关性分析模块,通过动态评估模态贡献度,提升其对模态间相关性的感知能力,实现对融合专家有效的决策能力;
3)根据模态贡献度组合,利用跨模态注意力机制设计了对应的融合专家,最大化利用互补模态的有效情感信息。通过在MER2024,CMU-MOSEI,CH-SIMS这3个数据集上进行对比实验和消融实验,验证了本文提出方法在多模态情感识别任务中的有效性。
情感识别研究最初主要围绕文本[12]、视觉[13]和音频[14]等单一模态展开。随着深度学习的发展,各模态均经历了从传统手工特征到深层特征的演进。其中文本模态从早期词袋模型,逐渐过渡到基于GloVe[15]的预训练词向量,再到BERT[16]等大规模预训练语言模型生成的上下文情感表征;视觉模态从人工设计的方向梯度直方图(histogram of oriented gradients,HOG)等图像特征,发展为通过卷积神经网络以及Vision Transformer[17]等,逐步提升面部表情和肢体动作的识别精度;音频模态也从传统的梅尔频谱系数(Mel-frequency cepstral coefficients,MFCC)等声学特征,发展到借助Hubert[18]和Whisper[19]等语音预训练模型,更高效地提取与情感相关的深度特征。这些单模态领域的突破为多模态情感识别研究提供了强大的特征提取工具,为多模态情感识别中的特征融合与交互建模奠定了基础。
多模态融合是多模态情感识别任务中的关键步骤,早期融合和后期融合是2种经典的多模态融合策略。早期融合在特征提取阶段将不同模态的特征进行简单拼接,形成一个统一的特征表示。后期融合在每个模态独立进行特征提取和预测后,将各模态的预测结果进行融合。尽管以上方法在一定程度上实现了多模态信息的整合,由于情感表达在多模态数据上传递的信息量并不相同,过于简单的特征处理无法使多模态融合得到最有效的信息。最近许多工作致力于分析模态之间的信息相关性来解决模态不平衡问题。Wei等人[20]将互信息与多模态情感识别结合,提出了一种层次互信息最大化框架,以减少与任务相关的信息丢失。Cai等人[21]通过计算模态之间的互信息并利用互信息神经估计方法,最大化不同模态之间特征建模的相似性。此外,一些框架尝试根据不同模态对最终情感任务的贡献设计融合方法。Kumar等人[22]提出一种条件门控机制,基于说话者的语言、语音和面部表情,选择性地学习不同模态的重要性权重。Majumder等人[23]则通过使用门控网络自适应调节各个模态特征的重要性权重,从而实现多模态融合。上述方法在多模态融合过程中实现了对不同模态的差异化侧重,但其特征融合层并没有根据不平衡的模态而设计个性化的融合方法。因此,本文通过动态分析模态间相关性,构建了差异化的互补融合方案。
MoE是一种结合多个专家子模型的集成学习方法,最初由Jacobs等人[24]提出。MoE的核心思想是希望将输入空间划分为多个类别,每个区域由一个专家模型负责处理,并通过路由机制动态选择最合适的专家进行后续任务。早期MoE主要应用于分类和回归任务,但随着深度学习的发展,MoE逐渐被融入更复杂的神经网络架构中。近年来,MoE在大规模预训练模型中的应用取得了显著进展。例如,Switch Transformer[25]是将MoE结构引入Transformer[26]模型的方案,通过稀疏激活机制,仅激活一小部分专家模型,显著提升了模型的参数效率和训练速度。随后,Gshard[27]和GlaM[28]等模型进一步优化了MoE的分布式训练策略,解决了专家负载不均衡等问题。此外,研究人员还在MoE的动态路由、专家数量优化和路由机制设计等方面进行了深入探索[29]。
随着多模态数据在人工智能应用中的重要性日益凸显,MoE架构已被广泛应用于各种多模态的下游任务中。例如,在多模态Transformer模型中,MoE结构被用于动态选择不同模态的专家网络[30],以增强模型对多模态数据的语义理解能力和泛化性能。然而,在情感识别任务中,现有的MoE架构在实现高质量的融合学习效果方面仍面临挑战。为了进一步研究针对性的模态特征融合专家,Lin等人[31]提出细粒度的专家融合,为每个模态单独构建视觉专家。Han等人[32]集成了MoE融合层与模态嵌入,并提供了多种路由器配置,以处理不同复杂度级别的多模态组合。尽管上述MoE模型考虑到多种模态数据的特性,但是缺乏对专家的针对性优化。Xiong等人[33]通过构建主模态与其他参考模态融合的专家模块,并引入模态重要性权重,在融合过程中选择性地关注来自特定模态的信息,这种方法在处理特定模态组合时表现出色。但在扩展到三模态或缺失模态时有局限性。因此,本文基于MoE框架,实现动态选择不同专业化的互补融合方法。
如图2(a)所示,整体情感识别过程主要包括多模态特征提取、MoMFE以及情感识别。其中MoMFE模块由ARM和MFEM组成,ARM中引入相关性分析模块,MFEM则包含多个不同的互补融合方法。整体工作流程如下:
输入数据包含视觉(v)、音频(a)和文本(t)这3种模态。首先,通过单模态编码器分别提取各模态的特征,得到$ {\boldsymbol{f}}_{\text{v}}\in {\mathbb{R}}^{{{l}_{\text{v}}}\times {{d}_{\text{v}}}} $,$ {\boldsymbol{f}}_{\text{a}}\in {\mathbb{R}}^{{{l}_{\text{a}}}\times {{d}_{\text{a}}}} $,$ {\boldsymbol{f}}_{\text{t}}\in {\mathbb{R}}^{{{l}_{\text{t}}}\times {{d}_{\text{t}}}} $,其中$ {l}_{m} $表示序列长度,$ {d}_{m} $表示特征向量的维度,$ m\in \{\text{v},\text{a},\text{t}\} $。随后,利用模态特定的多层感知机(multilayer perceptron,MLP)将多模态特征映射到同一维度d。接着,将各模态特征输入ARM,分析模态间的相关性并计算激活专家的概率logits,同时依据贡献度大小将多模态特征$ {\boldsymbol{f}}_{\text{v}} $,$ {\boldsymbol{f}}_{\text{a}} $,$ {\boldsymbol{f}}_{\text{t}} $排序为$ {\boldsymbol{f}}_{\text{1}} $,$ {\boldsymbol{f}}_{\text{2}} $,$ {\boldsymbol{f}}_{\text{3}} $。之后,根据计算得到的logits来激活MFEM中最优的融合专家(Top-1),对$ {\boldsymbol{f}}_{\text{1}} $,$ {\boldsymbol{f}}_{\text{2}} $,$ {\boldsymbol{f}}_{\text{3}} $进行互补融合。最终,融合特征$ {\boldsymbol{F}}_{m}\in {\mathbb{R}}^{{{l}_{m}}\times {{d}_{m}}} $被输入分类器,从而确定情感状态。
MoE模型中的路由机制通常在词(token)级别进行专家选择,通过计算输入词与各专家之间的匹配分数来分配数据处理任务[34]。而针对情感识别任务,特征级别的选择方式能够更好地整合不同模态的信息,现有基于Softmax路由或者基于加权混合路由,根据输入多模态特征直接根据输入多模态特征直接计算样本与专家的相关性。然而面对模态贡献度不平衡的数据时,这种专家选择策略缺乏对模态间相关性的深入考量,无法保障模态贡献度组合与所选专家强相关,导致融合策略与输入特性失配。因此,如图2(b)所示,本文设计自适应路由模块,其中模态间相关性分析部分采用互信息系数估计方法计算模态特征的相互依赖性来动态量化各模态的贡献度,然后再计算与专家的匹配分数,从而优化路由选择过程。随后,专家选择基于模态可靠性优先原则,通过建立模态可靠性评估、特征重加权以及专家激活的级联机制,确保融合过程始终聚焦于最可信的模态组合。具体计算过程如下:
1)相关性分析模块
①计算相互依赖系数。首先分别计算模态间的相互表示$ {\boldsymbol{f}}_{\text{mu}}(\text{v},\text{a}) $,$ {\boldsymbol{f}}_{\text{mu}}(\text{v},\text{t}) $,$ {\boldsymbol{f}}_{\text{mu}}(\text{a},\text{t}) $,以视频模态$ {\boldsymbol{f}}_{\text{v}} $和语音模态$ {\boldsymbol{f}}_{\text{a}} $为例,使用跨模态的Transformer,其中交叉注意力(cross-attention)的查询输入为$ {\boldsymbol{f}}_{\text{v}} $,键和值输入为$ {\boldsymbol{f}}_{\text{a}} $。跨模态相互表示如式(1)所示。
$ {\boldsymbol{f}}_{\text{mu}}\left(\text{v},\text{a}\right)\boldsymbol=Transformer\left(\boldsymbol{Q}={\boldsymbol{f}}_{\text{v}},\boldsymbol{K}=\boldsymbol{V}={\boldsymbol{f}}_{\text{a}}\right)\;,$
其中Q表示查询;V表示值;K表示键。
然后使用MLP分别处理跨模态和单模态表示,通过计算二者输出的差值,获得视频和语音之间的相互依赖系数$ I(\text{v},\text{a}) $,如式(2)所示。
$ I\left(\text{v},\text{a}\right)=MLP\left(\left[{\boldsymbol{f}}_{\text{mu}}\left(\text{v},\text{a}\right)\oplus {\boldsymbol{f}}_{\text{a}}\right]\right)-\ln \left(MLP\left(\left[{\boldsymbol{f}}_{\text{v}}\oplus {\boldsymbol{f}}_{\text{a}}\right]\right)\right)\;,$
其中$ \left[{\boldsymbol{f}}_{\text{mu}}\left(\text{v},\text{a}\right)\oplus {\boldsymbol{f}}_{\text{a}}\right] $和$ \left[{\boldsymbol{f}}_{\text{v}}\oplus {\boldsymbol{f}}_{\text{a}}\right] $都表示对应位置特征相加。相应地,可以得到其他模态间的相互依赖系数$ I(\text{v},\text{t}) $和$ I(\text{t},\text{a}) $。
为了最大化每2个模态之间的相互依赖系数,引入损失$ {L}_{\text{mu}} $。通过最小化$ {L}_{\text{mu}} $,模型能够促使不同模态之间的特征表示进一步对齐,从而增强多模态信息的一致性,损失$ {L}_{\text{mu}} $计算如式(3)所示。
$ {L}_{\text{mu}}=-I\left(\text{a},\text{v}\right)-I\left(\text{v},\text{t}\right)-I\left(\text{t},\text{a}\right)\;。$
②计算可靠性系数。将同一模态相关的2个相互依赖系数相加定义为可靠性系数,以此衡量多个模态间的相关性以及对情感识别任务的表现能力。可靠性系数由式(4)计算。
$ \begin{cases}I\left(i\right)=I\left(i,{j}_{1}\right)+I\left(i,{j}_{2}\right)\;,\\i,{j}_{1},{j}_{2}\in \left\{\text{v},\text{a},\text{t}\right\};i\neq {j}_{1}\neq {j}_{2}\;。\end{cases} $
通过Softmax将可靠性系数归一化为$ {\boldsymbol{w}}_{i} $,将$ {\boldsymbol{w}}_{i} $作为权重对单模态特征进行加权,然后过归一化层重新映射为$ {\boldsymbol{g}}_{i} $,如式(5)所示。
$ \begin{cases}{\boldsymbol{w}}_{i}=Softmax\left(I\left(i\right)\right),i\in \left\{\text{v},\text{a},\text{t}\right\}\;,\\{\boldsymbol{g}}_{i}=Mean\left(GELU\left(RMSNorm\left({\boldsymbol{w}}_{i}\cdot {\boldsymbol{f}}_{i}\right)\right)\right)\;,\end{cases} $
其中RMSNorm为均方根归一化层,GELU为高斯误差线性单元层,Mean为均值处理。
2)融合专家及主导模态选择
最后将多模态特征相加为$ [{\boldsymbol{g}}_{\text{v}}\oplus {\boldsymbol{g}}_{\text{a}}\oplus {\boldsymbol{g}}_{\text{t}}] $,然后送入线性层处理,得到logits作为每个专家的激活率,并激活Top-1融合专家,如式(6)所示。
$ \boldsymbol{logits}=Linear\left([{\boldsymbol{g}}_{\text{v}}\oplus {\boldsymbol{g}}_{\text{a}}\oplus {\boldsymbol{g}}_{\text{t}}]\right)\;。$
对于主导模态的选择,按照可靠性系数的计算从大到小依次将模态特征排序为$ {\boldsymbol{f}}_{\text{1}} $,$ {\boldsymbol{f}}_{\text{2}} $,$ {\boldsymbol{f}}_{\text{3}} $:
$ \begin{cases}\boldsymbol{p}=Sort\left(I\left(i\right)\right),i\in \left\{\text{v},\text{a},\text{t}\right\}\;,\\{\boldsymbol{f}}_{\text{1}}=\boldsymbol{p}[\text{0}],{\boldsymbol{f}}_{\text{2}}=\boldsymbol{p}[1],{\boldsymbol{f}}_{\text{3}}=\boldsymbol{p}[2]\;。\end{cases} $
为了进一步引导路由激活相匹配的融合专家,引入专家引导损失$ {L}_{\text{na}} $。首先将专家激活率logits从大到小排序为$ E=\left\{{E}_{1},{E}_{2},{E}_{3}\right\} $,然后通过多模态可靠性系数$ {\boldsymbol{w}}_{i} $计算每个专家的置信度为$ S=\left\{{S}_{1},{S}_{2},{S}_{3}\right\} $,如式(8)所示。
$ \left\{\begin{aligned}&{r}_{\text{l}},{r}_{\text{m}},{r}_{\text{s}}=Sort({\boldsymbol{w}}_{i}),i\in \left\{\text{v},\text{a},\text{t}\right\}\;,\\&{S}_{\text{EF}}=1-avg({r}_{\text{l}},{r}_{\text{m}},{r}_{\text{s}})\;,\\&{S}_{\text{LF}}=1-\frac{{r}_{\text{s}-\text{l}}}{\max ({r}_{\text{l}-\text{s}},{r}_{\text{s}-\text{l}})}\;,\\&{S}_{\text{DF}}=1-\frac{{r}_{\mathrm{m}-\mathrm{s}}}{\max ({r}_{\text{l}-\textit{s}},{r}_{\text{m}-\text{s}})}\;。\end{aligned}\right. $
融合专家的E与S之间的引导损失定义为
$ {L}_{\text{na}}\left(E,S\right)=\sum\limits_{(i,j)\colon {S}_{i} \lt {S}_{j}}h\left({E}_{j}-{E}_{i}\right)\;, $
其中函数h是合页损失函数$ h\left(x\right)=\max \left(1-x,0\right) $,该函数为非递增函数。如果$ {S}_{j} \lt {S}_{i} $则$ {E}_{j} \lt {E}_{i} $,该函数惩罚E和S之间的反向配对,并鼓励E和S的顺序相同,从而引导路由选择置信度高的融合专家。
如图2(c)所示,本文利用跨模态注意力机制,构建了多种融合专家来处理各种多模态贡献度组合,包含单模态主导融合(dominant fusion,DF)专家、双模态协同融合(lack fusion,LF)专家、三模态均衡融合(equal fusion,EF)专家以及共享融合(shared fusion,SF)专家。其中,DF专家适用于某一模态特征的贡献程度显著高于其他模态;LF专家适用于某一模态特征贡献度较差的情况,此时应更加依赖其他2个模态进行融合;EF专家适用于所有模态贡献度均衡的情况;SF专家作为共享融合专家,旨在捕获独立于模态贡献度变化的通用情感特征,为模型提供基础的泛化表示能力。
1)DF专家
首先采用2个跨模态注意力模块基于主导模态进行特征增强,其中主导模态$ {\boldsymbol{f}}_{\text{1}} $被用作查询Q,以指定注意力机制的聚焦位置。随后,将增强后的模态特征进行拼接,并通过线性层进一步处理后与主导模态相加,最终得到融合特征$ {\boldsymbol{F}}_{{m}} $:
$ \left\{\begin{aligned}& Att\left({\boldsymbol{f}}_{1},{\boldsymbol{f}}_{2}\right)=Attention\left(\boldsymbol{Q}={\boldsymbol{f}}_{1},\boldsymbol{K}={\boldsymbol{f}}_{2},\boldsymbol{V}={\boldsymbol{f}}_{2}\right)\;,\\& Att\left({\boldsymbol{f}}_{1},{\boldsymbol{f}}_{3}\right)=Attention\left(\boldsymbol{Q}={\boldsymbol{f}}_{1},\boldsymbol{K}={\boldsymbol{f}}_{3},\boldsymbol{V}={\boldsymbol{f}}_{3}\right)\;,\\& {\boldsymbol F}'_{m}=Linear\left(Concat\left(Att\left({\boldsymbol{f}}_{1},{\boldsymbol{f}}_{2}\right),Att\left({\boldsymbol{f}}_{1},{\boldsymbol{f}}_{3}\right)\right)\right)\;,\\& {\boldsymbol{F}}_{m}={\boldsymbol{f}}_{1}+{\boldsymbol F}'_{m}\;。\end{aligned}\right. $
2)LF专家
采用一个跨模态注意力模块对贡献度较差的模态$ {\boldsymbol{f}}_{\text{3}} $进行补充学习。其中,注意力机制的Q由2个重要模态特征的拼接$ Concat\left({\boldsymbol{f}}_{1},{\boldsymbol{f}}_{2}\right) $构成,而K和V均来自$ {\boldsymbol{f}}_{\text{3}} $。随后,通过残差连接和线性层进一步处理,得到融合特征$ {\boldsymbol{F}}_{{m}} $:
$ \begin{cases}{{\boldsymbol{F}}}'_{m}}=Norm\left(Concat\left({\boldsymbol{f}}_{1},{\boldsymbol{f}}_{2}\right)\right)\;,\\Att({{\boldsymbol{F}}}' _{m}},{\boldsymbol{f}}_{3})=Attention\left(\boldsymbol{Q}={{\boldsymbol{F}}}'_{m}},\boldsymbol{K}={\boldsymbol{f}}_{3},\boldsymbol{V}={\boldsymbol{f}}_{3}\right)\;,\\{\boldsymbol{F}}_{m}}=Linear\left({{\boldsymbol{F}}}'_{m}}+Att\left({{{\boldsymbol{F}}}}'_{m}},{\boldsymbol{f}}_{3}\right)\right)\;。\end{cases} $
3)EF专家
通过Softmax计算模态间注意力分数进行加权融合,其计算方法如式(12)所示。其中$ {\boldsymbol{F}} _{{\mathrm{m}}}^{'{\mathrm{T}}} $表示3个模态特征拼接后的转置向量,$ {\boldsymbol{\alpha}}_{i} $为注意力分数,用于衡量不同模态特征的重要性权重。$ {\boldsymbol{W}}_{i} $,$ {\boldsymbol{b}}_{i} $分别为可训练的注意力矩阵和偏置项。最终,$ {\boldsymbol{F}}_{\text{m}} $是融合3个较高质量模态特征得到的融合特征:
$ \left\{\begin{aligned}&{{\boldsymbol{F}}}'_{{m}}=Concat\left({\boldsymbol{f}}_{1},{\boldsymbol{f}}_{2},{\boldsymbol{f}}_{3}\right)\;,\\&{\boldsymbol{\alpha}}_{i}=Softmax\left({{\boldsymbol{F}}}_{{m}}^{'\text{T}}\cdot {\boldsymbol{W}}_{i}+{\boldsymbol{b}}_{i}\right)\;,\\&{\boldsymbol{F}}_{{m}}={{\boldsymbol{F}}}'_{{m}}\cdot {\boldsymbol{\alpha}}_{i}\;。\end{aligned}\right. $
4)SF专家
虽然在前期训练可选择的专家时,强调各个专家的独特性,即它们所学习的知识侧重不同,但是由于专家之间没有共享机制,导致专家参数的冗余。受Deepseek-MoE[35]启发,引入共享的融合专家,不论路由选择哪个融合专家,多模态特征都将同时分配给共享的融合专家,融合结构与均衡模态融合专家一致。最终的融合结果如式(13)所示,其中α为共享融合专家的重要性权重。
$ {\boldsymbol{F}}_{\text{fusion}}={\boldsymbol{F}}_{{m}}+\alpha \cdot S haredFusion\left({\boldsymbol{f}}_{1},{\boldsymbol{f}}_{2},{\boldsymbol{f}}_{3}\right)\;。$
获得融合特征之后,将其输入情感分类器以得到情感识别的类别结果,如式(14)所示。
$ \hat{{\boldsymbol{y}}} =Classifier\left({\boldsymbol{F}}_{\text{fusion}}\right)\;。$
并使用交叉熵损失作为情感分类任务的损失:
$ {L}_{\text{task}}=-N\mathbf{\ln }\left(\boldsymbol{logit}{\boldsymbol{s}}_{\text{e}}\right)\;, $
其中N表示真实情感标签的索引,$ \boldsymbol{logit}{\boldsymbol{s}}_{\text{e}} $表示对所有情感类别预测的概率分布。
本文最终的损失函数计算如式(16)所示。
$ L={L}_{\text{task}}+\beta {L}_{\text{mu}}+\gamma {L}_{\text{na}}\;, $
其中$\beta $和$\gamma $表示为总损失中各部分贡献的权重系数。
本文在3个公开的多模态情感数据集上进行实验,都包含视频、语音、文本3个模态。分别是MER2024[5],CMU-MOSEI[36],CH-SIMS[37],数据集的统计细节如表1所示。
MER2024是第2届多模态情感识别挑战赛推出的数据集,重点关注开放词汇情感识别和半监督学习。MER2024-SEMI数据集包含约6199个有标签的视频片段,采用6类离散情感标签。数据集设计强调真实场景的复杂性,多来源于影视片段。
CMU-MOSEI是多模态情感分析领域的经典数据集,包含1000名不同讲述者的22856个带标注的YouTube视频评论片段,覆盖250个主题,总时长约65 h。每个片段均标注$ -3\sim +3 $的连续情感强度值,以及7种离散情绪类别。
CH-SIMS是一个中文多模态情感数据集,专注于提供细粒度的模态注释,支持单模态和多模态情感分析任务。包含2281个视频片段,采集自电影、电视剧和综艺节目,具有丰富的人物背景、年龄跨度大、数据质量高等特点。与前2个数据集不同的是,对于每个视频片段不仅进行多模态标注,还有单模态标注。
1)评价准则
本文以加权F1分数作为核心评估指标,应对样本类别不平衡的数据集。实验过程中,所有数据集均以F1分数作为模型选择的主要依据。为保持与其他研究的可比性,同时提供了准确率(Acc)、平均绝对误差(Mae)和相似度系数(Corr)等多项评估指标。对于MER2024数据集,由于官方未指定验证集,遵循官方的评价标准,对训练集采用5折交叉验证,并使用测试集进行模型测试。评价指标有加权平均F1分数和6分类准确率(Acc-6)。对于CMU-MOSEI数据集和CH-SIMS数据集,评价指标采用2级精度准确率(Acc-2),Mae,Corr,F1。不同的是,CH-SIMS还使用3级精度准确率(Acc-3)和5级精度准确率(Acc-5),而CMU-MOSEI还使用7级精度准确率(Acc-7)。除Mae外,其他指标的数值越高表明模型性能越好。
2)评价指标
Mae是一种衡量预测值与真实值之间绝对差异的回归指标,数值越小表示模型预测越精准,在多模态情感分析中常用于评估连续情感维度(如情感强度)的预测准确性。
Corr用于评估预测值与真实值之间的线性相关性,其值越接近1表示模型对情感趋势的捕捉能力越强。能有效反映模型是否保留了情感变化的相对顺序,但对绝对误差不敏感,常与Mae互补使用。
Acc是分类任务中最直观的评估指标,表示正确预测的样本比例,不同细粒度的准确率可反映模型从基础极性判别到细粒度分类的综合能力,但在数据类别不平衡时可能会掩盖模型对少数类的识别缺陷。
F1综合了精确率与召回率的调和平均值,适合处理类别不平衡的情感数据集,如某些情绪样本稀少的情况。F1通过按类别样本量加权,能更公平地评估模型对各类情感的识别均衡性,是多模态情感任务的核心指标,其提升通常直接反映模型实用性的增强。
实验环境基于Python3.9和PyTorch2.3构建,所有实验均在配备NVIDIA GeForce RTX 3090 GPU的设备上进行。模型训练采用Adam优化器,其权重衰减为0.02,初始学习率为0.0001,最大迭代次数为40,批大小设置为4。分类任务使用交叉熵损失函数,回归任务使用均方误差损失函数。其他超参数设置:各模态特征经MLP映射到特征维度d = 256,共享专家权重为α = 0.15,损失$ {L}_{\text{mu}} $权重为$\beta $ = 0.2,损失$ {L}_{\text{na}} $权重为$\gamma $ = 0.2。
对于视觉特征,先使用openFace工具箱[38]从视频中检测并对齐人脸,再通过CLIP-large图像编码器[39]提取深度特征。对于声学特征,与之前的工作[40]相同,使用HuBert-large模型[18]提取深度特征。对于文本特征,使用Baichuan2-13B模型[41]提取深度特征。视觉特征、音频特征和文本特征的维度分别为$ {d}_{\text{v}}= 768 $,$ {d}_{\text{a}}= 1\;024 $,$ {d}_{\text{t}}=5\;120 $。
本文设计了以下6方面的实验以全面验证所提方法的有效性:与其他前沿方法的对比实验、自适应路由有效性实验、多模态融合专家有效性实验、缺失模态下模型性能实验、模型噪声鲁棒性实验、超参数评估实验。
为全面评估本文方法的有效性,本文在MER2024,CMU-MOSEI,CH-SIMS这3个情感数据集上,与当前最先进的12种多模态融合方法进行了系统对比。对比方法涵盖3个方面:1)基于固定模态主导的方法(MPID[42],GFNPME[43],MR-CCL[44]);2)动态权重融合方法(SIMSUF[4],DWE[7],KuDA[8],LAA[40],AGT[45]);3)混合专家架构方法(GLoMo[10],TGMoE[11],FuseMoE[32],AC-MoE[46]),实验数据来源于所对比文章的公开结果。在MER2024数据集的实验结果如表2所示,在CMU-MOSEI和CH-SIMS数据集的实验结果如表3所示。本文方法MoMFE在情感识别准确率(Acc-2)、F1分数等核心指标上较所有对比方法均有提升。
1)与基于固定模态主导方法对比
与MR-CCL,MPID,GFNPME方法进行对比。其中,MR-CCL通过固定权重结合多模态和单模态分类器的输出;MPID首先以文本模态为主导分别整合音频和视觉模态,最后进行全局融合;GFNPME先利用其他模态增强文本表征,再逐步优化视觉和音频模态的特征。实验数据表明,在MER2024数据集中,与采用固定权重融合的MR-CCL方法相比,MoMFE在F1分数上提升了1.63个百分点。在CMU-MOSEI和CH-SIMS数据集中,MoMFE在Corr,Acc-2,F1分数等评价指标上均优于以文本为主导模态的固定融合方法(MPID和GFNPME)。这些结果说明本文提出的ARM能够有效地动态衡量各模态的贡献度,在多模态融合中实现更全面的评估。
2)与动态权重融合方法对比
与LAA,AGT,SIMSUF,KuDA,DWE等方法进行比较。LAA计算模态间注意力分数进行加权融合,AGT基于多头注意力机制计算模态间相似性,并在某一模态与其他模态相似性显著偏低时直接剔除该模态;SIMSUF,KuDA,DWE方法分别通过计算互信息系数、动态注意力融合模块和遗传算法,动态选择主导模态进行融合。从实验数据综合来看,MoMFE在3个数据集上多个指标均优于以上方法,其中在CH-SIMS数据集上提升更为明显,与单一的融合策略的KuDA方法相比,Acc-2和F1指标分别提升了3.59%和3.80%,这一结果说明了多种互补融合方法的设计为数据提供更灵活的选择。在MER2024数据集中,相较于AGT方法直接剔除贡献度低的模态,MoMFE方法通过特征补充使得F1分数提升了0.05%。这表明,MoMFE能够进一步挖掘低贡献度模态的潜在信息。
3)与混合专家架构方法对比
与AC-MoE,TGMoE,FuseMoE,GLoMo方法进行比较。这些方法都使用MoE层实现多模态融合,但各有侧重。AC-MoE中的MoE网络用于提取融合表示中非语言模态的信息;TGMoE中的MoE层用来增强以文本主导的融合模态中的情感信息;FuseMoE设计了特殊的路由来控制专家的细粒度的多模态数据输入;GLoMo中的MoE层仅用于单模态的局部表示模块。与以上几种方法相比,MoMFE在CMU-MOSEI数据集上Acc-2和F1指标都提高了0.3%以上,在CH-SIMS数据集上Acc-2和F1指标比AC-MoE方法提升2.28%和2.47%,充分证明了将MoE用于设计动态选择多模态融合方法的有效性。
综合来看,MoMFE在Acc-2和F1指标上取得了最优效果,同时在部分指标上与现有最优方法保持接近。具体而言,在CMU-MOSEI数据集上,其Corr指标低于FuseMoE,Acc-7指标与GLoMo的差距仅为0.28%;在CH-SIMS数据集上,虽然Mae值优于大多数对比方法,但与AC-MoE相比仍有差距。这些实验数据表明,MoMFE在连续情感识别指标和细粒度分类指标上仍有提升空间,但在F1分数这一关键指标上展现了显著优势,这一结果充分说明了MoMFE在处理类别不平衡数据集时的优越性能。值得注意的是,MoMFE在CH-SIMS数据集上的性能提升显著优于其他基准数据集(MER2024,CMU-MOSEI)。这一差异可能源于各数据集内的模态特性分布:CH-SIMS数据集场景复杂,呈现出更加均衡的跨模态贡献度分布;相比之下,MER2024数据集中的情感表达主要依赖于声学模态特征[5],而CMU-MOSEI数据集则存在明显的文本模态主导倾向[37,47]。MoMFE在CH-SIMS数据集上的结果说明其针对真实场景中多变的多模态融合挑战所具备的适应性和性能优势。
自适应路由中的关键组成部分是相关性分析模块,因此,为验证该模块的有效性,设计如下实验:
1)在MER2024和CH-SIMS数据集上,对比了3种路由设置的效果:传统的Softmax路由、本文提出的自适应路由机制(ARM),以及在ARM基础上引入专家引导损失$ {L}_{\text{na}} $。其中Softmax路由直接计算专家激活率logits,并不包含相关性分析模块,实验结果如表4所示。
表4系统地评估了3种路由机制在跨模态情感识别任务中的表现。首先,相较于传统Softmax路由(直接计算专家激活率而未深入考虑模态相关性),本文提出的ARM路由通过引入模态相关性分析模块,在2个数据集上都取得了性能提升:MER2024数据集的F1分数从87.45%提升至88.81%,CH-SIMS数据集从81.19%提升至83.11%。这一性能提升得益于ARM路由引入的相关性分析模块,能够基于模态间的深层交互实现更有效的专家选择。相比之下,Softmax路由仅基于浅层的模态理解直接进行专家选择,在处理模态间复杂交互时存在一定的局限性。其次,进一步引入$ {L}_{\text{na}} $损失函数后,模型性能获得持续提升,即MER2024和CH-SIMS数据集的F1分数分别达到89.88%和84.51%。这一结果说明,$ {L}_{\text{na}} $通过约束路由决策与专家能力的一致性,强化了“模态可靠性-专家专业化”的匹配关系,促使路由形成更清晰的决策边界。
2)为了更直观地观察路由的分配数据的能力,可视化不同路由设置下每个融合专家所处理的数据类型比率(单模态主导数据、双模态协同数据、三模态均衡数据),如图3所示。
从图3可以看出,采用Softmax路由时,数据分布没有与特定专家的能力相匹配。例如,三模态均衡类型样本被分散到3个专家,而单模态主导类型样本却更多被分配给了EF专家中,并未依据模态贡献度组合将样本输入到相应的融合专家中。然而,在使用引入了相关性分析模块的ARM路由后,可以明显观察到路由对模态间相关性的感知能力增强。例如,DF专家更多被单模态主导类型样本激活,LF专家更多被双模态协同类型样本激活,EF专家更多被三模态均衡类型样本激活。此外,这一数据也体现了在添加相关性模块之后,路由对专家选择的正确率有明显提升。整体来看,这一分配模式与各专家的设计初衷高度一致,证明了相关性分析模块能够增强路由对模态贡献模式的判别能力。
此外,提出的基于相关性分析模块的自适应路由机制,不仅是ARM路由性能提升的核心原因,其本身也仅引入极少的参数与计算开销。该模块通过轻量化的模态交互计算,显著提升了路由决策精度,为后续专家的高效调度奠定了坚实基础。
为验证多模态融合专家的性能,设计如下实验:
1)为评估本文构建的3个互补融合专家在处理不同模态贡献组合时的性能,将CH-SIMS数据集分为3类数据分别是:单模态主导类型、双模态协同类型、三模态均衡类型,各自输入到专家中进行特征融合,然后进行情感识别并计算F1分数。实验结果如图4所示。
如图4所示,三模态均衡类型样本在所有专家中的F1分数均高于其他类型样本,这表明模态贡献度的不平衡确实对情感识别任务产生了影响。具体来看,每种数据类型在最适合其特点的融合方法下性能最佳。例如,EF专家在三模态均衡数据上获得83.25%的F1分数,显著高于其在其他数据类型的表现。这归因于其采用的多模态交叉注意力机制能够平等地整合所有模态信息,避免因强调单一模态而损失互补信息。LF专家在双模态协同数据上表现最佳,F1分数为82.07%,这是由于其设计的双模态交互的注意力机制能够识别并强化2个主导模态间的协同效应。DF专家在单模态主导数据上取得81.96%的F1分数,这得益于其主导模态增强的注意力机制,突出高贡献模态的特征的同时抑制噪声模态的干扰。这些结果证明,本文设计多样化互补融合方法能够最大化利用贡献度不均衡的数据,提升情感识别的准确性。
2)为评估SF专家的性能,在CH-SIMS数据集上进行消融实验,包含4种融合专家。首先在完整专家组合(DF+LF+EF)基础上进行SF专家的消融对照,其次针对专家两两组合(LF+EF,DF+EF,DF+LF)分别进行SF专家的消融测试,进一步全面考察SF专家在不同专家组合条件下的性能表现。实验结果如表5所示。
如表5所示,实验结果展现了不同专家组合在CH-SIMS数据集上的性能差异。通过系统性地对比分析DF,LF,EF,SF这4种融合专家的组合效果,发现模型性能随着专家组合的完善而显著提升,其中在DF+LF+EF基础上添加SF专家后F1分数提升0.8%,并在3项指标上均达到最优,说明SF专家通过参数共享机制有效学习了跨专家的通用融合模式,从而增强了对常见情感表达规律的捕获能力,并提升了模型的泛化能力。值得注意的是,即便在部分专家缺失的情况下,引入SF的专家组合仍能维持相对稳定的表现,这表明SF专家可作为“安全专家”提供稳健的输出,有效缓解因专家缺失导致的性能下降,确保系统稳定性。
此外,得益于前置路由的高效分配,对于每个输入样本,模型仅激活最匹配的融合专家,而无需同时计算所有专家网络。这种专家激活机制,使得模型在绝大部分情况下实际参与计算的参数量被大幅缩减,从而在保证模型性能提升的同时,有效控制计算开销。
为测试MoMFE在不同缺失模态情况下的鲁棒性表现,进行如下实验:模拟现实场景中常见的模态不完整情况,在MER2024,CMU-MOSEI,CH-SIMS这3个数据集上分别设计单模态缺失和双模态缺失这2种场景,设置了10%,30%,50%这3种不同比例的随机缺失样本。该实验主要观察模型在Acc和F1这2个核心指标的性能变化。为确保实验结果的可靠性,所有实验条件均重复5次并取均值作为最终结果,实验结果如表6所示。
从表6看出,不同数据集对模态缺失的敏感性存在显著差异。MER2024和CMU-MOSEI数据集受模态缺失的影响更为明显,在单模态50%缺失情况下,MoMFE的F1分数在MER2024数据集上相比基线下降了3.75%,相比CMU-MOSEI数据集下降了2.32%。而相比CH-SIMS数据集仅下降1.67%。这可能是由于MER2024和CMU-MOSEI数据集对单模态的依赖性更强,当这些关键模态缺失时,模型难以从剩余模态中获得足够的互补信息。且双模态缺失的情况会进一步加剧指标下降,这在3个数据集的实验结果中都得到了验证。同时,即便在严重的模态缺失条件下,MoMFE方法仍能保持相对稳定的性能表现,特别是在CH-SIMS数据集上展现出较强的鲁棒性,这充分证明了MoMFE在非理想数据条件下的适应能力和实用价值。
为测试MoMFE在噪声环境下的鲁棒性,进行如下实验:在3个模态上采用与MER2024官方一致的加噪方式,对于视觉模态采用高斯模糊模拟图像退化,对于语音模态通过添加环境噪声再现真实场景中的语音干扰,而文本模态保留ASR自动识别引入的固有噪声数据。为模拟现实场景中的噪声情况,实验在MER2024数据集上分别设计单模态加噪以及多模态加噪,并设置了10%,30%,50%共3种不同的随机加噪的样本比例。为确保实验结果的可靠性,所有实验条件均重复5次并取均值作为最终结果,实验结果如表7所示。
表7所示的MER2024数据集实验结果表明,MoMFE在不同噪声条件下均表现出良好的鲁棒性。语音模态作为该数据集的重要模态,其加噪10%~50%时模型性能指标F1分数从88.21%降至85.95%,下降幅度较视觉模态加噪情况更为明显。值得注意的是,即使在语音+视觉双模态同时加噪且样本比例为50%的情况下,模型指标F1分数仍能保持85.62%的稳定表现,仅比单语音模态加噪时下降约0.4个百分点。这一结果表明,虽然语音模态的噪声对模型性能影响较大,但整体而言,模型在不同类型和程度的噪声干扰下都表现出较为稳定的适应能力。
为了研究特征维度d以及共享专家权重α对MoMFE模型性能的影响,本文设置特征维度d取值为128,256,512,α范围为0.1~0.3。图5展示了不同设置下MoMFE模型的F1分数。
图5的实验数据展示了特征维度d和共享专家权重α对模型性能的影响。当特征维度d=256时,模型整体表现最优,最高F1分数达84.51%,表明该特征维度在表征能力和计算效率之间较为平衡:较小的特征维度(d=128)导致特征表达能力不足,最高F1分数仅83.68%,而过大的特征维度(d=512)则可能引入噪声,性能波动明显,最高84.20%降至最低81.67%。对于共享专家权重α,实验显示在0.1~0.2表现效果显著,其中α=0.15时在d=256条件下取得峰值;当α超过0.2后,性能呈单调下降趋势,说明过大的专家共享权重会影响各融合专家的特异性,而过分依赖共享融合专家的输出会影响情感识别的准确性。
本文提出一种基于混合专家框架的多模态情感识别方法,旨在解决多模态数据中情感特征质量不平衡的核心问题,提升情感识别任务的准确性与结果一致性。该方法的核心在于设计了多模态融合专家层,该层级由自适应路由模块与多模态重要性融合专家模块构成。一方面,自适应路由模块通过引入模态间相关性分析度量,实现对各模态动态贡献度的量化,同时结合专家引导损失函数优化路由决策机制,从而高效完成对最优融合专家的自适应选择;另一方面,针对不同模态在情感表征中的贡献差异,多模态重要性融合专家模块通过强化模态间互补信息的挖掘与利用,有效缓解了多模态数据中情感特征质量不均衡的问题,进一步提升了模型对复杂情感场景的表征能力。实验结果表明,相较于现有方法,本文所提基于混合专家的多模态情感识别方法在多模态情感识别任务中展现出显著的性能优势,验证了该方法在提升情感识别准确性与结果稳定性方面的有效性,为多模态情感分析领域提供了具有参考价值的技术思路。

参考文献 引证文献
排序方式:
1
Das R, Singh T D. Multimodal sentiment analysis: A survey of methods, trends, and challenges[J]. ACM Computing Surveys, 2023, 55(13): 1−38
2
Poria S, Hazarika D, Majumder N, et al. Beneath the tip of the iceberg: Current challenges and new directions in sentiment analysis research[J]. IEEE Transactions on Affective Computing, 2020, 14(1): 108−132
3
Islam M S, Kabir M N, Ghani N A, et al. Challenges and future in deep learning for sentiment analysis: A comprehensive review and a proposed novel hybrid approach[J]. Artificial Intelligence Review, 2024, 57(62): 1−79
4
Huang Jian, Ji Yanli, Qin Zhen, et al. Dominant single-modal supplementary fusion (SIMSUF) for multimodal sentiment analysis[J]. IEEE Transactions on Multimedia, 2023, 26: 8383−8394
5
Lian Zhen, Sun Haiyang, Sun Licai, et al. MER2024: Semi-supervised learning, noise robustness, and open-vocabulary multimodal emotion recognition[C]//Proc of the 2nd Int Workshop on Multimodal and Responsible Affective Computing. New York: ACM, 2024: 41−48
6
罗渊贻, 吴锐, 刘家锋, 等. 面向情感语义不一致的多模态情感分析方法[J]. 计算机研究与发展, 2025, 62(2): 374−382
Luo Yuanyi, Wu Rui, Liu Jiafeng, et al. Multimodal sentiment analysis method for sentimental semantic inconsistency[J]. Journal of Computer Research and Development, 2025, 62(2): 374−382 (in Chinese)
7
Liang Yang. A dynamic weighted fusion model for multimodal sentiment analysis[J]. Signal, Image and Video Processing, 2025, 19(8): 609−623
8
Feng Xinyu, Lin Yuming, He Lihua, et al. Knowledge-Guided dynamic modality attention fusion framework for multimodal sentiment analysis[J]. arXiv preprint, arXiv: 2410.04491, 2024
9
申锦涛. 基于模态交互和动态融合的多模态情感识别研究[D]. 沈阳: 辽宁大学, 2024
Shen Jintao. Research on multimodal emotion recognition based on modal interaction and dynamic fusion[D]. Shenyang: Liaoning University, 2024 (in Chinese)
10
Zhuang Yan, Zhang Yanru, Hu Zheng, et al. GLoMo: Global-local modal fusion for multimodal sentiment analysis[C]//Proc of the 32nd ACM Int Conf on Multimedia. New York: ACM, 2024: 1800−1809
11
Zhao Xueliang, Wang Mingyang, Tan Yingchun, et al. TGMoE: A text guided mixture-of-experts model for multimodal sentiment analysis[J]. International Journal of Advanced Computer Science & Applications, 2024, 15(8): 1227−1234
12
贾熹滨, 曾檬, 米庆, 等. 领域对齐对抗的无监督跨领域文本情感分析算法[J]. 计算机研究与发展, 2022, 59(6): 1225−1270
Jia Xibin, Zeng Meng, Mi Qing, et al. Domain alignment adversarial unsupervised cross-domain text sentiment analysis algorithm[J]. Journal of Computer Research and Development, 2022, 59(6): 1225−1270 (in Chinese)
13
Chen Haifeng, Zhang Peng, Guo Chujia, et al. Facial action unit representation based on self-supervised learning with ensembled priori constraints[J]. IEEE Transactions on Image Processing, 2024, 33: 5045−5069
14
Wang Yuhua, Shen Guang, Xu Yuezhu, et al. Learning mutual correlation in multimodal transformer for speech emotion recognition[C/OL]//Proc of the 2021 Interspeech. 2021: 4518−4522.[2025-04-16]. https://www.isca-archive.org/interspeech_2021/wang21ga_interspeech.html
15
Pennington J, Socher R, Manning C, et al. GloVe: Global vectors for word representation[C]//Proc of the 2014 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2014: 1532−1543
16
Devlin J, Chang Mingwei, Lee K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]//Proc of the 2019 Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, PA: ACL, 2019: 4171−4186
17
Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv preprint, arXiv: 2010.11929, 2020
18
Hsu W N, Bolte B, Tsai Y H, et al. Hubert: Self-supervised speech representation learning by masked prediction of hidden units[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 3451−3460
19
Radford A, Kim J W, Xu Tao, et al. Robust speech recognition via large-scale weak supervision[C]//Proc of the 38th Int Conf on Machine Learning. New York: ACM, 2023: 28492−28518
20
Wei Han, Chen Hui, Poria S. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis[C]//Proc of the 2021 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2021: 9180–9192
21
Cai Yunrui, Wu Zhiyong, Jia Jia, et al. Lora-Mer: Low-rank adaptation of pre-trained speech models for multimodal emotion recognition using mutual information[C/OL]//Proc of the 2024 Interspeech. 2024: 4658−4662.[2025-04-15]. https://www.isca-archive.org/interspeech_2024/cai24b_interspeech.html
22
Kumar A, Vepa J. Gated mechanism for attention based multi modal sentiment analysis[C]//Proc of the 2020 IEEE Int Conf on Acoustics, Speech and Signal Processing. Piscataway, NJ: IEEE, 2020: 4477−4481
23
Majumder N, Hazarika D, Gelbukh A, et al. Multimodal sentiment analysis using hierarchical fusion with context modeling[J]. Knowledge-Based Systems, 2018, 161: 124−133
24
Jacobs R A, Jordan M I, Nowlan S J, et al. Adaptive mixtures of local experts[J]. Neural Computation, 1991, 3(1): 79−87
25
Fedus W, Zoph B, Shazeer N. Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity[J]. Journal of Machine Learning Research, 2022, 23(120): 1−39
26
Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proc of the 31st Int Conf on Neural Information Processing Systems. New York: ACM, 2017: 6000−6010
27
Lepikhin D, Lee H J, Xu Yuanzhong, et al. Gshard: Scaling giant models with conditional computation and automatic sharding[C/OL]//Proc of the 9th Int Conf on Learning Representations. 2021: 1−23.[2025-04-23]. https://openreview.net/forum?id=qrwe7XHTmYb
28
Du Nan, Huang Yanping, Dai A M, et al. GlaM: Efficient scaling of language models with mixture-of-experts[C]//Proc of the 39th Int Conf on Machine Learning. New York: ACM, 2022: 5547−5569
29
Cai Weilin, Jiang Juyong, Wang Fan, et al. A survey on mixture of experts in large language models[J]. IEEE Transactions on Knowledge and Data Engineering, 2025, 37(7): 3896−3915
30
Sarkar R, Liang Hanxue, Fan Zhiwen, et al. Edge-MoE: Memory-efficient multi-task vision transformer architecture with task-level sparsity via mixture-of-experts[C]//Proc of the 2023 IEEE/ACM Int Conf on Computer Aided Design. Piscataway, NJ: IEEE, 2023: 1−9
31
Lin X V, Shrivastava A, Liang Luo, et al. Moma: Efficient early-fusion pre-training with mixture of modality-aware experts[J]. arXiv preprint, arXiv: 2407.21770, 2024
32
Han Xing, Nguyen H, Harris C, et al. FuseMoE: Mixture-of-experts transformers for fleximodal fusion[C]//Proc of the 38th Int Conf on Neural Information Processing Systems, New York: ACM, 2025: 67850–67900
33
Xiong Congxiong, Chen Hao, Zheng Hao, et al. Mome: Mixture of multimodal experts for cancer survival prediction[C]//Proc of the 27th Int Conf on Medical Image Computing and Computer-Assisted Intervention. Springer: Berlin, 2024: 318−328
34
Shazeer N, Mirhoseini A, Maziarz K, et al. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer[J]. arXiv preprint, arXiv: 1701.06538, 2017
35
Dai Damai, Deng Chengqi, Zhao Chenggang, et al. Deepseek-Moe: Towards ultimate expert specialization in mixture-of-experts language models[C]//Proc of the 62nd Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2024: 1290−1297
36
Zadeh A B, Liang P P, Poria S, et al. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph[C]//Proc of the 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2018: 2236−2246
37
Yu Wenmeng, Xu Hua, Meng Fanyang, et al. CH-SIMS: A Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality[C]//Proc of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 3718−3727
38
Baltrusaitis T, Zadeh A, Lim Y C, et al. OpenFace 2.0: Facial behavior analysis toolkit[C]//Proc of the 13th IEEE Int Conf on Automatic Face & Gesture Recognition. Piscataway, NJ: IEEE, 2018: 59−66
39
Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//Proc of the 38th Int Conf on Machine Learning. New York: ACM, 2021: 8748−8763
40
Zhao Zhixian, Chen Haifeng, Li Xi, et al. Improving multimodal emotion recognition by leveraging acoustic adaptation and visual alignment[C]//Proc of the 2nd Int Workshop on Multimodal and Responsible Affective Computing. New York: ACM, 2024: 67−71
41
Yang Aiyuan, Xiao Bin, Wang Bingning, et al. Baichuan 2: Open large-scale language models[J]. arXiv preprint, arXiv: 2309.10305, 2023
42
Li Tianyi, Liu Daming. MPID: A modality-preserving and interaction-driven fusion network for multimodal sentiment analysis[C]//Proc of the 31st Int Conf on Computational Linguistics. Stroudsburg, PA: ACL, 2025: 4313−4322
43
Xin Feng, Gong Wenjuan, Shi Tingbo, et al. Gated fusion network with progressive modality enhancement for sentiment analysis[J]. IEEE Access, 2024, 12: 165810−165821
44
Fan Qi, Li Yong, Xin Yi, et al. Leveraging contrastive learning and self-training for multimodal emotion recognition with limited labeled samples[C]//Proc of the 2nd Int Workshop on Multimodal and Responsible Affective Computing. New York: ACM, 2024: 72−77
45
Gao Fei, Shi Pujin. Audio-Guided fusion techniques for multimodal emotion analysis[C]//Proc of the 2nd Int Workshop on Multimodal and Responsible Affective Computing. New York: ACM, 2024: 62−66
46
Lei Ling, He Wangjun, Zheng Qiuyan, et al. Multimodal sentiment analysis based on expert mixing of subtask representations[J]. IEEE Access, 2025, 13: 44278−44287
47
Liu Yihe, Yuan Ziqi, Mao Huisheng, et al. Make acoustic and visual cues matter: CH-SIMS v2. 0 dataset and av-mixup consistent module[C]//Proc of the 24th Int Conf on Multimodal Interaction. New York: ACM, 2022: 247−258
2026年第63卷第6期
89
43
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202550544
  • 接收时间:2025-07-20
  • 首发时间:2026-11-17
  • 出版时间:2026-06-01
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2025-07-20
  • 修回日期:2025-12-15
  • 录用日期:2026-01-07
基金
作者信息
    1陕西科技大学电子信息与人工智能学院 西安 710021
    2西北工业大学计算机学院 西安 710072
    3陕西科技大学电气与控制工程学院 西安 710021
    4西安邮电大学计算机学院 西安 710061
    5鹏城实验室 广东深圳 518055

通讯作者:

参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202550544
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏