Article(id=1325493573717721504, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1325493569112388051, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202440896, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=research-article, receivedDate=1731859200000, receivedDateStr=2024-11-18, revisedDate=1751212800000, revisedDateStr=2025-06-30, acceptedDate=1752681600000, acceptedDateStr=2025-07-17, onlineDate=1793951894454, onlineDateStr=2026-11-06, pubDate=1769875200000, pubDateStr=2026-02-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1793951894454, onlineIssueDateStr=2026-11-06, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1793951894454, creator=13701087609, updateTime=1793951894454, updator=13701087609, issue=Issue{id=1325493569112388051, tenantId=1146029695717560320, journalId=1324759914146529478, year='2026', volume='63', issue='2', pageStart='275', pageEnd='538', issueExtLink='null', onlineDate='null', pubDate='1769875200000', pubDateStr='2026-02-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1793951893355, creator='13701087609', updateTime=1793952511329, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext={EN=IssueExt(id=1325496161167090054, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1325493569112388051, language=EN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=), CN=IssueExt(id=1325496161167090055, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1325493569112388051, language=CN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=)}, issueFiles=null, downloadFileDto=null}, startPage=305, endPage=323, ext={EN=ArticleExt(id=1325493574002934177, articleId=1325493573717721504, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=Adversarial Soft-Contrast Modulated Dynamic Graph Learning for Multimodal Emotion Recognition in Conversations, columnId=1325489891789910943, journalTitle=Journal of Computer Research and Development, columnName=Artificial Intelligence, runingTitle=null, highlight=null, articleAbstract=

Multimodal emotion recognition in conversations (MERC) has emerged as a prominent research focus in the domain of human-computer intelligent interaction. It demonstrates extensive applicability across diverse scenarios, including affective dialogue systems and conversational recommendation systems, thereby substantially improving user’s experience through enhanced emotional engagement. Identifying abstract emotional semantics in multimodal conversation scenarios poses a significant challenge. Current research predominantly relies on contrastive learning frameworks to extract discriminative feature. Although these approaches effectively enforce intra-class feature consistency, they inherently constrain fine-grained feature diversity. This limitation consequently impairs the model’s generalization capacity, particularly when processing samples from minority classes. In addition, current dialogue context graph learning only captures the dependence with a fixed window, neglecting the dynamic cross-correlation of the conversation flow, which can lead to either redundancy or insufficient of extracted context information. To address the aforementioned limitations, the adversarial soft-contrast modulated dynamic graph learning is proposed for MERC. Specifically, a dynamic discourse graph is firstly constructed according to the number of utterances emitted by the speaker in the conversation, which allows for dynamically modeling the conversational dependency range of the utterance in each modality to extract rich contextual information more precisely. Secondly, the adversarial soft-contrast training mechanism is designed to enhance the discrimination and robustness of the network learning. Among them, the class sample space is expanded by adding perturbations to the hidden layers of different modal feature extractors to generate adversarial samples, and the soft-contrast learning is exploited to maximize label semantic consistency of the original samples and the adversarial samples. Finally, a bi-stream graph learning strategy for different modalities is constructed, which not only explores the cross-modal consistent feature interaction guided by text semantics, but also learns the context dynamic dependencies within multimodal dialogue flow to collaboratively facilitates the complementary fusion of multimodal conversations data. Extensive experiments conducted on the IEMOCAP and MELD benchmark datasets have demonstrated that our proposed model achieves competitive results in terms of effectiveness for MERC, compared with current state-of-the-art methods.

, authors=Shunjie Wang, Guoyong Cai*, Guangrui Lü, authorsList=Shunjie Wang, Guoyong Cai, Guangrui Lü, authorCompany=null, correspAuthors=Guoyong Cai, authorNote=

born in 1996. PhD candidate. Her main research interests include deep learning and multi-modal affective computing

born in 1971, PhD, professor, PhD supervisor. Distinguished member of CCF. His main research interests include multi-modal affective computing, trustable AI theory and techniques

born in 1989. PhD, master supervisor. Member of CCF. His main research interests include computer vision and multi-modal information processing

, correspAuthorsNote=null, copyrightStatement=Copyright © 2026 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1325493577769419186, articleId=1325493573717721504, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=对抗软对比调制动态图学习的多模态会话情绪识别方法, columnId=1325489892104483745, journalTitle=计算机研究与发展, columnName=人工智能, runingTitle=null, highlight=null, articleAbstract=

多模态会话情绪识别(multimodal emotion recognition in conversations,MERC)已成为人机智能交互的研究热点,目前被广泛应用于情感对话机器人及对话推荐等多个场景。在多模态会话场景中识别抽象的情感语义是困难的,大多现有研究利用对比学习来提取判别特征,其尽管约束了类内特征的一致性,但损害了精细的多样性表征,导致模型泛化性降低,尤其不利于少类样本的学习。此外,目前的会话上下文学习多建模固定窗口的语境依赖,忽略了会话信息流的动态互相关。为了解决上述限制,提出对抗软对比调制动态图学习(adversarial soft-contrast modulated dynamic graph learning,ASDG)方法。具体地:首先根据会话场景中说话人的话语数量构建话语图实现在各自模态内动态地建模话语的会话依赖范围,以精准地提取丰富的语境信息;其次,设计对抗性软对比训练机制,通过在不同模态特征提取器的隐藏层中添加扰动生成对抗样本来扩展类样本空间,并使用软对比学习在原始样本和对抗样本上最大化类扩展样本的标签语义一致性以增强模型学习的判别性与鲁棒性;最后,构建针对不同模态的双流图学习策略,以协同地促进各模态会话数据的互补融合。在IEMOCAP和MELD的多模态会话基准数据集上进行广泛的实验,结果表明提出的方法与目前先进的方法相比,在MERC任务上取得了具有竞争力的效果。

, authors=王顺杰, 蔡国永*, 吕光瑞, authorsList=王顺杰, 蔡国永, 吕光瑞, authorCompany=null, correspAuthors=蔡国永, authorNote=

1996年生。博士研究生。主要研究方向为深度学习、多模态情感计算

1971年生。博士,教授,博士生导师。CCF杰出会员。主要研究方向为多模态情感计算、可信AI理论技术

1989年生。博士,硕士生导师。CCF会员。主要研究方向为计算机视觉、多模态信息处理

, correspAuthorsNote=
蔡国永()
, copyrightStatement=版权所有 © 2026 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=tnhp+6CH9ZpBXr1yOqBt4w==, magXml=NDmaAXX8eO/PpX9zDJAC+g==, pdfUrl=null, pdf=dxrfxTAWMo6y6fe/3zo6YQ==, pdfFileSize=2640835, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=odt8GJ5fiafn8hoEJhXJpw==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=EO7Oo9CoBpZ66ArmYlRSMw==, mapNumber=null, fund=null)}, authors=[Author(id=1325493578285318586, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=wsj12130@foxmail.com, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325493578377593275, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, authorId=1325493578285318586, language=EN, stringName=Shunjie Wang, firstName=Shunjie, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"content":"

born in 1996. PhD candidate. Her main research interests include deep learning and multi-modal affective computing

"}, bioImg=null, bioContent=

born in 1996. PhD candidate. Her main research interests include deep learning and multi-modal affective computing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325493578474062268, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, authorId=1325493578285318586, language=CN, stringName=王顺杰, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"img":"EfDEKRK25FOsN4sGYbYblg==","content":"

1996年生。博士研究生。主要研究方向为深度学习、多模态情感计算

"}, bioImg=EfDEKRK25FOsN4sGYbYblg==, bioContent=

1996年生。博士研究生。主要研究方向为深度学习、多模态情感计算

, aboutCorrespAuthor=null)}, companyList=null), Author(id=1325493578557948350, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=ccgycai@guet.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1325493578641834431, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, authorId=1325493578557948350, language=EN, stringName=Guoyong Cai, firstName=Guoyong, middleName=null, lastName=Cai, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=null, bio={"content":"

born in 1971, PhD, professor, PhD supervisor. Distinguished member of CCF. His main research interests include multi-modal affective computing, trustable AI theory and techniques

"}, bioImg=null, bioContent=

born in 1971, PhD, professor, PhD supervisor. Distinguished member of CCF. His main research interests include multi-modal affective computing, trustable AI theory and techniques

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325493578729914816, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, authorId=1325493578557948350, language=CN, stringName=蔡国永, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=null, bio={"img":"kfxwXQSG7jspWV5+TgsbBg==","content":"

1971年生。博士,教授,博士生导师。CCF杰出会员。主要研究方向为多模态情感计算、可信AI理论技术

"}, bioImg=kfxwXQSG7jspWV5+TgsbBg==, bioContent=

1971年生。博士,教授,博士生导师。CCF杰出会员。主要研究方向为多模态情感计算、可信AI理论技术

, aboutCorrespAuthor=null)}, companyList=null), Author(id=1325493578805412290, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=lvguangrui_rio@163.com, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325493578876715459, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, authorId=1325493578805412290, language=EN, stringName=Guangrui Lü, firstName=Guangrui, middleName=null, lastName=Lü, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"content":"

born in 1989. PhD, master supervisor. Member of CCF. His main research interests include computer vision and multi-modal information processing

"}, bioImg=null, bioContent=

born in 1989. PhD, master supervisor. Member of CCF. His main research interests include computer vision and multi-modal information processing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325493578948018628, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, authorId=1325493578805412290, language=CN, stringName=吕光瑞, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"img":"BNzvNSDFdB9/5VhgsCuQWA==","content":"

1989年生。博士,硕士生导师。CCF会员。主要研究方向为计算机视觉、多模态信息处理

"}, bioImg=BNzvNSDFdB9/5VhgsCuQWA==, bioContent=

1989年生。博士,硕士生导师。CCF会员。主要研究方向为计算机视觉、多模态信息处理

, aboutCorrespAuthor=null)}, companyList=null)], keywords=[Keyword(id=1325493579044487621, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=EN, orderNo=1, keyword=multimodal emotion recognition in conversations (MERC)), Keyword(id=1325493579136762310, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=EN, orderNo=2, keyword=dynamic discourse graph), Keyword(id=1325493579199676871, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=EN, orderNo=3, keyword=adversarial soft-contrast), Keyword(id=1325493579266785736, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=EN, orderNo=4, keyword=bi-stream graph learning), Keyword(id=1325493579350671817, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=EN, orderNo=5, keyword=semantic consistency), Keyword(id=1325493579417780682, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=CN, orderNo=1, keyword=多模态会话情绪识别), Keyword(id=1325493579501666763, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=CN, orderNo=2, keyword=动态话语图), Keyword(id=1325493579581358540, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=CN, orderNo=3, keyword=对抗性软对比), Keyword(id=1325493579635884493, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=CN, orderNo=4, keyword=双流图学习), Keyword(id=1325493579698799054, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=CN, orderNo=5, keyword=语义一致性)], refs=[Reference(id=1325493583041659361, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=Singh G V, Firdaus M, Ekbal A, et al. EmoInt-Trans: A multimodal transformer for identifying emotions and intents in social conversations[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022, 31: 290−300, articleTitle=null, refAbstract=null), Reference(id=1325493583125545442, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhou Hao, Huang Minlie, Zhang Tianyang, et al. Emotional chatting machine: Emotional conversation generation with internal and external memory[C]//Proc of the 32nd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2018: 730−738, articleTitle=null, refAbstract=null), Reference(id=1325493583192654307, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Rosa R L, Schwartz G M, Ruggiero W V, et al. A knowledge-based recommendation system that includes sentiment analysis and deep learning[J]. IEEE Transactions on Industrial Informatics, 2018, 15(4): 2124−2135, articleTitle=null, refAbstract=null), Reference(id=1325493583251374564, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Chatterjee A, Narahari K N, Joshi M, et al. SemEval−2019 task 3: EmoContext contextual emotion detection in text[C]//Proc of the 13th Int Workshop on Semantic Evaluation. Stroudsburg, PA: ACL, 2019: 39−48, articleTitle=null, refAbstract=null), Reference(id=1325493583318483429, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Bhavan A, Chauhan P, Shah R R. Bagged support vector machines for emotion recognition from speech[J]. Knowledge-Based Systems, 2019, 184(C): 104886, articleTitle=null, refAbstract=null), Reference(id=1325493583389786598, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Shou Yuntao, Tao Meng, Wei Ai, et al. Adversarial alignment and graph fusion via information bottleneck for multimodal emotion recognition in conversations[J]. Information Fusion, 2024, 112: 102590, articleTitle=null, refAbstract=null), Reference(id=1325493583465284071, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Ruobing, Zha Daren, Zhao Qingfei, et al. RW-MMDCG: Muti-modal via rolling-Window directed graph network for conversational emotion recognition[C]//Proc of the 26th Int Conf on Computer Supported Cooperative Work in Design. Piscataway, NJ: IEEE, 2023: 606−611, articleTitle=null, refAbstract=null), Reference(id=1325493583544975848, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Jiang, Wang Xiaoping, Lv Guoqing, et al. GraphCFC: A directed graph based cross-modal feature complementation approach for multimodal conversational emotion recognition[J]. IEEE Transactions on Multimedia, 2023, 26: 77−89, articleTitle=null, refAbstract=null), Reference(id=1325493583633056233, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Jiang, Wang Xiaoping, Lv Guoqing, et al. GraphMFT: A graph network based multimodal fusion technique for emotion recognition in conversation[J]. Neurocomputing, 2023, 550(suppl-C): 126427, articleTitle=null, refAbstract=null), Reference(id=1325493583704359402, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Lu Nannan, Han Zhiyuan, Han Min, et al. Bi-stream graph learning based multimodal fusion for emotion recognition in conversation[J]. Information Fusion, 2024, 106: 102272, articleTitle=null, refAbstract=null), Reference(id=1325493583775662571, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Ma Hongchao, Wang Zhongqing, Zhou Xiabing, et al. Emotion recognition with conversational generation transfer[J]. Transactions on Asian and Low-Resource Language Information Processing, 2022, 21(4): 1−17, articleTitle=null, refAbstract=null), Reference(id=1325493583842771437, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Ren Minjie, Huang Xiangdong, Liu Jing, et al. MALN: Multimodal adversarial learning network for conversational emotion recognition[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(11): 6965−6980, articleTitle=null, refAbstract=null), Reference(id=1325493583909880303, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Abonizio H Q, Paraiso E C, Barbon S. Toward text data augmentation for sentiment analysis[J]. IEEE Transactions on Artificial Intelligence, 2021, 3(5): 657−668, articleTitle=null, refAbstract=null), Reference(id=1325493583976989168, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Gunel B, Du Jingfei, Conneau A, et al. Supervised contrastive learning for pre-trained language model fine-tuning[C]//Proc of the 9th Int Conf on Learning Representations. Washington: ICLR, 2021: 1−15, articleTitle=null, refAbstract=null), Reference(id=1325493584144761329, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Faliang, Li Xuelong, Yuan Changan, et al. Attention-emotion-enhanced convolutional LSTM for sentiment analysis[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 33(9): 4332−4345, articleTitle=null, refAbstract=null), Reference(id=1325493584203481586, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Lu Yi, Mak M W. Improving speech emotion recognition with adversarial data augmentation network[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 33(1): 172−184, articleTitle=null, refAbstract=null), Reference(id=1325493584287367667, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Essa I A. Analysis, interpretation and synthesis of facial expressions[D]. Cambridge, MA: Massachusetts Institute of Technology, 1995, articleTitle=null, refAbstract=null), Reference(id=1325493584362865140, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Hanqing, Song Dawei. Towards contrastive context-aware conversational emotion recognition[J]. IEEE Transactions on Affective Computing, 2022, 13(4): 1879−1891, articleTitle=null, refAbstract=null), Reference(id=1325493584425779701, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Huan Ruohong, Shu Jia, Bao Shenglin, et al. Video multimodal emotion recognition based on Bi-GRU and attention fusion[J]. Multimedia Tools and Applications, 2021, 80(6): 8213−8240, articleTitle=null, refAbstract=null), Reference(id=1325493584492888566, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Jianyou, Xue M, Culhane R, et al. Speech emotion recognition with dual-sequence LSTM architecture[C]//Proc of the 2020 IEEE Int Conf on Acoustics, Speech and Signal Processing. Piscataway, NJ: IEEE, 2020: 6474−6478, articleTitle=null, refAbstract=null), Reference(id=1325493584559997431, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Chao, Bao Zhongtian, Li Linhao, et al. Exploring temporal representations by leveraging attention-based bidirectional LSTM-RNNs for multi-modal emotion recognition[J]. Information Processing & Management, 2020, 57(3): 102185, articleTitle=null, refAbstract=null), Reference(id=1325493584627106296, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Ho N H, Yang H J, Kim S H, et al. Multimodal approach of speech emotion recognition using multi-level multi-head fusion attention-based recurrent neural network[J]. IEEE Access, 2020, 8: 61672−61686, articleTitle=null, refAbstract=null), Reference(id=1325493586317410809, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Shen Xudong, Huang Xianying, Zou Shihao, et al. Multimodal knowledge-enhanced interactive network with mixed contrastive learning for emotion recognition in conversation[J]. Neurocomputing, 2024, 582(C): 127550, articleTitle=null, refAbstract=null), Reference(id=1325493586384519674, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=Fu Changzeng, Qian Fengkui, Su Kaifeng, et al. HiMul-LGG: A hierarchical decision fusion-based local–global graph neural network for multimodal emotion recognition in conversation[J]. Neural Networks, 2025, 181: 106764, articleTitle=null, refAbstract=null), Reference(id=1325493586460017147, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=Hu Jingwen, Liu Yuchen, Zhao Jinming, et al. MMGCN: Multimodal fusion via deep graph convolution network for emotion recognition in conversation[C]//Proc of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int Joint Conference on Natural Language Processing. Stroudsburg, PA: ACL, 2021: 5666−5675, articleTitle=null, refAbstract=null), Reference(id=1325493586535514620, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Jiang, Wang Xiaoping, Lv Guoqing, et al. GA2MIF: Graph and attention based two-stage multi-source information fusion for conversational emotion detection[J]. IEEE Transactions on Affective Computing, 2023, 15(1): 130−143, articleTitle=null, refAbstract=null), Reference(id=1325493586602623485, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=26, authorNames=null, journalName=null, refType=null, unstructuredReference=黄学坚, 马廷淮, 王根生. 基于样本内外协同表示和自适应融合的多模态学习方法[J]. 计算机研究与发展, 2024, 61(5): 1310−1324, articleTitle=null, refAbstract=null), Reference(id=1325493586678120958, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=27, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Xuejian, Ma Tinghuai, Wang Gensheng. Multimodal learning method based on intra- and inter-sample cooperative representation and adaptive fusion[J]. Journal of Computer Research and Development, 2024, 61(5): 1310−1324 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1325493586757812735, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=28, rfOrder=28, authorNames=null, journalName=null, refType=null, unstructuredReference=Xie B, Sidulova M, Park C H. Robust multimodal emotion recognition from conversation with transformer-based crossmodality fusion[J]. Sensors, 2021, 21(14): 4913, articleTitle=null, refAbstract=null), Reference(id=1325493586841698816, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=29, rfOrder=29, authorNames=null, journalName=null, refType=null, unstructuredReference=Wen Huanglu, You Shaodi, Fu Ying. Cross-modal dynamic convolution for multi-modal emotion recognition[J]. Journal of Visual Communication and Image Representation, 2021, 78(Suppl C): 103178, articleTitle=null, refAbstract=null), Reference(id=1325493586908807681, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=30, rfOrder=30, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Jian, Tao Jianhua, Liu Bin, et al. Multimodal transformer fusion for continuous emotion recognition[C]//Proc of the 2020 IEEE Int Conf on Acoustics, Speech and Signal Processing. Piscataway, NJ: IEEE, 2020: 3507−3511, articleTitle=null, refAbstract=null), Reference(id=1325493587089162754, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=31, rfOrder=31, authorNames=null, journalName=null, refType=null, unstructuredReference=Wei Ai, Shou Yuntao, Meng Tao, et al. DER-GCN: Dialog and event relation-aware graph convolutional neural network for multimodal dialog emotion recognition[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024, 36(3): 4908−4921, articleTitle=null, refAbstract=null), Reference(id=1325493587181437443, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=32, rfOrder=32, authorNames=null, journalName=null, refType=null, unstructuredReference=杜金明, 孙媛媛, 林鸿飞, 等. 融入知识图谱和课程学习的对话情绪识别[J]. 计算机研究与发展, 2024, 61(5): 1299−1309, articleTitle=null, refAbstract=null), Reference(id=1325493587252740612, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=32, rfOrder=33, authorNames=null, journalName=null, refType=null, unstructuredReference=Du Jinming, Sun Yuanyuan, Lin Hongfei, et al. Conversational emotion recognition incorporating knowledge graph and curriculum learning[J]. Journal of Computer Research and Development, 2024, 61(5): 1299−1309 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1325493587340820997, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=33, rfOrder=34, authorNames=null, journalName=null, refType=null, unstructuredReference=Kalantidis Y, Sariyildiz M B, Pion N, et al. Hard negative mixing for contrastive learning[C]//Proc of the 34th Int Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2020: 21798−21809, articleTitle=null, refAbstract=null), Reference(id=1325493587529564678, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=34, rfOrder=35, authorNames=null, journalName=null, refType=null, unstructuredReference=Xu Yuan, Yang Meng. MCM-CSD: Multi-Granularity context modeling with contrastive speaker detection for emotion recognition in real-time conversation[C]//Proc of the 2024 IEEE Int Conf on Acoustics, Speech and Signal Processing. Piscataway, NJ: IEEE, 2024: 11956−11960, articleTitle=null, refAbstract=null), Reference(id=1325493587617645063, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=35, rfOrder=36, authorNames=null, journalName=null, refType=null, unstructuredReference=Tu Geng, Xiong Feng, Ling Bin, et al. Multimodal eemotion recognition calibration in conversations[C]//Proc of the 32nd ACM Int Conf on Multimedia. New York: ACM, 2024: 9621−9630, articleTitle=null, refAbstract=null), Reference(id=1325493587923829256, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=36, rfOrder=37, authorNames=null, journalName=null, refType=null, unstructuredReference=Hu Guimin, Lin Tingen, Zhao Yi, et al. UniMSE: Towards unified multimodal sentiment analysis and emotion recognition[C]//Proc of the 2022 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2022: 7837−7851, articleTitle=null, refAbstract=null), Reference(id=1325493588007715337, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=37, rfOrder=38, authorNames=null, journalName=null, refType=null, unstructuredReference=Lu Jiajun, Sibai H, Fabry E. Adversarial examples that fool detectors[J]. arXiv preprint, arXiv: 1712.02494, 2017, articleTitle=null, refAbstract=null), Reference(id=1325493588070629898, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=38, rfOrder=39, authorNames=null, journalName=null, refType=null, unstructuredReference=Szegedy C, Zaremba W, Sutskever I, et al. Intriguing properties of neural networks[J]. arXiv preprint, arXiv: 1312.6199, 2013, articleTitle=null, refAbstract=null), Reference(id=1325493588498448907, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=39, rfOrder=40, authorNames=null, journalName=null, refType=null, unstructuredReference=Goodfellow I J, Shlens J, Szegedy C. Explaining and harnessing adversarial examples[J]. arXiv preprint, arXiv: 1412.6572, 2014, articleTitle=null, refAbstract=null), Reference(id=1325493588582334988, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=40, rfOrder=41, authorNames=null, journalName=null, refType=null, unstructuredReference=Miyato T, Dai A M, Goodfellow I. Adversarial training methods for semisupervised text classification[C]//Proc of the 5th Int Conf on Learning Representations. Washington: ICLR, 2017: 24−26, articleTitle=null, refAbstract=null), Reference(id=1325493588645249549, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=41, rfOrder=42, authorNames=null, journalName=null, refType=null, unstructuredReference=Meng Tao, Shou Yuntao, Ai Wei, et al. Deep imbalanced learning for multimodal emotion recognition in conversations[J]. IEEE Transactions on Artificial Intelligence, 2024, 5(12): 2691−4581, articleTitle=null, refAbstract=null), Reference(id=1325493588783661582, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=42, rfOrder=43, authorNames=null, journalName=null, refType=null, unstructuredReference=Poria S, Hazarika D, Majumder N, et al. MELD: A multimodal multi-party dataset for emotion recognition in conversations[C]//Proc of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2019: 527−536, articleTitle=null, refAbstract=null), Reference(id=1325493588859159055, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=43, rfOrder=44, authorNames=null, journalName=null, refType=null, unstructuredReference=Busso C, Bulut M, Lee C C, et al. IEMOCAP: Interactive emotional dyadic motion capture database[J]. Language Resources and Evaluation, 2008, 42(4): 335−359, articleTitle=null, refAbstract=null), Reference(id=1325493588947239440, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=44, rfOrder=45, authorNames=null, journalName=null, refType=null, unstructuredReference=Zheng Wenjie, Yu Jianfei, Xia Rui, et al. A facial expression-aware multimodal multi-task learning framework for emotion recognition in multi-party conversations[C]//Proc of the 61st Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2023: 15445−15459, articleTitle=null, refAbstract=null), Reference(id=1325493589177926161, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=45, rfOrder=46, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Lichen, Wu Jiaxiang, Huang Shaolun, et al. An efficient approach to informative feature extraction from multimodal data[C]//Proc of the 2019 AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 5281−5288, articleTitle=null, refAbstract=null), Reference(id=1325493590910173715, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=46, rfOrder=47, authorNames=null, journalName=null, refType=null, unstructuredReference=Mądry A, Makelov A, Schmidt L, et al. Towards deep learning models resistant to adversarial attacks[J]. arXiv preprint, arXiv: 1706.06083, 2017, articleTitle=null, refAbstract=null), Reference(id=1325493591002448404, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=47, rfOrder=48, authorNames=null, journalName=null, refType=null, unstructuredReference=Ghosal D, Majumder N, Gelbukh A, et al. COSMIC: Commonsense knowledge for emotion identification in conversations[C]//Proc of the 2020 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2020: 2470–2481, articleTitle=null, refAbstract=null), Reference(id=1325493591291855381, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=48, rfOrder=49, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Zajing, Tang Fengxiao, Zhao Ming, et al. EmoCaps: Emotion capsule based model for conversational emotion recognition[C]//Proc of the 60th Association for Computational Linguistics. Stroudsburg, PA: ACL, 2022: 1610−1618, articleTitle=null, refAbstract=null), Reference(id=1325493591371547158, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=49, rfOrder=50, authorNames=null, journalName=null, refType=null, unstructuredReference=Tu Geng, Xie Tian, Liang Bin, et al. Adaptive graph learning for multimodal conversational emotion detection[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 19089−19097, articleTitle=null, refAbstract=null), Reference(id=1325493591484793367, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=50, rfOrder=51, authorNames=null, journalName=null, refType=null, unstructuredReference=Van der Maaten L, Hinton G. Visualizing data using t-SNE[J]. Journal of Machine Learning Research, 2008, 9(11): 2579−2605, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1325493578042048947, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, xref=null, ext=[AuthorCompanyExt(id=1325493578058826164, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, companyId=1325493578042048947, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=School of Computer and Information Security, Guilin University of Electronic Technology, Guilin, Guangxi 541004), AuthorCompanyExt(id=1325493578067214773, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, companyId=1325493578042048947, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=桂林电子科技大学计算机与信息安全学院 广西桂林 541004)]), AuthorCompany(id=1325493578180460982, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, xref=null, ext=[AuthorCompanyExt(id=1325493578188849591, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, companyId=1325493578180460982, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Guangxi Key Laboratory of Trusted Software (Guilin University of Electronic Technology), Guilin, Guangxi 541004), AuthorCompanyExt(id=1325493578197238200, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, companyId=1325493578180460982, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=广西可信软件重点实验室 (桂林电子科技大学) 广西桂林 541004)])], figs=[ArticleFig(id=1325493579807850959, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=EN, label=Figure 1, caption=The process of constructing discourse graph, figureFileSmall=xrfaM0wv+QR2wGzojpMXdQ==, figureFileBig=odt8GJ5fiafn8hoEJhXJpw==, tableContent=null), ArticleFig(id=1325493579883348432, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=CN, label=图1, caption=构建话语图的过程, figureFileSmall=xrfaM0wv+QR2wGzojpMXdQ==, figureFileBig=odt8GJ5fiafn8hoEJhXJpw==, tableContent=null), ArticleFig(id=1325493580055314897, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=EN, label=Figure 2, caption=Overall architecture of adversarial soft-contrast modulated bi-stream dynamic graph learning, figureFileSmall=L4mnw25LUXa/2Y5WJ9Mjkw==, figureFileBig=brjnWJJk+dCxESy1jobhpQ==, tableContent=null), ArticleFig(id=1325493581716259282, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=CN, label=图2, caption=对抗软对比调制双流动态图学习的总体架构, figureFileSmall=L4mnw25LUXa/2Y5WJ9Mjkw==, figureFileBig=brjnWJJk+dCxESy1jobhpQ==, tableContent=null), ArticleFig(id=1325493581795951059, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=EN, label=Figure 3, caption=Model details of adversarial soft-contrast modulated dynamic graph learning method, figureFileSmall=bO4l5c/+SINqyaqBHcDgtg==, figureFileBig=jCmqbaUIi9osxMDFG8BcvQ==, tableContent=null), ArticleFig(id=1325493581875642836, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=CN, label=图3, caption=对抗软对比调制动态图学习方法的模型细节, figureFileSmall=bO4l5c/+SINqyaqBHcDgtg==, figureFileBig=jCmqbaUIi9osxMDFG8BcvQ==, tableContent=null), ArticleFig(id=1325493581946946005, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=EN, label=Figure 4, caption=t-SNE visualization of multimodal representations of different emotion classes on IEMOCAP dataset, figureFileSmall=u970Xh/JIRDaUmDmE7jsVw==, figureFileBig=vD1PGPYCwE6LYeL1rLL4Qg==, tableContent=null), ArticleFig(id=1325493582030832086, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=CN, label=图4, caption=在IEMOCAP数据集上不同情绪类别的多模态表示的t-SNE可视化, figureFileSmall=u970Xh/JIRDaUmDmE7jsVw==, figureFileBig=vD1PGPYCwE6LYeL1rLL4Qg==, tableContent=null), ArticleFig(id=1325493582110523863, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=EN, label=Figure 5, caption=Confusion matrix on IEMOCAP and MELD test sets, figureFileSmall=ztJHOzJta3UIJly2eFwe8A==, figureFileBig=/0GcDTSYqOU6RcQNiLPCJQ==, tableContent=null), ArticleFig(id=1325493582186021336, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=CN, label=图5, caption=IEMOCAP和MELD测试集上的混淆矩阵, figureFileSmall=ztJHOzJta3UIJly2eFwe8A==, figureFileBig=/0GcDTSYqOU6RcQNiLPCJQ==, tableContent=null), ArticleFig(id=1325493582269907417, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=EN, label=Table 1, caption=Statistics of Datasets, figureFileSmall=null, figureFileBig=null, tableContent=
数据集会话数量话语数量类别会话者人数
训练+验证测试训练+验证测试
IEMOCAP120315 8101 623610
MELD1 15328011 0982 61079
), ArticleFig(id=1325493582366376410, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=CN, label=表1, caption=

数据集的统计

, figureFileSmall=null, figureFileBig=null, tableContent=
数据集会话数量话语数量类别会话者人数
训练+验证测试训练+验证测试
IEMOCAP120315 8101 623610
MELD1 15328011 0982 61079
), ArticleFig(id=1325493582471234011, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=EN, label=Table 2, caption=Performance of ASDG and Baseline Models on IEMOCAP Dataset %, figureFileSmall=null, figureFileBig=null, tableContent=
方法情感类别
喜悦悲伤中性愤怒兴奋沮丧WAAWAF1
ACCF1ACCF1ACCF1ACCF1ACCF1ACCF1
注:黑体数值表示最优结果。
EmoCaps71.9185.0664.4868.9978.4166.7671.77
UniMSE70.5670.66
GraphCFC43.0884.9964.7071.3578.8663.7069.1368.91
GA2MIF46.1584.5068.3870.2975.9966.4969.7570.00
HiMul-LGG53.9579.9271.6667.5672.0068.4670.1270.22
MALN55.481.864.169.178.071.470.8
CBERL58.8467.3463.3172.8456.4260.7575.3273.5170.3270.7778.2171.1969.3669.27
AdaIGN53.0481.4771.2665.8776.3467.7970.74
AGF-IB71.8869.9674.6481.5867.2563.8073.7968.3782.6679.1560.4563.9570.4670.36
DER-GCN60.758.875.979.866.561.571.372.171.173.366.167.869.769.4
ASDG(本文)64.5859.6276.7381.7477.0875.5165.8864.7478.9379.3364.3065.4272.0972.25
), ArticleFig(id=1325493582559314396, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=CN, label=表2, caption=

在IEMOCAP数据集上ASDG和基线模型的性能

, figureFileSmall=null, figureFileBig=null, tableContent=
方法情感类别
喜悦悲伤中性愤怒兴奋沮丧WAAWAF1
ACCF1ACCF1ACCF1ACCF1ACCF1ACCF1
注:黑体数值表示最优结果。
EmoCaps71.9185.0664.4868.9978.4166.7671.77
UniMSE70.5670.66
GraphCFC43.0884.9964.7071.3578.8663.7069.1368.91
GA2MIF46.1584.5068.3870.2975.9966.4969.7570.00
HiMul-LGG53.9579.9271.6667.5672.0068.4670.1270.22
MALN55.481.864.169.178.071.470.8
CBERL58.8467.3463.3172.8456.4260.7575.3273.5170.3270.7778.2171.1969.3669.27
AdaIGN53.0481.4771.2665.8776.3467.7970.74
AGF-IB71.8869.9674.6481.5867.2563.8073.7968.3782.6679.1560.4563.9570.4670.36
DER-GCN60.758.875.979.866.561.571.372.171.173.366.167.869.769.4
ASDG(本文)64.5859.6276.7381.7477.0875.5165.8864.7478.9379.3364.3065.4272.0972.25
), ArticleFig(id=1325493582664171997, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=EN, label=Table 3, caption=Performance of ASDG and Baseline Models on MELD Dataset %, figureFileSmall=null, figureFileBig=null, tableContent=
方法情绪类别
中性惊讶恐惧悲伤喜悦厌恶愤怒WAAWAF1
ACCF1ACCF1ACCF1ACCF1ACCF1ACCF1ACCF1
注:黑体数值表示最优结果。
EmoCaps77.1263.193.0342.5257.507.6957.5464.00
UniMSE65.0965.51
FacialMMT80.1359.6319.1841.9964.8818.1856.0066.58
GraphCFC76.9849.3626.8951.8847.5961.4258.86
GA2MIF76.9249.0827.1851.8748.5261.6558.94
HiMul-LGG66.2165.18
MALN82.058.621.243.064.317.666.9
AdaIGN66.79
AGF-IB81.1081.1956.1657.246.905.0647.4137.3265.9265.922.942.9442.2845.2264.1464.01
DER-GCN76.880.650.551.014.810.456.741.569.364.317.210.352.557.466.866.1
ASDG(本文)82.8880.1155.8759.5822.0023.1639.4243.9764.4365.9032.3529.5353.0452.9767.2466.83
), ArticleFig(id=1325493582756446686, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=CN, label=表3, caption=

在MELD数据集上ASDG和基线模型的性能

, figureFileSmall=null, figureFileBig=null, tableContent=
方法情绪类别
中性惊讶恐惧悲伤喜悦厌恶愤怒WAAWAF1
ACCF1ACCF1ACCF1ACCF1ACCF1ACCF1ACCF1
注:黑体数值表示最优结果。
EmoCaps77.1263.193.0342.5257.507.6957.5464.00
UniMSE65.0965.51
FacialMMT80.1359.6319.1841.9964.8818.1856.0066.58
GraphCFC76.9849.3626.8951.8847.5961.4258.86
GA2MIF76.9249.0827.1851.8748.5261.6558.94
HiMul-LGG66.2165.18
MALN82.058.621.243.064.317.666.9
AdaIGN66.79
AGF-IB81.1081.1956.1657.246.905.0647.4137.3265.9265.922.942.9442.2845.2264.1464.01
DER-GCN76.880.650.551.014.810.456.741.569.364.317.210.352.557.466.866.1
ASDG(本文)82.8880.1155.8759.5822.0023.1639.4243.9764.4365.9032.3529.5353.0452.9767.2466.83
), ArticleFig(id=1325493582857109983, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=EN, label=Table 4, caption=Results of Ablation Studies on IEMOCAP and MELD %, figureFileSmall=null, figureFileBig=null, tableContent=
方法IEMOCAPMELD
WAAWAF1WAAWAF1
ASDG(本文)72.0972.2567.2466.83
ASDG-
based model
-w/o CCGL68.7069.1365.9465.69
-w/o CFDG72.0272.0366.5566.23
-w/o DG70.6770.5565.2965.56
-w/o PE70.3670.3566.7466.54
-w/o SE71.171.3466.3265.69
训练机制-w/o ASCL69.9970.0866.2165.58
-w/o AT69.7569.7766.5565.45
-w/o SHGR71.9071.9666.3965.94
Modality文本68.4468.4765.2564.96
音频59.0957.8446.0234.95
视频38.2030.8144.3332.31
文本 + 音频70.4370.5765.8265.45
文本 + 视频68.5268.8366.7465.89
音频 + 视频62.8562.2347.3932.46
), ArticleFig(id=1325493582945190368, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, language=CN, label=表4, caption=

在IEMOCAP和MELD上的消融实验结果

, figureFileSmall=null, figureFileBig=null, tableContent=
方法IEMOCAPMELD
WAAWAF1WAAWAF1
ASDG(本文)72.0972.2567.2466.83
ASDG-
based model
-w/o CCGL68.7069.1365.9465.69
-w/o CFDG72.0272.0366.5566.23
-w/o DG70.6770.5565.2965.56
-w/o PE70.3670.3566.7466.54
-w/o SE71.171.3466.3265.69
训练机制-w/o ASCL69.9970.0866.2165.58
-w/o AT69.7569.7766.5565.45
-w/o SHGR71.9071.9666.3965.94
Modality文本68.4468.4765.2564.96
音频59.0957.8446.0234.95
视频38.2030.8144.3332.31
文本 + 音频70.4370.5765.8265.45
文本 + 视频68.5268.8366.7465.89
音频 + 视频62.8562.2347.3932.46
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202440896, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202440896, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202440896, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202440896, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1769875200000, fullTextJson=null, articleText=null, reference=null)
收藏切换
对抗软对比调制动态图学习的多模态会话情绪识别方法
王顺杰 , 蔡国永 * , 吕光瑞
计算机研究与发展 | 人工智能 2026,63(2): 305-323
收起
收藏切换
计算机研究与发展 |人工智能 2026 , 63 (2) : 305 -323
对抗软对比调制动态图学习的多模态会话情绪识别方法
全屏
[Author(id=1325493578285318586, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=wsj12130@foxmail.com, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325493578377593275, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, authorId=1325493578285318586, language=EN, stringName=Shunjie Wang, firstName=Shunjie, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"content":"

born in 1996. PhD candidate. Her main research interests include deep learning and multi-modal affective computing

"}, bioImg=null, bioContent=

born in 1996. PhD candidate. Her main research interests include deep learning and multi-modal affective computing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325493578474062268, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, authorId=1325493578285318586, language=CN, stringName=王顺杰, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"img":"EfDEKRK25FOsN4sGYbYblg==","content":"

1996年生。博士研究生。主要研究方向为深度学习、多模态情感计算

"}, bioImg=EfDEKRK25FOsN4sGYbYblg==, bioContent=

1996年生。博士研究生。主要研究方向为深度学习、多模态情感计算

, aboutCorrespAuthor=null)}, companyList=null), Author(id=1325493578557948350, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=ccgycai@guet.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1325493578641834431, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, authorId=1325493578557948350, language=EN, stringName=Guoyong Cai, firstName=Guoyong, middleName=null, lastName=Cai, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=null, bio={"content":"

born in 1971, PhD, professor, PhD supervisor. Distinguished member of CCF. His main research interests include multi-modal affective computing, trustable AI theory and techniques

"}, bioImg=null, bioContent=

born in 1971, PhD, professor, PhD supervisor. Distinguished member of CCF. His main research interests include multi-modal affective computing, trustable AI theory and techniques

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325493578729914816, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, authorId=1325493578557948350, language=CN, stringName=蔡国永, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=null, bio={"img":"kfxwXQSG7jspWV5+TgsbBg==","content":"

1971年生。博士,教授,博士生导师。CCF杰出会员。主要研究方向为多模态情感计算、可信AI理论技术

"}, bioImg=kfxwXQSG7jspWV5+TgsbBg==, bioContent=

1971年生。博士,教授,博士生导师。CCF杰出会员。主要研究方向为多模态情感计算、可信AI理论技术

, aboutCorrespAuthor=null)}, companyList=null), Author(id=1325493578805412290, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=lvguangrui_rio@163.com, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325493578876715459, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, authorId=1325493578805412290, language=EN, stringName=Guangrui Lü, firstName=Guangrui, middleName=null, lastName=Lü, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"content":"

born in 1989. PhD, master supervisor. Member of CCF. His main research interests include computer vision and multi-modal information processing

"}, bioImg=null, bioContent=

born in 1989. PhD, master supervisor. Member of CCF. His main research interests include computer vision and multi-modal information processing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325493578948018628, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493573717721504, authorId=1325493578805412290, language=CN, stringName=吕光瑞, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"img":"BNzvNSDFdB9/5VhgsCuQWA==","content":"

1989年生。博士,硕士生导师。CCF会员。主要研究方向为计算机视觉、多模态信息处理

"}, bioImg=BNzvNSDFdB9/5VhgsCuQWA==, bioContent=

1989年生。博士,硕士生导师。CCF会员。主要研究方向为计算机视觉、多模态信息处理

, aboutCorrespAuthor=null)}, companyList=null)]
王顺杰 , 蔡国永* , 吕光瑞
作者信息
  • 桂林电子科技大学计算机与信息安全学院 广西桂林 541004
  • 广西可信软件重点实验室 (桂林电子科技大学) 广西桂林 541004
通讯作者:
蔡国永()
作者简介:

1996年生。博士研究生。主要研究方向为深度学习、多模态情感计算

1971年生。博士,教授,博士生导师。CCF杰出会员。主要研究方向为多模态情感计算、可信AI理论技术

1989年生。博士,硕士生导师。CCF会员。主要研究方向为计算机视觉、多模态信息处理

Adversarial Soft-Contrast Modulated Dynamic Graph Learning for Multimodal Emotion Recognition in Conversations
Shunjie Wang , Guoyong Cai* , Guangrui Lü
Affiliations
  • School of Computer and Information Security, Guilin University of Electronic Technology, Guilin, Guangxi 541004
  • Guangxi Key Laboratory of Trusted Software (Guilin University of Electronic Technology), Guilin, Guangxi 541004
  • born in 1996. PhD candidate. Her main research interests include deep learning and multi-modal affective computing

    born in 1971, PhD, professor, PhD supervisor. Distinguished member of CCF. His main research interests include multi-modal affective computing, trustable AI theory and techniques

    born in 1989. PhD, master supervisor. Member of CCF. His main research interests include computer vision and multi-modal information processing

出版时间: 2026-02-01 doi: 10.7544/issn1000-1239.202440896
文章导航
收藏切换

多模态会话情绪识别(multimodal emotion recognition in conversations,MERC)已成为人机智能交互的研究热点,目前被广泛应用于情感对话机器人及对话推荐等多个场景。在多模态会话场景中识别抽象的情感语义是困难的,大多现有研究利用对比学习来提取判别特征,其尽管约束了类内特征的一致性,但损害了精细的多样性表征,导致模型泛化性降低,尤其不利于少类样本的学习。此外,目前的会话上下文学习多建模固定窗口的语境依赖,忽略了会话信息流的动态互相关。为了解决上述限制,提出对抗软对比调制动态图学习(adversarial soft-contrast modulated dynamic graph learning,ASDG)方法。具体地:首先根据会话场景中说话人的话语数量构建话语图实现在各自模态内动态地建模话语的会话依赖范围,以精准地提取丰富的语境信息;其次,设计对抗性软对比训练机制,通过在不同模态特征提取器的隐藏层中添加扰动生成对抗样本来扩展类样本空间,并使用软对比学习在原始样本和对抗样本上最大化类扩展样本的标签语义一致性以增强模型学习的判别性与鲁棒性;最后,构建针对不同模态的双流图学习策略,以协同地促进各模态会话数据的互补融合。在IEMOCAP和MELD的多模态会话基准数据集上进行广泛的实验,结果表明提出的方法与目前先进的方法相比,在MERC任务上取得了具有竞争力的效果。

多模态会话情绪识别  /  动态话语图  /  对抗性软对比  /  双流图学习  /  语义一致性

Multimodal emotion recognition in conversations (MERC) has emerged as a prominent research focus in the domain of human-computer intelligent interaction. It demonstrates extensive applicability across diverse scenarios, including affective dialogue systems and conversational recommendation systems, thereby substantially improving user’s experience through enhanced emotional engagement. Identifying abstract emotional semantics in multimodal conversation scenarios poses a significant challenge. Current research predominantly relies on contrastive learning frameworks to extract discriminative feature. Although these approaches effectively enforce intra-class feature consistency, they inherently constrain fine-grained feature diversity. This limitation consequently impairs the model’s generalization capacity, particularly when processing samples from minority classes. In addition, current dialogue context graph learning only captures the dependence with a fixed window, neglecting the dynamic cross-correlation of the conversation flow, which can lead to either redundancy or insufficient of extracted context information. To address the aforementioned limitations, the adversarial soft-contrast modulated dynamic graph learning is proposed for MERC. Specifically, a dynamic discourse graph is firstly constructed according to the number of utterances emitted by the speaker in the conversation, which allows for dynamically modeling the conversational dependency range of the utterance in each modality to extract rich contextual information more precisely. Secondly, the adversarial soft-contrast training mechanism is designed to enhance the discrimination and robustness of the network learning. Among them, the class sample space is expanded by adding perturbations to the hidden layers of different modal feature extractors to generate adversarial samples, and the soft-contrast learning is exploited to maximize label semantic consistency of the original samples and the adversarial samples. Finally, a bi-stream graph learning strategy for different modalities is constructed, which not only explores the cross-modal consistent feature interaction guided by text semantics, but also learns the context dynamic dependencies within multimodal dialogue flow to collaboratively facilitates the complementary fusion of multimodal conversations data. Extensive experiments conducted on the IEMOCAP and MELD benchmark datasets have demonstrated that our proposed model achieves competitive results in terms of effectiveness for MERC, compared with current state-of-the-art methods.

multimodal emotion recognition in conversations (MERC)  /  dynamic discourse graph  /  adversarial soft-contrast  /  bi-stream graph learning  /  semantic consistency
王顺杰, 蔡国永, 吕光瑞. 对抗软对比调制动态图学习的多模态会话情绪识别方法. 计算机研究与发展, 2026 , 63 (2) : 305 -323 . DOI: 10.7544/issn1000-1239.202440896
Shunjie Wang, Guoyong Cai, Guangrui Lü. Adversarial Soft-Contrast Modulated Dynamic Graph Learning for Multimodal Emotion Recognition in Conversations[J]. Journal of Computer Research and Development, 2026 , 63 (2) : 305 -323 . DOI: 10.7544/issn1000-1239.202440896
随着智能设备的发展以及会话应用场景的增加,增强机器对人类情感的感知和识别能力使得对话系统能够产生连贯且共情响应,进而提高人机交互的舒适感。近年来,多模态会话情绪识别(multimodal emotion recognition in conversations,MERC)在人工智能领域引起了越来越多研究者的兴趣[1],旨在识别说话人在文本、视频和音频等多模态信息下的情绪及衍变。目前,MERC的应用正扩展到对话生成[2]、推荐系统[3]、意见挖掘[4]、医疗诊断[5]等领域。尽管MERC取得了一定的进展,但会话情景中的多模态数据相互关联,情感表达的隐晦性更强,使得MERC具有极大的挑战性。首先,由于情感是一种语用级的抽象表达,从数据中提取与情感标签相关的泛化表示是困难的[6],且会话情绪识别涉及不同会话者,使得在会话情绪识别任务中提取判别且泛化的特征更加严峻;其次,由于不同的话语所依赖的会话语境范围是有差异的,然而较多研究设计固定会话窗口的规则来学习上下文依赖关系,其不能随着话语的改变而动态调整,导致获取的上下文信息缺失或者冗余[7-9];此外,会话情境中不同的模态通常表达相同的情感,但有时由于情感的刻意隐藏而使会话中异构模态的表达是矛盾的,因而如何有效地建模会话场景中异构模态的语义关联是必要且挑战的[10]。
泛化的模态表示学习对于提取稳健的判别性特征是至关重要的。Ma等人[11]使用迁移学习来提高模型的泛化性,通过在相同的数据集中训练会话生成模型,并将生成模型的中间层参数迁移到会话情感判断任务,但该策略可能会过度适应特定的任务,从而影响模型的稳定性。Ren等人[12]利用生成对抗网络来学习数据的泛化表示,该方法可以缓解模态间的异构问题,但会使得模态间的判别性降低,不利于挖掘模态间的互补性。Abonizio等人[13]通过对原始数据添加噪声的方式生成对抗样本以学习更加鲁棒的特征表示,但是在会话数据中添加噪声会破坏会话语义的连贯性。此外,结合交叉熵损失的软对比学习能够捕获类内样例的相似性,并将其与不同类的样例进行对比以学习特征的泛化表示[14],但是由于相同的情感具有相似的上下文和重叠的特征空间,直接压缩每个类的特征空间可能会损坏类内的细粒度特征,并且大大降低对少样本的分类能力,从而限制了模型的泛化性。因此,为了既保护会话上下文的语义连贯性又提高模态特征的泛化性,本文提出对抗性软对比训练机制。具体地,通过将噪声施加在特征提取的隐藏层以生成对抗样例,不仅能避免破坏会话上下文的语义关联,而且可扩展每个模态的类表示空间,并在迭代学习中联合原始样本和对抗样本进行软对比优化以最大化模态内相同标签样本的一致性,从而尽可能地保留细粒度的类内特征,以达到更好的泛化效果。
在人际交谈中,情绪随语境流动,并通过语言表述、音频特征(语调、语速等)及面部表情等多模态方式而体现[15-17],因此针对会话中多模态数据的上下文建模也是一个关键挑战[18]。起初,针对多模态会话上下文建模的研究多通过构建循环神经网络来实现[19-22],尽管这些工作取得了令人鼓舞的成果,但这些方法不利于远距离的会话上下文依赖关系的学习。目前,研究者更多关注利用图神经网络(graph neural network,GNN)来建模多模态会话的远程上下文依赖性[6-9,23-25],这是由于图结构能够依据边关系从局部到全局逐步地传播图中任意关联节点间的信息。例如Shou等人[6]和Hu等人[25]通过构建密集连接的图结构来自适应学习上下文关联,尽管构建了会话中任一节点与其他所有节点间的连接,但是容易挖掘到冗余的特征。此外,一些方法[8-9]通过设置超参数来构建固定的窗口以建模会话的语境信息,然而这也不太适合会话的动态特性,同样限制了从会话场景中自适应地获取精确上下文信息的能力。为此,本文提出能够精确建模上下文的动态话语图构建机制,如图1所示,使得当前话语能够根据会话中的讲话者密度来动态调整窗口,不仅能够关注话语较少的说话者所表达的内容,且不会对话语较多的说话者产生过多的关注,以此获得充足的语境信息,且能有效避免上下文信息的冗余。
除了动态建模会话场景中任一话语的各自模态上下文涵盖范围,不同模态的数据交互融合对MERC而言也是至关重要的。一些研究基于GNN能够捕捉节点间复杂关系结构的优势,实现MERC中的多模态交互融合。如Hu等人[25]构建了一个大范围多模态密集图来学习多模态交互和模态内的上下文信息,节点表示不同的模态特征,则每个节点的邻居节点同质或异质,尽管实现了不同模态数据的信息融合,但忽略了不同模态数据中存在的异质性。为了缓解不同模态的异构性,Li等人[8-9]构造了3个异构图,每个异构图仅融合2种模态,但当中心节点与邻居节点是异质模态时,在信息传播过程中异构信息仍然影响不同模态信息的有效交互。此外,Li等人[26]使用跨模态注意机制实现多模态密集交互,但其不能精确界定多模态话语级的上下文依赖范围,这会导致用于情绪判断的多模态特征的冗余。为了缓解以上问题,本文将首先构建动态话语图来准确建模各自模态内的语境依赖范围,然后设计文本语义引导的一致性跨模态图学习以降低不同模态信息的异质性并实现模态间信息的高效传递与融合,同时,构建会话流动态图学习来更精确地挖掘多模态话语级上下文依赖范围以进一步减轻多模态特征的冗余。
综上,本文提出对抗软对比调制动态图学习(adversarial soft-contrast modulated dynamic graph learning,ASDG)方法用于多模态会话情绪识别,所设计的总体结构如图2所示。为了提高会话语境依赖的上下文学习的泛化性,设计对抗训练策略自适应地生成对抗样本,通过将扰动$\boldsymbol r $作用在模态特征提取器的隐藏层,不仅能避免直接生成对抗样例干扰话语之间的相关性,还能丰富有限的会话数据样本,使得模型能够提取更多样化的特征。尤其,在原始样本和对抗样本上分别进行软对比学习,使得模型能够学习到更鲁棒的情感特征。同时,为了动态建模各自模态的上下文依赖信息和模态间的交互信息,依次设计单模态的会话上下文动态图学习和互补双流图学习。单模态的会话上下文动态图学习根据会话中的讲话者密度分别构建3个模态的话语图,精确建模会话中每个话语的语境依赖范围,实现模态内上下文语义信息的有效传递。而在互补双流图学习中,文本语义引导的一致性跨模态图学习用于实现文本语义为中心约束的跨模态特征交互融合,以缓解异质模态特征的鸿沟;而会话流动态图学习仍基于动态话语图的策略建模多模态话语级的动态上下文依赖,以减轻多模态特征的冗余,然后聚合双流图学习分支用于MERC任务。在2个公共基准数据集上进行实验,实验结果一致表明本文所设计的方法优于其他基线方法。
本文的主要贡献概括如下:
1)提出对抗性软对比训练机制。在模型迭代优化时从上下文依赖的会话中生成每个模态特征的对抗样例,并在原始样例和对抗样例上利用软对比学习最大化标签一致性以丰富多样化的类特征,从而增强模型的鲁棒性。
2)提出单模态的会话上下文动态图学习策略。会话场景中每个话语根据其所在的位置以及讲话者数量计算其话语密度,动态地调整话语所关注的前向与后向的窗口大小,从而构建不同模态的动态话语图,以准确地提取会话上下文的语义依赖。
3)提出互补双流图学习机制。文本语义引导的一致性跨模态图学习通过在文本语义主导的约束下挖掘文本-音频跨模态交互和文本-视频跨模态交互,而会话流动态图学习自适应地建模多模态话语级的上下文依赖,两者协同地促进多模态特征的有机融合。
多模态数据的上下文建模是MERC任务的一个关键挑战,通常当前话语的语境对情绪感知至关重要。最初的方法多使用基于循环神经网络(recurrent neural network,RNN)的模型来建模远程上下文依赖用于提取整体语义信息。随着Transformer结构的出现,一些研究者开始利用其强大的全局上下文信息建模能力以实现模态内或模态间的特征融合。此外,由于图结构具有复杂的关系建模能力,使得研究者也多关注通过GNN来构建多模态的相互依赖关系。
在基于RNN的多模态会话情绪识别研究中,Huan等人[19]提出了一种基于双向门控循环单元和注意力融合的视频多模态情绪识别方法。Wang等人[20]设计了一个基于长短期记忆网络(long short-term memory,LSTM)的双层模型来处理从音频中提取的每个话语的梅尔频率倒谱数据以建模时序上的语境依赖关系。Li等人[21]提出了基于注意力的多模态双向LSTM网络,该方法将自注意力机制与LSTM相结合自适应地学习上下文依赖的多模态特征。Ho等人[22]提出了一种结合多头注意力机制的RNN多层结构来分别建模语音和文本的上下文依赖用于多模态语音情绪识别。尽管基于RNN的相关方法在上下文语义建模的情绪识别任务中取得了较好的效果,但是其对远程上下文信息建模能力不足,且当会话的轮次较多时,此类型的网络结构难以满足会话情绪分析的需求,因此,最近的一些研究工作通常将循环神经网络(如LSTM)作为初步的上下文依赖特征提取器。
多模态的表示和融合是多模态机器学习的关键任务[27]。最近,许多研究利用Transformer结构来建模远程上下文交互以及用于多模态会话情绪识别任务,如Xie等人[28]利用Transformer结构实现了以各自模态为中心的跨模态交互。Wen等人[29]也利用Transformer结构实现了模态间的两两双向交互,并进一步使用Transformer来两两聚合跨模态交互的特征用于情绪识别。Huang等人[30]首先利用Transformer结构分别获得各自模态的长期依赖关系,然后构建跨模态Transformer结构来实现模态间的融合。Ren等人[12]通过使用传统的Transformer和跨模态注意层来交换不同模态之间的信息以捕获模态间的互补信息。综上,虽然Transformer结构已被广泛用于模态内的上下文依赖学习和模态间的跨模态特征交互以提取丰富的语义信息,但是仅仅使用这些密集的Transformer结构,或不可避免地导致模态特征的冗余。
目前,研究者多关注基于GNN的多模态会话情绪识别研究,如Hu等人[25]基于相同模态内节点两两互连、同一话语的不同模态节点也相连的原则,构建了一个音频、视频、文本3模态的密集连接图,尽管能充分挖掘模态内与模态间的信息,但这会导致模型学习到冗余的语境信息,且忽略了多模态融合中的异构问题。随后,Li等人[9]基于上述构图原则构建了3个异质图,每个图只考虑2种模态的信息,通过减少图中的异构节点,一定程度上缓解了异构模态间的差异。与此同时,Li等人[8]也使用相同构图方式构建了一个异质图和一个同质图,但却采用固定的超参数来控制上下文窗口的大小,且在构图过程中定义了讲话者内和讲话者间以及模态内和模态间的边的概念,一定程度上减少了冗余特征的学习。此外,Li等人[26]提出了一种2阶段的多模态融合方法,首先为每个模态构建密集连接的图结构来提取模态内的上下文信息,并使用固定的超参数设定上下文窗口的依赖范围,然后利用多头跨模态注意网络实现不同模态的交互和融合。Fu等人[24]提出了一个基于层次化融合的全局-局部图融合网络,以讲话者关系(讲话者内和讲话者间)构建了3个双向的单模态图,在每个单模态图中,采用固定窗口的图网络来渐进地学习模态内的上下文依赖。而Wei等人[31]基于事件关系构建了图结构以建模在不同的事件关系下多模态话语级特征之间的关联,并引入了自监督掩码的自编码器来提高图节点特征和结构的鲁棒性。2024年,Shen等人[23]提出了一个知识增强的多模态网络,基于讲话者的互动关系构建图,并挖掘话语间的常识知识来丰富文本模态的上下文表示,并使用Transformer结构来实现跨模态交互,但其仅建立了一个针对文本的知识聚合图来捕获话语之间的常识知识依赖,却忽略了对视频和音频的上下文依赖建模。显然,以上研究均通过构建不同的图结构来实现模态内的上下文依赖或模态间的特征交互,但这些方法在构建图结构时往往忽略了会话的动态性,多通过设计基于任意节点或固定前后向窗口的密集连接来实现话语级的上下文关联聚合,导致当前话语关联的上下文信息出现缺失或冗余。此外,一些方法还忽略了语音、视频模态内的上下文依赖挖掘,显然不利于实现多模态信息的有效交互与融合。然而会话是动态的且每个话语所依赖的上下文范围是不同的,则如何自适应地学习图关系以及有效挖掘模态内与模态间的关系是必要的。本文将构建同质的动态图来自适应学习模态内以及话语级多模态特征的上下文依赖,且构建双流图模块来缓解模态间的异构性。
在会话情绪识别任务中,会话所表达出的情绪语义较为相似[32],而对比学习能够通过不断缩小正样本之间以及扩大负样本之间的差异来学习不同类别之间可区分的类别边界信息,是一种探索判别嵌入的表示学习技术[33]。为了缓解数据分布不平衡的问题,Wei等人[31]采用欧氏距离将话语的正类样本和负类样本进行对比学习,缩小正样例与该话语的差距,并扩大负样例与该话语的差距,以学习更具判别性的类边界信息。为了考虑会话者自身的历史影响,Xu等人[34]利用余弦相似度将与该话语是同一讲话者的会话内容与其他话语进行对比以区分当前说话人与之前说话人是否为同一人。Tu等人[35]利用余弦相似度将该话语的伪标签正负样本进行对比学习以捕获话语间的细微相关性和判别性。Shou等人[6]利用互信息将该样例的正负类样例进行对比学习以获得更具判别性的嵌入表示。Hu等人[36]通过乘积操作将多模态融合特征与各个模态进行对比,以增强模态间的相互影响,并放大样本与融合表征的差异。这些对比学习方法通过利用标签信息将属于同一类样例的特征聚集,而不属于同一类的远离。但是这类方法往往通过直接压缩类内特征空间,不可避免地损坏细粒度的类内特征,从而降低模型的泛化能力。另外,通常仅使用分类交叉熵损失函数用于MERC[8]来学习模态的判别性,难以有效应对数据中的类别不平衡的问题。为了缓解上述问题,本文将利用由对比损失和交叉熵损失构成的软对比损失来学习更具判别性的模态表示,并结合对抗学习策略来提高模型的泛化能力。
泛化的多模态表示一直是MERC任务中的一个严峻挑战。对抗训练是一种广泛使用的正则化方法,往往通过使用最小-最大的博弈训练来提高模型的鲁棒性[37]。例如,Szegedy等人[38]通过在原始样例上添加扰动生成对抗样例,并在对抗样例和原始样例的混合样例上以对抗的方式训练神经网络。为了缓解一些神经网络通常错误地分类扰动样例,Goodfellow等人[39]进一步提出了一种快速梯度符号方法,该方法基于模型的梯度信息生成对抗样本。而Miyato等人[40]将对抗性和虚拟对抗性训练扩展到文本域,主要在词嵌入层上应用扰动。生成对抗网络也被引入多模态会话情绪识别任务中,如Ren等人[12]为每个模态设置一个编码器和判别器,所有模态共享1个生成器,在每个模态上进行对抗训练来挖掘多模态信息的共性和多样性。而Meng等人[41]提出了一个类边界增强表示学习方法,首先在各个模态上加噪生成对抗样例,然后利用生成器和反向生成器间的一致性损失来确保每个模态的对抗样例和原始样例的一致性,但对模态的异构性考虑不足。为了缓解模态的异构问题,Shou等人[6]为3个模态分别构建了生成器和判别器,其中每个模态的生成器输入是其他2种模态,而对应模态的判别器输入是这3种模态的特征,以此通过对抗训练来缓解不同模态特征之间的数据分布差异,进而增强模型的鲁棒性。显然,扰动对抗和生成对抗是从不同的角度来进行对抗训练以提高模型的鲁棒性,但考虑到会话上下文的连贯属性,虽然生成对抗网络缓解了不同模态的异构性,但是会损害会话的连贯性。因此,本文将通过采用扰动对抗来提高模型的泛化,以在扰动对抗学习过程中确保会话的连贯性,具体地,通过将扰动应用于单模态特征提取器的隐层中以生成对抗样本,并在原始样本和对抗样本上联合使用软对比损失来有效地学习类扩展表征和标签一致性特征,以提高模型的泛化能力。
形式上,一个会话U是由N个连续的语句$ \{ {u_{\text{1}}}, {u_{\text{2}}},…,{u_N}\} $和S个说话人$ \{ {p_{\text{1}}},{p_{\text{2}}},…,{p_S}\} $组成。每个话语$ {u_i} $由参与方$ {p_{\phi ({u_i})}} \in \{ {p_{\text{1}}},{p_{\text{2}}},…,{p_S}\} $说出,其中ϕ是将语句映射到对应的说话人。此外,$ {u_i} $涉及文本(T)、音频(A)和视频(V)模态,其特征表示分别为$ {\boldsymbol{u}}_i^{\text{T}} \in {\mathbb{R}^{{d_{\text{T}}}}} $,$ {\boldsymbol{u}}_i^{\text{A}} \in {\mathbb{R}^{{d_{\rm A}}}} $和$ {\boldsymbol{u}}_i^{\text{V}} \in {\mathbb{R}^{{d_{\text{V}}}}} $。会话中所有话语的文本模态、音频模态和视频模态的序列表示为$ {{\boldsymbol U}_{\text{T}}} = ({\boldsymbol u}_{\text{1}}^{\text{T}},{\boldsymbol u}_{\text{2}}^{\text{T}},…,{\boldsymbol u}_N^{\text{T}}) \in {\mathbb{R}^{N \times {d_{\text{T}}}}} $,$ {{\boldsymbol U}_{\text{A}}} = ({\boldsymbol u}_{\text{1}}^{\text{A}},{\boldsymbol u}_{\text{2}}^{\text{A}},…,{\boldsymbol u}_N^{\text{A}}) \in {\mathbb{R}^{N \times {d_{\text{A}}}}} $且$ {{\boldsymbol U}_{\text{V}}} = ({\boldsymbol u}_1^{\text{V}},{\boldsymbol u}_2^{\text{V}},…,{\boldsymbol u}_N^{\text{V}}) \in {\mathbb{R}^{N \times {d_{\text{V}}}}} $。给定一个话语,MERC旨在基于该话语的多模态表示$ \{ {\boldsymbol u}_i^{\text{T}},{\boldsymbol u}_i^{\text{A}},{\boldsymbol u}_i^{\text{V}}\} $以及会话上下文预测每个话语的情感标签$ {y_i} $。通常,每个话语代表一种情绪,由于不同数据集的会话主题不一致,其包含的情感丰富度也不同,因此,不同的数据集具有不同的情绪类别,MELD数据集[42]包含7种情绪标签,IEMOCAP数据集[43]包含6种情绪标签。
图3给出了本文提出的ASDG方法的模型细节展示。具体地,基于预训练模型提取话语级的单模态特征表示后,利用双向LSTM来初步建模有关文本、音频和视频的会话上下文依赖关系以及讲话者依赖关系,然后同时构建文本话语图、语音话语图以及视频话语图来自适应地建模文本、音频、视频的模态内动态依赖,且设计文本语义引导的一致性约束的跨模态图学习来探索不同模态间特征的有机融合,主要利用Transformer结构实现文本-音频、文本-视频的跨模态交互;同时设计会话流动态图学习来发掘多模态话语级语境的动态依赖,具体使用跨模态注意力机制来分别实现文本对语音的引导以及文本对视频的引导,并进一步构建话语图来实现信息的传播,最后集成双流图的输出用于会话情绪判断。
此外,针对模型的优化,提出使用软对比训练策略来学习相同类别的一致性以及不同类别的判别性,并将其作用在3个模态的话语图输出上以及互补双流图的输出上。另外,利用对抗训练策略生成每个话语的对抗样本以扩大类空间表示,在对抗样本上也使用软对比学习来进行训练,并使对抗样本的模型训练和原始样本保持一致,这种联合原始样本和对抗样本进行软对比学习使得类内特征更具判别性,从而提高模型的泛化性和鲁棒性。接下来,将详细阐述ASDG模型的细节。
为了捕获会话的语境感知依赖信息和说话者感知依赖信息,以及处理维度不一致的模态数据,使用双重双向LSTM对文本、音频和视频数据分别进行会话上下文特征编码和讲话者依赖特征编码。以文本为例来阐述特征编码的过程。
在会话中讲话者的情感状态不仅受到其他讲话者的影响,还受自身影响。为了建模说话人自身的上下文依赖,首先分别抽取各个讲话者在会话中说的话语,然后使用双向LSTM来学习说话人感知特征。但是如果说话人的话语数小于预定义的整数阈值$ \xi $,这些说话人的特征直接由共享的通用说话人向量o表示。因此,说话人感知特征$ {\boldsymbol{c}}_i^{{\text{sp}}} $的计算过程如下:
$ {\boldsymbol{c}}_i^{{\text{sp}}},{\boldsymbol{h}}_{s,j}^{{\text{sp}}} = \left\{ \begin{gathered}{\boldsymbol{o}},\;\;{\text{无, }}\;\;{\rm{if}}\; | {{\boldsymbol{U}}_s} | < \xi \; , \\{\overleftrightarrow {LSTM}_{{\text{sp}}}}({{\boldsymbol{u}}_i},{\boldsymbol{h}}_{s,j-1}^{{\text{sp}}}),j \in [1,\left| {{{\boldsymbol{U}}_s}} \right|],\;\;{ { 其他}} \; , \\ \end{gathered} \right.$
其中$ {\overleftrightarrow {LSTM}_{{\text{sp}}}} $表示通过双向LSTM获取说话人感知依赖特征,$ {\boldsymbol{h}}_{s,j}^{{\text{sp}}} $是说话人$ {p_s} $发出的第j个话语的隐状态。$ s = \phi ({{\boldsymbol{u}}_i}) $是讲话者身份。$ {{\boldsymbol{U}}_s} $指一个会话中$ {p_s} $说的所有话语。语境感知特征$ {\boldsymbol{c}}_i^{{\text{co}}} $的计算过程为
$ {\boldsymbol{c}}_i^{{\text{co}}},{\boldsymbol{h}}_i^{{\text{co}}} = {\overleftrightarrow {LSTM}_{{\text{co}}}}({{\boldsymbol{u}}_i},{\boldsymbol{h}}_{i-1}^{{\text{co}}}) \; , $
其中$ {\overleftrightarrow {LSTM}_{{\text{co}}}} $表示通过双向LSTM获取语境感知特征,$ {\boldsymbol{h}}_i^{{\text{co}}} $是话语$ {{\boldsymbol{u}}_i} $的隐状态。
然后将说话人感知特征和语境感知特征进行拼接以形成话语的文本上下文表示$ {\boldsymbol c}_i^{\text{T}} = [{\boldsymbol c}_i^{{\text{sp}}};{\boldsymbol c}_i^{{\text{co}}}] $。如此,针对音频和视频模态经过同样操作,以生成音频的会话上下文表示$ {\boldsymbol c}_i^{\text{A}} $以及视频的会话上下文表示$ {\boldsymbol c}_i^{\text{V}} $。
为了提高ASDG的泛化能力,通过加噪的方式生成对抗样例来扩展每个类的表示空间以缓解类分布不平衡的问题,这尤其对少类样例的训练是很有必要的。但是对于会话上下文依赖的场景,直接在样本上添加扰动噪声会很大程度上摧毁不同话语之间的连贯性,因此将扰动噪声作用于经过原始样本训练后的双向LSTM的隐藏层,从而借助上下文依赖的隐层输出以得到平滑的表示空间。特别强调的是,原始样本和对抗样本都需经过ASDG来学习,以联合的形式有效地学习类内的一致性特征,并通过软对比学习保留细粒度的精细类内特征,以达到更有效的泛化效果,具体见2.4节。
当前基于GNN的会话情绪识别研究往往以任一节点为中心构建密集连接的图[6-25]或设置固定超参数窗口大小的图[7-9],然而这种固定形式的构图方式会导致获取的上下文语境信息冗余或者不足,且难以适应动态的会话场景。为了精准地获取任一话语关联的上下文信息并模拟实际会话中上下文的相互影响,设计一个动态话语图来实现各自模态话语间的情感信息扩散,使得每个话语可以动态地学习上下文依赖范围。单模态的会话上下文动态图学习包含文本话语图、音频话语图、视频话语图。由于每个话语图都有一一对应的3个模态信息,因此该3个话语图的图结构是一致的。下面以文本话语图的构建过程来展开阐述。
在话语图的构建过程中,话语之间的边是根据话语人的话语密度来动态建立的,其定义是说话人所说的话语数占整个话语总数的比例。根据该规则,对任意一个会话计算每个说话人的话语密度,并记为$ {\boldsymbol{D}} = (d_1,d_2,…,d_p) $,其中p是该会话的参与者数量,$ d_i $是说话人i的话语密度。在图1所示的对话中,第5个话语(讲话者A所说)的前向语境是其前面3个说话人形成的4个话语,讲话者A,B,C的话语数量为2,1,1,此时他们的话语密度为1/2,1/4,1/4。同理,第4个话语的前向语境是其前面2个说话人形成的3个话语,讲话者A,B的话语数量为2,1,此时他们的话语密度为2/3,1/3。
动态话语图表示为$ G = ({\boldsymbol{C}},{\boldsymbol{E}},{\boldsymbol{W}}) $,节点是会话中的语句,即$ {\boldsymbol{C}} = ({\boldsymbol c}_1^{\text{T}},{\boldsymbol c}_2^{\text{T}},…,{\boldsymbol c}_N^{\text{T}}) $,且边$ (i,j,{e_{i,j}}) \in {\boldsymbol{E}} $表示信息从节点$ {\boldsymbol c}_i^{\text{T}} $传播到节点$ {\boldsymbol c}_j^{\text{T}} $,其中$ {e_{i,j}} \in \{ 0,1\} $表示边是否存在,其中$ {\boldsymbol{W}} $是边权重,构建话语图的过程如下:
1)节点。话语$ {\boldsymbol c}_i^{\text{T}} $代表话语图中的初始节点。
2)边。以下规则确定动态话语图中边的连接。
①基于目标话语在会话序列中的相对位置,可将其上下文划分为前向语境和后向语境。
②针对这2个语境片段,需要分别统计各说话人在每个片段中的话语分布密度, $ d_{{\text{min}}}^{\text{f}} = {\mathrm{min}}( d_1^{\text{f}},d_2^{\text{f}},…,d_{p1}^{\text{f}} ) $,$ d_{{\text{min}}}^{\text{b}} = {\mathrm{min}}( d_1^{\text{b}},d_2^{\text{b}},…,d_{p2}^{\text{b}} ) $,其中$ d_{{\text{min}}}^{\text{f}} $和$ d_{{\text{min}}}^{\text{b}} $分别是前向和后向会话中的最小话语密度值,其中$ p1 $表示前向会话涉及的讲话者数量,$ p2 $代表后向会话包含的讲话者数量。如图1所示,在前向对话分析中,第5句话的最小话语密度为1/4,而第4句话的对应数值为1/3。
③以前向注意过程为例来阐述目标话语需要关注的前向话语。首先,初始化一个空列表h,并依次将目标话语之前的且按时间顺序最邻近的语句加入该列表,然后计算h中出现的讲话人的话语密度,列表h中的最小话语密度值用$ d_{t,{\text{min}}}^{\text{f}} $表示,如果某个说话人的话语未出现在h中,则该说话人的话语密度记为0。随着话语连续添加到列表h,$ d_{t,{\text{min}}}^{\text{f}} $的值也随之而改变。如果h中的最小话语密度值满足$ d_{t{\text{,min}}}^{\text{f}} $< $ d_{{\text{min}}}^{\text{f}} $,则继续搜索。而当一个话语添加在h中后并且$ d_{t{\text{,min}}}^{\text{f}} \geqslant d_{{\text{min}}}^{\text{f}} $时,则停止搜索,此时列表h中的话语是目标话语需要关注的上下文。如图1左侧所示,在计算第5个话语的前向会话语境时,首先将第4个话语添加到h,此时3个说话人的最小话语密度是0,低于阈值1/4,因此第4个话语被标记为需要关注的内容,并继续执行搜索过程,直到将讲话者B所说的话语添加到列表h时,3个说话人的最小话语密度为1/3,超过了阈值1/4,此时搜索过程终止。那么,第5句话要关注的前向语境是第4句话、第3句话以及第2句话。类似地,如图1右侧所示,第4个话语要关注的话语是第3和第2个话语。动态话语图的构建过程如算法1所示。
算法1. 动态话语图构建算法。
输入:完整会话列表{c1,c2,…,cn},会话中每个会话的说话人身份;
输出:所要关注的前向上下文列表H。
① 初始化空列表H;
② for i ← 1 to n−1 do /*话语0没有前向语境, 从话语1开始遍历*/
③ 空列表h;
④ Pi =p0~i−1;/*在轮次0~i−1出现的讲话者*/
⑤ NPi = $ np_{{P_i}}^{0\sim i-1} $;/*在轮次0~i−1会话中每个 讲话者的话语数量*/
⑥ $ d_{{\text{min}}}^{\text{f}} $← min (NPi/i);/*话语i的前向会话列 表中的最小讲话者密度*/
⑦ for j ← i−1 to 0 do
⑧ h. append(cj);/*话语cj添加到列表h*/
⑨ $ NP_i^t $ = $ np_{{P_i}}^{j\sim i-1} $;/*时刻t时,h中每个讲话 者的话语数量*/
⑩ $ d_{t,{\text{min}}}^{\text{f}} $←min ($ NP_i^t $/i−j);/*时刻t时,h列表 中的最小讲话者密度*/
⑪ if $ d_{t,{\text{min}}}^{\text{f}} $ < $ d_{{\text{min}}}^{\text{f}} $ then /*时刻t时如果满足条 件则继续搜索*/
⑫ 继续;
⑬ else
⑭ 中止;
⑮ end if
⑯ end for
⑰ H. append(h);
⑱ end for
⑲ return H。
通过这种方法来构建动态话语图,可以使得每个话语感知到前向会话中所有出现的说话人表达的内容,每个话语能够充分获取上下文信息且能避免上下文信息的冗余。
3)边的权重。由于图注意力网络(graph attention network,GAT)可以自适应地学习节点之间的强弱关联性。因此,本文利用该网络来更新节点表示,其中边权重的计算过程为:
$ {\alpha _{i,j}} = \frac{{\exp (LeakyReLU({{\boldsymbol a}^{\text{T}}}[{{\boldsymbol W}^{\text{T}}}{\boldsymbol c}_i^{\text{T}}\|{{\boldsymbol W}^{\text{T}}}{\boldsymbol c}_j^{\text{T}}]))}}{{\displaystyle\sum\limits_{g \in {H_i}} {\exp (LeakyReLU({{\boldsymbol a}^{\text{T}}}[{{\boldsymbol W}^{\text{T}}}{\boldsymbol c}_i^{\text{T}}\|{{\boldsymbol W}^{\text{T}}}{\boldsymbol c}_g^{\text{T}}]))} }} \; , $
其中$ {{\boldsymbol{W}}^{\text{T}}} $是线性变换的权值。$ {H_i} $表示图中第i个话语所要关注的节点集合。LeakyReLU是激活函数,$ {\boldsymbol{a}} $是一个可训练的注意力权重。
4)图学习。将图节点和边权重以及边关系作为GAT的输入,并执行可学习的线性变换和多头注意力机制逐层更新节点。其中第i个节点经过l层GAT的聚合更新过程表示如下:
$ {\boldsymbol{x}}_i^{{\text{T}}(l)} = \mathop {\|}\limits_{head = 1}^{Head} \sigma \left(\sum\limits_{g \in {H_i}} {\alpha _{i,g}^{head}{\boldsymbol{Wc}}_g^{{\text{T}}(l-1)}} \right) \; , $
其中$ l $代表第$ l $层GAT。Head是注意力机制的头的数量。$\| $为级联运算,$ \sigma $为非线性激活函数。通过以上操作,每个话语能够动态地感知语境信息,从而获得精准且充分的文本模态上下文特征$ {{\boldsymbol X}^{\text{T}}} $。
与此类似,针对音频模态和视频模态,利用与文本模态相同的话语图构建方式以及图注意力学习方式,来生成音频模态的上下文特征$ {{\boldsymbol X}^{\text{A}}} $以及视频模态的上下文特征$ {{\boldsymbol X}^{\text{V}}} $。
在2.1节中,我们仅充分挖掘了各自不同模态的上下文依赖特征,但没有关注会话上下文中不同模态的有效融合。为此,进一步设计互补双流图学习架构,该架构主要由2个分支流构成,如图3所示,即文本语义引导的一致性跨模态图学习分支和会话流动态图学习分支。一方面从模态的一致性角度来挖掘以文本语义为中心的跨模态交互,缓解不同模态特征融合中的异质鸿沟,另一方面从会话的动态性角度建模会话中多模态话语级信息流的动态关联,随后集成双流分支的输出进一步促进模态间的交互。
由于语、视、音模态信息存在本质上的异构特性,使得各个模态的数据分布不一致,显然如何挖掘3者之间的一致性语义是相对困难的。目前研究者倾向于使用Transformer结构实现跨模态交互[44],但其往往忽略了模态间相关性挖掘的主导性探索。另外,一些相关研究已经证明文本模态的语义信息比音频模态或视频模态更丰富[8,26],因此,以文本为锚点来分别探索文本与音频的相关性以及文本与视频的相关性,具体地,使用改进的SHGR(soft hirschfeld Gebelein-Rényi)一致性损失[45]来探索模态间的相关性:
$ {l_{{\text{SHGR}}}} =-\sum\limits_{M \in \{ {\text{A,V}}\} } {E{\text{[}}{{\boldsymbol X}^{\text{T}}}{{\boldsymbol X}^M}{{]-}}\frac{1}{2}} {\rm{tr}}(cov({{\boldsymbol X}^{\text{T}}})cov({{\boldsymbol{X}}^M})) \; , $
其中$ {{\boldsymbol X}^{\text{T}}} $,$ {{\boldsymbol X}^{\text{A}}} $,$ {{\boldsymbol X}^{\text{V}}} $分别表示从动态图中获取的文本特征、音频特征以及视频特征。E代表样本均值,cov代表样本方差。
在一致性损失式(5)的约束下,实现文本与音频的跨模态交互以及文本与视频的跨模态交互。由于话语序列是有时间信息存在的且不同的话语是由不同的说话人发出,则明确话语的位置信息以及赋予话语的说话者属性是非常有必要的。因此,利用话语的位置信息和讲话者嵌入信息来增强模态的上下文特征,具体地,位置嵌入的计算为:
$ \begin{gathered}p{e_{(pos,2i)}} = \sin \left(\frac{{pos}}{{{{10\;000}^{2i/d}}}}\right)\; , \\p{e_{(pos,2i + 1)}} = \cos \left(\frac{{pos}}{{{{10\;000}^{2i/d}}}}\right)\; , \\ \end{gathered} $
其中pos是话语索引,i是维度索引。同时,将说话人$ {p_j} $映射为一个向量,如式(7)所示:
$ {{\boldsymbol{se}}_j} = {{\boldsymbol V}_{{\boldsymbol{se}}}}one({p_j}),{\text{ }}j{{ = 1,2,}} … {\text{,}}S \; , $
其中$ {{\boldsymbol V}_{{\boldsymbol{se}}}} $是一个可训练的讲话者嵌入矩阵,$ one({p_j}) $是说话人$ {p_j} $的独热编码。因此,一个会话的讲话者嵌入可以表示为$ {\boldsymbol{S E}}{{ = }}({{\boldsymbol{se}}_{\phi ({u_1})}}{\text{,}}{{\boldsymbol{se}}_{\phi ({u_2})}},…,{{\boldsymbol{se}}_{\phi ({u_N})}}{{}}) $。
由此可得到模态M的话语位置嵌入和说话人嵌入的上下文表征序列$ {{\boldsymbol Q}^M} $,如式(8)所示。
$ {{\boldsymbol Q}^M}{{ = }}{{\boldsymbol X}^M}{ { + }}{\boldsymbol{PE }}+ {\boldsymbol{SE}} \; 。 $
然后基于$ {{\boldsymbol Q}^M} $使用Transformer编码器来实现跨模态交互,具体将$ {{\boldsymbol Q}^{\text{T}}} $作为查询,$ {{\boldsymbol Q}^M} $作为键和值,计算过程为
$ {{\boldsymbol Q}^{{\text{T}} \to M}} = Transformer({{\boldsymbol Q}^{\text{T}}},{{\boldsymbol Q}^M},{{\boldsymbol Q}^M}) \in {\mathbb{R}^{N \times d}} \; , $
其中$ {{\boldsymbol Q}^{\text{T}}} $代表文本模态,$ M \in \{ {\text{A,V}}\} $。以此实现文本-音频以及文本-视频的跨模态交互。最后将$ {{\boldsymbol Q}^{\text{T}}} $,$ {{\boldsymbol Q}^{{\text{T}} \to {\text{A}}}} $,$ {{\boldsymbol Q}^{{\text{T}} \to {\text{V}}}} $进行拼接得到最后的输出表征$ {{\boldsymbol Q}^{{\text{up}}}} $。
文本语义引导的一致性跨模态图学习更多关注跨模态交互融合,却忽略了会话中话语级多模态信息的流动性,因此,进一步设计会话流动态图学习来建模话语级多模态特征间的动态语境关联。
针对会话流动态图的构建,多模态会话流动态图$ G = ({\boldsymbol{Z}},{\boldsymbol{E}},{\boldsymbol{W}}) $,节点Z是每个话语的文本、音频、视频3模态注意力拼接后的特征,且边$ (i,j,{e_{i,j}}) \in {\boldsymbol{E}} $表示2节点之间的信息传播,其中$ {e_{i,j}} \in \{ 0,1\} $表示边是否存在,$ {\boldsymbol{W}} $是边的权重,构建会话流动态图的规则如下:
1)节点。多模态会话流动态图中的每个节点是话语级多模态表示,若直接将3模态表示进行拼接,既不能实现话语中细粒度的模态间对齐与融合,也难以有效应对融合时出现的冲突问题,显然不利于后续关于会话流的上下文建模。此外,由于文本比音频和视频的语义更丰富,因此,将构建以文本为中心的跨模态注意力机制来分别实现文本-音频的语义对齐融合以及文本-视频的语义对齐融合。
具体而言,利用跨模态注意力机制来自适应地学习话语级文本模态引导音频的关联对齐以及话语级文本模态引导视频的关联对齐。将$ {{\boldsymbol {C}}^{\text{T}}} $作为查询,$ {{\boldsymbol{C}}^M} $作为键和值,计算如下所示:
$ {{\boldsymbol Z}^M}{{ = }}\mathop {{\text{||}}}\limits_{head = 1}^{Head} softmax\left(\frac{{{{({\boldsymbol W}_1^{head}{{\boldsymbol {C}}^{\text{T}}})}^{\text{T}}}({\boldsymbol W}_2^{head}{\boldsymbol C^M})}}{{\sqrt {d/Head} }}\right){\text{(}}{\boldsymbol W}_3^{head}{\boldsymbol C^M}{{\text{)}}^{\text{T}}} \; , $
其中$ M \in \{ {\text{A,V}}\} $,Head是跨模态注意头的数量,$ {\boldsymbol W}_1^{head}, {\boldsymbol W}_2^{head},{\boldsymbol W}_3^{head} \in {\mathbb{R}^{d/Head \times N}} $是查询、键和值的权重矩阵。$ {{\boldsymbol {C}}^{\text{T}}} $,$ {{\boldsymbol{C}}^M} $是双向LSTM的输出。然后再聚合$ {{\boldsymbol {C}}^{\text{T}}} $,$ {{\boldsymbol Z}^{\text{A}}} $,$ {{\boldsymbol Z}^{\text{V}}} $作为会话流动态图的节点表示。
2)边。根据单模态动态图学习模块中的算法1创建节点之间的连接,以动态地建模每个话语级多模态特征的依赖范围。
3)边的权重。根据单模态动态图学习模块中的方法计算2节点之间的权重,如式(3)所示。
4)图学习。根据单模态动态图学习模块中的图聚合方式来逐层更新节点表示,如式(4)所示。
通过会话流动态图的学习,得到了文本-音频-视频的上下文动态关联特征$ {{\boldsymbol P}^{{\text{do}}}} $。
文本语义引导的一致性跨模态图学习和会话流动态图学习从不同的角度实现会话中的多模态交互。为了更有效地发挥双分支流图的功能,进一步将其整合以挖掘它们之间的协同效应与互补优势,如下:
$ {{\boldsymbol H}^{{\text{do}} \leftrightarrow {\text{up}}}} = [{{\boldsymbol P}^{{\text{do}}}}\|{{\boldsymbol Q}^{{\text{up}}}}] \in {\mathbb{R}^{N \times 6 \times d}} \; 。 $
至此,获得了针对会话场景的多模态序列特征表示$ {{\boldsymbol H}^{{\text{do}} \leftrightarrow {\text{up}}}} = ({{\boldsymbol h}_1},{{\boldsymbol h}_2},…,{{\boldsymbol h}_N}) $。
为了计算会话中每个多模态话语的情绪类别,将$ {{\boldsymbol H}^{{\text{do}} \leftrightarrow {\text{up}}}} $输入到带有全连接层和softmax层的分类器:
$ \overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{{\boldsymbol{Y}}} = softmax({{\boldsymbol W}_{\rm e}}{{\boldsymbol H}^{{\text{do}} \leftrightarrow {\text{up}}}} + {{\boldsymbol{b}}_{\rm e}}) \; , $
其中$ {{\boldsymbol W}_{\rm e}} $和$ {{\boldsymbol{b}}_{\rm e}} $是可训练参数。$ \overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{\boldsymbol Y} = ({\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{\boldsymbol y} _1},{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{\boldsymbol y} _2},…,{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{\boldsymbol y} _N}) $,$ {\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{\boldsymbol y} _i} $是话语$ {{\boldsymbol{u}}_i} $的情感概率。最后,选择$ \arg \max ({\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}}{{\boldsymbol{y}}} _i}) $作为$ {{\boldsymbol{u}}_i} $的预测的情绪标签。
学习泛化且鲁棒的判别性表征一直是情感分析相关任务的难点。由于MERC任务的情绪类别较多,且情感语义较为相似,因此,需要设计更为合适的优化策略来提取更具判别性的多模态融合表征,以提高模型的泛化性。软对比学习通过捕捉同类样本间的相似性,将其与其他类的样本进行对比来提高模态特征的判别性。然而,多模态会话数据集存在严重的类别分布不平衡问题,仅用软对比学习会强制压缩每个类的特征空间,进而损害细粒度的多样性特征,尤其给少类样本的学习增加了困难。因此,考虑构建对抗训练的方式来扩展类空间,以增强模型的鲁棒性。综上,提出对抗性软对比训练机制,即首先在原始样本上利用软对比学习训练模型,再基于单模态特征提取器的隐层梯度来获取扰动信息,然后通过扰动对抗训练生成对抗样本以丰富样本特征,并在对抗样本上利用软对比学习进一步优化模型以提高泛化能力。
软对比学习的损失是交叉熵损失和带有权衡参数$ \lambda $的对比学习损失之和。给定一个小批量样本集Ω,定义$ \vartheta (i) = \{ e \in \varOmega \backslash \{ i\} :{y_e} = {y_i}\} $是Ω中与第i个样例标签相同的样本集合,但不包含i样本,软对比学习的过程如下所示:
$ {l_{{\text{soft - scl}}}} = l_{{\text{ce}}} + \lambda l_{{\text{scl}}} \; , $
$ {l_{{\text{ce}}}} =-\frac{1}{{\left| \varOmega \right|}}\sum\limits_{i = 1}^\varOmega {\sum\limits_{j = 1}^K {{{\boldsymbol{y}}_{i,j}}\lg \left({\hat{\boldsymbol{y}}}^{{\text{soft}}} _{i,j}\right)} } \; , $
$ { \overset{\frown} {\boldsymbol{Y}} ^{{\text{soft}}}} = softmax({{\boldsymbol{W}}^{{\text{soft}}}}{\boldsymbol{\varPsi}} + {{\boldsymbol{b}}^{{\text{soft}}}}) \; , $
$ {l_{{\text{scl}}}} =-\sum\limits_{i \in \varOmega } {\frac{{-1}}{{|\vartheta (i)|}}\sum\limits_{e \in \vartheta (i)} {\lg \frac{{\exp (sim({{\boldsymbol{\varphi}} _i},{{\boldsymbol{\varphi}} _e})/\tau )}}{{\displaystyle\sum\limits_{j \in \Delta (i)} {\exp (sim({{\boldsymbol{\varphi}} _i},{{\boldsymbol{\varphi}} _j})/\tau )} }}} } \; , $
其中$ {{\boldsymbol{W}}^{{\text{soft}}}} \in {\mathbb{R}^{d \times K}} $和$ {{\boldsymbol{b}}^{{\text{soft}}}} \in {\mathbb{R}^K} $是可训练的参数,K是情绪标签的类别数。$ {{\boldsymbol{y}}_{i,j}} $和$ { \overset{\frown} {\boldsymbol{y}}_{i,j}^{{\text{soft}}}}$分别表示类别索引j下的独热向量和概率向量。$ \Delta (i) = \varOmega \backslash \{ i\} $,$ {{\boldsymbol{\varphi}} _i} $是网络模块输出的第i个样本的特征表示。$ sim( \cdot , \cdot ) $是相似性函数,$ \tau $是控制类分散的一个标量。
然后,将软对比损失分别作用于单模态会话上下文动态图网络的3个输出上(如图3所示)来充分挖掘各自模态的判别性语义,便于后续在文本语义主导的一致性损失下,跨模态图学习能够充分且有效地挖掘不同模态的交互融合特征。此外,为了进一步使得话语级的多模态特征更具判别性,将软对比损失作用于互补双流图学习的融合输出上,进一步挖掘更为判别的话语级多模态融合特征。总体而言,本文提出的模型在原始样本上训练的总损失为
$ {l_{\rm or}} = {l_{{\text{SHGR}}}} + l_{{\text{soft-scl}}}^{\text{T}} + l_{{\text{soft-scl}}}^{\text{A}} + l_{{\text{soft-scl}}}^{\text{V}} + l_{{\text{soft-scl}}}^{{\text{TAV}}} \; , $
其中$ l_{{\text{soft-scl}}}^{\text{T}} $,$ l_{{\text{soft-scl}}}^{\text{A}} $,$ l_{{\text{soft-scl}}}^{\text{V}} $,$ l_{{\text{soft-scl}}}^{{\text{TAV}}} $分别代表文本话语图输出、音频话语图输出、视频话语图输出以及分类器上的软对比损失。
在每个批训练步过程,首先在原始样本上利用$ {l_{\text{o}}} $损失来训练网络,然后采用扰动对抗训练策略生成关于原始样本的对抗样本特征以扩展类样本空间。现有研究将噪声扰动信息放在原始样本上[41],然而这会摧毁上下文的语义连贯性,使得语义难以理解。为了在扰动对抗过程中保护会话的语义连贯性,受到PGD对抗训练策略[46]的启发,本文将扰动$ \boldsymbol r $放在双向LSTM的上下文感知的隐藏层,其包括3个门控层和1个记忆细胞层。给定$ {L_q} $范式球和$\boldsymbol r $的半径$ \varepsilon $,通过线性近似,并在软对比损失目标下,对抗样例的训练过程如下所示:
$ {\boldsymbol{r}} =-\varepsilon {{{\boldsymbol{g}} \mathord{\left/{\vphantom {{\boldsymbol{g}} {\|{\boldsymbol{g}}\|}}} \right.} {\|{\boldsymbol{g}}\|}}_q}\;。 \\$
式(18)中的内部最大化是生成上下文级别的最坏情况对抗样本,外部最小化是针对对抗样本训练一个鲁棒的网络。在引入PGD对抗之后,软对比损失可以从上下文依赖的输入中进一步学习更多样化的特征和平滑的表示空间,并增强模型对上下文扰动的鲁棒性。对抗样例上的优化损失$ l_{{\text{ad}}}^{{\text{adv}}} $与原始样例上的优化损失$ {l_{{\text{or}}}} $计算一致,在对抗样本上的总损失$ l_{{\text{ad}}}^{{\text{adv}}} $如下所示:
$ l_{{\text{ad}}}^{{\text{adv}}} = l_{{\text{SHGR}}}^{{\text{adv}}} + l_{{\text{soft-scl}}}^{{\text{T,adv}}} + l_{{\text{soft-scl}}}^{{\text{A,adv}}} + l_{{\text{soft-scl}}}^{{\text{V,adv}}} + l_{{\text{soft-scl}}}^{{\text{TAV,adv}}} \; , $
其中$ l_{{\text{SHGR}}}^{{\text{adv}}} $,$ l_{{\text{soft-scl}}}^{{\text{T,adv}}} $,$ l_{{\text{soft-scl}}}^{{\text{A,adv}}} $,$ l_{{\text{soft-scl}}}^{{\text{V,adv}}} $,$l_{{\text{soft-scl}}}^{{\text{TAV,adv}}} $分别是对抗样例上一致性损失、文本话语图输出、音频话语图输出、视频话语图输出以及互补双流图输出上的软对比损失。
综上,本文模型ASDG的优化总目标函数为
$ l = {l_{{\text{or}}}} + l_{{\text{ad}}}^{{\text{adv}}} 。$
使用MELD[42]和IEMOCAP[43]这2个数据集来验证所提模型ASDG的有效性,这些数据集目前在MERC的研究中得到了广泛的使用[8,25],数据集的详细信息如表1所示,为了更公平地比较ASDG的性能,本文采用现有MERC研究使用的数据集划分设置。
1)IEMOCAP。该数据集由151个对话和7 433个话语组成,每句话都被标记为6种情绪快乐、悲伤、中性、愤怒、兴奋和沮丧中的1种,每个会话涉及2个讲话者。
2)MELD。该数据集是从电视剧《老友记》中搜集的9人对话数据集,包含1 433个对话和13 708个句子,包含中性、惊讶、恐惧、悲伤、喜悦、厌恶和愤怒7种情绪,每句话被标注为其中的一种情绪。
为了评估ASDG在多模态会话情绪识别任务上的性能,参照现有MERC研究中[8,25]广泛使用的评估标准,具体使用以下评估指标:1)准确率(accuracy,ACC);2)加权平均准确率(weighted average accuracy,WAA);3)F1-score;4)加权平均F1-score(weighted average F1-score,WAF1)。
根据文献[47],采用RoBERTa大语言模型提取文本特征。RoBERTa是一个使用多层Transformer编码器架构的预训练模型,将最后一层[CLS]嵌入来获得文本特征。此外,根据文献[25],使用特征提取工具包openSMILE进行音频特征提取,然后通过全连接层来降低音频特征维度,以及使用在面部表情识别数据集上预训练的稠密卷积网络DenseNet进行视觉特征提取,DenseNet由多个密集块组成,每个块包含多层CNN。
为了评估ASDG的性能,本文选择一些当前较有竞争力的MERC模型作为基线用于对比分析,大致将模型分为3类,分别是基于循环神经网络的模型、基于图神经网络的模型以及基于对比学习的模型,具体描述如下:
1)EmoCaps (emotion capsule)[48]。EmoCaps通过Emoformer提取不同模态的特征,并将不同模态进行维度拼接形成话语级多模态表示,然后输入LSTM网络学习上下文依赖。
2)GraphCFC (graph based cross-modal feature complementation)[8]。GraphCFC是一个基于有向图的跨模态特征互补模型,将每个话语的语、视、音特征分别作为一个节点,构建了一个包含讲话者信息以及多种边关系的异构图,使得GNN能够同时获取模态内上下文信息和模态间的交互信息。
3)GA2MIF (graph and attention based two-stage multi-source information fusion)[26]。GA2MIF是基于图注意力的2阶段多模态信息融合方法,将会话中的每个模态的所有上下文分别构建一个图,避免了将异构图作为模型输入的问题。
4)HiMul-LGG (hierarchical decision fusion-ba- sed local-global graph neural network)[24]。HiMul-LGG采用层次化的决策融合策略,构建局部-全局图神经网络架构来增强说话人在模态间和模态内的依赖。
5)AdaIGN (adaptive interactive graph neural network)[49]。AdaIGN采用Gumbel softmax技术在所构建的异构图中自适应选择节点和边,增强模态内和跨模态交互,并设计特定任务的损失函数以优先考虑文本模态和说话人自身依赖的上下文选择。
6)UniMSE (unified multimodal sentiment analysis and emotion recognition)[36]。UniMSE将情感分类和情绪识别任务相结合,相互提供多模态情感知识,并引入对比学习来放大样例间的差异。
7)FacialMMT (facial expression aware multimo-dal multi-task learning)[44]。FacialMMT是2阶段学习框架。第1阶段利用无监督人脸聚类和人脸匹配算法提取每个会话的人脸序列;第2阶段利用面部表情识别来获取情绪分布,从而辅助情绪识别任务。
8)CBERL(class boundary enhanced represent-ation learning)[41]。CBERL利用生成对抗网络的数据增强方式生成新样本,并将原始样本和新样本输入变分自编码器进行跨模态融合,然后将多模态特征输入到多任务图神经网络中优化类边界的分布表示,以缓解类分布不平衡的问题。
9)MALN (multimodal adversarial learning network)[12]。MALN是一个多模态对抗学习网络,以对抗的方式提取多模态的共性信息,同时学习多样性信息,从而对多模态数据进行综合挖掘。
10)AGF-IB (adversarial alignment and graph fusion via information bottleneck)[6]。AGF-IB是一种基于对抗对齐和图融合的会话情感识别方法,通过对抗学习消除模态间的异构性,并引入图对比学习来捕获模态内和模态间的互补语义信息。
11)DER-GCN (dialog and event relation-aware graph convolutional neural network)[31]。DER-GCN构建了一个带权的多关系图来同时捕捉对话中说话人与事件关系之间的依赖关系,引入了自监督掩码图自编码器来提高特征和结构的融合表示能力,提出一种基于对比学习的损失优化策略,以增强少数类特征的表示能力。
使用Pytorch实现所提出的ASDG模型,并且使用AdamW作为优化器在IEMOCAP数据集上的学习率为6E−5,批大小为1,对应的文本、音频和视觉模态的维度分别设置为1 024,1 582,342。在MELD数据集上的学习率为8E−5,批大小为12,对应的文本、音频和视觉模态的维度分别设置为1024,300,342。对于Transformer编码器,隐藏层的维度大小、注意力头数量和层数分别设置为1024,1,1。对于GAT网络,注意力头和层数分别设置为4,2。为了防止过拟合,将L2权重衰减设置为1E−5,并将神经元丢失率dropout设置为0.5。
表2和表3分别展示了在IEMOCAP和MELD数据集上ASDG方法与3.3节中采用的一系列基线方法在各类情绪上的ACC和F1值,以及总体分类的WAA和WAF1。
如表2所示,在IEMOCAP数据集上,ASDG方法相较于基线方法取得了更具优势的识别效果,其WAA和WAF1值分别达到了72.09%和72.25%,且在“兴奋”和“中性”2个类别下的F1值也是最高的,这表明本文构建的对抗软对比调制动态图学习的MERC方法是有效的。其中基于循环神经网络的EmoCaps方法取得的效果仅次于ASDG方法,其WAF1值为71.77%,但是EmoCaps在各个情绪类别的ACC相对较差,这是由于EmoCaps方法尽管能够学习话语级的上下文关联,但是其难以自适应挖掘当前话语的上下文信息,且不能关注判别性特征的学习,而ASDG方法不仅构建动态话语图来自适应地建模上下文依赖,还设计对抗性软对比训练机制来提高模型的泛化性以及判别性。针对基于图结构的模型,如GraphCFC方法取得的WAA和WAF1值分别为69.13%和68.91%,其虽然利用独立和共享编码器考虑了模态的异构问题,但是其构建图过程中的窗口大小是固定的,不能自适应地确定上下文的依赖范围。虽然GA2MIF和HiMul-LGG通过分阶段融合来缓解模态间的异构性,但是同样面临固定窗口的限制,导致性能提升有限。DER-GCN利用图重构的思想来自适应地丢弃或增加节点与边,而AdaIGN利用Gumbel softmax自适应地选择节点和边,尽管DER-GCN和AdaIGN都采用一定程度的自适应构图的思路,但是它们均缺乏考虑模态表征的泛化设计,使得效果明显劣于ASDG方法。此外,ASDG也明显优于最近提出的基于对比思想的方法,例如,UniMSE仅实施不同模态特征间的对比来缓解异构问题,但是并未考虑情感语义上的标签级的对比。AGF-IB不仅考虑了标签级的对比,并对文本、音频、视频3模态分别构建生成器和判别器,而MALN仅对3模态构建一个共同的生成器,虽然AGF-IB方法和MALN方法能不同程度地缓解模态间的异构问题,但是过于强调模态间的相似性,不利于挖掘模态间的互补性。CBERL方法通过在3模态的特征级上加噪来提高模态的泛化性,但是这会摧毁会话的上下文语义连贯性,而ASDG方法通过在特征抽取的LSTM隐藏层加噪来生成对抗样本,并在原始样本和对抗样本上进行联合软对比学习,来扩展每个模态的类表示空间和最大化模态内具有相同标签样本的一致性,从而达到更可靠的泛化效果。
不同于IEMOCAP数据集,MELD数据集的每个会话的轮次更少,但每个会话中涉及的说话者更多,且MELD数据集呈现出极其不平衡的类分布特点,这使得在MELD数据集上开展MERC任务更加困难。如表3所示,ASDG方法在MELD数据集上的WAA和WAF1值分别为67.24%,66.83%,而对比的EmoCaps方法的WAF1值仅达到了64.00%,这是由于MELD数据集牵扯9个讲话者,而EmoCaps方法没有考虑讲话者特征,但ASDG方法既涵盖讲话者依赖特征又嵌入讲话者身份的独热编码,能有效地帮助识别讲话者的话语语义关联。此外,ASDG方法要优于所列举的不同形式的基于GNN的方法,如GraphCFC,GA2MIF,DER-GCN,AdaIGN,这是因为MELD数据集涉及的讲话者多但会话轮次少,导致能够利用的上下文信息较少,因此,精准地学习上下文依赖范围显得尤为重要,这更表明ASDG方法中设计的动态会话图学习策略更适合于精准界定上下文关联范围。尤其,ASDG在少类样本上取得了最好的效果,如在“恐惧”“悲伤”“厌恶”上F1值分别达到了23.16%,43.97%,29.53%,相较于基于对比思想的MALN方法,分别提高了1.96个百分点,0.97个百分点,11.93个百分点。这说明ASDG方法中的对抗性软对比训练机制对少类样本的处理较有优势,能够获得更具判别的特征。
综上实验可知,ASDG方法在样本量、会话长度、讲话者数量等分布差异较大的IEMOCAP数据集和MELD数据集上均能有效识别多模态会话中的情绪类别,这表明本文所设计的基于密度的自适应图学习策略能有效地捕获模态内以及跨模态的动态语境依赖,且在原始样例和设计的基于加噪策略生成的对抗样例上联合进行对抗性软对比训练以调制构建的双流图结构,最终使得构建的模型能够提取更泛化和鲁棒的模态表征。
为了评估ASDG中各模块对情绪识别效果的影响,具体在IEMACAP和MELD这2个数据集上从模型组件、训练方式以及不同模态贡献度这3个角度展开模型的消融实验,表4给出了不同消融设置下的实验结果,并选择WAA和WAF1作为评价指标。
1)-w/o CCGL (consistent cross-modal graph learning guided by textual semantics)。移除文本语义引导的一致性跨模态图学习,但保留一致性损失。
2)-w/o CFDG (conversation flow dynamic graph learning)。移除会话流动态图学习分支。
3)-w/o DG (density graph)。移除单模上下文动态图网络和会话流动态图学习中的图学习,直接进行特征维度的拼接。
4)-w/o PE (positional embedding)。在CCGL模块中移除位置嵌入。
5)-w/o SE (speaker embedding)。在CCGL模块中移除讲话者嵌入。
以上组件是ASDG的5个关键组成部分,通过每次只移除1个组件来评估该组件的有效性。从表4中可以看出:1)本文所提出的这些组件都是有效的,因为任意移除其中1个组件都会导致性能的下降。2)文本语义引导的一致性跨模态图学习对模型性能的影响最大,当移除该模块时,模型性能在2个数据集上WAA分别下降了3.39个百分点,1.3个百分点,这说明在语义一致性损失的约束下实现跨模态交互可以充分挖掘不同模态的语义信息。3)会话流动态图学习在IEMOCAP数据集上的影响不是很大,但是在MELD数据集上影响较大,这是由于MELD数据集中的会话轮次少且信息量少,通过充分地挖掘多模态话语级的会话动态上下文依赖,是能显著提高模型对多模态数据语义的理解能力。综合2)和3)可以看出以双流结构的学习方式进行跨模态交互可以增强对会话中抽象情感语义的学习能力,并有效地缓解模态异构问题。4)动态话语图对2个数据集的情绪识别都有相当大的影响,这说明自适应地建模上下文范围可以精准地界定会话流的依赖范围,从而减轻特征冗余或者信息不足的情况。5)位置嵌入和说话者嵌入对2个数据集的情绪识别也有一定的影响,这意味着捕获会话场景中的话语顺序信息和讲话者信息是有价值的。
1)-w/o ASCL (adversarial soft-contrast learning)。为了验证对抗性软对比的有效性,移除对抗性软对比训练机制,但将软对比损失替换为交叉熵损失。
2)-w/o AT (adversarial training)。为了验证对抗性训练的有效性,在对抗性软对比训练机制中,移除对抗性训练,仅保留软对比学习。
3)-w/o SHGR。移除SHGR一致性损失。
从表4中可以看出:1)本文所提出的对抗性软对比训练机制是有效的,因为当移除对抗性软对比训练机制时,评估指标WAF1在IEMOCAP数据集上下降了2.17个百分点,且在MELD数据集上下降了1.25个百分点,这说明了本文设计的对抗性软对比训练机制比仅用交叉熵损失来协同优化,更能使得学习到的多模态特征具有判别性。2)本文设计对抗训练机制也有较大的效用,因为当仅移除对抗训练时,评估指标WAF1在IEMOCAP数据集和MELD数据集上分别下降了2.48个百分点和1.38个百分点,样本和对抗样本进行联合优化,使得模型能学习到更泛化的模态表示,且能有效地避免生成的对抗样本面临语义不连贯的问题。3)当不使用SHGR损失时,模型的WAF1指标在2个数据集上分别下降了0.29个百分点和0.89个百分点,这说明了使用本文设计的文本语义引导的跨模态一致性学习来迫使音频模态和视频模态分别与文本模态的语义保持一致性是合理的,能够促进跨模态特征的有效交互。
为了体现多模态会话情绪识别中不同模态的贡献度,每次移除1个或者2个模态来进行实验。从表4中可以得出3个结论:1)对于单模态会话情绪识别的实验结果,文本模态的效果要远远优于其他2种模态,表明文本模态比音频模态和视频模态的语义要丰富,这更说明本文所设计的文本语义引导的一致性跨模态图学习中以文本为中心来引导和集成音频和视频的设计是合理的。2)在双模态的会话情绪识别中,文本+音频的效果要好于文本+视频的效果,这说明在情绪识别中音频信息要比视频信息更重要,语音中含有的语调等信息可能比图像更能反映人的情感,而图像中的背景噪声往往较大。3)同时使用文本、音频和视频这3种模态可以获得最佳的会话情绪识别性能,这说明结合多种模态的信息更能反映人的情绪及其变化,但是合理地融合多模态信息进行情绪识别是至关重要的,因为本文提出的多模态会话信息融合模型相比其他的融合模型的情绪识别效果更好,如表2和表3所示。
为了更直观地验证本文所提出的对抗性软对比训练机制的有效性,分别在去除和保留对抗性软对比损失的情况下训练模型,并在IEMOCAP的测试集中可视化模态特征的分布情况,如图4(b)(c)所示。首先图4(a)直观展示了原始数据中不同情绪类别的分布状况。具体地,从IEMOCAP的测试集中提取每个话语的不同模态表征,并整合这些不同模态表征以形成初始的话语级多模态表征,然后利用t-SNE算法[50]将这些多模态表征降维并投影至2维空间进行可视化。通过比较图4 (a)(c),可以看出相较于原始的多模态表征,本文提出的基于对抗性软对比调制双流图学习模型所习得的表征在聚类效果上有了显著提升。通过分析图4(b),可以看出在没有应用对抗性软对比策略学习的情况下,多模态特征显得过于离散,即便是同一情绪类别的多模态特征也难以有效聚合。通过对比分析图4(b)(c),可以清晰地观察到,在引入对抗性软对比学习策略后,在图4(c)中相同情绪类别的多模态表征能够更加紧密地聚集在一起,而不同情绪类别的表征则较少地混杂在一起。通过可视化结果表明,本文设计的对抗性软对比的调制策略能够帮助构建的模型学习到更鲁棒且泛化的模态表征。
表2和表3可知,ASDG方法相较对比方法在IEMOCAP数据集(如“悲伤”“兴奋”) 和 MELD数据集(如“恐惧”“悲伤”“厌恶”“愤怒”)的少样例类别上取得了较好的结果,这表明ASDG方法能够提取更具泛化且鲁棒的特征表示。尽管ASDG在MERC任务上展现出了相对优越的性能,但它仍面临一些局限,尤其在处理情绪类别语义相近的样例上识别效果仍存在一定的不足。为此,在IEMOCAP和MELD的测试集上通过混淆矩阵展开具体情绪类别上的性能分析。具体地,如图5所示,ASDG方法在处理语义边界模糊的情绪时仍存在样例的混淆,例如,在IEMOCAP数据集中,模型倾向于将“喜悦”类的样本误判为“兴奋”,以及将“愤怒”类的样本误识别为“沮丧”。类似地,在MELD数据集中,模型对“恐惧”情绪的识别也常常偏离,错误地将其归类为“悲伤”。诸如这种高级抽象语义的混淆表明ASDG方法在区分极其细微情绪差异时存在挑战。此外,ASDG方法还展现出对“中性”情绪类别的偏好,这与数据集中存在的情绪类别不平衡问题有关,尤其“中性”情绪的样本在数据集中往往占据较大的比例,使得模型在学习过程中可能过度拟合这类情绪的特征,进而影响对少类样例的情绪识别。
本文针对当前多模态会话研究难以提取泛化且鲁棒的特征表示以及不能动态调整会话窗口导致提取的语境信息冗余或不足的问题,提出对抗软对比调制动态图学习的方法,不仅能实现会话语境的动态建模,还能有效融合多模态异构特征,并显著提升模型的鲁棒性与泛化性。ASDG首先通过LSTM初步提取各自模态的上下文特征,并构建基于动态话语图的GAT来动态地建模各自模态内的精准上下文依赖,然后,设计互补双流图学习机制来分别建模跨模态交互和话语级多模态上下文的交互关联,其中,文本语义引导的跨模态流图学习通过Transformer架构来集成3个模态的图特征表示以探索跨模态的异构融合,而会话流动态图学习利用跨模态注意力来学习文本引导的音频和视频表示,并将其融合作为话语图GAT的输入以自适应地学习多模态话语级的上下文依赖。尤其,在模型优化上设计了对抗性软对比训练机制,通过在LSTM隐藏层加入扰动来生成对抗样本,并在原始样本和对抗样本上利用软对比学习来扩展类样本空间,以学习到更泛化的类特征表示。在IEMOCAP和MELD这2个不同代表性的数据集上进行了实验,实验结果表明ASDG方法的性能优于相关的对比方法,并通过消融实验证明了ASDG中的不同模块以及优化设置的有效性。在未来的研究中,将在会话场景上下文学习中探索因果关联式的自适应上下文的学习,从而进一步探索多模态会话情绪识别的可解释性。
作者贡献声明:王顺杰负责研究方案的构建,并完成实验和撰写论文;蔡国永提出了指导意见并修改论文;吕光瑞提出了算法思路和实验方案。

参考文献 引证文献
排序方式:
1
Singh G V, Firdaus M, Ekbal A, et al. EmoInt-Trans: A multimodal transformer for identifying emotions and intents in social conversations[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022, 31: 290−300
2
Zhou Hao, Huang Minlie, Zhang Tianyang, et al. Emotional chatting machine: Emotional conversation generation with internal and external memory[C]//Proc of the 32nd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2018: 730−738
3
Rosa R L, Schwartz G M, Ruggiero W V, et al. A knowledge-based recommendation system that includes sentiment analysis and deep learning[J]. IEEE Transactions on Industrial Informatics, 2018, 15(4): 2124−2135
4
Chatterjee A, Narahari K N, Joshi M, et al. SemEval−2019 task 3: EmoContext contextual emotion detection in text[C]//Proc of the 13th Int Workshop on Semantic Evaluation. Stroudsburg, PA: ACL, 2019: 39−48
5
Bhavan A, Chauhan P, Shah R R. Bagged support vector machines for emotion recognition from speech[J]. Knowledge-Based Systems, 2019, 184(C): 104886
6
Shou Yuntao, Tao Meng, Wei Ai, et al. Adversarial alignment and graph fusion via information bottleneck for multimodal emotion recognition in conversations[J]. Information Fusion, 2024, 112: 102590
7
Wang Ruobing, Zha Daren, Zhao Qingfei, et al. RW-MMDCG: Muti-modal via rolling-Window directed graph network for conversational emotion recognition[C]//Proc of the 26th Int Conf on Computer Supported Cooperative Work in Design. Piscataway, NJ: IEEE, 2023: 606−611
8
Li Jiang, Wang Xiaoping, Lv Guoqing, et al. GraphCFC: A directed graph based cross-modal feature complementation approach for multimodal conversational emotion recognition[J]. IEEE Transactions on Multimedia, 2023, 26: 77−89
9
Li Jiang, Wang Xiaoping, Lv Guoqing, et al. GraphMFT: A graph network based multimodal fusion technique for emotion recognition in conversation[J]. Neurocomputing, 2023, 550(suppl-C): 126427
10
Lu Nannan, Han Zhiyuan, Han Min, et al. Bi-stream graph learning based multimodal fusion for emotion recognition in conversation[J]. Information Fusion, 2024, 106: 102272
11
Ma Hongchao, Wang Zhongqing, Zhou Xiabing, et al. Emotion recognition with conversational generation transfer[J]. Transactions on Asian and Low-Resource Language Information Processing, 2022, 21(4): 1−17
12
Ren Minjie, Huang Xiangdong, Liu Jing, et al. MALN: Multimodal adversarial learning network for conversational emotion recognition[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(11): 6965−6980
13
Abonizio H Q, Paraiso E C, Barbon S. Toward text data augmentation for sentiment analysis[J]. IEEE Transactions on Artificial Intelligence, 2021, 3(5): 657−668
14
Gunel B, Du Jingfei, Conneau A, et al. Supervised contrastive learning for pre-trained language model fine-tuning[C]//Proc of the 9th Int Conf on Learning Representations. Washington: ICLR, 2021: 1−15
15
Huang Faliang, Li Xuelong, Yuan Changan, et al. Attention-emotion-enhanced convolutional LSTM for sentiment analysis[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 33(9): 4332−4345
16
Lu Yi, Mak M W. Improving speech emotion recognition with adversarial data augmentation network[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 33(1): 172−184
17
Essa I A. Analysis, interpretation and synthesis of facial expressions[D]. Cambridge, MA: Massachusetts Institute of Technology, 1995
18
Zhang Hanqing, Song Dawei. Towards contrastive context-aware conversational emotion recognition[J]. IEEE Transactions on Affective Computing, 2022, 13(4): 1879−1891
19
Huan Ruohong, Shu Jia, Bao Shenglin, et al. Video multimodal emotion recognition based on Bi-GRU and attention fusion[J]. Multimedia Tools and Applications, 2021, 80(6): 8213−8240
20
Wang Jianyou, Xue M, Culhane R, et al. Speech emotion recognition with dual-sequence LSTM architecture[C]//Proc of the 2020 IEEE Int Conf on Acoustics, Speech and Signal Processing. Piscataway, NJ: IEEE, 2020: 6474−6478
21
Li Chao, Bao Zhongtian, Li Linhao, et al. Exploring temporal representations by leveraging attention-based bidirectional LSTM-RNNs for multi-modal emotion recognition[J]. Information Processing & Management, 2020, 57(3): 102185
22
Ho N H, Yang H J, Kim S H, et al. Multimodal approach of speech emotion recognition using multi-level multi-head fusion attention-based recurrent neural network[J]. IEEE Access, 2020, 8: 61672−61686
23
Shen Xudong, Huang Xianying, Zou Shihao, et al. Multimodal knowledge-enhanced interactive network with mixed contrastive learning for emotion recognition in conversation[J]. Neurocomputing, 2024, 582(C): 127550
24
Fu Changzeng, Qian Fengkui, Su Kaifeng, et al. HiMul-LGG: A hierarchical decision fusion-based local–global graph neural network for multimodal emotion recognition in conversation[J]. Neural Networks, 2025, 181: 106764
25
Hu Jingwen, Liu Yuchen, Zhao Jinming, et al. MMGCN: Multimodal fusion via deep graph convolution network for emotion recognition in conversation[C]//Proc of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int Joint Conference on Natural Language Processing. Stroudsburg, PA: ACL, 2021: 5666−5675
26
Li Jiang, Wang Xiaoping, Lv Guoqing, et al. GA2MIF: Graph and attention based two-stage multi-source information fusion for conversational emotion detection[J]. IEEE Transactions on Affective Computing, 2023, 15(1): 130−143
27
黄学坚, 马廷淮, 王根生. 基于样本内外协同表示和自适应融合的多模态学习方法[J]. 计算机研究与发展, 2024, 61(5): 1310−1324
Huang Xuejian, Ma Tinghuai, Wang Gensheng. Multimodal learning method based on intra- and inter-sample cooperative representation and adaptive fusion[J]. Journal of Computer Research and Development, 2024, 61(5): 1310−1324 (in Chinese)
28
Xie B, Sidulova M, Park C H. Robust multimodal emotion recognition from conversation with transformer-based crossmodality fusion[J]. Sensors, 2021, 21(14): 4913
29
Wen Huanglu, You Shaodi, Fu Ying. Cross-modal dynamic convolution for multi-modal emotion recognition[J]. Journal of Visual Communication and Image Representation, 2021, 78(Suppl C): 103178
30
Huang Jian, Tao Jianhua, Liu Bin, et al. Multimodal transformer fusion for continuous emotion recognition[C]//Proc of the 2020 IEEE Int Conf on Acoustics, Speech and Signal Processing. Piscataway, NJ: IEEE, 2020: 3507−3511
31
Wei Ai, Shou Yuntao, Meng Tao, et al. DER-GCN: Dialog and event relation-aware graph convolutional neural network for multimodal dialog emotion recognition[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024, 36(3): 4908−4921
32
杜金明, 孙媛媛, 林鸿飞, 等. 融入知识图谱和课程学习的对话情绪识别[J]. 计算机研究与发展, 2024, 61(5): 1299−1309
Du Jinming, Sun Yuanyuan, Lin Hongfei, et al. Conversational emotion recognition incorporating knowledge graph and curriculum learning[J]. Journal of Computer Research and Development, 2024, 61(5): 1299−1309 (in Chinese)
33
Kalantidis Y, Sariyildiz M B, Pion N, et al. Hard negative mixing for contrastive learning[C]//Proc of the 34th Int Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2020: 21798−21809
34
Xu Yuan, Yang Meng. MCM-CSD: Multi-Granularity context modeling with contrastive speaker detection for emotion recognition in real-time conversation[C]//Proc of the 2024 IEEE Int Conf on Acoustics, Speech and Signal Processing. Piscataway, NJ: IEEE, 2024: 11956−11960
35
Tu Geng, Xiong Feng, Ling Bin, et al. Multimodal eemotion recognition calibration in conversations[C]//Proc of the 32nd ACM Int Conf on Multimedia. New York: ACM, 2024: 9621−9630
36
Hu Guimin, Lin Tingen, Zhao Yi, et al. UniMSE: Towards unified multimodal sentiment analysis and emotion recognition[C]//Proc of the 2022 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2022: 7837−7851
37
Lu Jiajun, Sibai H, Fabry E. Adversarial examples that fool detectors[J]. arXiv preprint, arXiv: 1712.02494, 2017
38
Szegedy C, Zaremba W, Sutskever I, et al. Intriguing properties of neural networks[J]. arXiv preprint, arXiv: 1312.6199, 2013
39
Goodfellow I J, Shlens J, Szegedy C. Explaining and harnessing adversarial examples[J]. arXiv preprint, arXiv: 1412.6572, 2014
40
Miyato T, Dai A M, Goodfellow I. Adversarial training methods for semisupervised text classification[C]//Proc of the 5th Int Conf on Learning Representations. Washington: ICLR, 2017: 24−26
41
Meng Tao, Shou Yuntao, Ai Wei, et al. Deep imbalanced learning for multimodal emotion recognition in conversations[J]. IEEE Transactions on Artificial Intelligence, 2024, 5(12): 2691−4581
42
Poria S, Hazarika D, Majumder N, et al. MELD: A multimodal multi-party dataset for emotion recognition in conversations[C]//Proc of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2019: 527−536
43
Busso C, Bulut M, Lee C C, et al. IEMOCAP: Interactive emotional dyadic motion capture database[J]. Language Resources and Evaluation, 2008, 42(4): 335−359
44
Zheng Wenjie, Yu Jianfei, Xia Rui, et al. A facial expression-aware multimodal multi-task learning framework for emotion recognition in multi-party conversations[C]//Proc of the 61st Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2023: 15445−15459
45
Wang Lichen, Wu Jiaxiang, Huang Shaolun, et al. An efficient approach to informative feature extraction from multimodal data[C]//Proc of the 2019 AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 5281−5288
46
Mądry A, Makelov A, Schmidt L, et al. Towards deep learning models resistant to adversarial attacks[J]. arXiv preprint, arXiv: 1706.06083, 2017
47
Ghosal D, Majumder N, Gelbukh A, et al. COSMIC: Commonsense knowledge for emotion identification in conversations[C]//Proc of the 2020 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2020: 2470–2481
48
Li Zajing, Tang Fengxiao, Zhao Ming, et al. EmoCaps: Emotion capsule based model for conversational emotion recognition[C]//Proc of the 60th Association for Computational Linguistics. Stroudsburg, PA: ACL, 2022: 1610−1618
49
Tu Geng, Xie Tian, Liang Bin, et al. Adaptive graph learning for multimodal conversational emotion detection[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 19089−19097
50
Van der Maaten L, Hinton G. Visualizing data using t-SNE[J]. Journal of Machine Learning Research, 2008, 9(11): 2579−2605
2026年第63卷第2期
354
130
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202440896
  • 接收时间:2024-11-18
  • 首发时间:2026-11-06
  • 出版时间:2026-02-01
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2024-11-18
  • 修回日期:2025-06-30
  • 录用日期:2025-07-17
基金
作者信息
    桂林电子科技大学计算机与信息安全学院 广西桂林 541004
    广西可信软件重点实验室 (桂林电子科技大学) 广西桂林 541004

通讯作者:

蔡国永()
参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202440896
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏