Article(id=1327012451545100502, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1327012311115608215, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202330199, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=research-article, receivedDate=1679846400000, receivedDateStr=2023-03-27, revisedDate=1705939200000, revisedDateStr=2024-01-23, acceptedDate=null, acceptedDateStr=null, onlineDate=1794314023146, onlineDateStr=2026-11-10, pubDate=1738339200000, pubDateStr=2025-02-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1794314023146, onlineIssueDateStr=2026-11-10, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1794314023146, creator=13701087609, updateTime=1794314023146, updator=13701087609, issue=Issue{id=1327012311115608215, tenantId=1146029695717560320, journalId=1324759914146529478, year='2025', volume='62', issue='2', pageStart='285', pageEnd='541', issueExtLink='null', onlineDate='null', pubDate='1738339200000', pubDateStr='2025-02-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1794313989665, creator='13701087609', updateTime=1794376696965, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext={EN=IssueExt(id=1327275325060395748, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1327012311115608215, language=EN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=), CN=IssueExt(id=1327275325060395749, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1327012311115608215, language=CN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=)}, issueFiles=null, downloadFileDto=null}, startPage=374, endPage=382, ext={EN=ArticleExt(id=1327012451717066967, articleId=1327012451545100502, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=Multimodal Sentiment Analysis Method for Sentimental Semantic Inconsistency, columnId=1325489891789910943, journalTitle=Journal of Computer Research and Development, columnName=Artificial Intelligence, runingTitle=null, highlight=null, articleAbstract=

Multimodal sentiment analysis is a multimodal task that uses multiple modalities of subjective information to analyze sentiment. In some scenarios, the sentimental expression in different modalities is inconsistent, even contradictory, which will weaken the effect of multimodal collaborative decision-making. In this paper, a multimodal learning method is proposed to learn the modal feature representations with consistent sentimental semantics. In order to improve the common feature representation of different modalities and learn the dynamic interaction between modalities without affecting the original information, we first learn the common feature representation of each modality, and then use cross attention to enable one modality to effectively obtain auxiliary information from the common feature representations of other modalities. In multimodal fusion, we propose a multimodal attention, which is used to weighted concatenate modal feature representations, in order to increase the expression of contributed modalities and suppress the influence of weak modalities. The experimental results of the proposed method on the sentiment analysis datasets MOSI, MOSEI, CH-SIMS are better than those of the compared models, indicating the necessity and rationality of considering the problem of sentimental semantic inconsistency in multimodal sentiment analysis.

, authors=Yuanyi Luo, Rui Wu*, Jiafeng Liu, Xianglong Tang, authorsList=Yuanyi Luo, Rui Wu, Jiafeng Liu, Xianglong Tang, authorCompany=null, correspAuthors=Rui Wu, authorNote=

born in 1996. PhD candidate. His main research interest includes multimodal learning

born in 1976. PhD, associate professor, PhD supervisor. His main research interests include pattern recognition and multimodal learning

born in 1968. PhD, associate professor. His main research interests include pattern recognition and machine learning

born in 1960. PhD, professor, PhD supervisor. His main research interests include pattern recognition and computer vision

, correspAuthorsNote=null, copyrightStatement=Copyright © 2025 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1327012453193461990, articleId=1327012451545100502, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=面向情感语义不一致的多模态情感分析方法, columnId=1325489892104483745, journalTitle=计算机研究与发展, columnName=人工智能, runingTitle=null, highlight=null, articleAbstract=

多模态情感分析是利用多种模式的主观信息对情感进行分析判断的一种多模态任务. 情感表达具有主观性,在某些场景下不同模态的情感表达不一致,甚至存在相悖的情况,这会削弱多模态协同决策的效果. 针对不同模态间情感语义不一致的问题,提出一种多模态学习方法,学习情感语义表达一致的模态特征表示. 为了在不影响模态原始信息的同时,提高各模态的共性特征表达并增加模态间的动态交互,首先学习每个模态的共性特征表示,然后利用交叉注意力使单个模态能有效从其余模态的共性特征表示中获取辅助信息. 在模态融合模块,以软注意力机制为基础提出模态注意力,对情感语义表达一致的各模态特征表示进行加权连接,以增大强模态的表达,抑制弱模态对任务的影响. 提出的模型在情感分析数据集MOSI,MOSEI,CH-SIMS上的实验结果均优于对比模型,表明在多模态情感分析任务中考虑情感语义不一致问题的必要性与合理性.

, authors=罗渊贻, 吴锐*, 刘家锋, 唐降龙, authorsList=罗渊贻, 吴锐, 刘家锋, 唐降龙, authorCompany=null, correspAuthors=吴锐, authorNote=

1996年生. 博士研究生. 主要研究方向为多模态学习

1976年生. 博士,副教授,博士生导师. 主要研究方向为模式识别、多模态学习

1968年生. 博士,副教授. 主要研究方向为模式识别、机器学习

1960年生. 博士,教授,博士生导师. 主要研究方向为模式识别、计算机视觉

, correspAuthorsNote=
吴锐()
, copyrightStatement=版权所有 © 2025 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=QvIkqOUcnfQPWBKBhilytg==, magXml=Y4jq2abGr4UL12fgzjFOTQ==, pdfUrl=null, pdf=W3WCSnWHy2Im3Js0Sv6i1w==, pdfFileSize=1064152, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=rgkmpsFP7L/QCcYZ270oNg==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=tXuaI4VQCvTAbMRYVXr33w==, mapNumber=null, fund=null)}, authors=[Author(id=1327012453558366443, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=luoyy@stu.hit.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012453667418349, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, authorId=1327012453558366443, language=EN, stringName=Yuanyi Luo, firstName=Yuanyi, middleName=null, lastName=Luo, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=College of Computer Science and Technology, Harbin Institute of Technology, Harbin 150006, bio={"content":"

born in 1996. PhD candidate. His main research interest includes multimodal learning

"}, bioImg=null, bioContent=

born in 1996. PhD candidate. His main research interest includes multimodal learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012453726138606, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, authorId=1327012453558366443, language=CN, stringName=罗渊贻, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=哈尔滨工业大学计算机科学与技术学院 哈尔滨 150006, bio={"img":"m0zPg7xcidyAY+C3o1tbnQ==","content":"

1996年生. 博士研究生. 主要研究方向为多模态学习

"}, bioImg=m0zPg7xcidyAY+C3o1tbnQ==, bioContent=

1996年生. 博士研究生. 主要研究方向为多模态学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012453457703143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, xref=null, ext=[AuthorCompanyExt(id=1327012453474480360, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, companyId=1327012453457703143, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Institute of Technology, Harbin 150006), AuthorCompanyExt(id=1327012453487063273, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, companyId=1327012453457703143, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工业大学计算机科学与技术学院 哈尔滨 150006)])]), Author(id=1327012453814218992, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=simple@hit.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1327012453998768370, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, authorId=1327012453814218992, language=EN, stringName=Rui Wu, firstName=Rui, middleName=null, lastName=Wu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=College of Computer Science and Technology, Harbin Institute of Technology, Harbin 150006, bio={"content":"

born in 1976. PhD, associate professor, PhD supervisor. His main research interests include pattern recognition and multimodal learning

"}, bioImg=null, bioContent=

born in 1976. PhD, associate professor, PhD supervisor. His main research interests include pattern recognition and multimodal learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012454091043059, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, authorId=1327012453814218992, language=CN, stringName=吴锐, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=哈尔滨工业大学计算机科学与技术学院 哈尔滨 150006, bio={"img":"PH5J3LEicyxrWYmDOhRR3A==","content":"

1976年生. 博士,副教授,博士生导师. 主要研究方向为模式识别、多模态学习

"}, bioImg=PH5J3LEicyxrWYmDOhRR3A==, bioContent=

1976年生. 博士,副教授,博士生导师. 主要研究方向为模式识别、多模态学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012453457703143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, xref=null, ext=[AuthorCompanyExt(id=1327012453474480360, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, companyId=1327012453457703143, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Institute of Technology, Harbin 150006), AuthorCompanyExt(id=1327012453487063273, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, companyId=1327012453457703143, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工业大学计算机科学与技术学院 哈尔滨 150006)])]), Author(id=1327012454174929141, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012454267203831, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, authorId=1327012454174929141, language=EN, stringName=Jiafeng Liu, firstName=Jiafeng, middleName=null, lastName=Liu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=College of Computer Science and Technology, Harbin Institute of Technology, Harbin 150006, bio={"content":"

born in 1968. PhD, associate professor. His main research interests include pattern recognition and machine learning

"}, bioImg=null, bioContent=

born in 1968. PhD, associate professor. His main research interests include pattern recognition and machine learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012454334312696, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, authorId=1327012454174929141, language=CN, stringName=刘家锋, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=哈尔滨工业大学计算机科学与技术学院 哈尔滨 150006, bio={"img":"YK2/i++s2qGvYmOApDBAyw==","content":"

1968年生. 博士,副教授. 主要研究方向为模式识别、机器学习

"}, bioImg=YK2/i++s2qGvYmOApDBAyw==, bioContent=

1968年生. 博士,副教授. 主要研究方向为模式识别、机器学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012453457703143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, xref=null, ext=[AuthorCompanyExt(id=1327012453474480360, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, companyId=1327012453457703143, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Institute of Technology, Harbin 150006), AuthorCompanyExt(id=1327012453487063273, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, companyId=1327012453457703143, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工业大学计算机科学与技术学院 哈尔滨 150006)])]), Author(id=1327012454460141818, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012454539833596, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, authorId=1327012454460141818, language=EN, stringName=Xianglong Tang, firstName=Xianglong, middleName=null, lastName=Tang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=College of Computer Science and Technology, Harbin Institute of Technology, Harbin 150006, bio={"content":"

born in 1960. PhD, professor, PhD supervisor. His main research interests include pattern recognition and computer vision

"}, bioImg=null, bioContent=

born in 1960. PhD, professor, PhD supervisor. His main research interests include pattern recognition and computer vision

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012454606942461, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, authorId=1327012454460141818, language=CN, stringName=唐降龙, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=哈尔滨工业大学计算机科学与技术学院 哈尔滨 150006, bio={"img":"uAy4ykO7JsICd1XwAzDbmA==","content":"

1960年生. 博士,教授,博士生导师. 主要研究方向为模式识别、计算机视觉

"}, bioImg=uAy4ykO7JsICd1XwAzDbmA==, bioContent=

1960年生. 博士,教授,博士生导师. 主要研究方向为模式识别、计算机视觉

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012453457703143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, xref=null, ext=[AuthorCompanyExt(id=1327012453474480360, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, companyId=1327012453457703143, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Institute of Technology, Harbin 150006), AuthorCompanyExt(id=1327012453487063273, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, companyId=1327012453457703143, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工业大学计算机科学与技术学院 哈尔滨 150006)])])], keywords=[Keyword(id=1327012454695022846, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=EN, orderNo=1, keyword=multimodal sentiment analysis), Keyword(id=1327012454757937407, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=EN, orderNo=2, keyword=common features), Keyword(id=1327012454829240576, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=EN, orderNo=3, keyword=sentimental semantic inconsistency), Keyword(id=1327012454896349441, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=EN, orderNo=4, keyword=attention mechanism), Keyword(id=1327012454971846914, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=CN, orderNo=1, keyword=多模态情感分析), Keyword(id=1327012455038955779, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=CN, orderNo=2, keyword=共性特征), Keyword(id=1327012455106064644, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=CN, orderNo=3, keyword=情感语义不一致), Keyword(id=1327012455185756421, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=CN, orderNo=4, keyword=注意力机制)], refs=[Reference(id=1327012457429709085, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=陈龙,管子玉,何金红,等. 情感分类研究进展[J]. 计算机研究与发展,2017,54(6):1150−1170, articleTitle=null, refAbstract=null), Reference(id=1327012457505206558, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Long, Guan Ziyu, He Jinhong, et al. A survey on sentiment classification[J]. Journal of Computer Research and Development, 2017, 54(6): 1150−1170 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1327012457597481247, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Yu Wenmeng, Xu Hua, Meng Fanyang, et al. CH-SIMS: A Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality[C]//Proc of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 3718−3727, articleTitle=null, refAbstract=null), Reference(id=1327012457672978720, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Tsai Y, Bai S, Liang P P, et al. Multimodal transformer for unaligned multimodal language sequences[C]//Proc of the 57th Int Conf on Association for Computational Linguistics. Stroudsburg, PA: ACL, 2019: 6558−6569, articleTitle=null, refAbstract=null), Reference(id=1327012457731698977, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Bertasius G, Wang Heng, Torresani L. Is space-time attention all you need for video understanding[J]. arXiv preprint, arXiv: 2102.05095, 2021, articleTitle=null, refAbstract=null), Reference(id=1327012457790419234, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Zellinger W, Grubinger T, Lughofer E, et al. Central moment discrepancy (CMD) for domain-invariant representation learning[J]. arXiv preprint, arXiv: 1702.08811, 2017, articleTitle=null, refAbstract=null), Reference(id=1327012457853333795, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Sahay S, Kumar S H, Xia Rui, et al. Multimodal relational tensor network for sentiment and emotion classification[C]//Proc of the 1st Int Conf on Association for Computational Linguistics. Stroudsburg, PA: ACL, 2018: 20−27, articleTitle=null, refAbstract=null), Reference(id=1327012457916248356, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Zadeh A, Chen Minghai, Poria S, et al. Tensor fusion network for multimodal sentiment analysis[C]//Proc of the Int Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2017: 1103−1114, articleTitle=null, refAbstract=null), Reference(id=1327012457995940133, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Zhun, Shen Ying, Lakshminarasimhan V B, et al. Efficient low-rank multimodal fusion with modality-specific factors[C]//Proc of the Int Conf on Association for Computational Linguistics. Stroudsburg, PA: ACL, 2018: 2247−2256, articleTitle=null, refAbstract=null), Reference(id=1327012458071437606, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Hazarika D, Zimmermann R, Poria S. MISA: Modality-invariant and -specific representations for multimodal sentiment analysis[J]. arXiv preprint, arXiv: 2005.03545, 2020, articleTitle=null, refAbstract=null), Reference(id=1327012458146935079, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Rahman W, Hasan M K, Lee S, et al. Integrating multimodal information in large pretrained transformers[C]//Proc of the 58th Conf on Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 2359−2369, articleTitle=null, refAbstract=null), Reference(id=1327012459820462376, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Yu Wenmeng, Xu Hua, Yuan Ziqi, et al. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis[C]//Proc of the Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2021: 10790−10797, articleTitle=null, refAbstract=null), Reference(id=1327012460005011753, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=程艳,尧磊波,张光河,等. 基于注意力机制的多通道CNN和BiGRU的文本情感倾向性分析[J]. 计算机研究与发展,2020,57(12):2583−2595, articleTitle=null, refAbstract=null), Reference(id=1327012460197949738, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Cheng Yan, Yao Leibo, Zhang Guanghe, et al. Text sentiment orientation analysis of multi-channels CNN and BiGRU based on attention mechanism[J]. Journal of Computer Research and Development, 2020, 57(12): 2583−2595 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1327012460319584555, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate[J]. arXiv preprint, arXiv: 1409.0473, 2014, articleTitle=null, refAbstract=null), Reference(id=1327012460432830764, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Mnih V, Heess N, Graves A, et al. Recurrent models of visual attention[C]//Proc of the 27th Int Conf on Neural Information Processing Systems. Cambridge, MA: NeurIPS, 2014: 2204–2212, articleTitle=null, refAbstract=null), Reference(id=1327012460520911149, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Zhizheng, Lan Cuiling, Zeng Wenjun, et al. Relation-aware global attention for person re-identification[C]//Proc of the IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 3183–3192, articleTitle=null, refAbstract=null), Reference(id=1327012460600602926, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Xu K, Ba J L, Kiros R, et al. Show, attend and tell: Neural image caption generation with visual attention[C]//Proc of the Int Conf on Machine Learning. New York: ACM, 2015: 2048−2057, articleTitle=null, refAbstract=null), Reference(id=1327012460667711791, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Guo Menghao, Xu Tianxing, Liu Jiangjiang, et al. Attention mechanisms in computer vision: A survey[J]. Computational Visual Media, 2022, 8(3): 331−368, articleTitle=null, refAbstract=null), Reference(id=1327012460734820657, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Jianing, Wang Jingdong, Gao Wen, et al. Global–local temporal representations for video person re-identification[C]//Proc of the IEEE Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 3957–3966, articleTitle=null, refAbstract=null), Reference(id=1327012460801929522, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Xiang Long, Gan Chuang, Melo G, et al. Multimodal keyless attention fusion for video classification[C]//Proc of the 32nd Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2018: 7202−7209, articleTitle=null, refAbstract=null), Reference(id=1327012460864844083, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Ghosal D, Akhtar M S, Chauhan D, et al. Contextual inter-modal attention for multi-modal sentiment analysis[C]//Proc of the 2018 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2018: 3454−3466, articleTitle=null, refAbstract=null), Reference(id=1327012460927758644, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Ye Junjie, Zhou Jie, Tian Junfeng, et al. Sentiment-aware multimodal pre-training for multimodal sentiment analysis[J]. Knowledge-Based Systems, 2022, 258: 110021, articleTitle=null, refAbstract=null), Reference(id=1327012461011644725, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=Sun Zhongkai, Sarma P, Sethares W, et al. Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis[C]//Proc of the Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 8992−8999, articleTitle=null, refAbstract=null), Reference(id=1327012461108113718, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=Devlin J, Chang Mingwei, Lee K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]//Proc of the Conf on Association for Computational Linguistics. Stroudsburg, PA: ACL, 2019: 4171–4186, articleTitle=null, refAbstract=null), Reference(id=1327012461208777015, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=Zadeh A, Zellers R, Pincus E, et al. MOSI: Multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos[J]. arXiv preprint, arXiv: 1606.06259, 2016, articleTitle=null, refAbstract=null), Reference(id=1327012461280080184, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=26, authorNames=null, journalName=null, refType=null, unstructuredReference=Zadeh A, Liang P P, Vanbriesen J, et al. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph[C]//Proc of the 56th Conf on Association for Computational Linguistics. Stroudsburg, PA: ACL, 2018: 2236−2246, articleTitle=null, refAbstract=null), Reference(id=1327012461347189049, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=27, authorNames=null, journalName=null, refType=null, unstructuredReference=Sun Hao, Wang Hongyi, Liu Jiaqing, et al. CubeMLP: An MLP-based model for multimodal sentiment analysis and depression estimation[C]//Proc of the 30th ACM Int Conf on Multimedia. New York: ACM, 2022: 3722−3729, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1327012453457703143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, xref=null, ext=[AuthorCompanyExt(id=1327012453474480360, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, companyId=1327012453457703143, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Institute of Technology, Harbin 150006), AuthorCompanyExt(id=1327012453487063273, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, companyId=1327012453457703143, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工业大学计算机科学与技术学院 哈尔滨 150006)])], figs=[ArticleFig(id=1327012455303196934, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=EN, label=Figure 1, caption=Multimodal sentiment analysis model, figureFileSmall=QljLCmHYEadBM1k+rmkUlg==, figureFileBig=rgkmpsFP7L/QCcYZ270oNg==, tableContent=null), ArticleFig(id=1327012455382888711, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=CN, label=图1, caption=多模态情感分析模型, figureFileSmall=QljLCmHYEadBM1k+rmkUlg==, figureFileBig=rgkmpsFP7L/QCcYZ270oNg==, tableContent=null), ArticleFig(id=1327012455580021000, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=EN, label=Figure 2, caption=Structure diagram of modal attention, figureFileSmall=kRN/DlXRTHMyxHOb++/kDA==, figureFileBig=ARR8l5S1r/BKSvvI/UFXnw==, tableContent=null), ArticleFig(id=1327012455718433033, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=CN, label=图2, caption=模态注意力结构图, figureFileSmall=kRN/DlXRTHMyxHOb++/kDA==, figureFileBig=ARR8l5S1r/BKSvvI/UFXnw==, tableContent=null), ArticleFig(id=1327012455798124810, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=EN, label=Table 1, caption=Influence of Sample Data with Sentimental Semantic Inconsistency Between Modalities on Multimodal Sentiment Analysis Tasks, figureFileSmall=null, figureFileBig=null, tableContent=
模型CH-SIMS数据集
数据样本2分类3分类F1MAECorr
LF-DNNorg0.7920.6440.8020.4580.558
(–)d0.8300.6690.8310.4080.659
(re)$ {\mathrm{d}}_{170} $0.8040.6680.8110.4180.639
(re)$ {\mathrm{d}}_{228} $0.7930.6670.8020.4320.619
(re)$ {\mathrm{d}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}} $0.6990.5880.7120.5300.403
MulTorg0.7840.6470.7990.4920.555
(–)d0.7900.6340.8070.4920.589
(re)$ {\mathrm{d}}_{228} $0.7700.6280.7900.5150.544
), ArticleFig(id=1327012455861039371, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=CN, label=表1, caption=

模态间情感语义不一致的样本数据对多模态情感分析任务的影响

, figureFileSmall=null, figureFileBig=null, tableContent=
模型CH-SIMS数据集
数据样本2分类3分类F1MAECorr
LF-DNNorg0.7920.6440.8020.4580.558
(–)d0.8300.6690.8310.4080.659
(re)$ {\mathrm{d}}_{170} $0.8040.6680.8110.4180.639
(re)$ {\mathrm{d}}_{228} $0.7930.6670.8020.4320.619
(re)$ {\mathrm{d}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}} $0.6990.5880.7120.5300.403
MulTorg0.7840.6470.7990.4920.555
(–)d0.7900.6340.8070.4920.589
(re)$ {\mathrm{d}}_{228} $0.7700.6280.7900.5150.544
), ArticleFig(id=1327012455936536844, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=EN, label=Table 2, caption=Hyper-Parameter Settings in Dataset, figureFileSmall=null, figureFileBig=null, tableContent=
超参数MOSIMOSEISIMS
注:“batch size”是单次训练的样本个数;“learning rate”是神经网络的学习率;“hidden size”是隐藏层的维度大小;“depth”是时空注意力和时间注意力中的学习层数;“epoch”是训练迭代次数;“weight”是共性特征表示学习的损失函数系数;“early stop”是为防止过拟合而设置的提前结束循环条件.
batch size326464
learning rate0.00010.00010.0001
hidden size256256256
depth121218
epoch60100100
weight0.50.50.8
early stop202020
), ArticleFig(id=1327012456024617229, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=CN, label=表2, caption=

数据集的超参数设置

, figureFileSmall=null, figureFileBig=null, tableContent=
超参数MOSIMOSEISIMS
注:“batch size”是单次训练的样本个数;“learning rate”是神经网络的学习率;“hidden size”是隐藏层的维度大小;“depth”是时空注意力和时间注意力中的学习层数;“epoch”是训练迭代次数;“weight”是共性特征表示学习的损失函数系数;“early stop”是为防止过拟合而设置的提前结束循环条件.
batch size326464
learning rate0.00010.00010.0001
hidden size256256256
depth121218
epoch60100100
weight0.50.50.8
early stop202020
), ArticleFig(id=1327012456100114703, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=EN, label=Table 3, caption=Comparison Results of Methods Based on CMU-MOSI Dataset, figureFileSmall=null, figureFileBig=null, tableContent=
模型2分类5分类7分类F1MAECorr
注:上标为“1”的数据来源于文献[9],上标为“2”的数据来源于文献[11],上标为“3”的数据来源于文献[26],上标为“*”的数据结果是根据可获取的开源代码进行复现得出.
TFN10.8080.8070. 9010.698
LMF10.8250.8240. 9170.695
MulT10.8340.8280.8710.698
MISA10.8340.8360.7830.761
MAG-BERT20.8430.8430.7310.789
Self-MM20.8600.8600.7130.798
CubeMLP30.8560.4550.8550.7700.767
MAG-BERT*0.8350.5010.4380.8360.7490.783
Self-MM*0.8400.5260.4580.8410.7240.791
本文模型0.8660.5320.4740.8670.7090.796
), ArticleFig(id=1327012456242721040, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=CN, label=表3, caption=

基于CMU-MOSI数据集的方法对比结果

, figureFileSmall=null, figureFileBig=null, tableContent=
模型2分类5分类7分类F1MAECorr
注:上标为“1”的数据来源于文献[9],上标为“2”的数据来源于文献[11],上标为“3”的数据来源于文献[26],上标为“*”的数据结果是根据可获取的开源代码进行复现得出.
TFN10.8080.8070. 9010.698
LMF10.8250.8240. 9170.695
MulT10.8340.8280.8710.698
MISA10.8340.8360.7830.761
MAG-BERT20.8430.8430.7310.789
Self-MM20.8600.8600.7130.798
CubeMLP30.8560.4550.8550.7700.767
MAG-BERT*0.8350.5010.4380.8360.7490.783
Self-MM*0.8400.5260.4580.8410.7240.791
本文模型0.8660.5320.4740.8670.7090.796
), ArticleFig(id=1327012456318218513, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=EN, label=Table 4, caption=Comparison Results of Models Based on CMU-MOSEI Dataset, figureFileSmall=null, figureFileBig=null, tableContent=
模型2分类5分类7分类F1MAECorr
注:黑体数字表示最优结果. 上标为“1”的数据来源于文献[9],上标为“2”的数据来源于文献[11],上标为“3”的数据来源于文献[26],上标为“*”的数据结果是根据可获取的开源代码进行复现得出.
TFN10.8250.8210.5930.700
LMF10.8200.8210.6230.677
MulT10.8250.8230.5800.703
MISA20.8420.8400.5680.724
MAG-BERT20.8520.8510.5390.753
Self-MM20.8520.8530.5300.765
CubeMLP30.8510.5490.8450.5290.760
MAG-BERT*0.8510.5450.5270.8500.5430.760
Self-MM*0.8520.5450.5290.8510.5390.766
本文模型0.8540.5470.5310.8540.5370.767
), ArticleFig(id=1327012456456630546, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=CN, label=表4, caption=

基于CMU-MOSEI数据集的模型对比结果

, figureFileSmall=null, figureFileBig=null, tableContent=
模型2分类5分类7分类F1MAECorr
注:黑体数字表示最优结果. 上标为“1”的数据来源于文献[9],上标为“2”的数据来源于文献[11],上标为“3”的数据来源于文献[26],上标为“*”的数据结果是根据可获取的开源代码进行复现得出.
TFN10.8250.8210.5930.700
LMF10.8200.8210.6230.677
MulT10.8250.8230.5800.703
MISA20.8420.8400.5680.724
MAG-BERT20.8520.8510.5390.753
Self-MM20.8520.8530.5300.765
CubeMLP30.8510.5490.8450.5290.760
MAG-BERT*0.8510.5450.5270.8500.5430.760
Self-MM*0.8520.5450.5290.8510.5390.766
本文模型0.8540.5470.5310.8540.5370.767
), ArticleFig(id=1327012456582459667, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=EN, label=Table 5, caption=Comparison Results of Models Based on CH-SIMS Dataset, figureFileSmall=null, figureFileBig=null, tableContent=
模型多任务2分类F1
注:上标为“2”的数据来源于文献[11],上标为“*”的数据结果是根据可获取的开源代码进行复现得出.
TFN2×0.7980.802
LMF2×0.7940.787
Self-MM2√0.8070.808
Self-MM*√0.7920.789
本文模型√0.8120.825
), ArticleFig(id=1327012456725066004, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=CN, label=表5, caption=

基于CH-SIMS数据集的模型对比结果

, figureFileSmall=null, figureFileBig=null, tableContent=
模型多任务2分类F1
注:上标为“2”的数据来源于文献[11],上标为“*”的数据结果是根据可获取的开源代码进行复现得出.
TFN2×0.7980.802
LMF2×0.7940.787
Self-MM2√0.8070.808
Self-MM*√0.7920.789
本文模型√0.8120.825
), ArticleFig(id=1327012456804757781, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=EN, label=Table 6, caption=Analysis Results of Models Based on CH-SIMS Dataset, figureFileSmall=null, figureFileBig=null, tableContent=
样本实际值预测值
TAVMTAVM
注:“T”表示文本模态,“A”表示音频模态,“V”表示视频模态,“M”表示多模型.
N38–0.2–0.60.4–1.0–0.4–0.2–0.2–0.6
N63–0.6–0.20.4–0.6–0.3–0.2–0.2–0.5
N187–0.60.2–1.0–0.8–0.5–0.1–0.3–0.7
N732–0.8–0.60.6–0.6–0.4–0.2–0.2–0.6
N2081–0.20.6–0.4–1.0–0.1–0.2–0.2–0.2
), ArticleFig(id=1327012456876060950, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=CN, label=表6, caption=

基于CH-SIMS数据集的模型分析结果

, figureFileSmall=null, figureFileBig=null, tableContent=
样本实际值预测值
TAVMTAVM
注:“T”表示文本模态,“A”表示音频模态,“V”表示视频模态,“M”表示多模型.
N38–0.2–0.60.4–1.0–0.4–0.2–0.2–0.6
N63–0.6–0.20.4–0.6–0.3–0.2–0.2–0.5
N187–0.60.2–1.0–0.8–0.5–0.1–0.3–0.7
N732–0.8–0.60.6–0.6–0.4–0.2–0.2–0.6
N2081–0.20.6–0.4–1.0–0.1–0.2–0.2–0.2
), ArticleFig(id=1327012456947364119, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=EN, label=Table 7, caption=Cosine Similarity Comparison of Modality Features Before and After Training, figureFileSmall=null, figureFileBig=null, tableContent=
数据集训练前训练后
MOSI–0.01180.0314
MOSEI–0.03160.0176
CH-SIMS0.04990.0835
), ArticleFig(id=1327012457010278680, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=CN, label=表7, caption=

模态特征在训练前后的余弦相似度对比

, figureFileSmall=null, figureFileBig=null, tableContent=
数据集训练前训练后
MOSI–0.01180.0314
MOSEI–0.03160.0176
CH-SIMS0.04990.0835
), ArticleFig(id=1327012457094164761, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=EN, label=Table 8, caption=Comparison Results of Methods, figureFileSmall=null, figureFileBig=null, tableContent=
计算方法2分类F1MAECorr训练时间/s
CMD0.8660.8670.7090.79611
MMD0.8540.8540.7650.78715
KL0.8460.8470.7700.7649
), ArticleFig(id=1327012457178050842, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=CN, label=表8, caption=

方法对比结果

, figureFileSmall=null, figureFileBig=null, tableContent=
计算方法2分类F1MAECorr训练时间/s
CMD0.8660.8670.7090.79611
MMD0.8540.8540.7650.78715
KL0.8460.8470.7700.7649
), ArticleFig(id=1327012457261936923, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=EN, label=Table 9, caption=Results of Ablation Study, figureFileSmall=null, figureFileBig=null, tableContent=
模型2分类F1MAECorr
本文模型0.8670.8670.7090.796
(–)时空注意力0.8360.8350.7630.786
(–)交叉注意力0.8390.8390.7620.769
(–)模态注意力0.8450.8440.7470.788
), ArticleFig(id=1327012457333240092, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, language=CN, label=表9, caption=

消融实验结果

, figureFileSmall=null, figureFileBig=null, tableContent=
模型2分类F1MAECorr
本文模型0.8670.8670.7090.796
(–)时空注意力0.8360.8350.7630.786
(–)交叉注意力0.8390.8390.7620.769
(–)模态注意力0.8450.8440.7470.788
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202330199, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202330199, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202330199, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202330199, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1738339200000, fullTextJson=null, articleText=null, reference=null)
收藏切换
面向情感语义不一致的多模态情感分析方法
罗渊贻 , 吴锐 * , 刘家锋 , 唐降龙
计算机研究与发展 | 人工智能 2025,62(2): 374-382
收起
收藏切换
计算机研究与发展 |人工智能 2025 , 62 (2) : 374 -382
面向情感语义不一致的多模态情感分析方法
全屏
[Author(id=1327012453558366443, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=luoyy@stu.hit.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012453667418349, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, authorId=1327012453558366443, language=EN, stringName=Yuanyi Luo, firstName=Yuanyi, middleName=null, lastName=Luo, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=College of Computer Science and Technology, Harbin Institute of Technology, Harbin 150006, bio={"content":"

born in 1996. PhD candidate. His main research interest includes multimodal learning

"}, bioImg=null, bioContent=

born in 1996. PhD candidate. His main research interest includes multimodal learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012453726138606, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, authorId=1327012453558366443, language=CN, stringName=罗渊贻, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=哈尔滨工业大学计算机科学与技术学院 哈尔滨 150006, bio={"img":"m0zPg7xcidyAY+C3o1tbnQ==","content":"

1996年生. 博士研究生. 主要研究方向为多模态学习

"}, bioImg=m0zPg7xcidyAY+C3o1tbnQ==, bioContent=

1996年生. 博士研究生. 主要研究方向为多模态学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012453457703143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, xref=null, ext=[AuthorCompanyExt(id=1327012453474480360, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, companyId=1327012453457703143, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Institute of Technology, Harbin 150006), AuthorCompanyExt(id=1327012453487063273, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, companyId=1327012453457703143, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工业大学计算机科学与技术学院 哈尔滨 150006)])]), Author(id=1327012453814218992, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=simple@hit.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1327012453998768370, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, authorId=1327012453814218992, language=EN, stringName=Rui Wu, firstName=Rui, middleName=null, lastName=Wu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=College of Computer Science and Technology, Harbin Institute of Technology, Harbin 150006, bio={"content":"

born in 1976. PhD, associate professor, PhD supervisor. His main research interests include pattern recognition and multimodal learning

"}, bioImg=null, bioContent=

born in 1976. PhD, associate professor, PhD supervisor. His main research interests include pattern recognition and multimodal learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012454091043059, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, authorId=1327012453814218992, language=CN, stringName=吴锐, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=哈尔滨工业大学计算机科学与技术学院 哈尔滨 150006, bio={"img":"PH5J3LEicyxrWYmDOhRR3A==","content":"

1976年生. 博士,副教授,博士生导师. 主要研究方向为模式识别、多模态学习

"}, bioImg=PH5J3LEicyxrWYmDOhRR3A==, bioContent=

1976年生. 博士,副教授,博士生导师. 主要研究方向为模式识别、多模态学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012453457703143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, xref=null, ext=[AuthorCompanyExt(id=1327012453474480360, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, companyId=1327012453457703143, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Institute of Technology, Harbin 150006), AuthorCompanyExt(id=1327012453487063273, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, companyId=1327012453457703143, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工业大学计算机科学与技术学院 哈尔滨 150006)])]), Author(id=1327012454174929141, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012454267203831, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, authorId=1327012454174929141, language=EN, stringName=Jiafeng Liu, firstName=Jiafeng, middleName=null, lastName=Liu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=College of Computer Science and Technology, Harbin Institute of Technology, Harbin 150006, bio={"content":"

born in 1968. PhD, associate professor. His main research interests include pattern recognition and machine learning

"}, bioImg=null, bioContent=

born in 1968. PhD, associate professor. His main research interests include pattern recognition and machine learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012454334312696, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, authorId=1327012454174929141, language=CN, stringName=刘家锋, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=哈尔滨工业大学计算机科学与技术学院 哈尔滨 150006, bio={"img":"YK2/i++s2qGvYmOApDBAyw==","content":"

1968年生. 博士,副教授. 主要研究方向为模式识别、机器学习

"}, bioImg=YK2/i++s2qGvYmOApDBAyw==, bioContent=

1968年生. 博士,副教授. 主要研究方向为模式识别、机器学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012453457703143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, xref=null, ext=[AuthorCompanyExt(id=1327012453474480360, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, companyId=1327012453457703143, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Institute of Technology, Harbin 150006), AuthorCompanyExt(id=1327012453487063273, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, companyId=1327012453457703143, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工业大学计算机科学与技术学院 哈尔滨 150006)])]), Author(id=1327012454460141818, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012454539833596, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, authorId=1327012454460141818, language=EN, stringName=Xianglong Tang, firstName=Xianglong, middleName=null, lastName=Tang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=College of Computer Science and Technology, Harbin Institute of Technology, Harbin 150006, bio={"content":"

born in 1960. PhD, professor, PhD supervisor. His main research interests include pattern recognition and computer vision

"}, bioImg=null, bioContent=

born in 1960. PhD, professor, PhD supervisor. His main research interests include pattern recognition and computer vision

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012454606942461, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, authorId=1327012454460141818, language=CN, stringName=唐降龙, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=哈尔滨工业大学计算机科学与技术学院 哈尔滨 150006, bio={"img":"uAy4ykO7JsICd1XwAzDbmA==","content":"

1960年生. 博士,教授,博士生导师. 主要研究方向为模式识别、计算机视觉

"}, bioImg=uAy4ykO7JsICd1XwAzDbmA==, bioContent=

1960年生. 博士,教授,博士生导师. 主要研究方向为模式识别、计算机视觉

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012453457703143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, xref=null, ext=[AuthorCompanyExt(id=1327012453474480360, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, companyId=1327012453457703143, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Institute of Technology, Harbin 150006), AuthorCompanyExt(id=1327012453487063273, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012451545100502, companyId=1327012453457703143, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工业大学计算机科学与技术学院 哈尔滨 150006)])])]
罗渊贻 , 吴锐* , 刘家锋, 唐降龙
作者信息
  • 哈尔滨工业大学计算机科学与技术学院 哈尔滨 150006
通讯作者:
吴锐()
作者简介:

1996年生. 博士研究生. 主要研究方向为多模态学习

1976年生. 博士,副教授,博士生导师. 主要研究方向为模式识别、多模态学习

1968年生. 博士,副教授. 主要研究方向为模式识别、机器学习

1960年生. 博士,教授,博士生导师. 主要研究方向为模式识别、计算机视觉

Multimodal Sentiment Analysis Method for Sentimental Semantic Inconsistency
Yuanyi Luo , Rui Wu* , Jiafeng Liu, Xianglong Tang
Affiliations
  • College of Computer Science and Technology, Harbin Institute of Technology, Harbin 150006
  • born in 1996. PhD candidate. His main research interest includes multimodal learning

    born in 1976. PhD, associate professor, PhD supervisor. His main research interests include pattern recognition and multimodal learning

    born in 1968. PhD, associate professor. His main research interests include pattern recognition and machine learning

    born in 1960. PhD, professor, PhD supervisor. His main research interests include pattern recognition and computer vision

出版时间: 2025-02-01 doi: 10.7544/issn1000-1239.202330199
文章导航
收藏切换

多模态情感分析是利用多种模式的主观信息对情感进行分析判断的一种多模态任务. 情感表达具有主观性,在某些场景下不同模态的情感表达不一致,甚至存在相悖的情况,这会削弱多模态协同决策的效果. 针对不同模态间情感语义不一致的问题,提出一种多模态学习方法,学习情感语义表达一致的模态特征表示. 为了在不影响模态原始信息的同时,提高各模态的共性特征表达并增加模态间的动态交互,首先学习每个模态的共性特征表示,然后利用交叉注意力使单个模态能有效从其余模态的共性特征表示中获取辅助信息. 在模态融合模块,以软注意力机制为基础提出模态注意力,对情感语义表达一致的各模态特征表示进行加权连接,以增大强模态的表达,抑制弱模态对任务的影响. 提出的模型在情感分析数据集MOSI,MOSEI,CH-SIMS上的实验结果均优于对比模型,表明在多模态情感分析任务中考虑情感语义不一致问题的必要性与合理性.

多模态情感分析  /  共性特征  /  情感语义不一致  /  注意力机制

Multimodal sentiment analysis is a multimodal task that uses multiple modalities of subjective information to analyze sentiment. In some scenarios, the sentimental expression in different modalities is inconsistent, even contradictory, which will weaken the effect of multimodal collaborative decision-making. In this paper, a multimodal learning method is proposed to learn the modal feature representations with consistent sentimental semantics. In order to improve the common feature representation of different modalities and learn the dynamic interaction between modalities without affecting the original information, we first learn the common feature representation of each modality, and then use cross attention to enable one modality to effectively obtain auxiliary information from the common feature representations of other modalities. In multimodal fusion, we propose a multimodal attention, which is used to weighted concatenate modal feature representations, in order to increase the expression of contributed modalities and suppress the influence of weak modalities. The experimental results of the proposed method on the sentiment analysis datasets MOSI, MOSEI, CH-SIMS are better than those of the compared models, indicating the necessity and rationality of considering the problem of sentimental semantic inconsistency in multimodal sentiment analysis.

multimodal sentiment analysis  /  common features  /  sentimental semantic inconsistency  /  attention mechanism
罗渊贻, 吴锐, 刘家锋, 唐降龙. 面向情感语义不一致的多模态情感分析方法. 计算机研究与发展, 2025 , 62 (2) : 374 -382 . DOI: 10.7544/issn1000-1239.202330199
Yuanyi Luo, Rui Wu, Jiafeng Liu, Xianglong Tang. Multimodal Sentiment Analysis Method for Sentimental Semantic Inconsistency[J]. Journal of Computer Research and Development, 2025 , 62 (2) : 374 -382 . DOI: 10.7544/issn1000-1239.202330199
随着人工智能技术的迅速发展,智能机器逐渐进入大众生活,准确快速地对人类情感进行分析可以使智能机器为人类的日常生活提供更好的服务. 例如在智能汽车系统中,对驾驶员的情感变化进行实时分析,能有效减少意外事故的发生. 人的情感可以通过面部表情、声音等多种模态进行表现,多模态情感分析就是综合多个模态信息对人类情感进行分析决策的一种多模态处理任务[1].
本文的研究对象是以文本、语音、视频3种模态为基础的多模态情感分析任务. 在研究中我们发现,某些场景下不同模态表达的情感语义不一致,甚至相悖. 例如:一个人在悲伤的情况下说“天空真美”,当前场景下的视觉和音频模态表现出消极的情感,而文本表现出积极的情感,此种单模态间语义表达的不一致问题会降低多模态任务分析的准确性. 为直观地从数据方面体现模态情感语义不一致问题对多模态情感分析任务的影响,本文以中文情感数据集CH-SIMS[2]为基础,从2281个样本中筛选出228个包含模态情感语义不一致的样本,并选取2种现有的多模态情感分析模型:基于深度神经网络的后期融合模型(late fusion based on deep neural network,LF-DNN)和多模态转换器 (multimodal transformer,MulT)模型[3]进行对比实验分析,其中LF-DNN模型代码来源于文献[2]提供的可获取开源数据. CH-SIMS数据集包含了多模态标签和单模态标签,有助于分析单模态之间的相关性. 我们将单个样本中某个模态的情感极性与其余任一模态的情感极性相反作为筛选条件,获取情感语义不一致的样本. 分析实验结果如表1所示,其中org表示样本量为2281的原始数据集,其中训练数据为1368个样本、验证数据为456个样本、测试数据为457个样本,训练集、验证集、测试集的样本数比例约为3∶1∶1;(–)d表示从原始数据集中筛选出的228条包含情感语义不一致的样本数据;(re)$ {\mathrm{d}}_{170} $表示将170个情感语义不一致的样本以训练集、验证集、测试集的比例3∶1∶1与(–)d中的训练集、验证集、测试集数据进行替换(保证总样本数不变);(re)$ {\mathrm{d}}_{228} $是将228个情感语义不一致的样本以比例3∶1∶1与(–)d中的训练集、验证集、测试集数据进行替换;(re)$ {\mathrm{d}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}} $是将所有情感语义不一致的样本数据与(–)d中的测试集样本进行替换.
表1中的2分类是将情感分为积极和消极2类,范围分别为[1.0,0]和(0,–1.0];3分类是将情感分为积极、中性、消极,范围分别为[1.0,0.2),[0.2,–0.2],(–0.2,–1.0];MAE和Corr分别表示平均绝对误差(mean absolute error,MAE)和皮尔逊相关系数(Pearson correlation coefficient,Corr). 从表1中可以看出,模态间情感语义不一致的数据会降低多模态情感任务的分析准确度. 在以深度神经网络的后期融合模型LF-DNN为基础的对比实验中,将情感语义不一致的样本数据从原数据集中去除后,实验结果有大幅度提高,但随着将情感语义不一致的样本与其余样本数据逐渐进行替换,实验准确度不断下降,最终在测试集被全部替换后出现了大幅度下跌. 在以多模态转换器模型MulT为基础的对比实验中,表现出的实验现象与LF-DNN相同.
为解决上述问题,本文提出一种多模态情感分析方法以提升模态间情感语义的一致性. 在单模态特征提取方面,现有的多模态情感分析研究只单一地关注模态的时序性或空间性,采用长短期记忆(long short-term memory,LSTM)模型、时序卷积、空间卷积等方法获取模态的时序特征或空间特征表示. 然而视频模态中不仅含有时序信息,而且包含了空间信息[4],获取视频模态的时空信息能得到更具备表现力的特征信息. 由此,本文利用时间注意力和时空注意力分别获取音频模态和视频模态的特征表示. 据我们所知,目前没有多模态情感分析方面的研究单独采用时空注意力对视频模态进行特征提取. 为解决模态间情感语义不一致的问题,获取语义一致的模态特征表示,本文计算单模态间的中心距差异(central moment discrepancy,CMD)[5]作为共性学习损失,学习各模态的共性特征表示,并采用交叉注意力为单模态特征学习过程提供辅助信息. 此外,本文以软注意力机制为基础,提出一种多模态注意力,增大强模态的表达,抑制弱模态对任务的影响.
情感分析被广泛定义为对主观因素的计算研究,其中包括对人的意见、态度和情绪[6]. 多模态情感分析是利用多种形式的主观表达信息对情感进行分析判断的一种情感分析方法,当前的多模态情感分析方法主要针对多模态融合[7-8]和多模态表示进行研究 [9-11].
多模态融合方面,Zadeh等人[7]于2017年提出一种多模态张量融合方法对3种模态进行外积融合,得到的融合结果通过全连接深度网络进行最终的情感分析,该方法不仅能保存单个模态的内部信息,也能学习模态之间的互补信息;Tsai等人[3]于2019年提出一种多模态转换器结构,该结构利用交叉模态注意力将文本、音频、图像3种模态进行交叉融合,使单个模态可以从其他的模态中获取信息. 多模态表示方面,Hazarika等人[9]于2020年提出一种模态共性和特性的表示方法,利用损失函数学习单模态表示之间的共性和特性,将这些模态间的共性表示和特性表示进行融合,减少了模态之间的信息冗余;Rahman等人[10]于2020年提出一种多模态适应门的结构用于多模态模型的微调;Yu等人[11]于2021年设计了一种自监督学习策略的标签生成模块,并以多任务学习的方式分别学习模态间的一致性表示和差异性表示. 上述的多模态情感分析模型都是采用复杂的融合方式形成用于决策的多模态特征表示,没有考虑到模态间情感语义不一致问题对情感分析任务的影响. 本文以交叉注意力和软注意力为基础,解决模态间情感语义不一致的问题,并提高强模态对任务的作用.
在现实生活中,人们会选择性地关注信息的重要部分,同时忽略其他部分,并以不同的重要程度对获取的信息进行处理[12]. 注意力模型最初用于机器翻译领域,目前在多个领域都有大量的应用[13].
空间注意力是一种自适应的空间区域选择机制,在图像、视频分类方面都起到了重要作用. Mnih等人[14]提出循环注意力模型(recurrent attention model,RAM),采用循环神经网络和强化学习的方法使网络学习需要关注空间区域. Zhang等人[15]提出了一种有效的全局注意力模块,该模块可以掌握全局和局部的观测信息,能更好地进行空间注意力学习. 为了可视化图像字幕生成模型应该关注的位置和内容,Xu等人[16]引入了软注意力机制为输入的信息分配权值,做加权平均处理.
时间注意力是一种动态的时序选择机制,它决定了视频中每一帧的重要性[17]. Li等人[18]提出了一种全局局部时间表示来利用自注意力机制获取视频序列中的多尺度时间线索.
Xiang等人[19]于2018年以多种场景下的视频信息为基础,对RGB图、光流图、音频3种模态进行重要性分析,得出在不同场景下各模态体现的重要程度不同. Ghosal等人[20]发现,在多模态任务的研究中,不是合并所有可用的模态都能有利于提高性能,也不是所有的模态都能在训练中发挥平等的作用. 并针对这些问题提出基于多模态注意力框架的循环神经网络结构.
在多模态情感分析领域中,模态表示学习可以被分为单模态表示学习和多模态表示学习. 针对多模态表示学习,现有的研究往往会采用复杂的融合方法将多个模态融合为多模态表示,以获取模态间的交互关系[7-8]. Ye等人[21]采用对比学习的方式获取视觉模态和文本模态之间的交互语义信息,并在细粒度层面上融合2种模态信息. Sun等人[22]提出一种多模态关系表达方法,利用各模态间的典型相关性学习模态间的共享信息,并利用共享信息学习具体的情感语义表达. 针对单模态表示学习,Devlin等人[23]提出预训练文本模型BERT用于提供更好的语言特征表示. Hazarika等人[9]通过将每个模态映射到2个不同的子空间中学习各模态多种类型的特征表示.
本文针对情感分析任务中不同模态间情感语义不一致会对多模态情感分析任务造成影响的问题,提出一种多模态学习模型,学习情感语义表达一致的模态特征表示,该模型的具体结构如图1所示. 模型的具体学习过程如下:$ {\boldsymbol{X}}_{\mathrm{T}} $,$ {\boldsymbol{X}}_{\mathrm{A}} $,$ {\boldsymbol{X}}_{\mathrm{V}} $分别代表3种模态的特征表示,模型利用两两模态的中心距差异值(CMD)作为损失函数对单模态网络的学习过程进行监督,学习到各模态的共性特征表示$ {\boldsymbol{C}}_{\mathrm{T}}$,${\boldsymbol{C}}_{\mathrm{A}} $,${\boldsymbol{C}}_{\mathrm{V}} $;将原始的单模态特征表示$ {\boldsymbol{X}}_{\mathrm{T}} $,$ {\boldsymbol{X}}_{\mathrm{A}} $,$ {\boldsymbol{X}}_{\mathrm{V}} $作为主要特征表示,$ {\boldsymbol{C}}_{\mathrm{T}} $,${\boldsymbol{C}}_{\mathrm{A}} $,${\boldsymbol{C}}_{\mathrm{V}} $作为辅助特征表示输入到交叉注意力模块进行学习,最终的输出由模态注意力进行加权连接.
当前针对多模态情感分析的大多数研究仅单一地关注了模态中的时间信息或者空间信息,忽略了视频模态中包含的时空信息. 为了获取空间中局部与全局的关系与时序特征中时间序列的长依赖关系,本文采用时间注意力和时空注意力分别获取音频模态和视频模态的特征表示,结构如图1所示.
令视频模态的特征表示为$ {\boldsymbol{X}}_{\mathrm{V}}\in {\mathbb{R}}^{d\times F} $,其中$ d $是视频模态的特征维度,$ F $是视频中的图像帧数. 为了获取空间中的局部信息,将每帧图像的特征分解为$ N $个均等长度且不重叠的块$ \boldsymbol{p}\in {\mathbb{R}}^{{d}_{i}\times F} $,使得$ d=N{d}_{i} $. 对每一个块有$ {\boldsymbol{Q}}_{\boldsymbol{p}}={\boldsymbol{W}}_{\boldsymbol{Q}}\boldsymbol{p} $,$ {\boldsymbol{K}}_{\boldsymbol{p}}={\boldsymbol{W}}_{\boldsymbol{K}}\boldsymbol{p} $,$ {\boldsymbol{V}}_{\boldsymbol{p}}={\boldsymbol{W}}_{\boldsymbol{V}}\boldsymbol{p} $,则空间注意力可以表示为
$ {{\alpha }_{\boldsymbol{p}}^{\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{c}\mathrm{e}}}=\dfrac{{{\boldsymbol{Q}}_{\boldsymbol{p}}^{\rm T}}}{\sqrt{{d}_{i}}}\left[{\left\{{\boldsymbol{K}}_{{\boldsymbol{p}}^{n}}\right\}}_{n=1,2,… ,N}\right] ,$
$ {{\boldsymbol s }_{\boldsymbol{p}}^{\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{c}\mathrm{e}}}= \displaystyle\sum_{n = 1}^N {{\alpha }_{\boldsymbol{p}}^{\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{c}\mathrm{e}}}{{\boldsymbol V }_{\boldsymbol{p}^n}^{\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{c}\mathrm{e}}} . $
时间注意力可以表示为
$ {{\alpha }_{(\boldsymbol{p},t)}^{\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}}}=\dfrac{{{\boldsymbol{Q}}_{(\boldsymbol{p},t)}^{\rm T}}}{\sqrt{{d}_{i}}}\left[{\left\{{\boldsymbol{K}}_{(\boldsymbol{p},{t'})}\right\}}_{{t'}=1,2,… ,F}\right] ,$
$ {{\boldsymbol s }_{(\boldsymbol{p},t)}^{\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}}}=\displaystyle\sum\limits_{t' = 1}^F {{\alpha }_{(\boldsymbol{p},t)}^{\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}}}{{\boldsymbol V}_{(\boldsymbol{p},t')}^{\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}}}. $
时空注意力即为上述2种注意力的结合.
CMD可以通过匹配2个模态特征表示的阶数矩差来度量2个模态的分布之间的差异,其相比于KL散度包含了高阶矩信息,相较于最大平均差异(maximum mean discrepancy,MMD)减少了计算量,因此本文采用CMD学习模态间的共性特征表示. 2种模态间的相似性越高,对应的CMD就越小,即
$ {M}_{\mathrm{C}\mathrm{M}\mathrm{D}}\left(\boldsymbol{r},\boldsymbol{q}\right)\to 0\Rightarrow \boldsymbol{r}\stackrel{d}{\to }\boldsymbol{q} ,$
其中$ {M}_{\mathrm{C}\mathrm{M}\mathrm{D}}\left(\boldsymbol{r},\boldsymbol{q}\right) $代表$ \boldsymbol{r}, \boldsymbol{q}$这2个模态分布的中心距差异值,$ \stackrel{d}{\to } $代表在分布上近似. 当$ {M}_{\mathrm{C}\mathrm{M}\mathrm{D}}\left(\boldsymbol{r},\boldsymbol{q}\right)=0 $时,$ \boldsymbol{r},\boldsymbol{q} $这2个模态分布相等,即$ \boldsymbol{r}=\boldsymbol{q} $.
本文研究中采用两两模态之间的CMD值作为损失,用于在单模态表示学习的过程中学习到各模态的共性特征表示. 例如:采用$ {M}_{\mathrm{C}\mathrm{M}\mathrm{D}}(\mathrm{T},\mathrm{A}) $作为文本模态和音频模态的中心距差异值,则文本模态与音频模态的共性学习损失可以表示为$ {L}_{(\mathrm{T},\mathrm{A})}={M}_{\mathrm{C}\mathrm{M}\mathrm{D}}(\mathrm{T},\mathrm{A}) $(中心距差异的计算满足交换性[5],即$ {M}_{\mathrm{C}\mathrm{M}\mathrm{D}}\left(\mathrm{T},\mathrm{A}\right)= {M}_{\mathrm{C}\mathrm{M}\mathrm{D}}(\mathrm{A},\mathrm{T}) $). 在网络反向传播的过程中,随着$ {L}_{(\mathrm{T},\mathrm{A})} $的不断减小,文本模态表示和音频模态表示之间的一致性会不断增加. 与上述相似,在3种模态的表示学习过程中,$ {L}_{(\mathrm{T},\mathrm{A})} $,$ {L}_{(\mathrm{T},\mathrm{V})} $,$ {L}_{(\mathrm{A},\mathrm{V})} $覆盖了所有模态之间的共性学习损失,伴随着这3种损失的不断下降,最终学到的3种单模态共性特征表示为$ {\boldsymbol{C}}_{\mathrm{T}}$,${\boldsymbol{C}}_{\mathrm{A}} $,${\boldsymbol{C}}_{\mathrm{V}} $.
设有$ X $和$ Y $是2个独立且有界的随机样本,在区间$ {\left[a,b\right]}^{N} $上的概率分布为$ \boldsymbol{r} $和$ \boldsymbol{q} $. 则2个样本的中心矩差异为
$ \begin{split} {M}_{\mathrm{C}\mathrm{M}\mathrm{D}}\left(\boldsymbol{r},\boldsymbol{q}\right)=& \dfrac{1}{\left|b-a\right|}{\|{\boldsymbol E}\left(X\right)-{\boldsymbol E}\left(Y\right)\|}_{2}+\\[-2pt]& \displaystyle\sum\limits_{k' = 2}^K \dfrac{1}{|b-a|} {\|c_k(X)-c_k(Y)\|}_2, \end{split}$
其中$ \boldsymbol E\,(X)\,=\,\dfrac{1}{X}\displaystyle\sum\limits_{x \in X} \,x $是$ X $的期望向量,$ {c}_{k}\left(X\right)\,=\,{E}\,(({x}- {E}({X}))^{k}) $是所有第$ k $阶样本的中心矩向量.
本文分别计算3种模态特征表示$ {\boldsymbol{X}}_{\mathrm{T}}$,${\boldsymbol{X}}_{\mathrm{A}} $,${\boldsymbol{X}}_{\mathrm{V}} $两两之间的CMD值,并将其作为损失函数的一部分,学习3种模态的共性特征表示$ {\boldsymbol{C}}_{\mathrm{T}} $,${\boldsymbol{C}}_{\mathrm{A}} $,${\boldsymbol{C}}_{\mathrm{V}} $. 共性学习损失函数的具体表示如下:
$ L=\dfrac{1}{3}\displaystyle\sum\limits_{\begin{subarray}{l} ({m_1},{m_2}) \in \{ ({\rm T,A}), \\ ({\rm T,V}),({\rm A,V})\} \end{subarray}} M_{\rm CMD}(m_1,m_2). $
在单模态表示学习的过程中,为了在保留原始信息的情况下提高共性特征的表达,同时增加模态间的动态交互,本文采用交叉注意力使各模态的特征表示从其余模态的共性特征表示中获取辅助信息.
令文本模态、音频模态、视频模态分别为$ \mathrm{T}$,$\mathrm{A}$,$\mathrm{V} $. $ {\boldsymbol{X}}_{\mathrm{T}} $,$ {\boldsymbol{X}}_{\mathrm{A}} $,$ {\boldsymbol{X}}_{\mathrm{V}} $分别为3种模态的特征表示. $ {\boldsymbol{C}}_{{X}_{\mathrm{T}}} $,$ {\boldsymbol{C}}_{{X}_{\mathrm{A}}} $,$ {\boldsymbol{C}}_{{X}_{\mathrm{V}}} $分别为3种模态的共性特征表示. 将$ {\boldsymbol{X}}_{\mathrm{T}} $作为主特征,$ {\boldsymbol{C}}_{{X}_{\mathrm{A}}} $和$ {\boldsymbol{C}}_{{X}_{\mathrm{V}}} $作为辅助特征($ {\boldsymbol{C}}_{{X}_{\mathrm{A}}} $和$ {\boldsymbol{C}}_{{X}_{\mathrm{V}}} $中既包含了各自模态的原始特征信息,也包含了3种模态间的共性特征信息). 定义$ {\boldsymbol{Q}}_{\mathrm{T}} $=$ {\boldsymbol{X}}_{\mathrm{T}}{\boldsymbol{W}}_{{\boldsymbol{Q}}_{\mathrm{T}}} $,定义$ {\boldsymbol{K}}_{\mathrm{A}}={\boldsymbol{C}}_{{\boldsymbol{X}}_{\mathrm{A}}}{\boldsymbol{W}}_{{\boldsymbol{K}}_{\mathrm{A}}} $,$ {\boldsymbol{V}}_{\mathrm{A}}= {\boldsymbol{C}}_{{\boldsymbol{X}}_{\mathrm{A}}}{\boldsymbol{W}}_{{\boldsymbol{V}}_{\mathrm{A}}} $,$ {\boldsymbol{K}}_{\mathrm{V}}={\boldsymbol{C}}_{{\boldsymbol{X}}_{\mathrm{V}}}{\boldsymbol{W}}_{{\boldsymbol{K}}_{\mathrm{V}}} $,$ {\boldsymbol{V}}_{\mathrm{V}}={\boldsymbol{C}}_{{\boldsymbol{X}}_{\mathrm{V}}}{\boldsymbol{W}}_{{\boldsymbol{V}}_{\mathrm{V}}} $,其中$ \boldsymbol W $为权值系数,$ \boldsymbol{Q} $,$ \boldsymbol{K} $,$ \boldsymbol{V} $分别为特征经线性变换后的输出结果. 基于$ {\boldsymbol{Q}}_{\mathrm{T}},{\boldsymbol{K}}_{\mathrm{A}},{\boldsymbol{V}}_{\mathrm{A}},{\boldsymbol{K}}_{\mathrm{V}},{\boldsymbol{V}}_{\mathrm{V}} $,由音频模态$ \mathrm{A} $和视频模态$ \mathrm{V} $辅助文本模态$ \mathrm{T} $学习的过程可表示为:
$ {y}_{\mathrm{T}}=\dfrac{1}{2}\left({CrossM}_{\mathrm{A}\to \mathrm{T}}+{CrossM}_{\mathrm{V}\to \mathrm{T}}\right) ,$
$ {CrossM}_{\mathrm{A}\to \mathrm{T}}=so ftmax\left(\dfrac{{\boldsymbol{Q}}_{\mathrm{T}}{{\boldsymbol{K}}_{\mathrm{A}}^{\rm T}}}{\sqrt{{dim}_{{\boldsymbol{X}}_{\mathrm{T}}}}}\right){\boldsymbol{V}}_{\mathrm{A}} ,$
$ {CrossM}_{\mathrm{V}\to \mathrm{T}}=so ftmax\left(\dfrac{{\boldsymbol{Q}}_{\mathrm{T}}{{\boldsymbol{K}}_{\mathrm{V}}^{\rm T}}}{\sqrt{{dim}_{{\boldsymbol{X}}_{\mathrm{T}}}}}\right){\boldsymbol{V}}_{\mathrm{V}} ,$
其中$ {y}_{\mathrm{T}} $为文本模态特征提取网络的输出,其余模态的单模态表示学习过程与文本模态相似. $ {dim}_{\boldsymbol{X}} $为输入特征的维度,起到防止特征值过大的作用. $ so ftmax $是归一化指数函数,可表示为:
$ so ftmax(x_i)=\dfrac{{\rm e}^{x_i}}{\displaystyle\sum\limits_{j = 1}^J{\rm e}^{x_j} } . $
本文将情感语义表达一致的单模态特征进行融合,用于后期的任务决策. 为了增大强模态的表达,抑制弱模态的影响,将“软”注意力机制引入多模态融合的过程中,提出一种模态注意力方法对各模态的特征表示进行加权连接. 具体结构如图2所示.
图2中,T*,A*,V*分别为经交叉注意力辅助后的3种模态特征表示. $ {\alpha }_{\mathrm{T}},{\alpha }_{\mathrm{A}},{\alpha }_{\mathrm{V}} $分别为3种模态对任务的贡献度. 单模态对任务的贡献为
$ \boldsymbol{h}=\left(avepool\left[{\boldsymbol{T}}^{*},\;{\boldsymbol{A}}^{*},\;{\boldsymbol{V}}^{*}\right]\right){\boldsymbol{W}}^{3\times 3} ,$
$ \left[{\alpha }_{\mathrm{T}},\;{\alpha }_{\mathrm{A}},\;{\alpha }_{\mathrm{V}}\right]=sigmoid\left(\boldsymbol{h}{\boldsymbol{W}}^{3\times 3}\right) . $
用于任务决策的多模态表示为
$ y=\left[{\alpha }_{\mathrm{T}}{\boldsymbol{T}}^{*},\;{\alpha }_{\mathrm{A}}{\boldsymbol{A}}^{*},\;{\alpha }_{\mathrm{V}}{\boldsymbol{V}}^{*}\right] ,$
其中sigmoid为s型函数$ \dfrac{1}{1+{\mathrm{e}}^{-x}} $,[·]表示特征连接,avepool表示平均池化层.
针对多模态情感分析任务,本文采用的数据集均包含3种模态,分别为文本模态、音频模态、视频模态. 每个数据集的具体描述如下.
1)CMU-MOSI. 该数据集是由Zadeh等人[24]在2016年提出的第1个通过观点进行标注的情感分析数据集,它包括了独白、演讲、电影等多种形式的观点数据,共计93个视频、2198个视频片段. 这些视频片段被人为手工标注至[–3, 3]之间,作为情感评分,“–3”表示十分消极的情感,“3”表示十分积极的情感.
2)CMU-MOSEI. 该数据集是Zadeh等人[25]在2018年针对CMU-MOSI进行改进提出的情感分析数据集. CMU-MOSEI具有更多的样本数量,表达者和主题也具备了更大的多样性. 该数据集包含23453个视频片段,分别来自于5000个不同的视频.
3)CH-SIMS. 该数据集不仅包含了多模态表标签,而且包含了单模态标签,使研究人员可以研究单个模态之间的关系[2]. CH-SIMS包含了2281个视频片段,分别来自于60部不同种类的电影.
为体现本文提出模型的有效性,将本文模型与现有的多模态情感分析模型进行对比. 对比模型如下.
1)张量融合网络(tensor fusion network,TFN)[7]. 该网络通过创建一个多维的张量,端到端学习模态内和模态间的动态信息.
2)低秩多模态融合(low-rank multimodal fusion,LMF)网络[8]. 该网络能减少由张量运算带来的复杂度,并获取模态内和模态间的信息.
3)多模态转换器MulT[3]. 该方法利用交叉模态注意力机制获取多模态序列之间的远程交互,并使单模态可以从其他模态中获取信息.
4)模态的共性和特性表示(modality-invariant and specific representations,MISA)学习模型[9]. 该方法利用损失函数学习各个单模态表示之间的共性特征和特性特征,利用Transformer将学习到的特征进行融合.
5)BERT多模态自适应门限(the multimodal adaptation gate for BERT,MAG-BERT)[10]. 该方法提出了一个基于BERT和XLNet微调模型的多模态自适应门,允许在微调的过程中接受多模态非语言数据.
6)自监督多任务学习(self-supervised multi-task learning,Self-MM)[11]. 该方法设计了一种单模态标签自动生成器,为每个模态提供单独的单模态训练任务.
7)基于多层感知器的情感分析模型CubeMLP(an MLP-based model for multimodal sentiment analysis)[26]. 该方法通过3个独立的MLP单元输入并融合3种模态特征,并将混合的多模态特征以用于任务预测.
本文采用相同的评估标准对模型在CMU-MOSI和CMU-MOSEI这2种数据集上得到的测试结果进行评估:2分类(将数据标签分为积极和消极2类)、5分类(5类状态分别为消极、弱消极、中立、弱积极、积极)、7分类(2种数据集的标签都由人工标注为[–3, 3]的情感状态,其中7类状态由十分消极到十分积极逐渐变化)、F1分数(考虑了召回率和准确率的同等重要性)、MAE和Corr. 对于这些评估指标,除MAE外,值越高表示模型对于情感任务的分析性能越好.
为了获取合适的超参数,本文以验证集的结果为基础,采用网格搜索的方式对超参数进行选择. 最终实验的超参数设置如表2所示.
多模态情感分析的对比实验结果如表3~5所示.
在实验中,本文首先与现有文献提供的数据结果进行对比,其次为了突显本文模型在分类任务上的有效性,增加5分类和7分类实验进行对比. 从实验结果中可以看出,本文提出的多模态情感分析方法在CMU-MOSI,CMU-MOSEI,CH-SIMS这3个数据集中的表现均优于其余模型.
为了更好地利用CH-SIMS数据集包含单模态标签的特点,本文模型采用多任务处理的方式在该数据集上进行评估,即增加单模态训练任务,利用单模态任务提升单模态的特征表达能力,并辅助多模态模型训练. 实验结果如表5所示.
为验证本文模型对解决模态间情感语义不一致问题的有效性,在保留CH-SIMS数据集中训练集和验证集的基础上,从测试集中选出5个包含模态情感语义不一致的样本进行测试,测试结果如表6所示. 其中,“实际值”为标签数据,预测值为模型测试结果;“N38”表示CH-SIMS数据集中的第38个样本数据,以此类推.
从表6中可以看出,在第38和第63个样本中,文本模态和音频模态表现出消极的情感,与视频模态表现出的积极情感语义不一致. 在通过本文模型进行学习后,3种模态的情感语义趋于一致,解决了样本中模态情感语义不一致的问题. 在第732个样本中,视频模态表现出的情感与其余模态的情感不一致,且差距较大,而本文模型能有效利用其余模态对视频模态进行辅助,改变视频模态表达的情感语义,获取情感语义一致的多模态分析数据.
此外,本文采用余弦相似度对比模型训练前后的各模态差异. 余弦相似度的计算方法可由式(15)表示. 对比结果如表7所示.
$ sim({\boldsymbol{A}},{\boldsymbol{B}})=\dfrac{\displaystyle\sum\limits_{i = 1}^n A_iB_i}{ \sqrt {\displaystyle\sum\limits_{i = 1}^n (A_i)^2 } \sqrt {\displaystyle\sum\limits_{i = 1}^n (B_i)^2}} ,$
其中$ \boldsymbol{A} $,$ \boldsymbol{B} $是2个向量,$ sim(\boldsymbol{A},\boldsymbol{B}) $是2个向量的余弦相似度.
从表7中可以看出,本文模型能有效减小模态间的差异,并使不一致的情感语义趋于一致. 本文将情感语义一致的多模态特征数据用于多模态情感分析任务的后期决策.
本文研究中采用两两模态之间的CMD值作为损失对单模态网络的学习过程进行监督,学习各模态的共性特征表示. 为了探究CMD、MMD、KL散度分别对获取共性特征表示模块的影响,本文以MOSI数据集为基础,将MMD,KL散度分别替换CMD方法进行计算,结果如表8所示.
由表8可以看出,CMD由于减少了对核矩阵的计算,因此在运算时间上小于MMD. 此外,使用CMD作为共性特征提取模块的任务表现明显高于其余2种方法MMD和KL,证明了本文选择CMD方法的合理性.
为进一步研究本文模型中单个模块对整体模型的影响,本文以CMU-MOSI数据集为基础,对模型中的模块进行消融分析,结果如表9所示.
首先针对视频模态中时空注意力的使用进行消融分析. 消融实验中仅获取视频模态的时序信息而不获取空间信息. 从结果中看出,获取视频模态的时空信息更有利于任务的分析决策.
其次,针对交叉注意力模块进行消融分析. 消融实验中不利用共性特征进行辅助,而将学习到的共性特征作为输出. 从结果中可以看出,仅使用模态的共性特征表示会丢失大量的互补信息,导致实验结果下降.
最后,本文对提出的模态注意力模块进行消融分析. 消融实验中将各模态特征表示进行无权重连接,经线性变换后作为任务决策的多模态表示结果. 从表9中可以看出,增大强模态的权重并减小弱模态的影响有利于提升多模态任务的分析结果.
本文基于模态间情感语义表达不一致会对多模态情感分析任务造成影响的问题,提出一种多模态学习方法,学习情感语义表达一致的模态特征表示. 首先通过时间注意力和时空注意力分别提取音频模态和视频模态特征. 依据CMD计算量少、不需要计算核矩阵等优点,采用CMD计算模态之间的分布差距,学习各模态的共性特征表示. 利用交叉注意力使单个模态能从其余模态的共性特征表示中获取辅助信息,学习情感语义一致的单模态特征表示. 最后提出模态注意力对单模态特征表示进行加权连接,增大强模态的信息权重,减小弱模态对任务的影响. 本文将提出的多模态情感分析方法与现有的多模态情感分析模型在CMU-MOSI,CMU-MOSEI,CH-SIMS数据集上进行对比,结果显示本文提出的方法在所有评估标准中皆优于对比模型. 在未来的研究中,我们将进一步探索语义不一致问题对其他多模态任务的影响,并将该问题与不平衡的多模态学习问题进行结合.
作者贡献声明:罗渊贻提出了算法思路,完成实验并撰写论文;吴锐提出了实验方案并修改论文;刘家锋、唐降龙提出指导意见并修改论文.

参考文献 引证文献
排序方式:
1
陈龙,管子玉,何金红,等. 情感分类研究进展[J]. 计算机研究与发展,2017,54(6):1150−1170
Chen Long, Guan Ziyu, He Jinhong, et al. A survey on sentiment classification[J]. Journal of Computer Research and Development, 2017, 54(6): 1150−1170 (in Chinese)
2
Yu Wenmeng, Xu Hua, Meng Fanyang, et al. CH-SIMS: A Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality[C]//Proc of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 3718−3727
3
Tsai Y, Bai S, Liang P P, et al. Multimodal transformer for unaligned multimodal language sequences[C]//Proc of the 57th Int Conf on Association for Computational Linguistics. Stroudsburg, PA: ACL, 2019: 6558−6569
4
Bertasius G, Wang Heng, Torresani L. Is space-time attention all you need for video understanding[J]. arXiv preprint, arXiv: 2102.05095, 2021
5
Zellinger W, Grubinger T, Lughofer E, et al. Central moment discrepancy (CMD) for domain-invariant representation learning[J]. arXiv preprint, arXiv: 1702.08811, 2017
6
Sahay S, Kumar S H, Xia Rui, et al. Multimodal relational tensor network for sentiment and emotion classification[C]//Proc of the 1st Int Conf on Association for Computational Linguistics. Stroudsburg, PA: ACL, 2018: 20−27
7
Zadeh A, Chen Minghai, Poria S, et al. Tensor fusion network for multimodal sentiment analysis[C]//Proc of the Int Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2017: 1103−1114
8
Liu Zhun, Shen Ying, Lakshminarasimhan V B, et al. Efficient low-rank multimodal fusion with modality-specific factors[C]//Proc of the Int Conf on Association for Computational Linguistics. Stroudsburg, PA: ACL, 2018: 2247−2256
9
Hazarika D, Zimmermann R, Poria S. MISA: Modality-invariant and -specific representations for multimodal sentiment analysis[J]. arXiv preprint, arXiv: 2005.03545, 2020
10
Rahman W, Hasan M K, Lee S, et al. Integrating multimodal information in large pretrained transformers[C]//Proc of the 58th Conf on Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 2359−2369
11
Yu Wenmeng, Xu Hua, Yuan Ziqi, et al. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis[C]//Proc of the Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2021: 10790−10797
12
程艳,尧磊波,张光河,等. 基于注意力机制的多通道CNN和BiGRU的文本情感倾向性分析[J]. 计算机研究与发展,2020,57(12):2583−2595
Cheng Yan, Yao Leibo, Zhang Guanghe, et al. Text sentiment orientation analysis of multi-channels CNN and BiGRU based on attention mechanism[J]. Journal of Computer Research and Development, 2020, 57(12): 2583−2595 (in Chinese)
13
Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate[J]. arXiv preprint, arXiv: 1409.0473, 2014
14
Mnih V, Heess N, Graves A, et al. Recurrent models of visual attention[C]//Proc of the 27th Int Conf on Neural Information Processing Systems. Cambridge, MA: NeurIPS, 2014: 2204–2212
15
Zhang Zhizheng, Lan Cuiling, Zeng Wenjun, et al. Relation-aware global attention for person re-identification[C]//Proc of the IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 3183–3192
16
Xu K, Ba J L, Kiros R, et al. Show, attend and tell: Neural image caption generation with visual attention[C]//Proc of the Int Conf on Machine Learning. New York: ACM, 2015: 2048−2057
17
Guo Menghao, Xu Tianxing, Liu Jiangjiang, et al. Attention mechanisms in computer vision: A survey[J]. Computational Visual Media, 2022, 8(3): 331−368
18
Li Jianing, Wang Jingdong, Gao Wen, et al. Global–local temporal representations for video person re-identification[C]//Proc of the IEEE Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 3957–3966
19
Xiang Long, Gan Chuang, Melo G, et al. Multimodal keyless attention fusion for video classification[C]//Proc of the 32nd Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2018: 7202−7209
20
Ghosal D, Akhtar M S, Chauhan D, et al. Contextual inter-modal attention for multi-modal sentiment analysis[C]//Proc of the 2018 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2018: 3454−3466
21
Ye Junjie, Zhou Jie, Tian Junfeng, et al. Sentiment-aware multimodal pre-training for multimodal sentiment analysis[J]. Knowledge-Based Systems, 2022, 258: 110021
22
Sun Zhongkai, Sarma P, Sethares W, et al. Learning relationships between text, audio, and video via deep canonical correlation for multimodal language analysis[C]//Proc of the Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 8992−8999
23
Devlin J, Chang Mingwei, Lee K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]//Proc of the Conf on Association for Computational Linguistics. Stroudsburg, PA: ACL, 2019: 4171–4186
24
Zadeh A, Zellers R, Pincus E, et al. MOSI: Multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos[J]. arXiv preprint, arXiv: 1606.06259, 2016
25
Zadeh A, Liang P P, Vanbriesen J, et al. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph[C]//Proc of the 56th Conf on Association for Computational Linguistics. Stroudsburg, PA: ACL, 2018: 2236−2246
26
Sun Hao, Wang Hongyi, Liu Jiaqing, et al. CubeMLP: An MLP-based model for multimodal sentiment analysis and depression estimation[C]//Proc of the 30th ACM Int Conf on Multimedia. New York: ACM, 2022: 3722−3729
2025年第62卷第2期
117
51
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202330199
  • 接收时间:2023-03-27
  • 首发时间:2026-11-10
  • 出版时间:2025-02-01
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2023-03-27
  • 修回日期:2024-01-23
基金
作者信息
    哈尔滨工业大学计算机科学与技术学院 哈尔滨 150006

通讯作者:

吴锐()
参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202330199
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏