Article(id=1325493596878680717, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1325493569112388051, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202550273, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=research-article, receivedDate=1745164800000, receivedDateStr=2025-04-21, revisedDate=1754496000000, revisedDateStr=2025-08-07, acceptedDate=1756742400000, acceptedDateStr=2025-09-02, onlineDate=1793951899975, onlineDateStr=2026-11-06, pubDate=1769875200000, pubDateStr=2026-02-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1793951899975, onlineIssueDateStr=2026-11-06, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1793951899975, creator=13701087609, updateTime=1793951899975, updator=13701087609, issue=Issue{id=1325493569112388051, tenantId=1146029695717560320, journalId=1324759914146529478, year='2026', volume='63', issue='2', pageStart='275', pageEnd='538', issueExtLink='null', onlineDate='null', pubDate='1769875200000', pubDateStr='2026-02-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1793951893355, creator='13701087609', updateTime=1793952511329, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext={EN=IssueExt(id=1325496161167090054, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1325493569112388051, language=EN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=), CN=IssueExt(id=1325496161167090055, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1325493569112388051, language=CN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=)}, issueFiles=null, downloadFileDto=null}, startPage=294, endPage=304, ext={EN=ArticleExt(id=1325493597331665550, articleId=1325493596878680717, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=A Multimodal Adaptive Entity Recognition Method Based on Reinforcement Strategy Feedback, columnId=1325489891789910943, journalTitle=Journal of Computer Research and Development, columnName=Artificial Intelligence, runingTitle=null, highlight=null, articleAbstract=

The core objective of the named entity recognition (NER) is to identify entities with specific semantic categories and types in unstructured text. With the rapid growth of social media, textual information is often accompanied by visual content, forming multimodal data. To improve the accuracy of entity recognition, multimodal NER (MNER) methods exploit semantic information from different modalities fully to achieve the complementary and deep fusion of cross-modal features. However, differences in the representation of modalities may introduce visual noise that interferes with entity recognition. Issues such as entity ambiguity or contextual semantic vagueness within the textual modality complicate recognition. To address these challenges, we propose a multimodal NER method based on a reinforcement strategy feedback with adaptive loss mechanism. First, the method adopts a three-stage chain of thought (COT) reasoning process based on GPT-4o to form a progressive reasoning framework that incorporates the adaptive feedback mechanism in reinforcement learning. The degree of matching between images and text is scored, and the interference of visual noise is effectively filtered using an adaptive decision function. Second, four task-specific loss functions are designed and jointly optimized through an adaptive weighted fusion strategy to alleviate the uncertainty caused by contextual ambiguity. Experiments on two representative public datasets (Twitter-2015 and Twitter-2017) show that the overall F1 scores of our proposed method are 86.45% and 93.80%, respectively, representing a significant improvement on current state-of-the-art baseline models.

, authors=MingHai Jiao, Benhang Fan, Jing Wang, Yuhuai Peng*, authorsList=MingHai Jiao, Benhang Fan, Jing Wang, Yuhuai Peng, authorCompany=null, correspAuthors=Yuhuai Peng, authorNote=

born in 1973. PhD, associate professor, master supervisor. Member of CCF. His main research interests include artificial intelligence, image recognition, and big data parsing and optimization

born in 2002. Master candidate. His main research interests include machine learning and multimodal information extraction

born in 1992. PhD candidate. Her main research interests include Internet of things and digital twin

born in 1983. PhD, professor, PhD supervisor. His main research interests include Internet of things, cyber physical systems, and intelligent information networks

, correspAuthorsNote=null, copyrightStatement=Copyright © 2026 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1325493601765044897, articleId=1325493596878680717, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=基于强化策略反馈的多模态自适应实体识别方法, columnId=1325489892104483745, journalTitle=计算机研究与发展, columnName=人工智能, runingTitle=null, highlight=null, articleAbstract=

命名实体识别(named entity recognition,NER)的核心目标是从非结构化文本中识别出具有特定语义类别的实体与类型。随着社交媒体的迅速发展,文本信息往往与视觉信息共同出现,形成多模态内容。为了提升实体识别的准确性,多模态命名实体识别(multi-modal NER,MNER)方法利用不同模态中的语义信息,实现信息互补与深度融合。然而,不同模态之间的表征差异可能引入视觉噪声,干扰实体识别。文本模态中存在实体指代不清或上下文语义模糊的问题,增加了识别难度。针对上述问题,提出了一种基于强化策略反馈与自适应损失机制的MNER方法。首先,该方法采用基于 GPT-4o的3阶段思维链(chain of thought,COT)推理流程,形成渐进式推理框架,融合强化学习中的自适应反馈机制,对图像与文本之间的匹配程度进行评分,并利用自适应决策函数有效过滤视觉噪声的干扰。其次,设计了4类面向具体任务的损失函数,并利用自适应加权融合策略进行优化,以缓解上下文模糊带来的识别不确定性。在2个公开数据集Twitter-2015和Twitter-2017上开展实验,结果表明所提方法的总体F1分数分别达到86.45%和93.80%,显著优于当前主流基线模型。

, authors=焦明海, 樊本航, 王静, 彭玉怀*, authorsList=焦明海, 樊本航, 王静, 彭玉怀, authorCompany=null, correspAuthors=彭玉怀, authorNote=

1973年生。博士,副教授,硕士生导师。CCF会员。主要研究方向为人工智能、图像识别、大数据分析与优化

2002年生。硕士研究生。主要研究方向为机器学习、多模态信息抽取

1992年生。博士研究生。主要研究方向为物联网、数字孪生

1983年生。博士,教授,博士生导师。主要研究方向为物联网、网络物理系统、智能信息网络

, correspAuthorsNote=
彭玉怀()
, copyrightStatement=版权所有 © 2026 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=rMbu40HtVb6tduRBB116vg==, magXml=qLYTS/yLwqAntfssIT/osw==, pdfUrl=null, pdf=2ln/VPr5vOB1AYX0aklEEw==, pdfFileSize=1313352, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=bU+xLoqBbHT+BrkKlw69RA==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=3jSczXoTm6sDfO0L5vBfsg==, mapNumber=null, fund=null)}, authors=[Author(id=1325493602029286054, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=mjiao@mail.neu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325493602096394920, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, authorId=1325493602029286054, language=EN, stringName=MingHai Jiao, firstName=MingHai, middleName=null, lastName=Jiao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=School of Computer Science and Engineering, Northeastern University, Shenyang 110167, bio={"content":"

born in 1973. PhD, associate professor, master supervisor. Member of CCF. His main research interests include artificial intelligence, image recognition, and big data parsing and optimization

"}, bioImg=null, bioContent=

born in 1973. PhD, associate professor, master supervisor. Member of CCF. His main research interests include artificial intelligence, image recognition, and big data parsing and optimization

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325493602163503785, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, authorId=1325493602029286054, language=CN, stringName=焦明海, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=东北大学计算机科学与工程学院 沈阳 110167, bio={"img":"UwOZVa7etpVCfLzbeBcRDw==","content":"

1973年生。博士,副教授,硕士生导师。CCF会员。主要研究方向为人工智能、图像识别、大数据分析与优化

"}, bioImg=UwOZVa7etpVCfLzbeBcRDw==, bioContent=

1973年生。博士,副教授,硕士生导师。CCF会员。主要研究方向为人工智能、图像识别、大数据分析与优化

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325493601957982882, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, xref=null, ext=[AuthorCompanyExt(id=1325493601962177187, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, companyId=1325493601957982882, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=School of Computer Science and Engineering, Northeastern University, Shenyang 110167), AuthorCompanyExt(id=1325493601970565796, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, companyId=1325493601957982882, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=东北大学计算机科学与工程学院 沈阳 110167)])]), Author(id=1325493602218029739, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325493602306110125, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, authorId=1325493602218029739, language=EN, stringName=Benhang Fan, firstName=Benhang, middleName=null, lastName=Fan, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=School of Computer Science and Engineering, Northeastern University, Shenyang 110167, bio={"content":"

born in 2002. Master candidate. His main research interests include machine learning and multimodal information extraction

"}, bioImg=null, bioContent=

born in 2002. Master candidate. His main research interests include machine learning and multimodal information extraction

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325493602364830382, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, authorId=1325493602218029739, language=CN, stringName=樊本航, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=东北大学计算机科学与工程学院 沈阳 110167, bio={"img":"CyOx8FhSOI1lQPEFXDgHXg==","content":"

2002年生。硕士研究生。主要研究方向为机器学习、多模态信息抽取

"}, bioImg=CyOx8FhSOI1lQPEFXDgHXg==, bioContent=

2002年生。硕士研究生。主要研究方向为机器学习、多模态信息抽取

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325493601957982882, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, xref=null, ext=[AuthorCompanyExt(id=1325493601962177187, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, companyId=1325493601957982882, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=School of Computer Science and Engineering, Northeastern University, Shenyang 110167), AuthorCompanyExt(id=1325493601970565796, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, companyId=1325493601957982882, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=东北大学计算机科学与工程学院 沈阳 110167)])]), Author(id=1325493602629071536, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325493604164186802, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, authorId=1325493602629071536, language=EN, stringName=Jing Wang, firstName=Jing, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=School of Computer Science and Engineering, Northeastern University, Shenyang 110167, bio={"content":"

born in 1992. PhD candidate. Her main research interests include Internet of things and digital twin

"}, bioImg=null, bioContent=

born in 1992. PhD candidate. Her main research interests include Internet of things and digital twin

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325493604256461491, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, authorId=1325493602629071536, language=CN, stringName=王静, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=东北大学计算机科学与工程学院 沈阳 110167, bio={"img":"wzlGDrcS/L9aT3GIgFM44w==","content":"

1992年生。博士研究生。主要研究方向为物联网、数字孪生

"}, bioImg=wzlGDrcS/L9aT3GIgFM44w==, bioContent=

1992年生。博士研究生。主要研究方向为物联网、数字孪生

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325493601957982882, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, xref=null, ext=[AuthorCompanyExt(id=1325493601962177187, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, companyId=1325493601957982882, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=School of Computer Science and Engineering, Northeastern University, Shenyang 110167), AuthorCompanyExt(id=1325493601970565796, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, companyId=1325493601957982882, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=东北大学计算机科学与工程学院 沈阳 110167)])]), Author(id=1325493604352930485, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=pengyuhuai@mail.neu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1325493604441010871, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, authorId=1325493604352930485, language=EN, stringName=Yuhuai Peng, firstName=Yuhuai, middleName=null, lastName=Peng, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=School of Computer Science and Engineering, Northeastern University, Shenyang 110167, bio={"content":"

born in 1983. PhD, professor, PhD supervisor. His main research interests include Internet of things, cyber physical systems, and intelligent information networks

"}, bioImg=null, bioContent=

born in 1983. PhD, professor, PhD supervisor. His main research interests include Internet of things, cyber physical systems, and intelligent information networks

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325493604529091256, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, authorId=1325493604352930485, language=CN, stringName=彭玉怀, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=东北大学计算机科学与工程学院 沈阳 110167, bio={"img":"Qf2UWsdjq7oM2Hlf99se6w==","content":"

1983年生。博士,教授,博士生导师。主要研究方向为物联网、网络物理系统、智能信息网络

"}, bioImg=Qf2UWsdjq7oM2Hlf99se6w==, bioContent=

1983年生。博士,教授,博士生导师。主要研究方向为物联网、网络物理系统、智能信息网络

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325493601957982882, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, xref=null, ext=[AuthorCompanyExt(id=1325493601962177187, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, companyId=1325493601957982882, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=School of Computer Science and Engineering, Northeastern University, Shenyang 110167), AuthorCompanyExt(id=1325493601970565796, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, companyId=1325493601957982882, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=东北大学计算机科学与工程学院 沈阳 110167)])])], keywords=[Keyword(id=1325493604633948857, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=EN, orderNo=1, keyword=named entity recognition), Keyword(id=1325493604692669114, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=EN, orderNo=2, keyword=unstructured text), Keyword(id=1325493604772360891, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=EN, orderNo=3, keyword=multimodal data), Keyword(id=1325493604843664060, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=EN, orderNo=4, keyword=reinforcement learning), Keyword(id=1325493604902384317, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=EN, orderNo=5, keyword=adaptive loss), Keyword(id=1325493604973687486, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=EN, orderNo=6, keyword=chain of thought), Keyword(id=1325493605028213439, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=CN, orderNo=1, keyword=命名实体识别), Keyword(id=1325493605112099520, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=CN, orderNo=2, keyword=非结构化文本), Keyword(id=1325493605179208385, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=CN, orderNo=3, keyword=多模态数据), Keyword(id=1325493605250511554, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=CN, orderNo=4, keyword=强化学习), Keyword(id=1325493605334397635, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=CN, orderNo=5, keyword=自适应损失), Keyword(id=1325493605405700804, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=CN, orderNo=6, keyword=思维链)], refs=[Reference(id=1325493610736661207, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=Mai Chengcheng, Liu Jian, Qiu Mengchuan, et al. Pronounce differently, mean differently: A multi-tagging-scheme learning method for Chinese NER integrated with lexicon and phonetic features[J]. Information Processing & Management, 2022, 59(5): 103041, articleTitle=null, refAbstract=null), Reference(id=1325493610820547289, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=He Qizhen, Wu Liang, Yin Yida, et al. Knowledge-graph augmented word representations for named entity recognition[C]//Proc of the 34th Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 7919−7926, articleTitle=null, refAbstract=null), Reference(id=1325493610908627674, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=王嘉诚, 王凯, 王昊奋, 等. 面向远程监督命名实体识别等噪声检测[J]. 计算机研究与发展, 2024, 61(4): 916−928, articleTitle=null, refAbstract=null), Reference(id=1325493610992513755, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Jiacheng, Wang Kai, Wang Haofen, et al. Noise detection for distant supervised named entity recongnition[J]. Journal of Computer Research and Development, 2024, 61(4): 916−928 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1325493611097371356, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Arshad O, Gallo I, Nawaz S, et al. Aiding intra-text representations with visual context for multimodal named entity recognition[C]//Proc of the IEEE Int Conf on Document Analysis and Recognition (ICDAR). Piscataway, NJ: IEEE, 2019: 337−342, articleTitle=null, refAbstract=null), Reference(id=1325493611172868829, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Xin, Yuan Jingling, Li Lin, et al. Reducing the bias of visual objects in multimodal named entity recognition[C]//Proc of the 16th ACM Int Conf on Web Search and Data Mining. New York: ACM, 2023: 958−966, articleTitle=null, refAbstract=null), Reference(id=1325493611277726430, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Wei J, Wang X, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models[J]. Advances in Neural Information Processing Systems, 2022, 35: 24824−24837, articleTitle=null, refAbstract=null), Reference(id=1325493611357418207, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Mnih V, Kavukcuoglu K, Silver D, et al. Playing Atari with deep reinforcement learning[J]. arXiv preprint, arXiv: 1312.5602, 2013, articleTitle=null, refAbstract=null), Reference(id=1325493613039334112, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Moon S, Neves L, Carvalho V. Multimodal named entity recognition for short social media posts[J]. arXiv preprint, arXiv: 1802.07862, 2018, articleTitle=null, refAbstract=null), Reference(id=1325493613190329057, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Qi, Fu Jinlan, Liu Xiaoyu, et al. Adaptive co-attention network for named entity recognition in tweets[C]//Proc of the 32nd Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2018: 5674−5681, articleTitle=null, refAbstract=null), Reference(id=1325493613270020834, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Lu D, Neves L, Carvalho V, et al. Visual attention model for name tagging in multimodal social media[C]//Proc of the 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2018: 1990−1999, articleTitle=null, refAbstract=null), Reference(id=1325493613542650595, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Yu Jianfei, Jiang Jing, Yang Li, et al. Improving multimodal named entity recognition via entity span detection with unified multimodal transformer[C]//Proc of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 3342−3352, articleTitle=null, refAbstract=null), Reference(id=1325493613613953764, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Dong, Wei Suzhong, Li Shoushan, et al. Multi-modal graph fusion for named entity recognition with targeted visual guidance[C]//Proc of the 35th Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2021: 14347−14355, articleTitle=null, refAbstract=null), Reference(id=1325493613685256933, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Xu Bo, Huang Shizhou, Sha Chaofeng, et al. MAF: A general matching and alignment framework for multimodal named entity recognition[C]//Proc of the 15th ACM Int Conf on Web Search and Data Mining. New York: ACM, 2022: 1215−1223, articleTitle=null, refAbstract=null), Reference(id=1325493613760754406, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Bhattacharjee A, Karami M, Liu H. Text transformations in contrastive self-supervised learning: A review[J]. arXiv preprint, arXiv: 2203.12000, 2022, articleTitle=null, refAbstract=null), Reference(id=1325493614146630375, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Xiang, Zhang Ningyu, Li Lei, et al. Good visual guidance makes a better extractor: Hierarchical visual prefix for multimodal entity and relation extraction[J]. arXiv preprint, arXiv: 2205.03521, 2022, articleTitle=null, refAbstract=null), Reference(id=1325493614255682280, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Sun Lin, Wang Jiquan, Zhang Kai, et al. RpBERT: A text-image relation propagation-based BERT model for multimodal NER[C]//Proc of the 35th Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2021: 13860−13868, articleTitle=null, refAbstract=null), Reference(id=1325493614356345577, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhou Baohang, Zhang Ying, Song Kehui, et al. A span-based multimodal variational autoencoder for semi-supervised multimodal named entity recognition[C]//Proc of the 2022 Conf on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg, PA: ACL, 2022: 6293−6302, articleTitle=null, refAbstract=null), Reference(id=1325493614431843050, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Sang E F, Veenstra J. Representing text chunks[C]//Proc of the 9th Conf on European Chapter of the Association for Computational Linguistics (EACL). Stroudsburg, PA: ACL, 1999: 173−179, articleTitle=null, refAbstract=null), Reference(id=1325493614507340523, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Brown T, Mann B, Ryder N, et al. Language models are few-shot learners[J]. Advances in Neural Information Processing Systems, 2020, 33: 1877−1901, articleTitle=null, refAbstract=null), Reference(id=1325493614570255084, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Devlin J, Chang Mingwei, Lee K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]//Proc of the Annual Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, PA: ACL, 2019: 4171−4186, articleTitle=null, refAbstract=null), Reference(id=1325493614645752557, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Yao Ruilin, Xiong Shengwu, Zhao Yichen, et al. Visualgrounding with multi-model conditional adaptation[C]//Proc of the 32nd ACM Int Conf on Multimedia. New York: ACM, 2024: 3877−3886, articleTitle=null, refAbstract=null), Reference(id=1325493614708667118, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Zhiheng, Xu Wei, Yu Kai. Bidirectional LSTM-CRF models for sequence tagging[J]. arXiv preprint, arXiv: 1508.01991, 2015, articleTitle=null, refAbstract=null), Reference(id=1325493614771581679, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=Ma Xuezhe, Hovy E. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF[J]. arXiv preprint, arXiv: 1603.01354, 2016, articleTitle=null, refAbstract=null), Reference(id=1325493614830301936, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Feng, Liu Jiajia, Ji Kaixiang, et al. Learning implicit entity-object relations by bidirectional generative alignment for multimodal NER[C]//Proc of the 31st ACM Int Conf on Multimedia. New York: ACM, 2023: 4555−4563, articleTitle=null, refAbstract=null), Reference(id=1325493614897410801, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Jinyuan, Li Han, Pan Zhuo, et al. Prompting ChatGPT in MNER: Enhanced multimodal named entity recognition with auxiliary refined knowledge[J]. arXiv preprint, arXiv: 2305.12212, 2023, articleTitle=null, refAbstract=null), Reference(id=1325493614960325362, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=26, authorNames=null, journalName=null, refType=null, unstructuredReference=Zheng Zihao, Zhang Zihan, Wang Zexin, et al. Decompose, prioritize, and eliminate: Dynamically integrating diverse representations for multimodal named entity recognition[C]//Proc of the 2024 Joint Int Conf on Computational Linguistics, Language Resources and Evaluation (LREC-COLING’ 2024). Stroudsburg, PA: ACL, 2024: 4498−4508, articleTitle=null, refAbstract=null), Reference(id=1325493615023239923, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=27, authorNames=null, journalName=null, refType=null, unstructuredReference=Bao Xigang, Tian Mengyuan, Zha Zhiyuan, et al. MPMRC-MNER: A unified MRC framework for multimodal named entity recognition based multimodal prompt[C]//Proc of the 32nd ACM Int Conf on Information and Knowledge Management. New York: ACM, 2023: 47−56, articleTitle=null, refAbstract=null), Reference(id=1325493615086154484, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=28, rfOrder=28, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Zhengjie, Zhang Yu, Mi S. Assisting multimodal named entity recognition by cross-modal auxiliary tasks[J]. Pattern Recognition Letters, 2023, 175: 52−58, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1325493601957982882, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, xref=null, ext=[AuthorCompanyExt(id=1325493601962177187, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, companyId=1325493601957982882, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=School of Computer Science and Engineering, Northeastern University, Shenyang 110167), AuthorCompanyExt(id=1325493601970565796, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, companyId=1325493601957982882, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=东北大学计算机科学与工程学院 沈阳 110167)])], figs=[ArticleFig(id=1325493605539918533, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=EN, label=Figure 1, caption=Image information assisted recognition, figureFileSmall=+kj5STqVQHFgud7Dz9gsLw==, figureFileBig=bU+xLoqBbHT+BrkKlw69RA==, tableContent=null), ArticleFig(id=1325493605615416006, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=CN, label=图1, caption=图像信息辅助识别, figureFileSmall=+kj5STqVQHFgud7Dz9gsLw==, figureFileBig=bU+xLoqBbHT+BrkKlw69RA==, tableContent=null), ArticleFig(id=1325493609285432007, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=EN, label=Figure 2, caption=The structure of RFAL, figureFileSmall=HLg0hPHpmMOCd38pnLQUEA==, figureFileBig=Mrk03GUyhJPoX/2V8feJUw==, tableContent=null), ArticleFig(id=1325493609352540872, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=CN, label=图2, caption=RFAL结构, figureFileSmall=HLg0hPHpmMOCd38pnLQUEA==, figureFileBig=Mrk03GUyhJPoX/2V8feJUw==, tableContent=null), ArticleFig(id=1325493609432232649, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=EN, label=Figure 3, caption=Comparison of experimental results on two datasets, figureFileSmall=L1x2uPX7XvZkFDRFrH1ftA==, figureFileBig=BHjXlIqDxfMx5TSE6mzBuw==, tableContent=null), ArticleFig(id=1325493609520313034, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=CN, label=图3, caption=2个数据集上的实验结果对比, figureFileSmall=L1x2uPX7XvZkFDRFrH1ftA==, figureFileBig=BHjXlIqDxfMx5TSE6mzBuw==, tableContent=null), ArticleFig(id=1325493609612587723, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=EN, label=Table 1, caption=Statistical Data of Two Datasets, figureFileSmall=null, figureFileBig=null, tableContent=
类型Twitter-2015Twitter-2017
训练集验证集测试集训练集验证集测试集
PER221755218162943626621
LOC20915221697731173178
ORG9282478391674357395
MISC940225726701150157
总计617615465078604913241351
推文数量4000100032573373723723
), ArticleFig(id=1325493609692279500, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=CN, label=表1, caption=

2个数据集的统计数据

, figureFileSmall=null, figureFileBig=null, tableContent=
类型Twitter-2015Twitter-2017
训练集验证集测试集训练集验证集测试集
PER221755218162943626621
LOC20915221697731173178
ORG9282478391674357395
MISC940225726701150157
总计617615465078604913241351
推文数量4000100032573373723723
), ArticleFig(id=1325493609771971277, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=EN, label=Table 2, caption=Training Hyper-Parameters, figureFileSmall=null, figureFileBig=null, tableContent=
超参数取值
batch_size4
epoch6
learning_rate1E−5
warmup_ratio0.03
bf16True
), ArticleFig(id=1325493609847468750, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=CN, label=表2, caption=

训练的超参数

, figureFileSmall=null, figureFileBig=null, tableContent=
超参数取值
batch_size4
epoch6
learning_rate1E−5
warmup_ratio0.03
bf16True
), ArticleFig(id=1325493609914577615, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=EN, label=Table 3, caption=Comparison of Experimental Results %, figureFileSmall=null, figureFileBig=null, tableContent=
方法Twitter-2015Twitter-2017
PERLOCORGMISCPRF1PERLOCORGMISCPRF1
注:加黑数值表示在当前指标下的最优值。
BiLSTM-CRF76.7772.5641.3326.868.1461.0964.4285.1272.6872.5052.5679.4273.4376.31
CNN-BiLSTM-CRF80.8675.3947.7732.6166.2468.0967.1587.9977.4474.0260.828078.7679.37
BERT-CRF84.7480.5160.2737.2969.2274.5971.8190.2583.0581.1362.2183.3283.5783.44
MAF84.6781.1863.3541.8271.8675.1073.4291.5185.8085.1068.7986.1386.3886.25
BGA-MNER86.883.6263.642.6578.6074.1676.3193.7185.5585.7171.0587.7187.7187.71
PGIM88.3484.2270.1552.3479.2179.4579.3396.4689.8989.0379.6290.8692.0191.43
DPE-MNER87.3184.3665.9348.4876.8678.2777.5692.3787.4786.3373.8388.4687.3487.9
MPMRC-MNER85.8883.0666.642.9977.1575.3976.2692.8086.8083.4072.7987.187.1687.13
CMAT86.1181.6263.5341.1873.2275.6374.3992.0286.1285.2570.2087.386.8987.09
RFAL(本文)90.4586.9170.0155.2888.8584.1986.4596.8288.2789.8575.3894.7292.8993.80
), ArticleFig(id=1325493609994269392, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=CN, label=表3, caption=

实验结果对比

, figureFileSmall=null, figureFileBig=null, tableContent=
方法Twitter-2015Twitter-2017
PERLOCORGMISCPRF1PERLOCORGMISCPRF1
注:加黑数值表示在当前指标下的最优值。
BiLSTM-CRF76.7772.5641.3326.868.1461.0964.4285.1272.6872.5052.5679.4273.4376.31
CNN-BiLSTM-CRF80.8675.3947.7732.6166.2468.0967.1587.9977.4474.0260.828078.7679.37
BERT-CRF84.7480.5160.2737.2969.2274.5971.8190.2583.0581.1362.2183.3283.5783.44
MAF84.6781.1863.3541.8271.8675.1073.4291.5185.8085.1068.7986.1386.3886.25
BGA-MNER86.883.6263.642.6578.6074.1676.3193.7185.5585.7171.0587.7187.7187.71
PGIM88.3484.2270.1552.3479.2179.4579.3396.4689.8989.0379.6290.8692.0191.43
DPE-MNER87.3184.3665.9348.4876.8678.2777.5692.3787.4786.3373.8388.4687.3487.9
MPMRC-MNER85.8883.0666.642.9977.1575.3976.2692.8086.8083.4072.7987.187.1687.13
CMAT86.1181.6263.5341.1873.2275.6374.3992.0286.1285.2570.2087.386.8987.09
RFAL(本文)90.4586.9170.0155.2888.8584.1986.4596.8288.2789.8575.3894.7292.8993.80
), ArticleFig(id=1325493610082349777, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=EN, label=Table 4, caption=Ablation Experiments of Model Structure %, figureFileSmall=null, figureFileBig=null, tableContent=
方法Twitter-2015Twitter-2017
PRF1PRF1
w/o I&A88.6379.6583.993.8487.9390.79
w/o INF88.7382.4385.4795.1892.0893.6
w/o AMLO89.9980.1584.7895.6790.0192.75
RFAL(本文)88.8584.1986.4594.7292.8993.8
), ArticleFig(id=1325493610187207378, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=CN, label=表4, caption=

模型结构消融实验

, figureFileSmall=null, figureFileBig=null, tableContent=
方法Twitter-2015Twitter-2017
PRF1PRF1
w/o I&A88.6379.6583.993.8487.9390.79
w/o INF88.7382.4385.4795.1892.0893.6
w/o AMLO89.9980.1584.7895.6790.0192.75
RFAL(本文)88.8584.1986.4594.7292.8993.8
), ArticleFig(id=1325493610271093459, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=EN, label=Table 5, caption=Ablation Experiments of Different Components in an Adaptive Multi-Type Loss Optimization Module %, figureFileSmall=null, figureFileBig=null, tableContent=
方法Twitter-2015Twitter-2017
PRF1PRF1
注:INF表示只使用图文噪声过滤层,INF+L1,L2,L3分别表示在图文噪声过滤层的基础上加入自适应多类型损失优化层中的焦点交叉熵损失函数、标签平滑的KL损失函数、隐式边界一致性损失函数。RFAL是在INF+L1+L2+L3的基础上加入类似对比学习的损失函数。
INF89.9980.1584.7895.6790.0192.75
INF+L189.1881.7485.3095.0590.9092.92
INF+L1+L289.2282.5985.7895.2391.7193.44
INF+L1+L2+L389.2183.2886.1494.9391.4993.18
RFAL(本文)88.8584.1986.4594.7292.8993.8
), ArticleFig(id=1325493610354979540, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=CN, label=表5, caption=

自适应多类型损失优化模块中各组件消融实验

, figureFileSmall=null, figureFileBig=null, tableContent=
方法Twitter-2015Twitter-2017
PRF1PRF1
注:INF表示只使用图文噪声过滤层,INF+L1,L2,L3分别表示在图文噪声过滤层的基础上加入自适应多类型损失优化层中的焦点交叉熵损失函数、标签平滑的KL损失函数、隐式边界一致性损失函数。RFAL是在INF+L1+L2+L3的基础上加入类似对比学习的损失函数。
INF89.9980.1584.7895.6790.0192.75
INF+L189.1881.7485.3095.0590.9092.92
INF+L1+L289.2282.5985.7895.2391.7193.44
INF+L1+L2+L389.2183.2886.1494.9391.4993.18
RFAL(本文)88.8584.1986.4594.7292.8993.8
), ArticleFig(id=1325493610451448533, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=EN, label=Table 6, caption=Entity Prediction, figureFileSmall=null, figureFileBig=null, tableContent=
方法视觉信息重要性图像噪声干扰问题
(a) [Sergio Ramos]1:
[Real Madrid]2 as
motivated as two years ago
(b) My [Future Hearts]1
Tour concert was
one year ago today
注:“√”表示该类型识别正确,“×”表示该类型识别错误。
BERT-CRF1-PER√2-MISC×1-MISC√
MAF1-PER√2-ORG√1-PER×
RFAL(本文)1-PER√2-ORG√1-MISC√
), ArticleFig(id=1325493610606637782, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, language=CN, label=表6, caption=

实体预测

, figureFileSmall=null, figureFileBig=null, tableContent=
方法视觉信息重要性图像噪声干扰问题
(a) [Sergio Ramos]1:
[Real Madrid]2 as
motivated as two years ago
(b) My [Future Hearts]1
Tour concert was
one year ago today
注:“√”表示该类型识别正确,“×”表示该类型识别错误。
BERT-CRF1-PER√2-MISC×1-MISC√
MAF1-PER√2-ORG√1-PER×
RFAL(本文)1-PER√2-ORG√1-MISC√
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202550273, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202550273, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202550273, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202550273, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1769875200000, fullTextJson=null, articleText=null, reference=null)
收藏切换
基于强化策略反馈的多模态自适应实体识别方法
焦明海 , 樊本航 , 王静 , 彭玉怀 *
计算机研究与发展 | 人工智能 2026,63(2): 294-304
收起
收藏切换
计算机研究与发展 |人工智能 2026 , 63 (2) : 294 -304
基于强化策略反馈的多模态自适应实体识别方法
全屏
[Author(id=1325493602029286054, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=mjiao@mail.neu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325493602096394920, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, authorId=1325493602029286054, language=EN, stringName=MingHai Jiao, firstName=MingHai, middleName=null, lastName=Jiao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=School of Computer Science and Engineering, Northeastern University, Shenyang 110167, bio={"content":"

born in 1973. PhD, associate professor, master supervisor. Member of CCF. His main research interests include artificial intelligence, image recognition, and big data parsing and optimization

"}, bioImg=null, bioContent=

born in 1973. PhD, associate professor, master supervisor. Member of CCF. His main research interests include artificial intelligence, image recognition, and big data parsing and optimization

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325493602163503785, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, authorId=1325493602029286054, language=CN, stringName=焦明海, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=东北大学计算机科学与工程学院 沈阳 110167, bio={"img":"UwOZVa7etpVCfLzbeBcRDw==","content":"

1973年生。博士,副教授,硕士生导师。CCF会员。主要研究方向为人工智能、图像识别、大数据分析与优化

"}, bioImg=UwOZVa7etpVCfLzbeBcRDw==, bioContent=

1973年生。博士,副教授,硕士生导师。CCF会员。主要研究方向为人工智能、图像识别、大数据分析与优化

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325493601957982882, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, xref=null, ext=[AuthorCompanyExt(id=1325493601962177187, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, companyId=1325493601957982882, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=School of Computer Science and Engineering, Northeastern University, Shenyang 110167), AuthorCompanyExt(id=1325493601970565796, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, companyId=1325493601957982882, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=东北大学计算机科学与工程学院 沈阳 110167)])]), Author(id=1325493602218029739, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325493602306110125, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, authorId=1325493602218029739, language=EN, stringName=Benhang Fan, firstName=Benhang, middleName=null, lastName=Fan, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=School of Computer Science and Engineering, Northeastern University, Shenyang 110167, bio={"content":"

born in 2002. Master candidate. His main research interests include machine learning and multimodal information extraction

"}, bioImg=null, bioContent=

born in 2002. Master candidate. His main research interests include machine learning and multimodal information extraction

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325493602364830382, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, authorId=1325493602218029739, language=CN, stringName=樊本航, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=东北大学计算机科学与工程学院 沈阳 110167, bio={"img":"CyOx8FhSOI1lQPEFXDgHXg==","content":"

2002年生。硕士研究生。主要研究方向为机器学习、多模态信息抽取

"}, bioImg=CyOx8FhSOI1lQPEFXDgHXg==, bioContent=

2002年生。硕士研究生。主要研究方向为机器学习、多模态信息抽取

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325493601957982882, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, xref=null, ext=[AuthorCompanyExt(id=1325493601962177187, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, companyId=1325493601957982882, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=School of Computer Science and Engineering, Northeastern University, Shenyang 110167), AuthorCompanyExt(id=1325493601970565796, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, companyId=1325493601957982882, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=东北大学计算机科学与工程学院 沈阳 110167)])]), Author(id=1325493602629071536, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325493604164186802, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, authorId=1325493602629071536, language=EN, stringName=Jing Wang, firstName=Jing, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=School of Computer Science and Engineering, Northeastern University, Shenyang 110167, bio={"content":"

born in 1992. PhD candidate. Her main research interests include Internet of things and digital twin

"}, bioImg=null, bioContent=

born in 1992. PhD candidate. Her main research interests include Internet of things and digital twin

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325493604256461491, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, authorId=1325493602629071536, language=CN, stringName=王静, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=东北大学计算机科学与工程学院 沈阳 110167, bio={"img":"wzlGDrcS/L9aT3GIgFM44w==","content":"

1992年生。博士研究生。主要研究方向为物联网、数字孪生

"}, bioImg=wzlGDrcS/L9aT3GIgFM44w==, bioContent=

1992年生。博士研究生。主要研究方向为物联网、数字孪生

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325493601957982882, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, xref=null, ext=[AuthorCompanyExt(id=1325493601962177187, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, companyId=1325493601957982882, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=School of Computer Science and Engineering, Northeastern University, Shenyang 110167), AuthorCompanyExt(id=1325493601970565796, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, companyId=1325493601957982882, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=东北大学计算机科学与工程学院 沈阳 110167)])]), Author(id=1325493604352930485, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=pengyuhuai@mail.neu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1325493604441010871, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, authorId=1325493604352930485, language=EN, stringName=Yuhuai Peng, firstName=Yuhuai, middleName=null, lastName=Peng, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=School of Computer Science and Engineering, Northeastern University, Shenyang 110167, bio={"content":"

born in 1983. PhD, professor, PhD supervisor. His main research interests include Internet of things, cyber physical systems, and intelligent information networks

"}, bioImg=null, bioContent=

born in 1983. PhD, professor, PhD supervisor. His main research interests include Internet of things, cyber physical systems, and intelligent information networks

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325493604529091256, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, authorId=1325493604352930485, language=CN, stringName=彭玉怀, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=东北大学计算机科学与工程学院 沈阳 110167, bio={"img":"Qf2UWsdjq7oM2Hlf99se6w==","content":"

1983年生。博士,教授,博士生导师。主要研究方向为物联网、网络物理系统、智能信息网络

"}, bioImg=Qf2UWsdjq7oM2Hlf99se6w==, bioContent=

1983年生。博士,教授,博士生导师。主要研究方向为物联网、网络物理系统、智能信息网络

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325493601957982882, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, xref=null, ext=[AuthorCompanyExt(id=1325493601962177187, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, companyId=1325493601957982882, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=School of Computer Science and Engineering, Northeastern University, Shenyang 110167), AuthorCompanyExt(id=1325493601970565796, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325493596878680717, companyId=1325493601957982882, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=东北大学计算机科学与工程学院 沈阳 110167)])])]
焦明海 , 樊本航, 王静, 彭玉怀*
作者信息
  • 东北大学计算机科学与工程学院 沈阳 110167
通讯作者:
作者简介:

1973年生。博士,副教授,硕士生导师。CCF会员。主要研究方向为人工智能、图像识别、大数据分析与优化

2002年生。硕士研究生。主要研究方向为机器学习、多模态信息抽取

1992年生。博士研究生。主要研究方向为物联网、数字孪生

1983年生。博士,教授,博士生导师。主要研究方向为物联网、网络物理系统、智能信息网络

A Multimodal Adaptive Entity Recognition Method Based on Reinforcement Strategy Feedback
MingHai Jiao , Benhang Fan, Jing Wang, Yuhuai Peng*
Affiliations
  • School of Computer Science and Engineering, Northeastern University, Shenyang 110167
  • born in 1973. PhD, associate professor, master supervisor. Member of CCF. His main research interests include artificial intelligence, image recognition, and big data parsing and optimization

    born in 2002. Master candidate. His main research interests include machine learning and multimodal information extraction

    born in 1992. PhD candidate. Her main research interests include Internet of things and digital twin

    born in 1983. PhD, professor, PhD supervisor. His main research interests include Internet of things, cyber physical systems, and intelligent information networks

出版时间: 2026-02-01 doi: 10.7544/issn1000-1239.202550273
文章导航
收藏切换

命名实体识别(named entity recognition,NER)的核心目标是从非结构化文本中识别出具有特定语义类别的实体与类型。随着社交媒体的迅速发展,文本信息往往与视觉信息共同出现,形成多模态内容。为了提升实体识别的准确性,多模态命名实体识别(multi-modal NER,MNER)方法利用不同模态中的语义信息,实现信息互补与深度融合。然而,不同模态之间的表征差异可能引入视觉噪声,干扰实体识别。文本模态中存在实体指代不清或上下文语义模糊的问题,增加了识别难度。针对上述问题,提出了一种基于强化策略反馈与自适应损失机制的MNER方法。首先,该方法采用基于 GPT-4o的3阶段思维链(chain of thought,COT)推理流程,形成渐进式推理框架,融合强化学习中的自适应反馈机制,对图像与文本之间的匹配程度进行评分,并利用自适应决策函数有效过滤视觉噪声的干扰。其次,设计了4类面向具体任务的损失函数,并利用自适应加权融合策略进行优化,以缓解上下文模糊带来的识别不确定性。在2个公开数据集Twitter-2015和Twitter-2017上开展实验,结果表明所提方法的总体F1分数分别达到86.45%和93.80%,显著优于当前主流基线模型。

命名实体识别  /  非结构化文本  /  多模态数据  /  强化学习  /  自适应损失  /  思维链

The core objective of the named entity recognition (NER) is to identify entities with specific semantic categories and types in unstructured text. With the rapid growth of social media, textual information is often accompanied by visual content, forming multimodal data. To improve the accuracy of entity recognition, multimodal NER (MNER) methods exploit semantic information from different modalities fully to achieve the complementary and deep fusion of cross-modal features. However, differences in the representation of modalities may introduce visual noise that interferes with entity recognition. Issues such as entity ambiguity or contextual semantic vagueness within the textual modality complicate recognition. To address these challenges, we propose a multimodal NER method based on a reinforcement strategy feedback with adaptive loss mechanism. First, the method adopts a three-stage chain of thought (COT) reasoning process based on GPT-4o to form a progressive reasoning framework that incorporates the adaptive feedback mechanism in reinforcement learning. The degree of matching between images and text is scored, and the interference of visual noise is effectively filtered using an adaptive decision function. Second, four task-specific loss functions are designed and jointly optimized through an adaptive weighted fusion strategy to alleviate the uncertainty caused by contextual ambiguity. Experiments on two representative public datasets (Twitter-2015 and Twitter-2017) show that the overall F1 scores of our proposed method are 86.45% and 93.80%, respectively, representing a significant improvement on current state-of-the-art baseline models.

named entity recognition  /  unstructured text  /  multimodal data  /  reinforcement learning  /  adaptive loss  /  chain of thought
焦明海, 樊本航, 王静, 彭玉怀. 基于强化策略反馈的多模态自适应实体识别方法. 计算机研究与发展, 2026 , 63 (2) : 294 -304 . DOI: 10.7544/issn1000-1239.202550273
MingHai Jiao, Benhang Fan, Jing Wang, Yuhuai Peng. A Multimodal Adaptive Entity Recognition Method Based on Reinforcement Strategy Feedback[J]. Journal of Computer Research and Development, 2026 , 63 (2) : 294 -304 . DOI: 10.7544/issn1000-1239.202550273
命名实体识别(named entity recognition,NER)[1-2]作为自然语言处理(natural language processing,NLP)领域的基石,对于信息检索、智能问答系统以及知识图谱的构建等多方面的应用具有至关重要的意义[3]。随着多模态数据日益丰富,多模态命名实体识别(multimodal named entity recognition,MNER)[4]应运而生,通过高效融合文本与视觉模态数据等跨模态信息,不仅能够提升对非结构化文本数据中实体定位与分类的准确性,而且在文本上下文信息不足或存在语义模糊性的情况下,借助视觉模态提供的额外线索,极大地增强了实体识别的精准度[5]。尤其是在一些特定情境下,如图1文本中的“鸟巢”既可以指代鸟类用于栖息繁殖的巢穴(被视为杂项(miscellaneous,MISC)实体),也可能是指代著名的体育场馆——国家体育场(视为组织(organization,ORG)实体)。在这种情况下,仅依靠文本信息难以准确区分其实体类型,而结合相关的视觉信息则能有效解决这一问题,实现更加精确的实体识别。然而,由于跨模态异构数据融合过程中可能引入大量的噪声干扰,以及视觉特征中的冗余信息会对文本序列中的非实体部分造成误判,因此精准的MNER方法是确保自然语言处理系统有效性和可靠性不可或缺的一环。
在早期阶段,一些研究者通常采用全局图像编码策略来引导文本中单词学习整幅图像的视觉感知表征,然而这种粗粒度对齐方式一方面会引发视觉模态噪声干扰,另一方面会造成细粒度实体特征丢失。随后,越来越多的研究开始关注文本与图像之间的细粒度语义对齐。许多研究者通过优化跨模态注意力模型,在图文特征融合方面取得了不错的效果,推动了跨模态特征之间的信息交互,实现了视觉模态特征在文本实体语义表征中的深度融合。
然而,现有的方法还存在2个问题:1)多模态间的表征差异易引发视觉噪声干扰,当图文语义存在显著偏差时,会严重误导实体识别的准确性。2)文本模态内部存在实体指代上下文模糊,具体表现为实体多义性和实体边界划分的模糊性。
针对上述MNER任务所面临的问题和挑战,我们提出了一种基于强化策略反馈与自适应损失(reinforcement strategy feedback and adaptive loss,RFAL)的MNER方法。具体而言,我们首先使用基于大语言模型(large language model,LLM)的思维链(chain of thought,COT)[6]并结合强化学习(reinforcement learning,RL)[7]中的策略优化思想,通过渐进式推理和自适应反馈精准过滤视觉信息所带来的噪声干扰,并设计基于多类动态损失的自适应机制以解决不同样本歧义特性问题,从而提高实体分类的准确性。
综上所述,本文的贡献有如下3点:
1)提出了一种基于强化策略反馈与自适应损失的MNER分类方法,旨在减少视觉噪声干扰和实体上下文模糊性的影响,在2个层面,如图像与文本、文本内部实现更一致的多模态对齐。
2)分别提出了图文噪声过滤层(image and text noise filtering layer,INF)和自适应多类型损失优化层(adaptive multi-class loss optimization layer,AMLO)这2层协同工作,从而提高实体的分类性能。
3)在2个目前广泛使用的MNER数据集上进行了大量的实验,以证明提出方法的有效性,实验结果表明,本文方法在这2个数据集上达到了SOTA的性能。
作为跨模态信息处理领域的重要研究方向,MNER旨在通过融合视觉信息与文本信息实现对实体信息的联合抽取,扩展了传统的基于文本的NER任务。相较于传统单模态实体识别方法,MNER需要实现对文本、图像等模态数据的对齐与整合,通过构建跨模态语义表征实现多源信息的语义统一,并基于联合特征对实体解码。如何通过跨模态表征学习实现视觉线索与实体语义的精准对齐,是当前需要面临的巨大挑战。
早期的方法主要通过全局视觉特征编码实现对图文模态的隐式交互。Moon等人[8]设计了基于LSTM-CNN的文本-图像联合建模框架。其采用长短期记忆(LSTM)网络提取文本语义特征,同时运用卷积神经网络(CNN)解析视觉信息,并通过模态注意力机制实现2种模态的信息整合;Zhang等人[9]提出了自适应协同注意力网络,动态调整这2种模态的融合权重。与上述使用全局图像处理方法不同,后续研究则更关注细粒度图文关联的优化方向。这类方法强调提取与文本语义高度相关的视觉区域特征。Lu等人[10]首先提出从图像中筛选出与文本最相关的视觉区域,构建基于注意力模型的跨模态隐式交互机制。Yu等人[11]提出的UMT模型引入了多模态交互模块,通过使用BERT和ResNet预训练模型,分别提取文本与视觉特征,并利用Transformer中Encoder结构实现跨模态语义的深层关联,有效捕捉到图像语义驱动的文本特征与文本引导的视觉特征之间的双向映射关系。Zhang等人[12]构建统一的多模态图结构,通过捕捉文本词元与图像区域节点间的跨模态关联,实现细粒度语义单元之间的交互。
尽管上述研究取得了显著的成果,然而在跨模态数据中融合不同模态的信息导致的模态表征对齐偏差问题尚未得到有效的解决。Xu等人[13]基于对比学习[14]构建了多模态匹配对齐框架,其可以自适应确定视觉信息的保留比例,建立跨模态一致性表征空间。Chen等人[15]提出基于动态门控网络的多模态学习方法,通过构建可学习的阈值单元对视觉特征进行选择性强化,在保留有效视觉特征的同时抑制噪声干扰,并与多模态关系抽取任务进行联合训练,从而得到具有跨模态对齐能力和优异泛化特性的多模态特征。Sun等人[16]将多模态关系传播引入到BERT预训练方法中,通过构建多任务学习框架,在大规模图文数据集上训练和验证关系传播对MNER任务的作用效果,并判别视觉信息与文本内容之间的语义相关性。Zhou等人[17]提出一种基于变分自编码器的多模态特征融合方法,通过构建统一的潜在表征空间实现图文数据的跨模态对齐。
综上所述,上述提到的大部分方法都采用全局图像信息与文本信息相融合,然而这些方法在进行MNER任务时依然存在以下的问题:1)视觉信息和文本信息不匹配会误导模型进行实体的识别,引发视觉信息的噪声干扰。2)文本模态中普遍存在实体上下文模糊,具体表现为多义词歧义和边界模糊的问题,影响实体的识别。针对以上问题,本文提出了RFAL方法,通过对视觉噪声过滤以及自适应多类型损失模块的融合,提升了在MNER任务上的识别性能。
本文所提出的RFAL方法聚焦于跨模态异构数据的细粒度精准匹配,结合跨模态特征的增强交互,实现复杂信息的高效对齐。RFAL方法主要由图文噪声过滤层以及自适应多类型损失优化层2部分组成。本文提出的整体方法框架如图2所示。
MNER旨在通过联合分析文本-图像对中的跨模态语义信息,对文本序列中的实体以及实体类型进行识别与正确分类。给定输入文本序列$ S=\{{w}_{1},{w}_{2},… , {w}_{n}\} $及其关联图像I,其中$ {w}_{i} $表示S中第i个字符,并且$ Y=\{{y}_{1},{y}_{2},…{,y}_{n}\} $表示文本序列中的所有字符对应的实体类型标签。标签集Y基于BIO2标签模式构建[18],覆盖4类预定义实体类型:人物(person,PER)、位置(location,LOC)、组织(ORG)和杂项(MISC),具体包含“O”(非实体) 、“B-PER”(人物实体开始)、“I-ORG”(组织实体内部)等细粒度标签。通过将MNER建模为序列标注任务,模型需要对每个实体分配唯一标签,以同时捕获文本语义与视觉模态的潜在关系,最终实现跨模态实体边界识别与类型判断。
图文噪声过滤层是基于GPT-4o构建具有3阶段结构的思维链推理流程,形成基于LLM思维链的多阶段递进式推理框架。同时,融合强化学习中的自适应反馈机制,动态评估图像与文本之间的语义匹配程度,借助自适应决策函数,实现噪声数据的多层次精准过滤。通过多阶段的验证体系,每个阶段均设置可量化的评价指标与动态调整的阈值参数,解决了MNER数据中存在的图文不匹配而引发的噪声干扰问题。
本文提出的基于COT 的多阶段递进式推理模块,包含语义验证、实体对齐和逻辑连贯评估3个评分机制,逐步消除图文噪声对MNER任务的影响,并结合动态阈值优化实现自适应决策。
步骤1. 在语义验证阶段,旨在解决文本描述和图像表征在空间关系上可能存在的显示冲突。由于图文模态在空间方位描述(如“左侧”“上方”)上的显性矛盾是构成初级噪声的主要来源,本步骤通过结构化提示工程[19]引导GPT-4o生成具有可解释性的推理过程。具体而言,首先让GPT-4o从文本模态中提取空间关系描述,并识别图像中对应物体的位置关系,然后根据图像和文本中空间关系的差异程度输出相应评分。该步骤的输出评分范围设定在0~40之间。评分函数为:
$ {S core}_{1}=\left\{\begin{aligned}&40,\;\;{\mathrm{if}}\;Match({T}_{\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{a}\mathrm{l}},{I}_{\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{a}\mathrm{l}})\;,\\ &20,\;\;{{\mathrm{if}}}\;{PartialMatch}({T}_{\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{a}\mathrm{l}},{I}_{\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{a}\mathrm{l}})\;,\\ &0,\;\;其他\;,\end{aligned}\right. $
其中$ {T}_{\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{a}\mathrm{l}} $为文本模态中的空间关系,$ {I}_{\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{a}\mathrm{l}} $为图像中的空间关系。
步骤2. 在实体对齐阶段,针对实体虚构(如文本实体在视觉模态中缺失)及表征模糊是次级噪声的核心来源,传统的二值化判断(存在/不存在)无法有效量化实体存在性概率分布,且忽视不同实体类型对语义完整性的差异贡献。本步骤通过统计显著性加权与置信度评估实现精准化过滤。首先使用词频-逆文档频率(TF-IDF)统计量筛选出关键实体即通过IDF降低高频普通实体的权重,突出低频关键实体的权重。TF-IDF的计算公式为
$ TF{\text{-}}IDF\left(e\right)=\left(\dfrac{{f}_{e}}{N}\right)\times \mathrm{ln}\left(\dfrac{D}{{d}_{e}+1}\right)\;, $
其中$ {f}_{e} $为实体频次,N为文本总词数,D为总数据量,$ {d}_{e} $为包含实体e的数据量。在此基础上,为4类实体类型分别赋予不同的权重优先级,例如赋予人物、地点等核心实体类型更高的权重,最后通过归一化加权融合得到最终实体权重。实体权重的函数为
$ {w}_{e}=\dfrac{TF\text-IDF\left(e\right)\times {\lambda }_{\mathrm{t}\mathrm{y}\mathrm{p}\mathrm{e}}\left(e\right)}{\displaystyle\sum\limits_{{e}'\in \mathcal{E}}TF\text-IDF\left({e}'\right)\times {\lambda }_{\mathrm{t}\mathrm{y}\mathrm{p}\mathrm{e}}\left({e}'\right)}\;,$
其中$ {w}_{e} $是实体e的归一化权重,$ TF\text{-}IDF\left(e\right) $表示实体e的TF-IDF值,$ \mathcal{E} $为文本中所有实体的集合,$ {\lambda }_{\mathrm{t}\mathrm{y}\mathrm{p}\mathrm{e}}\left(e\right) $为实体e的类型权重,$ \displaystyle\sum\limits_{{e}'\in \mathcal{E}}TF\text{-}IDF\left({e}'\right)\times {\lambda }_{\mathrm{t}\mathrm{y}\mathrm{p}\mathrm{e}}\left({e}'\right) $为文本中所有实体的加权和。在计算出每个实体权重后,通过结构化提示词(prompt)引导GPT-4o生成细粒度存在性置信度评估。模型输出连续概率值$ {p}_{e}\in \left[\mathrm{0,1}\right] $,其中$ {p}_{e} $量化文本实体在视觉模态中的语义对齐程度(如部分遮挡实体可赋值为$ {p}_{e} $=0.6)。最后进行加权求和得到该步骤得分,该步骤的输出评分范围在0~40之间。最终评分计算函数为:
$ \begin{array}{c}{S core}_{2}=40\times \displaystyle\sum\limits_{e\in \mathcal{E}}\left({w}_{e}\times {p}_{e}\right)\;,\end{array} $
其中$ {w}_{e} $为每个实体的权重,$ {p}_{e} $为GPT-4o对图像中是否存在实体e的置信度。
步骤3. 在逻辑连贯性评估中,主要是解决文本描述与视觉表征在事件发展逻辑及因果关联层面的隐性矛盾问题。尽管前序阶段已确保实体存在性与空间关系的局部一致性,但仍需要确保跨模态语义的全局连贯性以及实体之间的现实逻辑性。因此,本步骤提出基于双重因果图匹配的评估框架,核心在于构建文本与图像双模态的因果图结构,并通过多维相似度度量实现语义连贯性量化。具体而言,首先通过结构化prompt引导GPT-4o分别从文本和图像中提取关键实体和实体状态来构成因果图中的节点集合:$ {V}_{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}|\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}}=\left\{{v}_{i}\right|{v}_{i}=({e}_{i},{s}_{i})\} $,其中$ {e}_{i} $表示实体,$ {s}_{i} $表示实体状态。节点相似度的计算如式(5)所示。
$ \begin{array}{c}{S im}_{\mathrm{n}\mathrm{o}\mathrm{d}\mathrm{e}}=\dfrac{\left|{V}_{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}}\cap {V}_{\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}}\right|}{\left|{V}_{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}}\cup {V}_{\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}}\right|}\;,\end{array} $
其中$ {V}_{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}} $表示从文本中提取的节点集合,$ {V}_{\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}} $表示从图像中提取的节点集合。在边关系建模层面,分别构建文本和图像的边集合:$ {E}_{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}|\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}}= \{ ({v}_{p},{r}_{q},{v}_{r})\} $,其中$ {r}_{q} $表示因果谓词(如“导致”“影响”等)。边的相似度计算由边的结构相似度与边的语义相似度共同决定。边的结构相似度使用Jaccard通过比较边拓扑连接模式计算,边的语义相似度则由GPT-4o根据文本和图像的边集合关系输出$ \left[\mathrm{0,1}\right] $之间的相似概率,边的相似度计算公式如式(6)所示。
$ \begin{split}{S im}_{\mathrm{e}\mathrm{d}\mathrm{g}\mathrm{e}}= &\alpha \times Jaccard\left({E}_{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}},{E}_{\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}}\right)+ \\&\left(1-\alpha \right)\times {GPT}_{\mathrm{s}\mathrm{i}\mathrm{m}}\left({E}_{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}},{E}_{\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}}\right)\;,\end{split} $
其中$ \alpha $为人工预先设置的权重超参数,$ Jaccard ({E}_{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}}, {E}_{\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}} ) $为边的结构相似度,$ {GPT}_{\mathrm{s}\mathrm{i}\mathrm{m}}({E}_{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}},{E}_{\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}}) $为边的语义相似度。
最终该步骤的评分通过融合节点与边相似度的几何平均合成,输出评分范围在0~20之间。最终评分计算函数为
$ {S core}_{3}=\sqrt{{S im}_{\mathrm{n}\mathrm{o}\mathrm{d}\mathrm{e}}\times {S im}_{\mathrm{e}\mathrm{d}\mathrm{g}\mathrm{e}}}\times 20\;。 $
步骤4. 在完成多阶段评分计算后,最终会通过层次化决策函数实现图文匹配度的最终判定。该决策机制使用动态边界阈值与概率化决策相结合的混合范式。具体而言,首先随机抽取100条由人工审核的黄金标注样本构建基准评分分布:均值($ u $)、标准差($ \sigma$)。然后计算总评分$ S={S core}_{1}+{S core}_{2}+{S core}_{3} $,以此建立自适应阈值计算模型。阈值动态调整过程遵循渐进式策略优化原则,定义硬决策边界和软决策边界。硬决策阈值:$ {\theta }_{\mathrm{h}\mathrm{a}\mathrm{r}\mathrm{d}}=\mu -k\sigma $,其中严格系数$ k\in \left[\mathrm{0.5,1.5}\right] $,该阈值确保显著噪声样本的确定性过滤。软决策阈值:$ {\theta }_{\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}}={\theta }_{\mathrm{h}\mathrm{a}\mathrm{r}\mathrm{d}}-\varDelta $,其中缓冲区间$ \varDelta =0.2\sigma $,用于缓解决策函数在临界区域的敏感性。总体的决策函数设计为:
$ {D}{e}{c}{i}{s}{i}{o}{n}(S)=\left\{\begin{aligned}&\text{K}\text{e}\text{e}\text{p},\;\;\text{i}\text{f}\;S\ge {\theta }_{\text{hard}}\;,\\&\text{D}\text{i}\text{s}\text{c}\text{a}\text{r}\text{d},\;\;\text{i}\text{f}\;S < {\theta }_{\text{soft}}\;,\\&\text{D}\text{i}\text{s}\text{c}\text{a}\text{r}\text{d},\;\;\text{i}\text{f}\;{\theta }_{\text{soft}}\le S < {\theta }_{\text{hard}}\;且\;r\le p\;,\\ &\text{K}\text{e}\text{e}\text{p},\;\text{i}\text{f}\;\;{\theta }_{\text{soft}}\le S < {\theta }_{\text{hard}}\;且\; r > p\;,\end{aligned}\right. $
其中$ r\sim U\left(\mathrm{0,1}\right) $为均匀分布随机变量,$ p=1-\mathrm{e}\mathrm{x}\mathrm{p}(-\beta (S- {\theta }_{\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}}\left)\right) $ 为自适应图像丢弃概率,衰减系数$ \beta $控制决策边界的平滑度。通过在决策过程中引入概率化扰动,防止模型陷入局部最优,并且指数衰减函数确保接近硬阈值的样本具有更高的保留概率。
在社交媒体场景中的MNER任务中,实体的上下文模糊主要表现为2种典型特征:多义词歧义(如“Java”指编程语言或岛屿)和边界模糊性(如“New York”被截断为“York”)。针对传统固定权重的损失函数难以动态适应不同样本歧义特性的局限性而导致模型对复杂语境泛化能力不足的问题,本节提出一种基于动态损失的自适应机制,通过对Qwen2-VL-7B多模态大模型(multimodal large language model,MLLM)进行全量微调[20],并融合多维度精细化设计的损失函数,从而实现具有上下文感知能力的自适应优化机制。
基于对目标数据集的分析,我们发现该数据集中包含4类具有显著差异的实体类型。因此,我们提出假设:针对不同的实体类型设计差异化的损失函数可能会优化模型的分类效果,为了验证该假设,本文分别构建了4类针对性的损失函数。并且我们注意到不同的损失函数对各类实体的适用性可能存在差异,为避免单一损失函数主导优化过程,我们引入了自适应权重优化机制,在训练过程中,权重参数$ \alpha $会进行动态调整。通过观察测试集上的消融实验结果,验证了本节提出的方法的有效性。
社交媒体文本中的实体类型分布呈现显著的长尾特性。这种分布不均衡导致传统的交叉熵损失在训练过程中易被高频类别主导以及低频类别实体表征学习不充分。此外,部分实体因上下文模糊成为“困难样本”,其预测概率接近决策边界($ p=0.5 $),此类样本在标准交叉熵损失中的梯度贡献较低、优化效率不足。基于上述分析我们设计了类别平衡的焦点交叉熵损失函数。焦点交叉熵损失通过双重调节机制增强对困难样本与低频类别的关注。
$ {L}_{\mathrm{f}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{l}}=-\frac{1}{N}\sum _{i=1}^{N}\sum _{c=1}^{C}{\alpha }_{c}(1-{p}_{i,c}{)}^{\gamma }{y}_{i,c}\;\mathrm{l}\mathrm{n}\;{p}_{i,c}\;, $
其中$ N $为批次实体数量,$ C $为实体类别数,$ {y}_{i,c} $为真实标签,$ {p}_{i,c} $为模型输出的预测概率,$ {\alpha }_{c} $为类别权重(根据类别频率动态调整),$ \gamma $为调节因子。相比于传统交叉熵,该损失在训练后期持续优化困难样本,提升决策边界附近的分类精度。
为缓解模型对错误标签的过度自信而导致的泛化性不足问题,本文融合了对称标签平滑的KL损失函数。损失函数为
$ \begin{split}{L}_{\mathrm{s}\mathrm{k}\mathrm{l}}=\frac{1}{2N}\displaystyle\sum\limits_{i=1}^{N}\left[{D}_{\mathrm{K}\mathrm{L}}\left({\boldsymbol{q}}_{i}\| {\boldsymbol{p}}_{i}\right)+{D}_{\mathrm{K}\mathrm{L}}\left({\boldsymbol{p}}_{i}\| {\boldsymbol{q}}_{i}\right)\right]\;,\end{split} $
其中$ {\boldsymbol{q}}_{i} $为平滑后的标签分布,$ {q}_{i,c}=(1-\epsilon ){y}_{i,c}+\dfrac{\epsilon }{C} $,$ \epsilon $为超参数,通常取值为$\epsilon=0.1 $,$ C $为实体类别数。$ {\boldsymbol{p}}_{i} $为模型的预测概率分布。$ {D}_{\mathrm{K}\mathrm{L}} $为KL散度,$ {D}_{\mathrm{K}\mathrm{L}}({\boldsymbol{q}}_{i}\| {\boldsymbol{p}}_{i}) $= $ \displaystyle\sum\limits_{i\in C}{\boldsymbol{q}}_{i}\;\mathrm{l}\mathrm{n}\;\dfrac{{\boldsymbol{q}}_{i}}{{\boldsymbol{p}}_{i}} $。通过进一步分析发现,实体边界的模糊性通常会导致2种典型的错误:不完整检测(如将“New York City”截断为“York”)与嵌套冲突(如“Bank of America”包含子实体“America”),传统的方法在预测时未充分考虑边界位置的合理性,可能直接导致类型误分类。为此,我们进一步融合了隐式边界一致性损失,其利用实体类型标签隐式学习实体边界信息。该损失函数设计为
$ {L}_{\text{boundary}}=\frac{1}{T}\sum _{t=1}^{T}( {{\lambda }_{1}d}_{t}\times I\left({y}_{t}\ne {y}_{t+1}\right)- {{\lambda }_{2}d}_{t}\times I\left({y}_{t}={y}_{t+1}\right) )\;, $
其中$ T $为文本的序列长度,$ {\lambda }_{1} $和${\lambda }_{2} $为平衡权重,$ {d}_{t}=\|{\boldsymbol{p}}_{t}- {\boldsymbol{p}}_{t+1}\|_{1} $表示相邻token的L1距离,$ {\boldsymbol{p}}_{t}\in {\mathbb{R}}^{C} $表示第t个token的预测概率分布,${I}(\cdot ) $为指示函数(条件满足为1,否则为0)。该损失函数在计算时,若相邻token类型不同,则增大差异,否则减小差异。
最后,考虑到之前设计的损失函数缺少对同类实体的全局语义约束,我们通过对比学习的思想使同类实体的预测分布相近,异类实体的预测分布远离,从而增强模型对上下文语义的敏感度。该损失函数的计算公式为
$ \begin{split}{L}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{s}\mathrm{t}}=-\dfrac{1}{N}\displaystyle\sum\limits_{i=1}^{N}\mathrm{l}\mathrm{n}\dfrac{\mathrm{exp}\left(\dfrac{s\left({\boldsymbol{p}}_{i},{\boldsymbol{m}}_{{c}_{i}}\right)}{\tau }\right)}{\displaystyle\sum\limits_{c\in C}\mathrm{exp}\left(\dfrac{s\left({\boldsymbol{p}}_{i},{\boldsymbol{m}}_{c}\right)}{\tau }\right)}\;,\end{split} $
其中$ N $为批次内实体数量,$ {\boldsymbol{p}}_{i}\in {\mathbb{R}}^{C} $表示第$ {i} $个实体的预测概率分布,$ {\boldsymbol{m}}_{{c}_{i}} $为类型$ c $的类中心,$ {\boldsymbol{m}}_{c}=\dfrac{1}{\left|{B}_{c}\right|}\displaystyle\sum\limits_{j\in {B}_{c}}{\boldsymbol{p}}_{j}, s\left({\boldsymbol{p}}_{i},{\boldsymbol{m}}_{{c}_{i}}\right) $表示2个向量间的余弦相似度,$ \tau $为温度系数。
在完成4类损失函数的设计后,本文使用一种基于自适应权重优化[21]的动态融合策略。通过在模型微调过程中自适应调整其权重数值分布,实现对4类损失函数的平衡优化,具体设计如下:
$ \begin{split}{L}_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}={\alpha }_{1}{L}_{\mathrm{f}\mathrm{o}\mathrm{c}\mathrm{a}\mathrm{l}}+{\alpha }_{2}{L}_{\mathrm{s}\mathrm{k}\mathrm{l}}+{\alpha }_{3}{L}_{\mathrm{b}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{d}\mathrm{a}\mathrm{r}\mathrm{y}}+{\alpha }_{4}{L}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{s}\mathrm{t}}\;,\end{split} $
其中$ {\alpha }_{1},{\alpha }_{2},{\alpha }_{3},{\alpha }_{4} $为自适应优化的权重参数,采用高斯分布进行初始化,初始值为$ {\alpha }_{i}\sim \mathcal{N}(0,{0.1}^{2}) $,在模型微调过程中,权重参数通过反向传播自适应优化,并且其满足非负性与归一化约束:$ \displaystyle\sum\limits_{i=1}^4\alpha_i\ge0,\alpha_i=1 $。
为验证本文所提出方法的有效性,使用了从社交媒体上收集的、目前广泛使用的Twitter-2015[15],Twitter-2017[9]两个数据集进行了相关实验,模型只有在正确识别出文本中的实体跨度以及其对应的实体类型才算完全正确的识别,表明了本文方法在MNER任务上的有效性。其中Twitter-2015数据集包括4000/1000/3357的训练/验证/测试分割,Twitter-2017数据集包括3373/723/723的训练/验证/测试分割。2个数据集中的每一条数据都可能包含多个实体,并且由文本和其对应的图像组成。其中实体类型主要包含4类:人物(PER)、位置(LOC)、组织(ORG)、其他(MISC)。2种数据集统计数据如表1所示。
本文通过计算2个数据集的F1分数、召回率R (recall)、精确率P(precision),来评估我们的模型在MNER任务中的效果。在当前任务中,精确率和召回率会呈现负相关的关系,理想情况下期望实现两者的同步优化,但是在实际的应用场景中,召回率的提升会使精确率下降。为了有效平衡这2个指标,采用F1分数作为综合评价指标,该指标更能客观地反映出模型的性能。F1分数在需要双重验证模型效果的大多数自然语言处理任务中具有重要的评估价值。相应的计算如式(14)(15)(16)所示。
$ \begin{array}{c}P=\dfrac{TP}{TP+FP}\;,\end{array} $
$ \begin{array}{c}R=\dfrac{TP}{TP+FN}\;,\end{array} $
$ \begin{array}{c}F1=2\times \dfrac{P\times R}{P+R}\;,\end{array} $
其中TP为模型正确识别出文本中的实体跨度和其对应实体类型的实体数量,FP为模型错误识别实体跨度和实体类型的实体数量,FN为模型未识别出的实体数量。
本文提出的模型通过PyTorch框架在4张Nvidia GeForce RTX 4090 GPU上进行了MNER的相关实验,使用Qwen2-VL-7B基座多模态大模型进行全量微调。模型的所有超参数配置如表2所示。
为了验证本文提出的RFAL方法在MNER任务上的效果,本文与在Twitter-2015和Twitter-2017数据集上所提出的一些经典的MNER方法进行了对比,包含以下所提出的方法:
1)BiLSTM-CRF[22]。该方法提出将双向长短时记忆网络(bidirectional long short-term memory,BiLSTM)与条件随机场(conditional random field,CRF)相结合用于序列标注任务。通过BiLSTM编码器有效捕获文本的上下文特征,并利用CRF层实现实体序列的全局最优解码。
2)CNN⁃BiLSTM⁃CRF[23]。该方法在BiLSTM-CRF架构基础上引入CNN,通过CNN与BiLSTM提取文本特征,并利用CRF实现实体序列的全局最优解码,是一种经典的序列标注方法。
3)BERT-CRF[22]。该方法通过BERT捕捉文本深层语义表征,并采用CRF层替代传统softmax分类器进行最终的标签预测。
4)MAF[13]。该方法通过动态调节视觉信息保留比例与跨模态噪声抑制机制,提升了多模态表征的鲁棒性。
5)BGA-MNER[24]。该方法提出双向生成对齐方法,通过图像到文本和文本到图像的生成任务,直接学习实体与图像对象间的隐含关系,无需对应标注,有效缓解模态偏差问题。
6)PGIM[25]。该方法提出2阶段框架,利用ChatGPT作为隐式知识库,通过多模态相似示例引导生成辅助精炼知识,增强文本实体预测。该方法在MNER任务中表现出色,具有更强的鲁棒性和泛化能力。
7)DPE-MNER[26]。该方法提出动态融合多模态信息的迭代推理框架,通过“分解、优先、消除”策略分层融合多粒度文本和图像信息,逐步优化实体识别结果。
8)MPMRC-MNER[27]。该方法提出基于多模态提示的机器阅读理解框架,将文本查询转化为包含图像信息的多模态提示,并通过对比学习优化跨模态语义空间,提升MNER性能。
9)CMAT[28]。该方法提出一种多任务多模态学习架构,通过跨模态辅助任务助力MNER。该方法利用跨模态匹配和互信息最大化分离共享特征与任务特异性特征,并通过自适应融合优化跨模态特征。
本文所提出的RFAL方法与上述其他方法在Twitter-2015数据集和Twitter-2017数据集上的效果对比如表3所示。
首先,通过表3中的对比结果可以看出基于预训练语言模型的BERT-CRF框架在2个测试数据集上的效果相较于传统BiLSTM-CRF等方法实现了F1分数上的7个百分点左右的提升。原因在于基于海量文本预训练的BERT模型通过深度上下文建模捕获多维语义信息,显著提升了语义消歧能力。并且CRF层的序列约束机制利用标签转移概率优化,有效增强相邻标记间的关联一致性。
其次,通过实验观察发现,融合视觉模态的MNER模型在2个测试数据集上相较于单一文本模态的NER模型效果上表现出更优的性能。这表明,在社交媒体常见的短文本语境下,融合视觉信息更有助于实体及其类型的识别。
最后,本文提出的MNER方法在2个测试数据集上相较于其他的方法都取得了显著的性能提升,并且极大地缓解了因图文不匹配而带来的噪声干扰问题以及实体上下文模糊等问题,证明了RFAL的有效性。与基于文本的单模态命名实体识别方法(BERT-CRF)相比,RFAL在Twitter-2015和Twitter-2017数据集上的总体F1分数分别提高了14.64个百分点和10.36个百分点,并且在各个实体类型上的F1分数也都远远高于BERT-CRF。证明了RFAL能够有效融合与文本模态相关的视觉模态信息。除此之外,在与最近提出的多模态SOTA方法PGIM和DPE-MNER的比较中,RFAL在2个数据集上的效果要好于它们。具体而言,与PGIM相比,在Twitter-2015数据集和Twitter-2017数据集上的总体F1分数分别提高了7.12个百分点和2.37个百分点。与DPE-MNER相比,2个数据集上的总体F1分数分别提高了8.89个百分点和5.9个百分点。并且如图3所示,在Twitter-2015数据集上的效果提升远好于Twitter-2017数据集上的效果,这也说明当前的方法受不同数据分布的影响,泛化性不足。而RFAL方法在2个数据集上的表现都比较优异,这也证明了RFAL方法中的2个优势:1)图文噪声过滤层有效减少了因图文不匹配而带来的额外视觉信息噪声干扰问题,2)自适应多类型损失优化层有效解决了因文本中实体语义多歧而带来的语义模糊问题。
为了验证本文所提出的RFAL方法中的2层,即图文噪声过滤层和自适应多类型损失优化层的有效性,进行了如下的消融实验。具体分析如下:
如表4所示,其中w/o I&A表示消除图文噪声过滤层和自适应多类型损失优化层,w/o INF表示消除图文噪声过滤层,w/o AMLO表示消除自适应多类型损失优化层。可以观察到RFAL方法在2个数据集上的F1分数上明显优于其他方法,证明了RFAL的所有模块对提升MNER任务的性能都很重要。
与RFAL相比,消除图文噪声过滤层导致Twitter-2015和Twitter-2017在F1分数上分别下降了0.98个百分点和0.2个百分点,结果证明了引入图文噪声过滤层可以很好地缓解图文不匹配带来的噪声干扰问题。
此外,消除自适应多类型损失优化层导致2个数据集上的F1分数下降了1.67个百分点和1.05个百分点,证明了本文在第2节提出的假设的有效性可以很好缓解实体上下文模糊的问题。
在消除图文噪声过滤层和自适应多类型损失优化层后,可以观察到同时消除这2层会导致最终结果显著下降。这进一步证明了本文所提出的两大方法层对MNER任务是有效且十分重要的。
为了进一步验证本文所提出方法中自适应多类型损失优化层中各个组件的有效性,进行了如下的消融实验。
如表5所示,可以观察到在图文噪声过滤层的基础上分别加入各类损失函数之后,模型在2个数据集上的F1分数上均有提升,这说明不同类型的损失函数可以有效解决NER中普遍存在的实体类型呈现长尾特性、模型对错误标签过渡自信而导致泛化性不足、实体边界模糊以及上下文语义不敏感等问题,验证了本文所提出的自适应多类型损失优化层中4类具体任务损失函数的有效性,其能够针对当前NER任务中存在的问题发挥作用。
本文从基准测试集中选取了2个具有代表性的测试样本,用来比较不同方法的实体识别效果。
当文本模态中存在信息偏差或缺失时,融合视觉特征往往能展现出更好的效果。这是因为图像信息可以补充文本描述的盲区,从而为实体识别任务提供有价值的线索。如表6(a)中,在纯文本模态下,BERT-CRF无法正确识别出“皇马”是一个组织类型。然而在结合视觉信息后,多模态模型能正确识别出该实体,并判断出其实体类型。
图像噪声干扰问题存在的主要原因包括:1)非描述性视觉因素的存在(如抽象插画或隐喻图像);2)跨模态差异化(即视觉符号与文字描述呈现事件的维度不同)。在此类情景下,MNER方法可能要比单模态方法(如BERT-CRF)效果更差。在表6(b)中提供的图像会干扰模型对实体的识别,而多模态方法MAF并没有对图像噪声问题进行过滤,因此会将文本中“Future Hearts”的类型“MISC”错误识别为“PER”。RFAL方法由于充分考虑到图像不匹配而引发的噪声问题,以及实体上下文模糊的问题,所以正确识别了上述的所有实体以及其实体类型。
针对目前MNER任务中存在的由于多模态间的表征差异而引发的视觉噪声干扰问题以及文本内部存在的实体指代上下文模糊的问题,本文提出了基于强化策略反馈与自适应损失的MNER方法。具体地,本文首先使用基于GPT-4o的 COT方式,并融合强化学习的自适应反馈机制,实现对视觉噪声的精准过滤;然后设计了动态自适应加权损失层,根据实体上下文模糊问题,针对性地采取多类损失函数优化模型实体识别的性能,提升了最终的实验结果。在2个公开基础的数据集上的实验结果表明,本文提出的MNER的方法明显优于目前大多数方法。
在未来的工作中,计划推进以下两大方面:1)针对跨模态交互中的特征匹配问题,拟构建一种基于多层次特征对齐机制的深度神经网络架构。2)将目前的方法拓展至其他多模态任务,包括但不限于多模态情感分析、多模态人机对话等。
作者贡献声明:焦明海指导论文撰写,提供技术支持;樊本航提出研究思路,设计和实施实验,撰写论文;王静完成了相关文献梳理、实验数据整理,并完善实验方案;彭玉怀完成了论文框架设计、整体内容规划。

参考文献 引证文献
排序方式:
1
Mai Chengcheng, Liu Jian, Qiu Mengchuan, et al. Pronounce differently, mean differently: A multi-tagging-scheme learning method for Chinese NER integrated with lexicon and phonetic features[J]. Information Processing & Management, 2022, 59(5): 103041
2
He Qizhen, Wu Liang, Yin Yida, et al. Knowledge-graph augmented word representations for named entity recognition[C]//Proc of the 34th Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 7919−7926
3
王嘉诚, 王凯, 王昊奋, 等. 面向远程监督命名实体识别等噪声检测[J]. 计算机研究与发展, 2024, 61(4): 916−928
Wang Jiacheng, Wang Kai, Wang Haofen, et al. Noise detection for distant supervised named entity recongnition[J]. Journal of Computer Research and Development, 2024, 61(4): 916−928 (in Chinese)
4
Arshad O, Gallo I, Nawaz S, et al. Aiding intra-text representations with visual context for multimodal named entity recognition[C]//Proc of the IEEE Int Conf on Document Analysis and Recognition (ICDAR). Piscataway, NJ: IEEE, 2019: 337−342
5
Zhang Xin, Yuan Jingling, Li Lin, et al. Reducing the bias of visual objects in multimodal named entity recognition[C]//Proc of the 16th ACM Int Conf on Web Search and Data Mining. New York: ACM, 2023: 958−966
6
Wei J, Wang X, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models[J]. Advances in Neural Information Processing Systems, 2022, 35: 24824−24837
7
Mnih V, Kavukcuoglu K, Silver D, et al. Playing Atari with deep reinforcement learning[J]. arXiv preprint, arXiv: 1312.5602, 2013
8
Moon S, Neves L, Carvalho V. Multimodal named entity recognition for short social media posts[J]. arXiv preprint, arXiv: 1802.07862, 2018
9
Zhang Qi, Fu Jinlan, Liu Xiaoyu, et al. Adaptive co-attention network for named entity recognition in tweets[C]//Proc of the 32nd Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2018: 5674−5681
10
Lu D, Neves L, Carvalho V, et al. Visual attention model for name tagging in multimodal social media[C]//Proc of the 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2018: 1990−1999
11
Yu Jianfei, Jiang Jing, Yang Li, et al. Improving multimodal named entity recognition via entity span detection with unified multimodal transformer[C]//Proc of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 3342−3352
12
Zhang Dong, Wei Suzhong, Li Shoushan, et al. Multi-modal graph fusion for named entity recognition with targeted visual guidance[C]//Proc of the 35th Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2021: 14347−14355
13
Xu Bo, Huang Shizhou, Sha Chaofeng, et al. MAF: A general matching and alignment framework for multimodal named entity recognition[C]//Proc of the 15th ACM Int Conf on Web Search and Data Mining. New York: ACM, 2022: 1215−1223
14
Bhattacharjee A, Karami M, Liu H. Text transformations in contrastive self-supervised learning: A review[J]. arXiv preprint, arXiv: 2203.12000, 2022
15
Chen Xiang, Zhang Ningyu, Li Lei, et al. Good visual guidance makes a better extractor: Hierarchical visual prefix for multimodal entity and relation extraction[J]. arXiv preprint, arXiv: 2205.03521, 2022
16
Sun Lin, Wang Jiquan, Zhang Kai, et al. RpBERT: A text-image relation propagation-based BERT model for multimodal NER[C]//Proc of the 35th Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2021: 13860−13868
17
Zhou Baohang, Zhang Ying, Song Kehui, et al. A span-based multimodal variational autoencoder for semi-supervised multimodal named entity recognition[C]//Proc of the 2022 Conf on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg, PA: ACL, 2022: 6293−6302
18
Sang E F, Veenstra J. Representing text chunks[C]//Proc of the 9th Conf on European Chapter of the Association for Computational Linguistics (EACL). Stroudsburg, PA: ACL, 1999: 173−179
19
Brown T, Mann B, Ryder N, et al. Language models are few-shot learners[J]. Advances in Neural Information Processing Systems, 2020, 33: 1877−1901
20
Devlin J, Chang Mingwei, Lee K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]//Proc of the Annual Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, PA: ACL, 2019: 4171−4186
21
Yao Ruilin, Xiong Shengwu, Zhao Yichen, et al. Visualgrounding with multi-model conditional adaptation[C]//Proc of the 32nd ACM Int Conf on Multimedia. New York: ACM, 2024: 3877−3886
22
Huang Zhiheng, Xu Wei, Yu Kai. Bidirectional LSTM-CRF models for sequence tagging[J]. arXiv preprint, arXiv: 1508.01991, 2015
23
Ma Xuezhe, Hovy E. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF[J]. arXiv preprint, arXiv: 1603.01354, 2016
24
Chen Feng, Liu Jiajia, Ji Kaixiang, et al. Learning implicit entity-object relations by bidirectional generative alignment for multimodal NER[C]//Proc of the 31st ACM Int Conf on Multimedia. New York: ACM, 2023: 4555−4563
25
Li Jinyuan, Li Han, Pan Zhuo, et al. Prompting ChatGPT in MNER: Enhanced multimodal named entity recognition with auxiliary refined knowledge[J]. arXiv preprint, arXiv: 2305.12212, 2023
26
Zheng Zihao, Zhang Zihan, Wang Zexin, et al. Decompose, prioritize, and eliminate: Dynamically integrating diverse representations for multimodal named entity recognition[C]//Proc of the 2024 Joint Int Conf on Computational Linguistics, Language Resources and Evaluation (LREC-COLING’ 2024). Stroudsburg, PA: ACL, 2024: 4498−4508
27
Bao Xigang, Tian Mengyuan, Zha Zhiyuan, et al. MPMRC-MNER: A unified MRC framework for multimodal named entity recognition based multimodal prompt[C]//Proc of the 32nd ACM Int Conf on Information and Knowledge Management. New York: ACM, 2023: 47−56
28
Chen Zhengjie, Zhang Yu, Mi S. Assisting multimodal named entity recognition by cross-modal auxiliary tasks[J]. Pattern Recognition Letters, 2023, 175: 52−58
2026年第63卷第2期
201
104
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202550273
  • 接收时间:2025-04-21
  • 首发时间:2026-11-06
  • 出版时间:2026-02-01
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2025-04-21
  • 修回日期:2025-08-07
  • 录用日期:2025-09-02
基金
作者信息
    东北大学计算机科学与工程学院 沈阳 110167

通讯作者:

参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202550273
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏