Article(id=1329461025973318555, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1329460877864055544, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202330960, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=null, receivedDate=1701273600000, receivedDateStr=2023-11-30, revisedDate=1710172800000, revisedDateStr=2024-03-12, acceptedDate=null, acceptedDateStr=null, onlineDate=1794897808783, onlineDateStr=2026-11-17, pubDate=1714492800000, pubDateStr=2024-05-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1794897808783, onlineIssueDateStr=2026-11-17, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1794897808783, creator=13701087609, updateTime=1794897808783, updator=13701087609, issue=Issue{id=1329460877864055544, tenantId=1146029695717560320, journalId=1324759914146529478, year='2024', volume='61', issue='5', pageStart='1085', pageEnd='1347', issueExtLink='null', onlineDate='null', pubDate='1714492800000', pubDateStr='2024-05-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1794897773471, creator='13701087609', updateTime=1794897773471, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext=null, issueFiles=null, downloadFileDto=null}, startPage=1206, endPage=1217, ext={EN=ArticleExt(id=1329461026141090716, articleId=1329461025973318555, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=A Method for Generating Explanations of Offensive Memes Based on Multimodal Large Language Models, columnId=1329460878547727097, journalTitle=Journal of Computer Research and Development, columnName=Special Issue on Large Models and Safety, runingTitle=null, highlight=null, articleAbstract=

With the advancement of 5G technology, offensive speech has increasingly proliferated across social networks in the form of multimodal memes. Consequently, the detection and interpretive generation of offensive memes play a crucial role in enhancing content moderation effectiveness and maintaining a harmonious and healthy public discourse environment. Existing studies on the interpretive generation of offensive memes focus solely on the targets and content of offense, neglecting the societal background knowledge and metaphorical expressions embedded in memes. This oversight limits the ability to comprehensively and accurately interpret the meaning of offensive memes, thus constraining the applicability of interpretations. To address this challenge, we propose a method based on multimodal large language model for generating interpretations of offensive memes. By augmenting elements such as offense targets, the content of the offense, and metaphor recognition into the instruction tuning data, we can effectively improve the multimodal large model’s proficiency in interpretively generating offensive memes through instruction tuning. The experimental outcomes validate three key strengths of our method: first, it achieves a notable 19% enhancement in the BERTScore evaluation metric over baseline models; second, it incorporates comprehensive background knowledge pertinent to offensive metaphorical expressions within its interpretations; third, it exhibits strong generalization capabilities when handling previously unseen meme data.

, authors=Meng Lin1, 2, Chengwei Dai1, 2, Tao Guo1, *, authorsList=Meng Lin, Chengwei Dai, Tao Guo, authorCompany=null, correspAuthors=Tao Guo, authorNote=

born in 1991. PhD candidate. Her main research interests include multi-modal hate speech detection and multimodal semantic understanding

born in 2000. Master candidate. His main research interests include model extraction and large language model distillation

born in 1974. PhD, Professor, PhD supervisor. His main research interests include cybersecurity, vulnerability analysis and risk assessment

, correspAuthorsNote=null, copyrightStatement=Copyright © 2024 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1329461029286818733, articleId=1329461025973318555, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=基于多模态大语言模型的攻击性模因解释生成方法, columnId=1329460878694527739, journalTitle=计算机研究与发展, columnName=大模型与安全专题, runingTitle=null, highlight=null, articleAbstract=

随着5G的发展,攻击性言论逐渐以多模态的方式在社交网络上广泛传播. 因此,攻击性模因的检测与解释生成对于提高内容审核效果、维护和谐健康的舆论场环境有着重要的作用. 现有的攻击性模因解释生成研究只关注于攻击对象和攻击内容,忽略了模因包含的社会背景知识和隐喻表达手法,无法全面、准确地解释攻击性模因的含义,大大限制了解释的应用范围. 为了应对这一挑战,提出一种基于多模态大模型的攻击性模因解释生成方法,通过增强攻击目标、攻击内容和隐喻识别等多种指令数据,利用其微调多模态大模型,以提升大模型对攻击性模因的解释生成能力. 实验结果证实,该方法生成的解释具有3点优势:一是相比基线模型在BERTScore评估指标上提高了19%;二是解释中包含了攻击性隐喻表达的相关背景知识;三是在处理未见的模因数据时也表现出良好的泛化性能.

, authors=林萌1, 2, 戴程威1, 2, 郭涛1, *, authorsList=林萌, 戴程威, 郭涛, authorCompany=null, correspAuthors=郭涛, authorNote=

1991 年生. 博士研究生. 主要研究方向为多模态仇恨言论检测、多模态语义理解

2000 年生. 硕士研究生. 主要研究方向为模型提取、大模型蒸馏

1974 年生. 博士,教授,博士生导师. 主要研究方向为网络空间安全、漏洞分析与风险评估

, correspAuthorsNote=
郭涛()
, copyrightStatement=版权所有 © 2024 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=s8ZcxUmIt7Yf2MxnRGt7rQ==, magXml=iivGAnqV96BCwpPGqxzAow==, pdfUrl=null, pdf=ZLKIwfrF76goKkb2HxX3LQ==, pdfFileSize=1394419, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=AN06igNnqxBz30rSulOP8A==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=38uOgau8KtyOqp68wVI/Bg==, mapNumber=null, fund=null)}, authors=[Author(id=1329461143766152171, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=linmeng@iie.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329461143837455342, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, authorId=1329461143766152171, language=EN, stringName=Meng Lin, firstName=Meng, middleName=null, lastName=Lin, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100019
2School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049, bio={"content":"

born in 1991. PhD candidate. Her main research interests include multi-modal hate speech detection and multimodal semantic understanding

"}, bioImg=null, bioContent=

born in 1991. PhD candidate. Her main research interests include multi-modal hate speech detection and multimodal semantic understanding

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329461143912952815, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, authorId=1329461143766152171, language=CN, stringName=林萌, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1中国科学院信息工程研究所 北京 100019
2中国科学院大学网络空间安全学院 北京 100049, bio={"img":"GpVHZau6HB8jvretondWnA==","content":"

1991 年生. 博士研究生. 主要研究方向为多模态仇恨言论检测、多模态语义理解

"}, bioImg=GpVHZau6HB8jvretondWnA==, bioContent=

1991 年生. 博士研究生. 主要研究方向为多模态仇恨言论检测、多模态语义理解

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329461142067459044, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, xref=1, ext=[AuthorCompanyExt(id=1329461142071653349, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142067459044, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100019), AuthorCompanyExt(id=1329461142080041958, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142067459044, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院信息工程研究所 北京 100019)]), AuthorCompany(id=1329461142172316647, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, xref=2, ext=[AuthorCompanyExt(id=1329461142180705256, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142172316647, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049), AuthorCompanyExt(id=1329461142184899561, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142172316647, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学网络空间安全学院 北京 100049)])]), Author(id=1329461143992644593, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329461144126862324, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, authorId=1329461143992644593, language=EN, stringName=Chengwei Dai, firstName=Chengwei, middleName=null, lastName=Dai, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100019
2School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049, bio={"content":"

born in 2000. Master candidate. His main research interests include model extraction and large language model distillation

"}, bioImg=null, bioContent=

born in 2000. Master candidate. His main research interests include model extraction and large language model distillation

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329461144202359797, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, authorId=1329461143992644593, language=CN, stringName=戴程威, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1中国科学院信息工程研究所 北京 100019
2中国科学院大学网络空间安全学院 北京 100049, bio={"img":"3fvwRyYLSf5waVcqrJxScA==","content":"

2000 年生. 硕士研究生. 主要研究方向为模型提取、大模型蒸馏

"}, bioImg=3fvwRyYLSf5waVcqrJxScA==, bioContent=

2000 年生. 硕士研究生. 主要研究方向为模型提取、大模型蒸馏

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329461142067459044, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, xref=1, ext=[AuthorCompanyExt(id=1329461142071653349, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142067459044, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100019), AuthorCompanyExt(id=1329461142080041958, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142067459044, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院信息工程研究所 北京 100019)]), AuthorCompany(id=1329461142172316647, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, xref=2, ext=[AuthorCompanyExt(id=1329461142180705256, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142172316647, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049), AuthorCompanyExt(id=1329461142184899561, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142172316647, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学网络空间安全学院 北京 100049)])]), Author(id=1329461144282051575, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=guotao@iie.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329461144357549049, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, authorId=1329461144282051575, language=EN, stringName=Tao Guo, firstName=Tao, middleName=null, lastName=Guo, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, *, address=1Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100019, bio={"content":"

born in 1974. PhD, Professor, PhD supervisor. His main research interests include cybersecurity, vulnerability analysis and risk assessment

"}, bioImg=null, bioContent=

born in 1974. PhD, Professor, PhD supervisor. His main research interests include cybersecurity, vulnerability analysis and risk assessment

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329461144445629434, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, authorId=1329461144282051575, language=CN, stringName=郭涛, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, *, address=1中国科学院信息工程研究所 北京 100019, bio={"img":"IQ9A+zOfNAAoZrpN+u9HPw==","content":"

1974 年生. 博士,教授,博士生导师. 主要研究方向为网络空间安全、漏洞分析与风险评估

"}, bioImg=IQ9A+zOfNAAoZrpN+u9HPw==, bioContent=

1974 年生. 博士,教授,博士生导师. 主要研究方向为网络空间安全、漏洞分析与风险评估

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329461142067459044, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, xref=1, ext=[AuthorCompanyExt(id=1329461142071653349, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142067459044, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100019), AuthorCompanyExt(id=1329461142080041958, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142067459044, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院信息工程研究所 北京 100019)])])], keywords=[Keyword(id=1329461144646956027, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=EN, orderNo=1, keyword=offensive meme), Keyword(id=1329461144730842108, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=EN, orderNo=2, keyword=explanation generation), Keyword(id=1329461144814728189, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=EN, orderNo=3, keyword=multi-modal LLM), Keyword(id=1329461144890225662, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=EN, orderNo=4, keyword=data augment), Keyword(id=1329461144961528831, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=EN, orderNo=5, keyword=instruction fine-tuning), Keyword(id=1329461145024443392, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=CN, orderNo=1, keyword=攻击性模因), Keyword(id=1329461145099939840, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=CN, orderNo=2, keyword=解释生成), Keyword(id=1329461145158660097, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=CN, orderNo=3, keyword=多模态大语言模型), Keyword(id=1329461145238351874, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=CN, orderNo=4, keyword=数据增强), Keyword(id=1329461145297072131, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=CN, orderNo=5, keyword=指令微调)], refs=[Reference(id=1329461148988059678, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=虎嵩林,赵军,唐杰,等. 虚假信息检测专题前言[J]. 计算机研究与发展,2021,58(7):1351−1352, articleTitle=null, refAbstract=null), Reference(id=1329461149046779935, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Hu Songlin, Zhao Jun, Tang Jie, et al. Preface to the special issue on fake information detection[J]. Journal of Computer Research and Development, 2021, 58(7): 1351−1352 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1329461149109694496, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Kiela D, Firooz H, Mohan A, et al. The hateful memes challenge: Detecting hate speech in multimodal memes[J]. Advances in Neural Information Processing Systems, 2020, 33: 2611−2624, articleTitle=null, refAbstract=null), Reference(id=1329461149168414753, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Linhao, Jin Li, Sun Xian, et al. TOT: Topology-aware optimal transport for multimodal hate detection[C]//Proc of the AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023, 37(4): 4884−4892, articleTitle=null, refAbstract=null), Reference(id=1329461149222940706, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Cao Rui, Hee MS, Kuek A, et al. Pro-Cap: Leveraging a frozen vision-language model for hateful meme detection[C]//Proc of the 31st ACM Int Conf on Multimedia. New York: ACM, 2023: 5244−5252, articleTitle=null, refAbstract=null), Reference(id=1329461149281660963, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Hee M S, Chong W H, Lee R K W. Decoding the underlying meaning of multimodal hateful memes[C]//Proc of the 32nd Int Joint Conf on Artificial Intelligence. Freiburg: IJCAI, 2023: 5995−6003, articleTitle=null, refAbstract=null), Reference(id=1329461149344575524, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Sharma S, Agarwal S, Suresh T, et al. What do you MEME? Generating explanations for visual semantic role labelling in memes[C]//Proc of the AAAI Conf on Artificial Intelligence. Washington, DC: AAAI, 2023, 37(8): 9763−9771, articleTitle=null, refAbstract=null), Reference(id=1329461149403295781, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Scott K. Memes as multimodal metaphors: A relevance theory analysis[J]. Pragmatics & Cognition, 2021, 28(2): 277−298, articleTitle=null, refAbstract=null), Reference(id=1329461149487181862, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Pramanick S, Sharma S, Dimitrov D, et al. MOMENTA: A multimodal framework for detecting harmful memes and their targets[C]//Findings of the Association for Computational Linguistics: EMNLP 2021. Stroudsburg, PA: ACL, 2021: 4439−4455, articleTitle=null, refAbstract=null), Reference(id=1329461149550096423, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhu Ron. Enhance multimodal Transformer with external label and in-domain pretrain: Hateful meme challenge winning solution[J]. arXiv preprint, arXiv: 2012.08290, 2020, articleTitle=null, refAbstract=null), Reference(id=1329461149604622376, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Chuanpeng, Zhu Fuqing, Liu Guihua, et al. Multimodal hate speech detection via cross-domain knowledge transfer[C]//Proc of the 30th ACM Int Conf on Multimedia. New York: ACM, 2022: 4505−4514, articleTitle=null, refAbstract=null), Reference(id=1329461149663342633, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Lee R K W, Cao Rui, Fan Ziqing, et al. Disentangling hate in online memes[C]//Proc of the 29th ACM Int Conf on Multimedia. New York: ACM, 2021: 5138−5147, articleTitle=null, refAbstract=null), Reference(id=1329461149722062890, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Velioglu R, Rose J. Detecting hate speech in memes using multimodal deep learning approaches: Prize-winning solution to hateful memes challenge[J]. arXiv preprint, arXiv: 2012.12975, 2020, articleTitle=null, refAbstract=null), Reference(id=1329461149801754667, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Yin Shukang, Fu Chaoyou, Zhao Sirui, et al. A survey on multimodal large language models[J]. arXiv preprint, arXiv: 2306.13549, 2023, articleTitle=null, refAbstract=null), Reference(id=1329461149868863532, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Gupta T, Kembhavi A. Visual programming: Compositional visual reasoning without training[C]//Proc of the IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 14953−14962, articleTitle=null, refAbstract=null), Reference(id=1329461149931778093, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Shao Zhenwei, Yu Zhou, Wang Meng, et al. Prompting large language models with answer heuristics for knowledge-based visual question answering[C]//Proc of the IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 14974−14983, articleTitle=null, refAbstract=null), Reference(id=1329461149994692654, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Cao Rui, Lee R K W, Chong W H, et al. Prompting for multimodal hateful meme classification[C]//Proc of the 2022 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2022: 321−332, articleTitle=null, refAbstract=null), Reference(id=1329461150061801519, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Yinhan, Ott M, Goyal N, et al. RoBERTa: A robustly optimized BERT pretraining approach[J]. arXiv preprint, arXiv: 1907.11692, 2019, articleTitle=null, refAbstract=null), Reference(id=1329461150124716080, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Ji Junhui, Ren Wei, Naseem U. Identifying creative harmful memes via prompt based approach[C]//Proc of the ACM Web Conf 2023. New York: ACM, 2023: 3868−3872, articleTitle=null, refAbstract=null), Reference(id=1329461150275711025, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Hwang E J, Shwartz V. MemeCap: A dataset for captioning and interpreting memes[J]. arXiv preprint, arXiv: 2305.13703, 2023, articleTitle=null, refAbstract=null), Reference(id=1329461150334431282, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhu Deyao, Chen Jun, Shen Xiaoqian, et al. MiniGPT-4: Enhancing vision-language understanding with advanced large language models[J]. arXiv preprint, arXiv: 2304.10592, 2023, articleTitle=null, refAbstract=null), Reference(id=1329461150397345843, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Renhui, Han Jiaming, Liu Chris, et al. LLaMA-Adapter: Efficient fine-tuning of language models with zero-init attention[J]. arXiv preprint, arXiv: 2303.16199, 2023, articleTitle=null, refAbstract=null), Reference(id=1329461150460260404, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Gao Peng, Han Jiaming, Zhang Renrui, et al. LLaMA-Adapter V2: Parameter-efficient visual instruction model[J]. arXiv preprint, arXiv: 2304.15010, 2023, articleTitle=null, refAbstract=null), Reference(id=1329461150527369269, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Haotian, Li Chunyuan, Wu Qingyang, et al. Visual instruction tuning[J]. Advances in Neural Information Processing Systems, 2024. DOI: 10.48550/arXiv.2304.08485, articleTitle=null, refAbstract=null), Reference(id=1329461150602866742, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=Horawalavithana S, Munikoti S, Stewart I, et al. SciTune: Aligning large language models with scientific multimodal instructions[J]. arXiv preprint, arXiv: 2307.01139, 2023, articleTitle=null, refAbstract=null), Reference(id=1329461150678364215, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=Wei J, Zou Kai. EDA: Easy data augmentation techniques for boosting performance on text classification tasks[C]//Proc of the 2019 Conf on Empirical Methods in Natural Language Processing and the 9th Int Joint Conf on Natural Language Processing (EMNLP-IJCNLP). Stroudsburg, PA: ACL, 2019: 6382−6388, articleTitle=null, refAbstract=null), Reference(id=1329461150745473080, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=26, authorNames=null, journalName=null, refType=null, unstructuredReference=Wu Xing, Lv Shangwen, Zang Liangjun, et al. Conditional BERT contextual augmentation[C]// Proc of 19th Int Conf on Computational Science(CCS 2019). Berlin: Springer, 2019: 84−95, articleTitle=null, refAbstract=null), Reference(id=1329461150816776249, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=27, authorNames=null, journalName=null, refType=null, unstructuredReference=Kumar V, Choudhary A, Cho E. Data augmentation using pre-trained Transformer models[C]//Proc of the 2nd Workshop on Life-long Learning for Spoken Language Systems. Stroudsburg, PA: ACL, 2020: 18−26, articleTitle=null, refAbstract=null), Reference(id=1329461150879690810, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=28, rfOrder=28, authorNames=null, journalName=null, refType=null, unstructuredReference=Radford A, Jeffrey W, Child R, et al. Language models are unsupervised multitask learners[J]. OpenAI Blog, 2019, 1(8): 9−9, articleTitle=null, refAbstract=null), Reference(id=1329461151013908539, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=29, rfOrder=29, authorNames=null, journalName=null, refType=null, unstructuredReference=Kenton J D M W C, Toutanova L K. BERT: Pre-training of deep bidirectional Transformers for language understanding[C]//Proc of Annual Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, PA: ACL, 2019: 4171−4186, articleTitle=null, refAbstract=null), Reference(id=1329461152637104188, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=30, rfOrder=30, authorNames=null, journalName=null, refType=null, unstructuredReference=Lewis M, Liu Yinhan, Goyal N, et al. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[C]//Proc of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 7871−7880, articleTitle=null, refAbstract=null), Reference(id=1329461152725184573, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=31, rfOrder=31, authorNames=null, journalName=null, refType=null, unstructuredReference=Dai Haixing, Liu Zhengliang, Liao Wenxiong, et al. ChatAug: Leveraging ChatGPT for text data augmentation[J]. arXiv preprint, arXiv: 2302.13007, 2023, articleTitle=null, refAbstract=null), Reference(id=1329461152783904830, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=32, rfOrder=32, authorNames=null, journalName=null, refType=null, unstructuredReference=Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//Proc of Int Conf on Machine Learning. New York: PMLR, 2021: 8748−8763, articleTitle=null, refAbstract=null), Reference(id=1329461152863596607, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=33, rfOrder=33, authorNames=null, journalName=null, refType=null, unstructuredReference=Touvron H, Lavril T, Izacard G, et al. Llama: Open and efficient foundation language models[J]. arXiv preprint, arXiv: 2302.13971, 2023, articleTitle=null, refAbstract=null), Reference(id=1329461152939094080, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=34, rfOrder=34, authorNames=null, journalName=null, refType=null, unstructuredReference=Xu Bo, Li Tingting, Zheng Junzhe, et al. MET-Meme: A multimodal meme dataset rich in metaphors[C]//Proc of the 45th Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2022: 2887−2899, articleTitle=null, refAbstract=null), Reference(id=1329461153010397249, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=35, rfOrder=35, authorNames=null, journalName=null, refType=null, unstructuredReference=Dimitrov D, Ali B B, Shaar S, et al. Detecting propaganda techniques in memes[C]//Proc of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int Joint Conf on Natural Language Processing (ACL-IJCNLP 2021). Stroudsburg, PA: 2021: 6603−6617, articleTitle=null, refAbstract=null), Reference(id=1329461153102671938, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=36, rfOrder=36, authorNames=null, journalName=null, refType=null, unstructuredReference=Pramanick S, Sharma S, Dimitrov D, et al. MOMENTA: A multimodal framework for detecting harmful memes and their targets[C]//Findings of the Association for Computational Linguistics: EMNLP 2021. Stroudsburg, PA: ACL, 2021: 4439−4455, articleTitle=null, refAbstract=null), Reference(id=1329461153169780803, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=37, rfOrder=37, authorNames=null, journalName=null, refType=null, unstructuredReference=Cai Yitao, Cai Huiyu, Wan Xiaojun. Multi-modal sarcasm detection in twitter with hierarchical fusion model[C]//Proc of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2019: 2506−2515, articleTitle=null, refAbstract=null), Reference(id=1329461153228501060, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=38, rfOrder=38, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Junnan, Li Dongxu, Savarese S, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models[J]. arXiv preprint, arXiv: 2301.12597, 2023, articleTitle=null, refAbstract=null), Reference(id=1329461153299804229, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=39, rfOrder=39, authorNames=null, journalName=null, refType=null, unstructuredReference=Papineni K, Roukos S, Ward T, et al. BLEU: A method for automatic evaluation of machine translation[C]//Proc of the 40th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2002: 311−318, articleTitle=null, refAbstract=null), Reference(id=1329461153362718790, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=40, rfOrder=40, authorNames=null, journalName=null, refType=null, unstructuredReference=Lin Chin-Yew. Rouge: A package for automatic evaluation of summaries[C]//Text Summarization Branches Out. Stroudsburg, PA: ACL, 2004: 74−81, articleTitle=null, refAbstract=null), Reference(id=1329461153417244743, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=41, rfOrder=41, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Tianyi, Kishore V, Wu Felix, et al. BERTScore: Evaluating text generation with BERT[J]. arXiv preprint, arXiv: 1904.09675, 2019, articleTitle=null, refAbstract=null), Reference(id=1329461153471770696, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=42, rfOrder=42, authorNames=null, journalName=null, refType=null, unstructuredReference=Raffel C, Shazeer N, Roberts A, et al. Exploring the limits of transfer learning with a unified text-to-text transformer[J]. The Journal of Machine Learning Research, 2020, 21(1): 5485−5551, articleTitle=null, refAbstract=null), Reference(id=1329461153551462473, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=43, rfOrder=43, authorNames=null, journalName=null, refType=null, unstructuredReference=Fersini E, Gasparini F, Rizzi G, et al. SemEval-2022 Task 5: Multimedia automatic misogyny identification[C]//Proc of the 16th Int Workshop on Semantic Evaluation (SemEval-2022). Stroudsburg, PA: ACL, 2022: 533−549, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1329461142067459044, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, xref=1, ext=[AuthorCompanyExt(id=1329461142071653349, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142067459044, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100019), AuthorCompanyExt(id=1329461142080041958, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142067459044, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院信息工程研究所 北京 100019)]), AuthorCompany(id=1329461142172316647, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, xref=2, ext=[AuthorCompanyExt(id=1329461142180705256, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142172316647, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049), AuthorCompanyExt(id=1329461142184899561, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142172316647, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学网络空间安全学院 北京 100049)])], figs=[ArticleFig(id=1329461145490010116, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=EN, label=Figure 1, caption=Example of interpretive generation of offensive memes, figureFileSmall=m7rw3SuhDkFbksecFCY3OA==, figureFileBig=AN06igNnqxBz30rSulOP8A==, tableContent=null), ArticleFig(id=1329461145569701893, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=CN, label=图1, caption=攻击性模因解释生成示例, figureFileSmall=m7rw3SuhDkFbksecFCY3OA==, figureFileBig=AN06igNnqxBz30rSulOP8A==, tableContent=null), ArticleFig(id=1329461145712308230, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=EN, label=Figure 2, caption=Illustration of the proposed model, figureFileSmall=xxTsTlUnQgx9VQENasnf6w==, figureFileBig=hkOMxjY8VrKV9b4Yo2eHbA==, tableContent=null), ArticleFig(id=1329461145779417095, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=CN, label=图2, caption=所提模型的示意图, figureFileSmall=xxTsTlUnQgx9VQENasnf6w==, figureFileBig=hkOMxjY8VrKV9b4Yo2eHbA==, tableContent=null), ArticleFig(id=1329461145838137352, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=EN, label=Figure 3, caption=Augmenting explanatory examples with GPT-3.5, figureFileSmall=bn+s414Yz+OssmRFm8nKmw==, figureFileBig=s9y78kAGPePNpBhqKV3D2w==, tableContent=null), ArticleFig(id=1329461145905246217, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=CN, label=图3, caption=GPT-3.5解释增强示例, figureFileSmall=bn+s414Yz+OssmRFm8nKmw==, figureFileBig=s9y78kAGPePNpBhqKV3D2w==, tableContent=null), ArticleFig(id=1329461145984937994, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=EN, label=Figure 4, caption=Instruction set, figureFileSmall=s3f1YIzkFCAVufy2h/JeVA==, figureFileBig=lpVnImal5tL4+Knr6nFdlQ==, tableContent=null), ArticleFig(id=1329461146056241163, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=CN, label=图4, caption=指令集, figureFileSmall=s3f1YIzkFCAVufy2h/JeVA==, figureFileBig=lpVnImal5tL4+Knr6nFdlQ==, tableContent=null), ArticleFig(id=1329461146123350028, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=EN, label=Figure 5, caption=Comparison of three types of explanations with examples, figureFileSmall=743ZaqRFBzbGJCzaNYjkzA==, figureFileBig=ekSD4S0drIQeCGhUSW9xqw==, tableContent=null), ArticleFig(id=1329461146186264589, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=CN, label=图5, caption=举例说明3种解释的比较, figureFileSmall=743ZaqRFBzbGJCzaNYjkzA==, figureFileBig=ekSD4S0drIQeCGhUSW9xqw==, tableContent=null), ArticleFig(id=1329461146253373454, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=EN, label=Table 1, caption=

Statistical Information of the Datasets

, figureFileSmall=null, figureFileBig=null, tableContent=
数据集模因数量数据集说明隐喻标签模因解释是否攻击性模因
HatReD[5]3228仇恨模因解释生成无有是
MemeCap[19]6384模因解释生成有有否
MET-Meme[34]10045模因隐喻分类有有(增强后)否
Propaganda[35]950模因隐喻分类有有(增强后)否
HarMeme[36]3544有害模因分类无有(增强后)是
Sarcasm Meme[37]24635讽刺模因分类无有(增强后)是
), ArticleFig(id=1329461146312093711, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=CN, label=表1, caption=

数据集统计信息

, figureFileSmall=null, figureFileBig=null, tableContent=
数据集模因数量数据集说明隐喻标签模因解释是否攻击性模因
HatReD[5]3228仇恨模因解释生成无有是
MemeCap[19]6384模因解释生成有有否
MET-Meme[34]10045模因隐喻分类有有(增强后)否
Propaganda[35]950模因隐喻分类有有(增强后)否
HarMeme[36]3544有害模因分类无有(增强后)是
Sarcasm Meme[37]24635讽刺模因分类无有(增强后)是
), ArticleFig(id=1329461146375008272, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=EN, label=Table 2, caption=

Dataset Division Results

, figureFileSmall=null, figureFileBig=null, tableContent=
数据集训练集测试集
HatReD2982246
MemeCap63840
MET-Meme100450
Propaganda9500
HarMeme35440
Sarcasm Meme246350
总计48540246
), ArticleFig(id=1329461146433728529, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=CN, label=表2, caption=

数据集划分结果

, figureFileSmall=null, figureFileBig=null, tableContent=
数据集训练集测试集
HatReD2982246
MemeCap63840
MET-Meme100450
Propaganda9500
HarMeme35440
Sarcasm Meme246350
总计48540246
), ArticleFig(id=1329461146492448786, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=EN, label=Table 3, caption=

Experimental Results from Automated Evaluations

, figureFileSmall=null, figureFileBig=null, tableContent=
模型N-gram匹配基于BERTScore的评估
类别编码器解码器BLEUROUGE-LBERT-PBERT-RBERT-F
文本模型RoBERTa-baseGPT2-base0.0680.2220.1120.3270.218
RoBERTa-baseRoBERTa-base0.1770.3890.5080.4530.480
T5-largeT5-large0.1900.3920.4850.4730.479
图片
-文本模型
VisualBERTGPT2-base0.0650.2190.1000.3420.219
VisualBERTRoBERTa-base0.1790.3910.4990.4490.474
VL-T5VL-T50.1800.3780.4720.4090.446
本文方法0.0980.2500.6890.6680.677
本文方法与其他方法的差值−0.092−0.1420.1810.1950.197
), ArticleFig(id=1329461146567946259, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=CN, label=表3, caption=

自动评估实验结果

, figureFileSmall=null, figureFileBig=null, tableContent=
模型N-gram匹配基于BERTScore的评估
类别编码器解码器BLEUROUGE-LBERT-PBERT-RBERT-F
文本模型RoBERTa-baseGPT2-base0.0680.2220.1120.3270.218
RoBERTa-baseRoBERTa-base0.1770.3890.5080.4530.480
T5-largeT5-large0.1900.3920.4850.4730.479
图片
-文本模型
VisualBERTGPT2-base0.0650.2190.1000.3420.219
VisualBERTRoBERTa-base0.1790.3910.4990.4490.474
VL-T5VL-T50.1800.3780.4720.4090.446
本文方法0.0980.2500.6890.6680.677
本文方法与其他方法的差值−0.092−0.1420.1810.1950.197
), ArticleFig(id=1329461148182753300, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=EN, label=Table 4, caption=

Experimental Results from GPT-4 Evaluations

, figureFileSmall=null, figureFileBig=null, tableContent=
方法流畅度正确性背景分析能力隐喻识别能力客观性
本文方法4.903.744.423.864.74
T5-large4.702.082.342.484.32
HatReD人工标注答案4.804.663.223.004.90
), ArticleFig(id=1329461148258250773, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=CN, label=表4, caption=

GPT-4评估实验结果

, figureFileSmall=null, figureFileBig=null, tableContent=
方法流畅度正确性背景分析能力隐喻识别能力客观性
本文方法4.903.744.423.864.74
T5-large4.702.082.342.484.32
HatReD人工标注答案4.804.663.223.004.90
), ArticleFig(id=1329461148333748246, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=EN, label=Table 5, caption=

Ablation Experimental Results from GPT-4 Evaluations

, figureFileSmall=null, figureFileBig=null, tableContent=
方法流畅度正确性背景分析能力隐喻识别能力客观性
本文方法4.903.744.423.864.74
w/o 数据增强4.822.462.522.944.48
w/o 隐喻数据集4.863.123.743.244.36
w/o有害数据集4.822.943.363.184.78
), ArticleFig(id=1329461148426022935, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=CN, label=表5, caption=

消融实验GPT-4评估结果

, figureFileSmall=null, figureFileBig=null, tableContent=
方法流畅度正确性背景分析能力隐喻识别能力客观性
本文方法4.903.744.423.864.74
w/o 数据增强4.822.462.522.944.48
w/o 隐喻数据集4.863.123.743.244.36
w/o有害数据集4.822.943.363.184.78
), ArticleFig(id=1329461148493131800, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=EN, label=Table 6, caption=

Ablation Experimental Results from Automated Evaluations

, figureFileSmall=null, figureFileBig=null, tableContent=
方法N-gram匹配基于BERTScore的评价
BLEUROUGE-LBERT-PBERT-RBERT-F
本文方法0.0980.2500.6890.6680.677
w/o 数据增强0.1080.3350.6040.5690.583
w/o 隐喻数据集0.0900.3180.6610.6490.653
w/o有害数据集0.0740.2790.5810.6350.604
), ArticleFig(id=1329461148572823577, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=CN, label=表6, caption=

消融实验自动评估结果

, figureFileSmall=null, figureFileBig=null, tableContent=
方法N-gram匹配基于BERTScore的评价
BLEUROUGE-LBERT-PBERT-RBERT-F
本文方法0.0980.2500.6890.6680.677
w/o 数据增强0.1080.3350.6040.5690.583
w/o 隐喻数据集0.0900.3180.6610.6490.653
w/o有害数据集0.0740.2790.5810.6350.604
), ArticleFig(id=1329461148627349530, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=EN, label=Table 7, caption=

Experimental Results of Offensive Memes Classification Task %

, figureFileSmall=null, figureFileBig=null, tableContent=
数据集无生成原因有生成原因
ACCAUROCACCAUROC
HateMeme33.448.548.050.6
), ArticleFig(id=1329461148702847003, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=CN, label=表7, caption=

攻击性模因分类任务结果

, figureFileSmall=null, figureFileBig=null, tableContent=
数据集无生成原因有生成原因
ACCAUROCACCAUROC
HateMeme33.448.548.050.6
), ArticleFig(id=1329461148765761564, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=EN, label=Table 8, caption=

Experimental Results on Unseen Data

, figureFileSmall=null, figureFileBig=null, tableContent=
方法流畅度正确性背景分析能力隐喻识别能力客观性
本文方法4.903.544.264.124.85
T5-large4.322.482.732.784.60
), ArticleFig(id=1329461148841259037, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, language=CN, label=表8, caption=

在未见数据集上的实验结果

, figureFileSmall=null, figureFileBig=null, tableContent=
方法流畅度正确性背景分析能力隐喻识别能力客观性
本文方法4.903.544.264.124.85
T5-large4.322.482.732.784.60
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202330960, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202330960, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202330960, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202330960, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1714492800000, fullTextJson=null, articleText=null, reference=null)
收藏切换
基于多模态大语言模型的攻击性模因解释生成方法
林萌 1, 2 , 戴程威 1, 2 , 郭涛 1, *
计算机研究与发展 | 大模型与安全专题 2024,61(5): 1206-1217
收起
收藏切换
计算机研究与发展 |大模型与安全专题 2024 , 61 (5) : 1206 -1217
基于多模态大语言模型的攻击性模因解释生成方法
全屏
[Author(id=1329461143766152171, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=linmeng@iie.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329461143837455342, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, authorId=1329461143766152171, language=EN, stringName=Meng Lin, firstName=Meng, middleName=null, lastName=Lin, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100019
2School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049, bio={"content":"

born in 1991. PhD candidate. Her main research interests include multi-modal hate speech detection and multimodal semantic understanding

"}, bioImg=null, bioContent=

born in 1991. PhD candidate. Her main research interests include multi-modal hate speech detection and multimodal semantic understanding

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329461143912952815, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, authorId=1329461143766152171, language=CN, stringName=林萌, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1中国科学院信息工程研究所 北京 100019
2中国科学院大学网络空间安全学院 北京 100049, bio={"img":"GpVHZau6HB8jvretondWnA==","content":"

1991 年生. 博士研究生. 主要研究方向为多模态仇恨言论检测、多模态语义理解

"}, bioImg=GpVHZau6HB8jvretondWnA==, bioContent=

1991 年生. 博士研究生. 主要研究方向为多模态仇恨言论检测、多模态语义理解

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329461142067459044, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, xref=1, ext=[AuthorCompanyExt(id=1329461142071653349, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142067459044, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100019), AuthorCompanyExt(id=1329461142080041958, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142067459044, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院信息工程研究所 北京 100019)]), AuthorCompany(id=1329461142172316647, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, xref=2, ext=[AuthorCompanyExt(id=1329461142180705256, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142172316647, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049), AuthorCompanyExt(id=1329461142184899561, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142172316647, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学网络空间安全学院 北京 100049)])]), Author(id=1329461143992644593, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329461144126862324, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, authorId=1329461143992644593, language=EN, stringName=Chengwei Dai, firstName=Chengwei, middleName=null, lastName=Dai, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100019
2School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049, bio={"content":"

born in 2000. Master candidate. His main research interests include model extraction and large language model distillation

"}, bioImg=null, bioContent=

born in 2000. Master candidate. His main research interests include model extraction and large language model distillation

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329461144202359797, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, authorId=1329461143992644593, language=CN, stringName=戴程威, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1中国科学院信息工程研究所 北京 100019
2中国科学院大学网络空间安全学院 北京 100049, bio={"img":"3fvwRyYLSf5waVcqrJxScA==","content":"

2000 年生. 硕士研究生. 主要研究方向为模型提取、大模型蒸馏

"}, bioImg=3fvwRyYLSf5waVcqrJxScA==, bioContent=

2000 年生. 硕士研究生. 主要研究方向为模型提取、大模型蒸馏

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329461142067459044, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, xref=1, ext=[AuthorCompanyExt(id=1329461142071653349, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142067459044, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100019), AuthorCompanyExt(id=1329461142080041958, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142067459044, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院信息工程研究所 北京 100019)]), AuthorCompany(id=1329461142172316647, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, xref=2, ext=[AuthorCompanyExt(id=1329461142180705256, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142172316647, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049), AuthorCompanyExt(id=1329461142184899561, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142172316647, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学网络空间安全学院 北京 100049)])]), Author(id=1329461144282051575, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=guotao@iie.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329461144357549049, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, authorId=1329461144282051575, language=EN, stringName=Tao Guo, firstName=Tao, middleName=null, lastName=Guo, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, *, address=1Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100019, bio={"content":"

born in 1974. PhD, Professor, PhD supervisor. His main research interests include cybersecurity, vulnerability analysis and risk assessment

"}, bioImg=null, bioContent=

born in 1974. PhD, Professor, PhD supervisor. His main research interests include cybersecurity, vulnerability analysis and risk assessment

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329461144445629434, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, authorId=1329461144282051575, language=CN, stringName=郭涛, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, *, address=1中国科学院信息工程研究所 北京 100019, bio={"img":"IQ9A+zOfNAAoZrpN+u9HPw==","content":"

1974 年生. 博士,教授,博士生导师. 主要研究方向为网络空间安全、漏洞分析与风险评估

"}, bioImg=IQ9A+zOfNAAoZrpN+u9HPw==, bioContent=

1974 年生. 博士,教授,博士生导师. 主要研究方向为网络空间安全、漏洞分析与风险评估

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329461142067459044, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, xref=1, ext=[AuthorCompanyExt(id=1329461142071653349, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142067459044, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100019), AuthorCompanyExt(id=1329461142080041958, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461025973318555, companyId=1329461142067459044, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院信息工程研究所 北京 100019)])])]
林萌1, 2 , 戴程威1, 2, 郭涛1, *
作者信息
  • 1中国科学院信息工程研究所 北京 100019
  • 2中国科学院大学网络空间安全学院 北京 100049
通讯作者:
郭涛()
作者简介:

1991 年生. 博士研究生. 主要研究方向为多模态仇恨言论检测、多模态语义理解

2000 年生. 硕士研究生. 主要研究方向为模型提取、大模型蒸馏

1974 年生. 博士,教授,博士生导师. 主要研究方向为网络空间安全、漏洞分析与风险评估

A Method for Generating Explanations of Offensive Memes Based on Multimodal Large Language Models
Meng Lin1, 2 , Chengwei Dai1, 2, Tao Guo1, *
Affiliations
  • 1Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100019
  • 2School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049
  • born in 1991. PhD candidate. Her main research interests include multi-modal hate speech detection and multimodal semantic understanding

    born in 2000. Master candidate. His main research interests include model extraction and large language model distillation

    born in 1974. PhD, Professor, PhD supervisor. His main research interests include cybersecurity, vulnerability analysis and risk assessment

出版时间: 2024-05-01 doi: 10.7544/issn1000-1239.202330960
文章导航
收藏切换

随着5G的发展,攻击性言论逐渐以多模态的方式在社交网络上广泛传播. 因此,攻击性模因的检测与解释生成对于提高内容审核效果、维护和谐健康的舆论场环境有着重要的作用. 现有的攻击性模因解释生成研究只关注于攻击对象和攻击内容,忽略了模因包含的社会背景知识和隐喻表达手法,无法全面、准确地解释攻击性模因的含义,大大限制了解释的应用范围. 为了应对这一挑战,提出一种基于多模态大模型的攻击性模因解释生成方法,通过增强攻击目标、攻击内容和隐喻识别等多种指令数据,利用其微调多模态大模型,以提升大模型对攻击性模因的解释生成能力. 实验结果证实,该方法生成的解释具有3点优势:一是相比基线模型在BERTScore评估指标上提高了19%;二是解释中包含了攻击性隐喻表达的相关背景知识;三是在处理未见的模因数据时也表现出良好的泛化性能.

攻击性模因  /  解释生成  /  多模态大语言模型  /  数据增强  /  指令微调

With the advancement of 5G technology, offensive speech has increasingly proliferated across social networks in the form of multimodal memes. Consequently, the detection and interpretive generation of offensive memes play a crucial role in enhancing content moderation effectiveness and maintaining a harmonious and healthy public discourse environment. Existing studies on the interpretive generation of offensive memes focus solely on the targets and content of offense, neglecting the societal background knowledge and metaphorical expressions embedded in memes. This oversight limits the ability to comprehensively and accurately interpret the meaning of offensive memes, thus constraining the applicability of interpretations. To address this challenge, we propose a method based on multimodal large language model for generating interpretations of offensive memes. By augmenting elements such as offense targets, the content of the offense, and metaphor recognition into the instruction tuning data, we can effectively improve the multimodal large model’s proficiency in interpretively generating offensive memes through instruction tuning. The experimental outcomes validate three key strengths of our method: first, it achieves a notable 19% enhancement in the BERTScore evaluation metric over baseline models; second, it incorporates comprehensive background knowledge pertinent to offensive metaphorical expressions within its interpretations; third, it exhibits strong generalization capabilities when handling previously unseen meme data.

offensive meme  /  explanation generation  /  multi-modal LLM  /  data augment  /  instruction fine-tuning
林萌, 戴程威, 郭涛. 基于多模态大语言模型的攻击性模因解释生成方法. 计算机研究与发展, 2024 , 61 (5) : 1206 -1217 . DOI: 10.7544/issn1000-1239.202330960
Meng Lin, Chengwei Dai, Tao Guo. A Method for Generating Explanations of Offensive Memes Based on Multimodal Large Language Models[J]. Journal of Computer Research and Development, 2024 , 61 (5) : 1206 -1217 . DOI: 10.7544/issn1000-1239.202330960
多模态数据由于其丰富的表现形式,已经成为社交媒体上主要的信息传播载体. 其中由图像与文本结合而成的模因数据,逐渐成为社交媒体上散布攻击性言论的主要手段之一. 攻击性模因往往包含歧视和暴力倾向等负面内容,可能会引发虚假信息的传播[1],损害社会公众的认知和判断能力,助长网络欺凌的滋生,进一步加剧社会矛盾. 面对海量的模因数据,人工审核方式给社交平台带来了沉重的经济负担,亟需通过多模态内容理解技术,实现对攻击性模因的自动识别. 2020年,Facebook举办的多模态仇恨模因检测[2]的比赛,引发了对攻击性模因检测的研究高潮. 研究者从多个角度提出了不同的识别方法,并取得了良好的识别效果[3-4].
虽然仇恨言论识别的研究取得了较大的进展,但关于攻击性模因解释生成的研究目前仍处于起步阶段. 生成攻击性模因的解释有助于内容审核员更深入地理解模因的仇恨内涵,从而制定有效的审核策略. 同时详尽的解释也有助于提高分类模型对于潜在语义的理解,提升识别效果. 生成的解释还有助于减少由于文化偏差引发的用户无意传播,引导用户更加谨慎地参与社交媒体,进一步减少恶意内容的传播.
近年来,研究人员逐步认识到攻击性模因解释生成任务的重要性. Hee等人[5]在2020年仇恨模因分类数据集上,采用“贬低性动词+攻击目标+攻击内容”的模板方式,人工标注生成了简明扼要的仇恨模因解释,公布了目前第一个也是唯一一个开源的攻击性模因解释生成数据集HatReD,为攻击性模因解释生成研究提供了评测基准. 之后,Sharma等人[6]提出为攻击性模因中的语义角色生成解释的方法,该方法首先识别模因中的语义角色,再对其中的每一个语义角色生成模板化的简明解释.
已有研究生成的解释主要聚焦于攻击目标和攻击内容,而忽略了对攻击性模因隐喻表达方式和社会背景知识的解释,没有相关背景知识的人员理解其含义依然有一定门槛. Scott[7]的研究证明隐喻是模因尤其是攻击性模因的主要表达方式之一. 因此,识别攻击性模因中的隐喻表达手法和相关联的社会背景知识,对于生成完整准确、易于理解的解释至关重要. 如图1所示,只有结合德国纳粹在二战期间屠杀犹太人等背景知识,识别出“烹饪”隐喻了“屠杀”行为,才能正确理解这个模因所表达的深刻含义以及展现出的极端攻击性.
为了生成全面翔实的攻击性模因解释,本文构建了一个面向模因解释生成的指令微调数据集,并提出了一个基于多模态大语言模型的攻击性模因解释生成方法. 首先,基于已有的6个公开数据集(3个隐喻模因数据集和3个攻击性模因数据集),利用GPT-3.5丰富的常识知识与强大的语义理解能力,为模因数据生成相应的解释. 其次,根据增强后的数据,构建一个多模态指令微调数据集. 该指令既包括了攻击性模因的攻击性属性,比如分类标签、攻击对象、攻击内容等,也包含了隐喻相关属性,比如隐喻分类、隐喻标签和相关的社会背景知识等. 最后,利用构建的指令微调数据集对多模态大语言模型进行微调. 通过这3步,从而使本文方法能够深入学习模因中的隐喻宣传技巧,同时精确匹配视觉和文本中的仇恨特征,进而有效地生成攻击性模因全面且准确的解释.
综上,本文的贡献主要在于3个方面:
1)基于GPT-3.5构建了一组模因解释生成指令数据集,该数据集包含3个攻击性模因数据集和3个隐喻模因数据集,为攻击性模因解释生成研究提供了评测数据集和评测基准.
2)提出了一种针对攻击性模因解释生成任务的多模态大语言模型训练方法,该方法不仅学习到模因固有的隐喻表达方式,也对齐跨模态的视觉和文本攻击性特征,从而生成一个全面准确的攻击性模因解释.
3)在HatReD数据集上进行实验,实验结果表明本文方法在BERT-Score-F评测指标上较已有方法有19%的提升,并具备较强的隐喻表达识别能力与背景知识关联能力. 另外,本文模型在处理新的模因数据时同样表现出良好的泛化能力.
随着移动互联网的快速发展,攻击言论呈现出多样化趋势,特别是多模态形式的攻击言论日渐增多,使得其检测工作成为研究领域的新热点. Kiela等人[2]发布了一个精心设计的多模态仇恨模因数据集,该数据集结合了图像和文本2种模态,并特意加入了一些难以辨识的样本(单一模态下看似正常,但跨模态结合后则成为仇恨模因),这大大增加了检测的难度. 基于Kiela的工作,部分研究者[8-9]通过微调多模态预训练模型来增强特征表示能力,以提升有害模因检测的效果. 还有研究通过数据增强[10]、解耦表征[11]和模型融合[12]等方法来增强多模态模型在仇恨言论检测方面的能力
近年来,预训练大语言模型领域取得了重大进展,同时视觉基础模型在感知能力上也有显著的进步. 基于对这些模型互补优势的认识,应用预训练大语言模型解决多模态任务成为一种新的研究方向[13]. Gupta等人[14]利用GPT-3进行视觉推理任务,通过大语言模型的上下文学习能力生成类似Python的模块化程序,并执行这些程序以获得结果. Shao等人[15]探索了如何引导GPT-3进行多模态视觉问答的问题,他们首先在VQA数据集上训练VQA模型,生成2种类型的回答启发式信息,再将这些信息与问题和图像描述整合成格式化的提示,指导GPT-3预测答案. 在攻击性模因识别领域,Cao等人[16]构建了简单的提示并结合上下文示例,利用预训练RoBERTa[17]语言模型的隐式知识进行仇恨模因分类. Ji等人[18]指出,在传统的预训练模型中,同时使用文本和图像作为输入可能会在特征空间造成显著的语义差异,这样的差异可能对模型的性能造成不利影响. 鉴于此,他们提出首先生成视觉数据的描述性文本和关键属性信息,以此将多模态数据转化为纯文本模式. 随后,这些文本数据被融合进基于提示的预训练语言模型中,从而有效实现对多模态有害模因的检测. 这些研究主要聚焦于提升攻击性模因检测性能,并未涉足攻击性模因的解释生成研究.
由于模因通常包含大量隐喻的修辞手法,当前的图像描述生成方法主要致力于准确描述图像场景,而无法解释其蕴含的深层含义. 随着对多模态模因研究的深入,研究者们进一步探索模因的生成背景和传播环境,提出了模因的解释生成研究领域.
Hee等人[5]构建了一个名为HatReD的多模态仇恨模因解释生成数据集. 该数据集经过人工精心标注,旨在深化对仇恨模因背后含义的理解. 此外,他们利用先进的预训练语言模型对这一任务进行了基准性能评估,发现即便是当前领先的多模态预训练模型,在生成仇恨模因的解释方面,与人类专家相比也存在显著差距. 特别是在面对新领域的模因时,生成的解释也常常缺乏准确性和相关性. 这项研究为未来攻击性模因解释生成的领域研究提供了一个重要的基准和研究起点. Hwang等人[19]构建了一个名为MemeCap的数据集. 该数据集的每个模因都标注了模因的文字信息、图像描述、隐喻表达以及人工生成的模因解释. Hwang等人使用了最新的多模态模型对模因解释生成任务进行了广泛的实验. 实验结果表明,现有的模型在理解视觉隐喻方面仍然存在困难,并且性能远远低于人类水平. 模因的解释生成,尤其是攻击性模因的解释生成目前仍处于研究起步阶段,随着大模型的发展,该领域也逐渐受到更多关注.
多模态大语言模型是指利用大语言模型强大的推理能力,通过调整模型结构使其接收多模态信息并对其进行推理. 由于多模态大语言模型更符合人类感知世界的方法,由GPT-4展示的惊人效果,引发了多模态大语言模型的研究热潮.
Zhu等人[20]提出了MiniGPT-4模型,该模型由一个冻结的ViT和Q-Former的视觉编码器、一个线性投影层和冻结的Vicuna组成. MiniGPT-4只需要训练线性层来对齐视觉特征和Vicuna. Zhang等人[21]提出LLaMA-Adapter的多模态模型,该模型基于门控的零初始化注意力机制,自适应地将图像特征注入LLaMA,同时有效地保留其预训练的知识,该模型在下游任务中表现优越. 在LLaMA-Adapter基础上,Gao等人[22]提出了LLaMA-Adapter V2的模型,该模型是一种高效调参的视觉指令跟随多模态大模型,在微调参数更少的情况下取得了比LLaMA-Adapter更好的下游任务识别效果. Liu等人[23]提出LLaVA模型,通过请求GPT-4构造指令微调训练数据,然后在训练中加入视觉令牌进行指令微调,利用GPT-4的回复实现与用户意图的对齐. Horawalavithana等人[24]提出了LLaMA-SciTune模型. 构建了科学任务领域的专用多模态大语言模型,在科学问题回答任务上超越了人类的表现.
数据增强方法通常用于防止过度拟合并提高深度神经网络模型的泛化能力,是解决样本量有限任务的有效策略. 早期,Wei等人[25]提出用于提高文本分类任务性能的简单数据增强(easy data augmentation,EDA)技术. EDA由4种简单但功能强大的操作组成:同义词替换、随机插入、随机交换和随机删除. EDA对于较小的数据集表现出特别强大的结果. 后来,预训练语言模型PLMs受到广泛关注后,Wu等人[26]提出CBERT方法,通过掩码填词任务构建与原始输入文本具有高语义相关性的增强样本. Kumar等人[27]进一步探索了条件数据增强的自回归模型GPT2[28]、自动编码器模型BERT[29]和seq2seq模型BART[30]. 最近大语言模型展现出了强大的数据增强性能,Dai等人[31]提出ChatAug方法,利用ChatGPT将训练样本中的每个句子重新表述为多个概念相似但语义不同的样本,来增强下游模型训练.
本文方法的模型结构图如图2所示,主要包含3个模块,分别是多模态大模型基座、基于GPT-3.5的攻击性模因解释增强和多数据集指令微调. 多模态大模型基座提供了基础的多模态信息处理和解释生成能力. 基于GPT-3.5的模因解释增强模块为多个模因数据集提供了模因解释,以解决模因解释数据集的缺失的问题. 多数据集指令微调训练则进一步增强了通用多模态大模型基座对模因隐喻表达的理解,同时提高了跨模态仇恨识别能力.
LLaMA-Adapter V2[22]是一种基于适配器(adapter)的向预训练大语言模型注入图像知识的方法,在多个下游任务中取得了良好的效果. 由于该模型是一种轻量的高效参数微调方法,本文方法选择其作为训练的大模型基座. 图片编码器使用训练好的CLIP[32]提取多尺度的全局特征,然后将这些多尺度特征拼接起来,经过一个投影层得到全局的信息表征. 在整个训练过程中冻结图片编码器的参数,通过调整投影层和适配器的参数,使图片特征注入到LLaMA[33]模型中. 由于攻击性模因解释生成任务较为复杂,为了达到更好的生成效果,除了调整投影层和适配器的参数,本文方法也调整了LLaMA模型最后8层Transformer的参数.
模因解释数据集由于构建难度大和构建成本高,目前可以获得的开源数据集只有2个,分别是HatReD[5]仇恨模因解释生成数据集(3228个样本)和MemeCap[19]模因隐喻解释生成数据集(6384个样本). 有限的训练数据可能不足以让大模型学到模因的隐喻表达手法和多模态互补的攻击性特征. 而其他的开源模因数据集主要针对分类任务构建,缺少相应的模因解释,比如对隐喻方式进行分类的MET-Meme[34]数据集和Propaganda[35]数据集,以及识别有害模因的HarMeme[36]数据集和讽刺模因的Sarcasm Meme[37]数据集. 这些数据集有人工标注的不同属性特征,比如对应分类任务的类别、情感和攻击对象等特征. 本文结合人工标注的属性特征和多模态模型生成的模因图片描述信息,利用GPT-3.5强大的文本推理分析能力,自动生成开源模因数据集的文本解释,使其与模因的隐喻识别和攻击性概念对齐. 具体的数据集列表如表1所示.
由于GPT-3.5不接受多模态的输入,因此本文将模因的图片特征转换为文本形式的图像描述作为GPT-3.5的输入. LLaMA Adapter V2[22]和Blip-2[38]是目前效果较优的图像描述生成模型,我们在实验中发现LLaMA Adapter V2对图像细节的描述,比如表情、动作等方面刻画得比较丰富,但是在图像重要实体识别上有很多幻觉. Blip-2对图像重要实体的识别比较精准,但是基本上没有图像细节的描述. 在模因解释生成的过程中,实体特征和图像细节都很重要. 因此本文生成的图像描述拼接了Blip-2和LLaMA Adapter V2的生成结果. 图3给出了利用GPT-3.5生成模因解释的示例. 在Prompt构建上,本文方法加入了图像描述、文字内容以及数据集原有的人工标注的各类属性标签信息,来提示GPT-3.5去分析输入模因所要表达的深层含义. 其中,加入各类属性标签是为了更好地利用人工标注的信息,使得生成的内容与模因包含的隐喻和攻击性概念对齐. 从GPT-3.5生成结果来看,在图像描述完整如实地反映图像特征的情况下,结合模因原有的文本信息,基本可以推理出模因的内在含义,为后续指令微调提供扩充的训练数据.
多模态大语言模型在预训练阶段所使用的训练数据中,攻击性模因的数据量有限. 因此通用多模态大语言模型在识别隐喻以及生成攻击性模因解释的能力相对较弱. 为了增强这一能力,本文利用GPT-3.5增强后的6个模因解释数据集对LLaMA-Adapter V2多模型大模型基座进行指令微调,以提高通用多模态大语言模型生成攻击性模因解释的质量. 指令微调的训练目标包括让多模态大语言模型充分理解模因的隐喻表达手法,并生成相关的攻击性解释.
6类主流的模因数据集如表1所示,本文分别从任务相关的攻击性标签、任务相关的隐喻标签和生成的解释3个方面,将其转换为指令跟随数据集. 构建的指令${S}$包括指令模板${S_{\mathrm{I}}}$、模因图像${x_{\mathrm{V}}}$和模因文本${x_{\mathrm{T}}}$:
$ {S} = \{ {S_{\mathrm{I}}},{x_{\mathrm{V}}},{x_{\mathrm{T}}}\}. $
具体的指令模板如图4所示. 指令模板共包含3类指令:1)模因解释生成指令,此类指令与具体数据集无关;2)攻击性识别指令,此类指令与3个攻击性模因识别任务相关,对应HatRed、HarMeme和Sarcasm Meme数据集;3)隐喻识别指令,此类指令与3个模因隐喻识别任务相关,对应MemeCap、MET-Meme和Propaganda数据集. 通过融合这3类指令,包括判断、生成和选择等多种问题类别,基本覆盖了攻击性模因解释生成的主要要素,使得模型生成的模因解释具备隐喻分析和攻击性分析的能力.
训练的损失函数如式(2)所示:
$ {L_{{\mathrm{CE}}}} = - \sum\limits_{i = 1}^N {\sum\limits_{j = 1}^\ell {\log {\rho _\theta }} } (r_j^i|x_{\mathrm{T}}^i,x_{\mathrm{V}}^i,r_i^i,…,r_{j - 1}^i) \text{,} $
其中$ x_{\mathrm{T}}^i $表示第$ i $个模因的文本信息,$ x_{\mathrm{V}}^i $表示第$ i $个模因的图像信息,$ r_{}^i $表示第$ i $个模因的仇恨解释. 由于$ r_{}^i $由一段文字序列组成,可以将其表示为$ r_{}^i = r_1^i,…,r_\ell ^i $,其中$ \ell $表示序列的长度,$ \theta $表示多模态大模型的可调参数,$ N $表示训练集的模因个数.
本节先对实验设置进行介绍,包括数据集、基线方法、实验参数以及实验结果评估方法;之后对实验结果进行对比分析.
本文一共使用了6个数据集,分别是与攻击性相关的数据集HatReD[5],HarMeme[36]和Sarcasm Meme[37]. 以及与隐喻相关的数据集MEMECAP[19],MET-Meme[34]和Propaganda[35]. 使用HatReD数据集的训练集部分和其他5个数据集来训练模型,保留HatReD的测试集来测试模型效果. 具体的划分情况如表2所示.
HatReD是人工标注的多模态仇恨模因解释生成数据集,它提供了基于固定模板创建的解释. 该数据集包括3228个已标注的仇恨模因,其中训练集有2982个,测试集则有246个. 在本研究中,我们将HatReD的训练集并入我们的训练过程,并保留其测试集用于验证模型效果.
MemeCap是一个模因解释生成任务数据集. 该任务要求模型根据给定的模因和模因标题,生成一个简洁的描述来解释模因的含义,这个任务涉及理解模因中的视觉隐喻,并结合跨模态信息进行解释. 该数据集有6384个模因样本.
MET-Meme包含了10045个模因,并人工标注了隐喻类别、情感倾向、模因意图和冒犯程度等属性. 该数据集考虑了模因研究中的隐喻特征并实验证明了隐喻特征的重要性.
Propaganda数据集主要关注模因的隐喻手法识别,包括贬低性称呼、夸大和反讽等22种攻击隐喻宣传手段. 该数据集标注了950条数据.
HarMeme是一个多模态的有害模因识别数据集,包含3544个与COVID-19相关的有害模因. HarMeme数据集经过了两阶段注释过程:在第1阶段,将模因标记为非常有害、部分有害或无害;在第2阶段,进一步注释了每个有害模因所指向的目标类型.
Sarcasm Meme是一个多模态的讽刺模因分类数据集,经过人工标注以确定每个模因是否具有讽刺特征. 该数据集包含24635个模因,主要从Twitter社交媒体平台中收集.
本文采用了自动评估和GPT-4专家评估相结合的方法. 自动评估指标通过可计算的量化方式评估生成文本和参考文本之间的相似度. 专家评估指标从生成解释的正确性、流畅性、背景知识分析能力、隐喻识别能力和客观性5个方面评估生成质量.
在自动评估中,采用了自然语言生成任务中常见的度量标准,其中包括基于词语相似性的N-gram匹配以及基于嵌入的语义相似性度量. 对于N-gram匹配度量标准,本文采用了平均BLEU[39]和ROUGE-L[40]的评估指标. BLEU的得分范围在0~1,越接近1表示生成的结果与目标文本越相似. ROUGE-L是基于生成文本与参考文本之间的最长公共子序列的重叠度量,旨在衡量内容的保留程度. 基于嵌入的语义相似性度量上,本文采用BERTScore[41]评价方法. BERTScore通过计算2个句子在BERT模型中的编码向量之间的余弦相似度,来评估生成文本和参考文本的相似性. 特别是对于语义相似但表达不同的句子,BERTScore更能捕捉到它们之间的相似性. 自动评估使用的参考文本是HatReD人工标注答案.
由于攻击性模因需要丰富的背景知识帮助理解,且包含大量隐喻相关的表达方式,因此在评估生成解释的时候,除了要关注正确性和流畅性指标2个通用指标外,还需要关注社会背景和隐喻识别的解释生成情况,因此本文列出5类指标进行评估:
1)正确性. 生成的解释需准确无误地反映模因所蕴含的攻击对象与攻击内容,从而避免产生误解.
2)流畅性. 生成的解释应当明晰、逻辑严密且易于理解.
3)背景知识分析能力. 生成的解释应全面考量模因在其创建和传播过程中所处的社会政治、历史及文化环境,并考虑它们在理解模因深层意义时所起的关键作用.
4)隐喻识别能力. 生成的解释应阐明隐喻元素如何与模因的主题相互作用,并分析它们对模因整体信息或意图的影响.
5)客观性. 生成的解释应保持中立和尊重,避免宣传或支持任何形式的仇恨内容,确保公平对待涉及的所有个体或群体.
本文采用GPT-4专家评估的方法,采用5分量表法进行打分,分越高表示生成的解释越满足该项指标. 1分表示生成的解释与该指标不符,5分表示该生成的解释充分满足该项指标.
本文对比了2类方法,分别是基于预训练语言模型方法和多模态预训练模型的解释生成方法. 对于预训练语言模型的解释生成方法,本文对比了Hee等人[5]提出的基于T5-large[42]的方法、基于RoBERTa[17]的方法和基于RoBERTa-base-GPT2[28]的方法. 由于预训练语言模型仅接受文本输入,图像特征需转为由Blip-2生成的文字图像描述和原始文本信息. 对于多模态预训练模型的解释生成方法,本文对比了Hee等人[5]提出的VL-T5-VL-T5,VisualBERT-GPT2和VisualBERT-RoBERTa方法.
实验参数设置为:批次大小(batch size)为16,共进行20个训练周期(epoch). 每个批次的梯度更新次数为1,基础学习率为5E−4. 在前10个训练周期内,采用学习率的warm up策略,之后的周期中采用半周期余弦(half-cycle cosine)学习率调度策略. 本文使用了AdamW优化策略,其中衰减率β1设定为0.9,β2设定为0.95. 权重衰减(weight decay)设置为0.02,用于对模型参数进行正则化控制. 模型接受输入的最大长度序列长度为128. GPT-3.5使用的版本是GPT-3.5-turbo-0613.
在HatReD测试集上的实验结果如表3所示. 在所有6个基线模型中,采用GPT2-base解码器的2个基线模型(包括文本模型和多模态模型)效果相对较差,其余4个模型(包括文本模型和多模态模型)效果差别不大. 综上所述,文本模型和多模态模型在自动评估指标上差别不大.
本文模型的效果在基于BERTScore的评估标准上大大优于基线模型. BERT-P达到了0.689,比基线模型提升了0.181;BERT-R达到了0.668,比基线模型提升了0.195;BERT-F达到了0.677,比基线模型提升了0.197. 基于BERTScore的结果表明,本文方法生成的模因解释在语义上更接近人工标注的解释,证明了本文方法的有效性. 在基于N-gram的评价标准上,本文方法没有取得最优的性能. 原因在于本文方法生成的解释长度较长,平均长度为62.17个词;生成的解释不仅包括模因的攻击对象和攻击点,也深入分析了模因中多模态的隐喻手法和背景知识,可以更准确更全面地理解模因的含义. 而人工标注的解释依据模板生成,十分简短,平均长度为13.62个词.
本文采用GPT-4专家评分方法来对HatReD测试集的人工标注答案、本文方法以及T5-large模型生成的内容进行评分,实验结果如表4所示. 在正确性和客观性这2个指标上,人工标注答案的表现最佳. 正确性指标主要用于评估生成解释的攻击目标和攻击内容是否正确,是理解攻击性模因的关键指标. GPT-4在评分中对人工标注答案给出了最高分(4.66/5),显著优于其他方法. 同时,由于人工标注在处理负面内容时更加谨慎,因此在客观性指标上也获得了最高分(4.9/5). 这2个评价指标表明GPT-4和人工标注在理解攻击目标和内容方面基本一致. 然而,由于HatReD数据集人工标注答案没有考虑到隐喻识别和背景知识,因此这2项指标的得分明显低于本文方法. 本文方法在文本流畅度、背景分析能力和隐喻识别能力方面均表现出色,尽管在正确性指标上低于人工标注答案,但明显优T5-large模型,验证了本文方法的有效性.
消融分析实验旨在进一步探究各个模块对攻击性模因解释生成性能的影响. 具体设置为:
1)w/o数据增强. 使用原本有解释的2个数据集HatReD和MemeCap做解释生成任务,而其他4个数据集只做分类任务. 2)w/o隐喻数据集. 在训练集中去掉MemeCap,Propaganda和MET-Meme 3个关注隐喻分析的数据集. 3)w/o有害数据集. 在训练集中去掉Sarcasm Meme和HarMeme数据集. 由于HatReD数据集人工标注答案的局限性,特别是其模板化的解释生成方式,没有包含对模因隐喻的解释以及相关背景知识. 因此,为了更加客观地评估所生成的解释,本文主要采用了基于GPT-4的专家评估方法进行消融实验. 此外,将HatReD数据集的人工标注答案作为自动评估的参考文本,提供补充评估参考.
消融实验的GPT-4专家评分结果如表5所示. 当移除由GPT-3.5生成的增强数据后,在正确性、背景分析能力以及隐喻识别能力方面均出现了明显的下降,尤其是背景分析能力的下降最为显著,证明了GPT-3.5增强数据在生成模因解释过程中扮演了关键要素. 相比之下,移除有害数据集比移除隐喻数据集对正确性、背景分析能力和隐喻识别能力的影响更为显著,这表明在攻击性模因解释生成任务中,使用丰富的同类数据进行训练能够带来更加明显的效果提升. 表6展示了消融实验的自动评估结果,其与GPT-4专家评分的结果大体一致. 然而,由于自动评估方法采用的是HatReD数据集人工标注答案作为评估标准,并未覆盖模因的隐喻解释和背景知识. 因此,当移除GPT-3.5增强数据时,生成的解释更加简短且形式上更接近HatReD的标准答案,从而导致BLEU和ROUGE-L得分较高. 但由于缺乏其他数据的增强,基于语义的BERTScore分数出现了较大的下降,其BERT-F值比本文方法降低了0.094.
为了验证生成解释的有效性,我们将生成的解释作为提示加入到原生的LLaMA-Adapter V2,验证生成的解释是否对于zero-shot的分类有增益. 实验结果如表7所示. 可以看出增加了生成的解释后,模型zero-shot的分类效果有明显的提升,HateMeme数据集的准确度(accuracy,ACC)提升了14.6个百分点,受试者曲线面积(area under the receiver operating characteristic curve,AUROC)提升了2.1个百分点. 实验结果验证了本文方法生成解释对攻击性模因的分类有较大的增益效果.
为了评估本文方法的泛化性能,将本文方法在新数据集MAMI[43]上进行攻击性解释生成实验. MAMI数据集主要收集针对女性的仇恨模因. 由于MAMI是面向分类任务的数据集,没有人工编写的解释,因此本文采用GPT-4评估生成效果方法. 表8展示了在MAMI数据集上随机抽样50个模因生成解释的专家评估结果. 实验结果说明本文方法具有良好的泛化能力,面向新的数据也能做出全面和准确的攻击性解释生成.
除了定量分析的方法,本文通过实例说明本文方法生成解释的优势. 如图5所示,本文方法生成的解释既包括攻击的目标和攻击内容,也包括背景知识和隐喻解释,如图5的下划线所示内容,内容全面易懂. 相比之下,人工标注的解释比较简洁,采用“负面谓词+攻击目标+攻击手段”的模板化格式生成方式,并未探讨模因的意图、象征意义或背后的社会文化背景,理解依然有一定的难度. 而T5-large方法在理解攻击目标和攻击内容上存在一些错误. 比如图5的第1个例子中,人工标注的结果中,关于“coon”的说明仅有“derogatory term(诬称词)”,如果读者没有相关的背景知识依然无法准确理解其含义. 而T5-large生成的解释错误地将对“blacks(黑人)”的攻击解释为针对“Chinese(中国人)”的攻击. 本文方法生成的解释,在隐喻释义生成部分详细说明了“coon”的含义,解释了模因如何通过隐喻的手段来强化对黑人的负面刻板印象,展示了本文方法对模因解释的全面性和正确性.
本文提出了一种基于多模态大语言模型的攻击性模因解释生成方法,生成的解释不仅包括攻击性模因的攻击目标和攻击内容,而且涵盖了其隐喻表达方式和社会背景知识,从而形成了一个更加全面完整的解释. 实验结果表明,本文方法生成的解释在BERTScore评估标准下相较于基线模型提高了19%. 基于GPT-4的专家评估方法表明本文方法生成的解释在处理背景知识分析和隐喻识别方面超越了人工标注结果. 此外,实验结果也表明本文方法对新数据也展现出了很好的泛化性.
作者贡献声明:林萌提出了论文的核心思想,设计算法进行实验并撰写论文;戴程威实现了论文对比方法;郭涛提出指导意见并修改论文.

参考文献 引证文献
排序方式:
1
虎嵩林,赵军,唐杰,等. 虚假信息检测专题前言[J]. 计算机研究与发展,2021,58(7):1351−1352
Hu Songlin, Zhao Jun, Tang Jie, et al. Preface to the special issue on fake information detection[J]. Journal of Computer Research and Development, 2021, 58(7): 1351−1352 (in Chinese)
2
Kiela D, Firooz H, Mohan A, et al. The hateful memes challenge: Detecting hate speech in multimodal memes[J]. Advances in Neural Information Processing Systems, 2020, 33: 2611−2624
3
Zhang Linhao, Jin Li, Sun Xian, et al. TOT: Topology-aware optimal transport for multimodal hate detection[C]//Proc of the AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023, 37(4): 4884−4892
4
Cao Rui, Hee MS, Kuek A, et al. Pro-Cap: Leveraging a frozen vision-language model for hateful meme detection[C]//Proc of the 31st ACM Int Conf on Multimedia. New York: ACM, 2023: 5244−5252
5
Hee M S, Chong W H, Lee R K W. Decoding the underlying meaning of multimodal hateful memes[C]//Proc of the 32nd Int Joint Conf on Artificial Intelligence. Freiburg: IJCAI, 2023: 5995−6003
6
Sharma S, Agarwal S, Suresh T, et al. What do you MEME? Generating explanations for visual semantic role labelling in memes[C]//Proc of the AAAI Conf on Artificial Intelligence. Washington, DC: AAAI, 2023, 37(8): 9763−9771
7
Scott K. Memes as multimodal metaphors: A relevance theory analysis[J]. Pragmatics & Cognition, 2021, 28(2): 277−298
8
Pramanick S, Sharma S, Dimitrov D, et al. MOMENTA: A multimodal framework for detecting harmful memes and their targets[C]//Findings of the Association for Computational Linguistics: EMNLP 2021. Stroudsburg, PA: ACL, 2021: 4439−4455
9
Zhu Ron. Enhance multimodal Transformer with external label and in-domain pretrain: Hateful meme challenge winning solution[J]. arXiv preprint, arXiv: 2012.08290, 2020
10
Yang Chuanpeng, Zhu Fuqing, Liu Guihua, et al. Multimodal hate speech detection via cross-domain knowledge transfer[C]//Proc of the 30th ACM Int Conf on Multimedia. New York: ACM, 2022: 4505−4514
11
Lee R K W, Cao Rui, Fan Ziqing, et al. Disentangling hate in online memes[C]//Proc of the 29th ACM Int Conf on Multimedia. New York: ACM, 2021: 5138−5147
12
Velioglu R, Rose J. Detecting hate speech in memes using multimodal deep learning approaches: Prize-winning solution to hateful memes challenge[J]. arXiv preprint, arXiv: 2012.12975, 2020
13
Yin Shukang, Fu Chaoyou, Zhao Sirui, et al. A survey on multimodal large language models[J]. arXiv preprint, arXiv: 2306.13549, 2023
14
Gupta T, Kembhavi A. Visual programming: Compositional visual reasoning without training[C]//Proc of the IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 14953−14962
15
Shao Zhenwei, Yu Zhou, Wang Meng, et al. Prompting large language models with answer heuristics for knowledge-based visual question answering[C]//Proc of the IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 14974−14983
16
Cao Rui, Lee R K W, Chong W H, et al. Prompting for multimodal hateful meme classification[C]//Proc of the 2022 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2022: 321−332
17
Liu Yinhan, Ott M, Goyal N, et al. RoBERTa: A robustly optimized BERT pretraining approach[J]. arXiv preprint, arXiv: 1907.11692, 2019
18
Ji Junhui, Ren Wei, Naseem U. Identifying creative harmful memes via prompt based approach[C]//Proc of the ACM Web Conf 2023. New York: ACM, 2023: 3868−3872
19
Hwang E J, Shwartz V. MemeCap: A dataset for captioning and interpreting memes[J]. arXiv preprint, arXiv: 2305.13703, 2023
20
Zhu Deyao, Chen Jun, Shen Xiaoqian, et al. MiniGPT-4: Enhancing vision-language understanding with advanced large language models[J]. arXiv preprint, arXiv: 2304.10592, 2023
21
Zhang Renhui, Han Jiaming, Liu Chris, et al. LLaMA-Adapter: Efficient fine-tuning of language models with zero-init attention[J]. arXiv preprint, arXiv: 2303.16199, 2023
22
Gao Peng, Han Jiaming, Zhang Renrui, et al. LLaMA-Adapter V2: Parameter-efficient visual instruction model[J]. arXiv preprint, arXiv: 2304.15010, 2023
23
Liu Haotian, Li Chunyuan, Wu Qingyang, et al. Visual instruction tuning[J]. Advances in Neural Information Processing Systems, 2024. DOI: 10.48550/arXiv.2304.08485
24
Horawalavithana S, Munikoti S, Stewart I, et al. SciTune: Aligning large language models with scientific multimodal instructions[J]. arXiv preprint, arXiv: 2307.01139, 2023
25
Wei J, Zou Kai. EDA: Easy data augmentation techniques for boosting performance on text classification tasks[C]//Proc of the 2019 Conf on Empirical Methods in Natural Language Processing and the 9th Int Joint Conf on Natural Language Processing (EMNLP-IJCNLP). Stroudsburg, PA: ACL, 2019: 6382−6388
26
Wu Xing, Lv Shangwen, Zang Liangjun, et al. Conditional BERT contextual augmentation[C]// Proc of 19th Int Conf on Computational Science(CCS 2019). Berlin: Springer, 2019: 84−95
27
Kumar V, Choudhary A, Cho E. Data augmentation using pre-trained Transformer models[C]//Proc of the 2nd Workshop on Life-long Learning for Spoken Language Systems. Stroudsburg, PA: ACL, 2020: 18−26
28
Radford A, Jeffrey W, Child R, et al. Language models are unsupervised multitask learners[J]. OpenAI Blog, 2019, 1(8): 9−9
29
Kenton J D M W C, Toutanova L K. BERT: Pre-training of deep bidirectional Transformers for language understanding[C]//Proc of Annual Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, PA: ACL, 2019: 4171−4186
30
Lewis M, Liu Yinhan, Goyal N, et al. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[C]//Proc of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 7871−7880
31
Dai Haixing, Liu Zhengliang, Liao Wenxiong, et al. ChatAug: Leveraging ChatGPT for text data augmentation[J]. arXiv preprint, arXiv: 2302.13007, 2023
32
Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//Proc of Int Conf on Machine Learning. New York: PMLR, 2021: 8748−8763
33
Touvron H, Lavril T, Izacard G, et al. Llama: Open and efficient foundation language models[J]. arXiv preprint, arXiv: 2302.13971, 2023
34
Xu Bo, Li Tingting, Zheng Junzhe, et al. MET-Meme: A multimodal meme dataset rich in metaphors[C]//Proc of the 45th Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2022: 2887−2899
35
Dimitrov D, Ali B B, Shaar S, et al. Detecting propaganda techniques in memes[C]//Proc of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int Joint Conf on Natural Language Processing (ACL-IJCNLP 2021). Stroudsburg, PA: 2021: 6603−6617
36
Pramanick S, Sharma S, Dimitrov D, et al. MOMENTA: A multimodal framework for detecting harmful memes and their targets[C]//Findings of the Association for Computational Linguistics: EMNLP 2021. Stroudsburg, PA: ACL, 2021: 4439−4455
37
Cai Yitao, Cai Huiyu, Wan Xiaojun. Multi-modal sarcasm detection in twitter with hierarchical fusion model[C]//Proc of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2019: 2506−2515
38
Li Junnan, Li Dongxu, Savarese S, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models[J]. arXiv preprint, arXiv: 2301.12597, 2023
39
Papineni K, Roukos S, Ward T, et al. BLEU: A method for automatic evaluation of machine translation[C]//Proc of the 40th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2002: 311−318
40
Lin Chin-Yew. Rouge: A package for automatic evaluation of summaries[C]//Text Summarization Branches Out. Stroudsburg, PA: ACL, 2004: 74−81
41
Zhang Tianyi, Kishore V, Wu Felix, et al. BERTScore: Evaluating text generation with BERT[J]. arXiv preprint, arXiv: 1904.09675, 2019
42
Raffel C, Shazeer N, Roberts A, et al. Exploring the limits of transfer learning with a unified text-to-text transformer[J]. The Journal of Machine Learning Research, 2020, 21(1): 5485−5551
43
Fersini E, Gasparini F, Rizzi G, et al. SemEval-2022 Task 5: Multimedia automatic misogyny identification[C]//Proc of the 16th Int Workshop on Semantic Evaluation (SemEval-2022). Stroudsburg, PA: ACL, 2022: 533−549
2024年第61卷第5期
71
33
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202330960
  • 接收时间:2023-11-30
  • 首发时间:2026-11-17
  • 出版时间:2024-05-01
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2023-11-30
  • 修回日期:2024-03-12
基金
作者信息
    1中国科学院信息工程研究所 北京 100019
    2中国科学院大学网络空间安全学院 北京 100049

通讯作者:

郭涛()
参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202330960
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏