收藏切换
A Method for Generating Explanations of Offensive Memes Based on Multimodal Large Language Models
收藏切换
Meng Lin1, 2, Chengwei Dai1, 2, Tao Guo1, *
Journal of Computer Research and Development | 2024, 61(5) : 1206 - 1217
Less
收藏切换
Journal of Computer Research and Development | 2024, 61(5): 1206-1217
• Special Issue on Large Models and Safety •
A Method for Generating Explanations of Offensive Memes Based on Multimodal Large Language Models
Full
Meng Lin1, 2, Chengwei Dai1, 2, Tao Guo1, *
Affiliations
  • 1Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100019
  • 2School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049
  • born in 1991. PhD candidate. Her main research interests include multi-modal hate speech detection and multimodal semantic understanding

    born in 2000. Master candidate. His main research interests include model extraction and large language model distillation

    born in 1974. PhD, Professor, PhD supervisor. His main research interests include cybersecurity, vulnerability analysis and risk assessment

About Author:

born in 1991. PhD candidate. Her main research interests include multi-modal hate speech detection and multimodal semantic understanding

born in 2000. Master candidate. His main research interests include model extraction and large language model distillation

born in 1974. PhD, Professor, PhD supervisor. His main research interests include cybersecurity, vulnerability analysis and risk assessment

Published: 2024-05-01 doi: 10.7544/issn1000-1239.202330960
Outline
收藏切换

With the advancement of 5G technology, offensive speech has increasingly proliferated across social networks in the form of multimodal memes. Consequently, the detection and interpretive generation of offensive memes play a crucial role in enhancing content moderation effectiveness and maintaining a harmonious and healthy public discourse environment. Existing studies on the interpretive generation of offensive memes focus solely on the targets and content of offense, neglecting the societal background knowledge and metaphorical expressions embedded in memes. This oversight limits the ability to comprehensively and accurately interpret the meaning of offensive memes, thus constraining the applicability of interpretations. To address this challenge, we propose a method based on multimodal large language model for generating interpretations of offensive memes. By augmenting elements such as offense targets, the content of the offense, and metaphor recognition into the instruction tuning data, we can effectively improve the multimodal large model’s proficiency in interpretively generating offensive memes through instruction tuning. The experimental outcomes validate three key strengths of our method: first, it achieves a notable 19% enhancement in the BERTScore evaluation metric over baseline models; second, it incorporates comprehensive background knowledge pertinent to offensive metaphorical expressions within its interpretations; third, it exhibits strong generalization capabilities when handling previously unseen meme data.

offensive meme  /  explanation generation  /  multi-modal LLM  /  data augment  /  instruction fine-tuning
林萌, 戴程威, 郭涛. 基于多模态大语言模型的攻击性模因解释生成方法. 计算机研究与发展, 2024 , 61 (5) : 1206 -1217 . DOI: 10.7544/issn1000-1239.202330960
Meng Lin, Chengwei Dai, Tao Guo. A Method for Generating Explanations of Offensive Memes Based on Multimodal Large Language Models[J]. Journal of Computer Research and Development, 2024 , 61 (5) : 1206 -1217 . DOI: 10.7544/issn1000-1239.202330960
Year 2024 volume 61 Issue 5
70
33
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202330960
  • Receive Date:2023-11-30
  • Online Date:2026-11-17
  • Published:2024-05-01
Article Data
Affiliations
History
  • Received:2023-11-30
  • Revised:2024-03-12
Affiliations
    1Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100019
    2School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202330960
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT