收藏切换
Safety Analysis of Large Model Content Generation Based on Knowledge Editing
收藏切换
Mengru Wang1, Yunzhi Yao2, Zekun Xi1, Jintian Zhang1, Peng Wang1, Ziwen Xu1, Ningyu Zhang1, 2, *
Journal of Computer Research and Development | 2024, 61(5) : 1143 - 1155
Less
收藏切换
Journal of Computer Research and Development | 2024, 61(5): 1143-1155
• Special Issue on Large Models and Safety •
Safety Analysis of Large Model Content Generation Based on Knowledge Editing
Full
Mengru Wang1, Yunzhi Yao2, Zekun Xi1, Jintian Zhang1, Peng Wang1, Ziwen Xu1, Ningyu Zhang1, 2, *
Affiliations
  • 1School of Software Technology, Zhejiang University, Hangzhou 315048
  • 2College of Computer Science and Technology, Zhejiang University, Hangzhou 310027
  • born in 1996. PhD candidate. Member of CCF. Her main research interests include natural language processing, knowledge graph, and the safety of large language models

    born in 2000. PhD candidate. Member of CCF. His main research interests include natural language processing, knowledge graph, and machine learning

    born in 2001. Master. His main research interests include natural language processing and model editing

    born in 2001. Master candidate. His main research interest includes natural language processing

    born in 2001. Master. His main research interests include natural language processing, enhancement of large language models, and knowledge editing

    born in 2002. Undergraduate. His main research interest includes natural language processing

    born in 1989. PhD, associate professor. Senior member of CCF. His main research interests include knowledge graph and natural language processing

About Author:

born in 1996. PhD candidate. Member of CCF. Her main research interests include natural language processing, knowledge graph, and the safety of large language models

born in 2000. PhD candidate. Member of CCF. His main research interests include natural language processing, knowledge graph, and machine learning

born in 2001. Master. His main research interests include natural language processing and model editing

born in 2001. Master candidate. His main research interest includes natural language processing

born in 2001. Master. His main research interests include natural language processing, enhancement of large language models, and knowledge editing

born in 2002. Undergraduate. His main research interest includes natural language processing

born in 1989. PhD, associate professor. Senior member of CCF. His main research interests include knowledge graph and natural language processing

Published: 2024-05-01 doi: 10.7544/issn1000-1239.202330965
Outline
收藏切换

Although large language models (LLMs) have achieved remarkable success, they still face security problems in practical applications, and it is easy to generate toxic and harmful content under malicious induction. Existing methods to mitigate the unsafe behavior of LLMs often demand significant computational resources and incur high costs associated with secure data collection. Knowledge editing offers a novel approach to constrain the model’s behavior precisely for specific inputs without the need for retraining, thus saving substantial resources. This approach provides a new feasible avenue for optimizing large models to generate secure content. Nevertheless, existing datasets for mitigating the unsafe behavior of LLMs do not encompass all unsafe scenarios. Moreover, the toxicity issues in these datasets are nearly insurmountable for post-alignment LLMs’ security defenses, hindering the optimization of safety concerns in post-alignment LLMs. In light of these challenges, we introduce a new dataset called SafeGen and propose a novel evaluation framework to analyze the potential of knowledge editing in optimizing the generation of secure content by LLMs. Extensive experiments reveal that knowledge editing demonstrates broad applications in rectifying unsafe behaviors exhibited by LLMs, and editing parameters can enhance the internal safety beliefs of LLMs. However, the fluency of text generated by knowledge editing falls short of expectations, indicating the inherent difficulty of this task. We hope that our work provides insights for the large model security community.

large language model  /  safety  /  knowledge editing  /  content generation  /  jailbreak prompt  /  defense  /  dataset
王梦如, 姚云志, 习泽坤, 张锦添, 王鹏, 徐子文, 张宁豫. 基于知识编辑的大模型内容生成安全分析. 计算机研究与发展, 2024 , 61 (5) : 1143 -1155 . DOI: 10.7544/issn1000-1239.202330965
Mengru Wang, Yunzhi Yao, Zekun Xi, Jintian Zhang, Peng Wang, Ziwen Xu, Ningyu Zhang. Safety Analysis of Large Model Content Generation Based on Knowledge Editing[J]. Journal of Computer Research and Development, 2024 , 61 (5) : 1143 -1155 . DOI: 10.7544/issn1000-1239.202330965
Year 2024 volume 61 Issue 5
67
33
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202330965
  • Receive Date:2023-12-01
  • Online Date:2026-11-17
  • Published:2024-05-01
Article Data
Affiliations
History
  • Received:2023-12-01
  • Revised:2024-02-22
Affiliations
    1School of Software Technology, Zhejiang University, Hangzhou 315048
    2College of Computer Science and Technology, Zhejiang University, Hangzhou 310027
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202330965
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT