收藏切换
Robustness of GPT Large Language Models on Natural Language Processing Tasks
收藏切换
Xuanting Chen1, Junjie Ye1, Can Zu1, Nuo Xu1, Tao Gui2, *, Qi Zhang1
Journal of Computer Research and Development | 2024, 61(5) : 1128 - 1142
Less
收藏切换
Journal of Computer Research and Development | 2024, 61(5): 1128-1142
• Special Issue on Large Models and Safety •
Robustness of GPT Large Language Models on Natural Language Processing Tasks
Full
Xuanting Chen1, Junjie Ye1, Can Zu1, Nuo Xu1, Tao Gui2, *, Qi Zhang1
Affiliations
  • 1School of Computer Science, Fudan University, Shanghai 200433
  • 2Institute of Modern Languages and Linguistics, Fudan University, Shanghai 200433
  • born in 1999. Master candidate. Her main research interests include natural language processing and robust models

    born in 2001. PhD candidate. His main research interest includes natural language processing

    born in 2000. Master candidate. Her main research interests include large language models and information extraction

    born in 1998. Master candidate. Her main research interests include natural language processing and large language models

    born in 1989. PhD, associate professor, master supervisor. His main research interests include pre-training models, information extraction, and robust models

    born in 1981. PhD, professor, PhD supervisor. His main research interests include natural language processing and information retrieval

About Author:

born in 1999. Master candidate. Her main research interests include natural language processing and robust models

born in 2001. PhD candidate. His main research interest includes natural language processing

born in 2000. Master candidate. Her main research interests include large language models and information extraction

born in 1998. Master candidate. Her main research interests include natural language processing and large language models

born in 1989. PhD, associate professor, master supervisor. His main research interests include pre-training models, information extraction, and robust models

born in 1981. PhD, professor, PhD supervisor. His main research interests include natural language processing and information retrieval

Published: 2024-05-01 doi: 10.7544/issn1000-1239.202330801
Outline
收藏切换

The GPT models have demonstrated impressive performance in various natural language processing (NLP) tasks. However, their robustness and abilities to handle various complexities of the open world have not yet to be well explored, which is especially crucial in assessing the stability of models and is a key aspect of trustworthy AI. In this study, we perform a comprehensive experimental analysis of GPT-3 and GPT-3.5 series models, exploring their performance and robustness using 15 datasets (about 147000 original test samples) with 61 robust probing transformations from TextFlint covering 9 popular NLP tasks. Additionally, we analyze the model’s robustness across different transformation levels, including character, word, and sentence. Our findings reveal that while GPT models exhibit competitive performance in certain tasks like sentiment analysis, semantic matching, and reading comprehension, they exhibit severe confusion regarding information extraction tasks. For instance, GPT models exhibit severe confusion in relation extraction and even exhibit “hallucination” phenomena. Moreover, they experience significant degradation in robustness in terms of tasks and transformations, especially in classification tasks and sentence-level transformations. Furthermore, we validate the impact of the quantity and the form of demonstrations on performance and robustness. These findings reveal that GPT models are still not fully proficient in handling common NLP tasks, and highlight the difficulty in addressing robustness challenges through enhancing model performance or altering prompt content. By comparing the performance and robustness of the updated version of gpt-3.5-turbo, gpt-4, LLaMA2-7B and LLaMA2-13B, we further validate the experimental findings. Future studies on large language models should strive to enhance their capacities in information extraction and semantic understanding, while simultaneously bolstering overall robustness.

robustness  /  GPT models  /  large language models  /  natural language processing  /  reliability
陈炫婷, 叶俊杰, 祖璨, 许诺, 桂韬, 张奇. GPT系列大语言模型在自然语言处理任务中的鲁棒性. 计算机研究与发展, 2024 , 61 (5) : 1128 -1142 . DOI: 10.7544/issn1000-1239.202330801
Xuanting Chen, Junjie Ye, Can Zu, Nuo Xu, Tao Gui, Qi Zhang. Robustness of GPT Large Language Models on Natural Language Processing Tasks[J]. Journal of Computer Research and Development, 2024 , 61 (5) : 1128 -1142 . DOI: 10.7544/issn1000-1239.202330801
Year 2024 volume 61 Issue 5
92
45
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202330801
  • Receive Date:2023-10-10
  • Online Date:2026-11-17
  • Published:2024-05-01
Article Data
Affiliations
History
  • Received:2023-10-10
  • Revised:2024-03-12
Affiliations
    1School of Computer Science, Fudan University, Shanghai 200433
    2Institute of Modern Languages and Linguistics, Fudan University, Shanghai 200433
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202330801
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT