收藏切换
Large Language Models: Principles, Implementation, and Progress
收藏切换
Wentao Shu, Ruixiao Li, Tianxiang Sun, Xuanjing Huang, Xipeng Qiu*
Journal of Computer Research and Development | 2024, 61(2) : 351 - 361
Less
收藏切换
Journal of Computer Research and Development | 2024, 61(2): 351-361
• Artificial Intelligence and Pattern Recognition •
Large Language Models: Principles, Implementation, and Progress
Full
Wentao Shu, Ruixiao Li, Tianxiang Sun, Xuanjing Huang, Xipeng Qiu*
Affiliations
  • 1School of Computer Science, Fudan University, Shanghai 200433
  • born in 2002. Undergraduate. His main research interests include deep learning, natural language processing, and large language models

    born in 2001. Undergraduate. His main research interests include deep learning and natural language processing

    born in 1997. PhD candidate. His main research interests include deep learning and natural language processing

    born in 1972. PhD, professor, PhD supervisor. Distinguished member of CCF. Her main research interests include natural language processing and information retrieval

    born in 1983. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include artifitial intelligence, natural language processing, and large language models

About Author:

born in 2002. Undergraduate. His main research interests include deep learning, natural language processing, and large language models

born in 2001. Undergraduate. His main research interests include deep learning and natural language processing

born in 1997. PhD candidate. His main research interests include deep learning and natural language processing

born in 1972. PhD, professor, PhD supervisor. Distinguished member of CCF. Her main research interests include natural language processing and information retrieval

born in 1983. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include artifitial intelligence, natural language processing, and large language models

Published: 2024-02-02 doi: 10.7544/issn1000-1239.202330303
Outline
收藏切换

In recent years, the emergence and development of large language models (LLMs) have revolutionized the field of natural language processing and even artificial intelligence. With the increasing number of model parameters and training data, the perplexity of language models decreases in a predictable manner, which implies the improvement of performance on various natural language processing tasks. Therefore, scaling up language models has been a promising way to improve the system intelligence. In this survey, we first review the definition and scope of LLMs and provide a scale standard to distinguish “large” language models from the perspectives of performance and computing. Then, we review the development and representative work of LLMs in three dimensions: data, algorithm, and model architecture, showing how up-scaling in these dimensions drives the development of LLMs at different stages. Next, we discuss the emergent abilities of LLMs and possible interpretations behind them. We highlight three key emergent abilities, i.e., chain-of-thought prompting, in-context learning, and instruction-following, introducing their related advances and applications. Finally, we outline some potential directions and challenges of LLMs.

natural language processing  /  neural networks  /  large language models  /  pre-training  /  alignment
舒文韬, 李睿潇, 孙天祥, 黄萱菁, 邱锡鹏. 大型语言模型:原理、实现与发展. 计算机研究与发展, 2024 , 61 (2) : 351 -361 . DOI: 10.7544/issn1000-1239.202330303
Wentao Shu, Ruixiao Li, Tianxiang Sun, Xuanjing Huang, Xipeng Qiu. Large Language Models: Principles, Implementation, and Progress[J]. Journal of Computer Research and Development, 2024 , 61 (2) : 351 -361 . DOI: 10.7544/issn1000-1239.202330303
Year 2024 volume 61 Issue 2
84
42
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202330303
  • Receive Date:2023-04-07
  • Online Date:2026-11-17
  • Published:2024-02-02
Article Data
Affiliations
History
  • Received:2023-04-07
  • Revised:2023-09-27
Affiliations
    1School of Computer Science, Fudan University, Shanghai 200433
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202330303
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT