收藏切换
LingLong: A High-Quality Small-Scale Chinese Pre-trained Language Model
收藏切换
Dongwen Li1, Zhenyu Zhong1, Yufei Sun1, *, Junyu Shen1, Zizhi Ma1, Chuanyue Yu1, Yuzhi Zhang1, 2
Journal of Computer Research and Development | 2025, 62(3) : 682 - 693
Less
收藏切换
Journal of Computer Research and Development | 2025, 62(3): 682-693
• Artificial Intelligence •
LingLong: A High-Quality Small-Scale Chinese Pre-trained Language Model
Full
Dongwen Li1, Zhenyu Zhong1, Yufei Sun1, *, Junyu Shen1, Zizhi Ma1, Chuanyue Yu1, Yuzhi Zhang1, 2
Affiliations
  • 1College of Software, Nankai University, Tianjin 300450
  • 2Haihe Laboratory of Information Technology Application Innovation, Tianjin 300450
  • born in 1997. PhD candidate. Student member of CCF. Her main research interests include natural language processing and deep learning

    born in 1997. PhD candidate. His main research interests include natural language processing, high-performance computing, and artificial intelligence operations

    born in 1976. PhD, professor, master supervisor. Her main research interests include deep learning, heterogeneous computing, and artificial intelligence

    born in 2001. Master candidate.His main research interest includes natural language processing

    born in 2000. Master candidate. His main research interest includes natural language processing

    born in 2001. Master candidate. Her main research interest includes natural language processing

    born in 1964. PhD, professor, PhD supervisor. Member of CCF. His main research interests include deep learning and other artificial intelligence-related fields

About Author:

born in 1997. PhD candidate. Student member of CCF. Her main research interests include natural language processing and deep learning

born in 1997. PhD candidate. His main research interests include natural language processing, high-performance computing, and artificial intelligence operations

born in 1976. PhD, professor, master supervisor. Her main research interests include deep learning, heterogeneous computing, and artificial intelligence

born in 2001. Master candidate.His main research interest includes natural language processing

born in 2000. Master candidate. His main research interest includes natural language processing

born in 2001. Master candidate. Her main research interest includes natural language processing

born in 1964. PhD, professor, PhD supervisor. Member of CCF. His main research interests include deep learning and other artificial intelligence-related fields

Published: 2025-03-01 doi: 10.7544/issn1000-1239.202330844
Outline
收藏切换

In recent years, large-scale autoregressive Chinese pre-trained language models (PLMs) have demonstrated outstanding performance on various natural language processing (NLP) tasks. However, these models are computationally expensive, and their word-based vocabulary poses significant challenges for practical applications. In addition, most of them use only unidirectional context information, which may result in performance degradation on many tasks, especially tasks requiring a nuanced understanding of context. To address these challenges, we introduce LingLong, a high-quality small-scale Chinese pre-trained language model. LingLong stands out due to its modest scale, comprising only 317 million parameters, making it highly deployable and resource-efficient. We tokenize the training corpus with a character-based vocabulary to mitigate the negative impacts of unknown tokens and word segmentation errors. Moreover, we go beyond the conventional unidirectional context by introducing a novel backward model. This model is trained by reversing the input order of the training data. Combining LingLong and its backward version allows for the use of bidirectional information on downstream tasks. Extensive experimental results validate the effectiveness of LingLong across a diverse set of NLP tasks. LingLong outperforms similar-sized Chinese PLMs on six downstream tasks and surpasses popular large-scale Chinese PLMs on four downstream tasks. These findings underscore the versatility and efficiency of LingLong, opening up possibilities for practical applications and advancements in the Chinese NLP field.

Chinese pre-trained language model  /  small-scale  /  character-based model  /  backward model  /  bidirectional information
李东闻, 钟震宇, 孙羽菲, 申峻宇, 马子智, 于川越, 张玉志. 玲珑:一个小规模的高质量中文预训练语言模型. 计算机研究与发展, 2025 , 62 (3) : 682 -693 . DOI: 10.7544/issn1000-1239.202330844
Dongwen Li, Zhenyu Zhong, Yufei Sun, Junyu Shen, Zizhi Ma, Chuanyue Yu, Yuzhi Zhang. LingLong: A High-Quality Small-Scale Chinese Pre-trained Language Model[J]. Journal of Computer Research and Development, 2025 , 62 (3) : 682 -693 . DOI: 10.7544/issn1000-1239.202330844
Year 2025 volume 62 Issue 3
183
77
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202330844
  • Receive Date:2023-10-31
  • Online Date:2026-11-10
  • Published:2025-03-01
Article Data
Affiliations
History
  • Received:2023-10-31
  • Revised:2024-03-22
  • Accepted:2024-05-30
Affiliations
    1College of Software, Nankai University, Tianjin 300450
    2Haihe Laboratory of Information Technology Application Innovation, Tianjin 300450
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202330844
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT