收藏切换
Resilio: An Elastic Fault-tolerant Training System for Large Language Models
收藏切换
Yan Li1, Sile Yang1, Chengchun Liu1, Linmei Wang1, Yaolin Tian1, Xinhang Zhang1, Yu Zhu2, Chunpu Li1, Lei Sun1, Shengen Yan2, Limin Xiao1, *, Weifeng Zhang1
Journal of Computer Research and Development | 2025, 62(6) : 1380 - 1395
Less
收藏切换
Journal of Computer Research and Development | 2025, 62(6): 1380-1395
• High Performance Computing System •
Resilio: An Elastic Fault-tolerant Training System for Large Language Models
Full
Yan Li1, Sile Yang1, Chengchun Liu1, Linmei Wang1, Yaolin Tian1, Xinhang Zhang1, Yu Zhu2, Chunpu Li1, Lei Sun1, Shengen Yan2, Limin Xiao1, *, Weifeng Zhang1
Affiliations
  • 1Lenovo Research, Beijing 100094
  • 2Nanoscale Integrated Circuits and Systems Lab (Tsinghua University), Beijing 100084
  • born in 1985. PhD, senior engineer. Member of CCF. His main research interests include high performance computing, heterogeneous computing, and deep learning

    born in 1994. Master. His main research interests include heterogenous computing and network communication

    born in 1992. Master. His main research interests include high performance computing and heterogeneous computing

    born in 1985. Master. Member of CCF. Her main research interests include high performance computing and heterogeneous computing

    born in 1999. Master. Her main research interests include optimization of LLM, heterogeneous computing, and visual SLAM

    born in 2000. Master. His main research interest includes distributed training of deep learning

    born in 1998. PhD candidate. His main research interests include LLM training architecture design and multi-core architecture

    born in 1978. Bachelor. Her main research interests include heterogeneous computing and cloud computing

    born in 1980. Bachelor. His main research interests include heterogeneous computing and edge computing

    born in 1986. PhD, associate professor. His main research interests include heterogeneous computing, intelligent computing, and machine learning systems

    born in 1970. PhD. Distinguished member of CCF. Distinguished researcher at Lenovo Research. His main research interests include computer architecture, heterogeneous intelligent computing, high performance computing, and intelligent computing chip

    born in 1968. PhD. Corporate VP at Lenovo Group. His main research interests include computer architecture, heterogeneous intelligent computing, wireless communication, and intelligent computing chip

About Author:

born in 1985. PhD, senior engineer. Member of CCF. His main research interests include high performance computing, heterogeneous computing, and deep learning

born in 1994. Master. His main research interests include heterogenous computing and network communication

born in 1992. Master. His main research interests include high performance computing and heterogeneous computing

born in 1985. Master. Member of CCF. Her main research interests include high performance computing and heterogeneous computing

born in 1999. Master. Her main research interests include optimization of LLM, heterogeneous computing, and visual SLAM

born in 2000. Master. His main research interest includes distributed training of deep learning

born in 1998. PhD candidate. His main research interests include LLM training architecture design and multi-core architecture

born in 1978. Bachelor. Her main research interests include heterogeneous computing and cloud computing

born in 1980. Bachelor. His main research interests include heterogeneous computing and edge computing

born in 1986. PhD, associate professor. His main research interests include heterogeneous computing, intelligent computing, and machine learning systems

born in 1970. PhD. Distinguished member of CCF. Distinguished researcher at Lenovo Research. His main research interests include computer architecture, heterogeneous intelligent computing, high performance computing, and intelligent computing chip

born in 1968. PhD. Corporate VP at Lenovo Group. His main research interests include computer architecture, heterogeneous intelligent computing, wireless communication, and intelligent computing chip

Published: 2025-06-02 doi: 10.7544/issn1000-1239.202550147
Outline
收藏切换

Large language models with hundreds of billions of parameters are driving rapid technology innovations and business model transformations in artificial intelligence and heterogeneous computing. However, training such models requires prolonged occupation of extensive hardware resources, thus often incurring diverse high-frequency software/hardware failures. These failures not only are challenging to diagnose, but also lead to much longer training time due to unwanted computation waste and slow training convergence. Resilio, an elastic fault-tolerant system for training large language models is proposed, to provide an efficient automated fault recovery mechanism. It is designed to target multiple typical failure scenarios during training processes, such as network interruptions, node crashes, and process failures. Leveraging the characteristics of the parallel model training strategies and underlying hierarchical storage architectures, Resilio implements multi-layer optimizations on checkpoint read/write operations and Just-In-Time (JIT) recovery mechanisms. For models with 100 billion scale parameters, Resilio reduces the end-to-end recovery time under 10 minutes, while reducing the re-started computation after interruptions to the cost of a single training iteration. Upon variations of the computation resources, Resilio can quickly identify the cluster configurations to enable optimal parallel training strategies. Combined with the fault-tolerant scheduling capability, the system ensures adaptive and elastic resource allocations to greatly improve training efficiency and boost GPU utilization across large-scale computing clusters.

large-scale model training  /  deep learning  /  fault tolerance  /  failure detection  /  elastic training  /  automatic parallelization
李焱, 杨偲乐, 刘成春, 王林梅, 田瑶琳, 张信航, 朱昱, 李莼蒲, 孙磊, 颜深根, 肖利民, 张伟丰. Resilio:一种大模型弹性训练容错系统. 计算机研究与发展, 2025 , 62 (6) : 1380 -1395 . DOI: 10.7544/issn1000-1239.202550147
Yan Li, Sile Yang, Chengchun Liu, Linmei Wang, Yaolin Tian, Xinhang Zhang, Yu Zhu, Chunpu Li, Lei Sun, Shengen Yan, Limin Xiao, Weifeng Zhang. Resilio: An Elastic Fault-tolerant Training System for Large Language Models[J]. Journal of Computer Research and Development, 2025 , 62 (6) : 1380 -1395 . DOI: 10.7544/issn1000-1239.202550147
Year 2025 volume 62 Issue 6
108
49
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202550147
  • Receive Date:2025-03-01
  • Online Date:2026-11-10
  • Published:2025-06-02
Article Data
Affiliations
History
  • Received:2025-03-01
  • Revised:2025-04-10
Affiliations
    1Lenovo Research, Beijing 100094
    2Nanoscale Integrated Circuits and Systems Lab (Tsinghua University), Beijing 100084
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202550147
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT