收藏切换
AdaptDNN: An Adaptive and Scalable LLM Distributed Training System
收藏切换
Guodong Liu1, 2, Jiaqi Zhu1, 2, Ziyuan Gao1, 2, Yungang Bao1, 2, 3, Sa Wang1, 2, *
Journal of Computer Research and Development | 2025, 62(12) : 3060 - 3076
Less
收藏切换
Journal of Computer Research and Development | 2025, 62(12): 3060-3076
• Ditributed Computing •
AdaptDNN: An Adaptive and Scalable LLM Distributed Training System
Full
Guodong Liu1, 2, Jiaqi Zhu1, 2, Ziyuan Gao1, 2, Yungang Bao1, 2, 3, Sa Wang1, 2, *
Affiliations
  • 1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
  • 2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049
  • 3Beijing Institute of Open Source Chip, Beijing 100080
  • born in 1997. PhD candidate. His main research interests include distributed systems and machine learning systems

    born in 1999. Master candidate. His main research interests include operating systems and machine learning systems. ()

    born in 2001. Master candidate. His main research interests include machine learning system, virtualization, and cloud computing. ()

    born in 1980. PhD, professor. Member of CCF. His main research interests include data center architecture, agile design methodology of processor chips, and ecosystem of open-source processor chips. ()

    born in 1986. PhD, associate professor. Member of CCF. His main research interests include cloud computing, operating systems, and system modeling and performance analysis

About Author:

born in 1997. PhD candidate. His main research interests include distributed systems and machine learning systems

born in 1999. Master candidate. His main research interests include operating systems and machine learning systems. ()

born in 2001. Master candidate. His main research interests include machine learning system, virtualization, and cloud computing. ()

born in 1980. PhD, professor. Member of CCF. His main research interests include data center architecture, agile design methodology of processor chips, and ecosystem of open-source processor chips. ()

born in 1986. PhD, associate professor. Member of CCF. His main research interests include cloud computing, operating systems, and system modeling and performance analysis

Published: 2025-12-01 doi: 10.7544/issn1000-1239.202440560
Outline
收藏切换

As the number of parameters in deep learning models continues to increase, the cost of training also keeps rising. To reduce training costs, using spot instances provided by cloud service providers has become a viable solution. Spot instances are priced at only 30% of normal instances, which can significantly lower training costs. However, despite the low cost of elastic instances, there is a risk of them being reclaimed at any time, posing new challenges to the stability of the model training system. To address the fault tolerance issue in the context of elastic instances, existing work mainly falls into two categories: checkpoint-based fault tolerance and redundancy-based fault tolerance. Checkpoint-based solutions incur substantial overhead, while redundancy-based solutions impose certain limitations on the model’s parallelism strategy, leading to suboptimal training efficiency. We propose a solution for training in the context of spot instances that leverages the grace period of spot instances to back up training progress, thereby reducing fault tolerance overhead. It also employs the bottleneck alleviation approach to adjust the parallelism strategy, maximizing the use of available cluster resources and enhancing training efficiency. Experimental results show that this solution not only achieves low-cost fault tolerance but also ensures training efficiency, allowing for efficient completion of model training tasks in the context of spot instances and reducing overall training costs.

deep learning  /  neural network  /  distributed training  /  spot instance  /  fault tolerance
刘国栋, 朱家祺, 高梓源, 包云岗, 王卅. AdaptDNN:一个自适应可伸缩的大模型分布式训练系统. 计算机研究与发展, 2025 , 62 (12) : 3060 -3076 . DOI: 10.7544/issn1000-1239.202440560
Guodong Liu, Jiaqi Zhu, Ziyuan Gao, Yungang Bao, Sa Wang. AdaptDNN: An Adaptive and Scalable LLM Distributed Training System[J]. Journal of Computer Research and Development, 2025 , 62 (12) : 3060 -3076 . DOI: 10.7544/issn1000-1239.202440560
Year 2025 volume 62 Issue 12
86
39
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202440560
  • Receive Date:2024-06-21
  • Online Date:2026-11-10
  • Published:2025-12-01
Article Data
Affiliations
History
  • Received:2024-06-21
  • Revised:2025-05-09
  • Accepted:2025-06-11
Affiliations
    1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
    2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049
    3Beijing Institute of Open Source Chip, Beijing 100080
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202440560
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT