Article(id=1327017398063969045, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1327017337032651028, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202440560, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=research-article, receivedDate=1718899200000, receivedDateStr=2024-06-21, revisedDate=1746720000000, revisedDateStr=2025-05-09, acceptedDate=1749571200000, acceptedDateStr=2025-06-11, onlineDate=1794315202489, onlineDateStr=2026-11-10, pubDate=1764518400000, pubDateStr=2025-12-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1794315202489, onlineIssueDateStr=2026-11-10, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1794315202489, creator=13701087609, updateTime=1794315202489, updator=13701087609, issue=Issue{id=1327017337032651028, tenantId=1146029695717560320, journalId=1324759914146529478, year='2025', volume='62', issue='12', pageStart='2889', pageEnd='3146', issueExtLink='null', onlineDate='null', pubDate='1764518400000', pubDateStr='2025-12-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1794315187938, creator='13701087609', updateTime=1794377025235, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext={EN=IssueExt(id=1327276701995872766, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1327017337032651028, language=EN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=), CN=IssueExt(id=1327276701995872767, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1327017337032651028, language=CN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=)}, issueFiles=null, downloadFileDto=null}, startPage=3060, endPage=3076, ext={EN=ArticleExt(id=1327017398303044375, articleId=1327017398063969045, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=AdaptDNN: An Adaptive and Scalable LLM Distributed Training System, columnId=1327017398223352598, journalTitle=Journal of Computer Research and Development, columnName=Ditributed Computing, runingTitle=null, highlight=null, articleAbstract=

As the number of parameters in deep learning models continues to increase, the cost of training also keeps rising. To reduce training costs, using spot instances provided by cloud service providers has become a viable solution. Spot instances are priced at only 30% of normal instances, which can significantly lower training costs. However, despite the low cost of elastic instances, there is a risk of them being reclaimed at any time, posing new challenges to the stability of the model training system. To address the fault tolerance issue in the context of elastic instances, existing work mainly falls into two categories: checkpoint-based fault tolerance and redundancy-based fault tolerance. Checkpoint-based solutions incur substantial overhead, while redundancy-based solutions impose certain limitations on the model’s parallelism strategy, leading to suboptimal training efficiency. We propose a solution for training in the context of spot instances that leverages the grace period of spot instances to back up training progress, thereby reducing fault tolerance overhead. It also employs the bottleneck alleviation approach to adjust the parallelism strategy, maximizing the use of available cluster resources and enhancing training efficiency. Experimental results show that this solution not only achieves low-cost fault tolerance but also ensures training efficiency, allowing for efficient completion of model training tasks in the context of spot instances and reducing overall training costs.

, authors=Guodong Liu1, 2, Jiaqi Zhu1, 2, Ziyuan Gao1, 2, Yungang Bao1, 2, 3, Sa Wang1, 2, *, authorsList=Guodong Liu, Jiaqi Zhu, Ziyuan Gao, Yungang Bao, Sa Wang, authorCompany=null, correspAuthors=Sa Wang, authorNote=

born in 1997. PhD candidate. His main research interests include distributed systems and machine learning systems

born in 1999. Master candidate. His main research interests include operating systems and machine learning systems. ()

born in 2001. Master candidate. His main research interests include machine learning system, virtualization, and cloud computing. ()

born in 1980. PhD, professor. Member of CCF. His main research interests include data center architecture, agile design methodology of processor chips, and ecosystem of open-source processor chips. ()

born in 1986. PhD, associate professor. Member of CCF. His main research interests include cloud computing, operating systems, and system modeling and performance analysis

, correspAuthorsNote=null, copyrightStatement=Copyright © 2025 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1327017405081039681, articleId=1327017398063969045, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=AdaptDNN:一个自适应可伸缩的大模型分布式训练系统, columnId=1327017398395319064, journalTitle=计算机研究与发展, columnName=分布式计算, runingTitle=null, highlight=null, articleAbstract=

随着深度学习模型参数量的不断增加,训练成本也在不断上升. 为了减少训练成本,使用云服务厂商提供的弹性实例训练模型成为了一个可行的解决方案. 弹性实例的价格仅为正常实例的30%,可以有效降低训练成本. 虽然弹性实例价格低廉,但随时都有被回收的风险,对模型训练系统的稳定性提出了新的挑战. 为了解决弹性实例场景下的容错问题,现有的工作主要有2类解决方案,分别是基于存盘点的容错和基于冗余性的容错. 基于存盘点的方案开销较大,而基于冗余性的方案则对模型的并行策略有一定的限制,导致训练效率并非最优. AdaptDNN是一种自适应可伸缩的大模型分布式训练系统. 在弹性实例训练场景,AdaptDNN利用弹性实例宽限期完成训练进度的备份,降低容错开销;并利用“瓶颈消除”思想调整模型并行策略,最大化利用集群可用资源,提升训练效率. 实验结果表明,AdaptDNN既能实现低成本容错,又能保证模型训练效率,从而可以在弹性实例场景高效完成模型训练任务,降低模型训练成本.

, authors=刘国栋1, 2, 朱家祺1, 2, 高梓源1, 2, 包云岗1, 2, 3, 王卅1, 2, *, authorsList=刘国栋, 朱家祺, 高梓源, 包云岗, 王卅, authorCompany=null, correspAuthors=王卅, authorNote=

1997年生. 博士研究生. 主要研究方向为分布式系统、机器学习系统

1999年生. 硕士研究生. 主要研究方向为操作系统、机器学习系统

2001年生. 硕士研究生. 主要研究方向为机器学习系统、虚拟化、云计算

1980年生. 博士,研究员. CCF会员. 主要研究方向为数据中心体系结构、处理器芯片敏捷设计方法论、开源芯片生态

1986年生. 博士,副研究员. CCF会员. 主要研究方向为云计算、操作系统、系统建模与性能分析

, correspAuthorsNote=
王卅()
, copyrightStatement=版权所有 © 2025 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=gkHQZ2JKXzc1z1ZefH2iNA==, magXml=7b6BAQm8L46UYOzzLnzY1Q==, pdfUrl=null, pdf=7mP6AAkVYsp37ptnVaMe1A==, pdfFileSize=1717681, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=jLHYlTcMToAbKo/OM3DfTg==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=2nKMC+1/PqQLVW+JEy2zCg==, mapNumber=null, fund=null)}, authors=[Author(id=1327017405626299212, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=liuguodong19z@ict.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327017405701796687, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017405626299212, language=EN, stringName=Guodong Liu, firstName=Guodong, middleName=null, lastName=Liu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049, bio={"content":"

born in 1997. PhD candidate. His main research interests include distributed systems and machine learning systems

"}, bioImg=null, bioContent=

born in 1997. PhD candidate. His main research interests include distributed systems and machine learning systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327017405785682768, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017405626299212, language=CN, stringName=刘国栋, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190
2中国科学院大学计算机科学与技术学院 北京 100049, bio={"img":"kqpEJ9QjvXTxp0ddc18OCA==","content":"

1997年生. 博士研究生. 主要研究方向为分布式系统、机器学习系统

"}, bioImg=kqpEJ9QjvXTxp0ddc18OCA==, bioContent=

1997年生. 博士研究生. 主要研究方向为分布式系统、机器学习系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327017405328503618, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=1, ext=[AuthorCompanyExt(id=1327017405336892227, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1327017405345280836, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)]), AuthorCompany(id=1327017405404001093, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=2, ext=[AuthorCompanyExt(id=1327017405412389702, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049), AuthorCompanyExt(id=1327017405420778311, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学计算机科学与技术学院 北京 100049)])]), Author(id=1327017405848597330, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=zhujiaqi22s@ict.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327017405953454933, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017405848597330, language=EN, stringName=Jiaqi Zhu, firstName=Jiaqi, middleName=null, lastName=Zhu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049, bio={"content":"

born in 1999. Master candidate. His main research interests include operating systems and machine learning systems. ()

"}, bioImg=null, bioContent=

born in 1999. Master candidate. His main research interests include operating systems and machine learning systems. ()

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327017406033146710, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017405848597330, language=CN, stringName=朱家祺, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190
2中国科学院大学计算机科学与技术学院 北京 100049, bio={"img":"Dybo/kwYC4LzvNaihwOjgw==","content":"

1999年生. 硕士研究生. 主要研究方向为操作系统、机器学习系统

"}, bioImg=Dybo/kwYC4LzvNaihwOjgw==, bioContent=

1999年生. 硕士研究生. 主要研究方向为操作系统、机器学习系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327017405328503618, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=1, ext=[AuthorCompanyExt(id=1327017405336892227, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1327017405345280836, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)]), AuthorCompany(id=1327017405404001093, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=2, ext=[AuthorCompanyExt(id=1327017405412389702, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049), AuthorCompanyExt(id=1327017405420778311, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学计算机科学与技术学院 北京 100049)])]), Author(id=1327017406133810008, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=gaoziyuan23s@ict.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327017406221890395, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017406133810008, language=EN, stringName=Ziyuan Gao, firstName=Ziyuan, middleName=null, lastName=Gao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049, bio={"content":"

born in 2001. Master candidate. His main research interests include machine learning system, virtualization, and cloud computing. ()

"}, bioImg=null, bioContent=

born in 2001. Master candidate. His main research interests include machine learning system, virtualization, and cloud computing. ()

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327017406288999260, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017406133810008, language=CN, stringName=高梓源, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190
2中国科学院大学计算机科学与技术学院 北京 100049, bio={"img":"aFfixB295d9HZRPRXvmtnA==","content":"

2001年生. 硕士研究生. 主要研究方向为机器学习系统、虚拟化、云计算

"}, bioImg=aFfixB295d9HZRPRXvmtnA==, bioContent=

2001年生. 硕士研究生. 主要研究方向为机器学习系统、虚拟化、云计算

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327017405328503618, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=1, ext=[AuthorCompanyExt(id=1327017405336892227, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1327017405345280836, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)]), AuthorCompany(id=1327017405404001093, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=2, ext=[AuthorCompanyExt(id=1327017405412389702, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049), AuthorCompanyExt(id=1327017405420778311, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学计算机科学与技术学院 北京 100049)])]), Author(id=1327017406389662558, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=baoyg@ict.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327017406557434722, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017406389662558, language=EN, stringName=Yungang Bao, firstName=Yungang, middleName=null, lastName=Bao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, 3, address=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049
3Beijing Institute of Open Source Chip, Beijing 100080, bio={"content":"

born in 1980. PhD, professor. Member of CCF. His main research interests include data center architecture, agile design methodology of processor chips, and ecosystem of open-source processor chips. ()

"}, bioImg=null, bioContent=

born in 1980. PhD, professor. Member of CCF. His main research interests include data center architecture, agile design methodology of processor chips, and ecosystem of open-source processor chips. ()

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327017406620349283, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017406389662558, language=CN, stringName=包云岗, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, 3, address=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190
2中国科学院大学计算机科学与技术学院 北京 100049
3北京开源芯片研究院 北京 100080, bio={"img":"7P6RQM1rfHT6kQbd8KJsVQ==","content":"

1980年生. 博士,研究员. CCF会员. 主要研究方向为数据中心体系结构、处理器芯片敏捷设计方法论、开源芯片生态

"}, bioImg=7P6RQM1rfHT6kQbd8KJsVQ==, bioContent=

1980年生. 博士,研究员. CCF会员. 主要研究方向为数据中心体系结构、处理器芯片敏捷设计方法论、开源芯片生态

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327017405328503618, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=1, ext=[AuthorCompanyExt(id=1327017405336892227, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1327017405345280836, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)]), AuthorCompany(id=1327017405404001093, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=2, ext=[AuthorCompanyExt(id=1327017405412389702, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049), AuthorCompanyExt(id=1327017405420778311, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学计算机科学与技术学院 北京 100049)]), AuthorCompany(id=1327017405504664392, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=3, ext=[AuthorCompanyExt(id=1327017405517247305, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405504664392, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3Beijing Institute of Open Source Chip, Beijing 100080), AuthorCompanyExt(id=1327017405525635914, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405504664392, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3北京开源芯片研究院 北京 100080)])]), Author(id=1327017406683263845, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=wangsa@ict.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1327017406771344232, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017406683263845, language=EN, stringName=Sa Wang, firstName=Sa, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, *, address=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049, bio={"content":"

born in 1986. PhD, associate professor. Member of CCF. His main research interests include cloud computing, operating systems, and system modeling and performance analysis

"}, bioImg=null, bioContent=

born in 1986. PhD, associate professor. Member of CCF. His main research interests include cloud computing, operating systems, and system modeling and performance analysis

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327017406846841705, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017406683263845, language=CN, stringName=王卅, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, *, address=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190
2中国科学院大学计算机科学与技术学院 北京 100049, bio={"img":"zanVPMzIJcEekagYZAutzw==","content":"

1986年生. 博士,副研究员. CCF会员. 主要研究方向为云计算、操作系统、系统建模与性能分析

"}, bioImg=zanVPMzIJcEekagYZAutzw==, bioContent=

1986年生. 博士,副研究员. CCF会员. 主要研究方向为云计算、操作系统、系统建模与性能分析

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327017405328503618, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=1, ext=[AuthorCompanyExt(id=1327017405336892227, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1327017405345280836, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)]), AuthorCompany(id=1327017405404001093, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=2, ext=[AuthorCompanyExt(id=1327017405412389702, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049), AuthorCompanyExt(id=1327017405420778311, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学计算机科学与技术学院 北京 100049)])])], keywords=[Keyword(id=1327017406943310698, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, orderNo=1, keyword=deep learning), Keyword(id=1327017407006225259, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, orderNo=2, keyword=neural network), Keyword(id=1327017407060751212, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, orderNo=3, keyword=distributed training), Keyword(id=1327017407115277165, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, orderNo=4, keyword=spot instance), Keyword(id=1327017407169803118, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, orderNo=5, keyword=fault tolerance), Keyword(id=1327017407224329071, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, orderNo=1, keyword=深度学习), Keyword(id=1327017407308215152, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, orderNo=2, keyword=神经网络), Keyword(id=1327017407396295537, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, orderNo=3, keyword=分布式训练), Keyword(id=1327017409057239922, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, orderNo=4, keyword=弹性实例), Keyword(id=1327017409124348787, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, orderNo=5, keyword=容错)], refs=[Reference(id=1327017411825480602, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=OpenAI. AI and compute[EB/OL]. (2018-05-16)[2024-05-30]. https://openai.com/index/ai-and-compute/, articleTitle=null, refAbstract=null), Reference(id=1327017411892589467, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Mu, Andersen D, Park J, et al. Scaling distributed machine learning with the parameter server[C]//Proc of the 11th USENIX Symp on Operating Systems Design and Implementation. Berkeley, CA: USENIX Association, 2014: 583−598, articleTitle=null, refAbstract=null), Reference(id=1327017413591282588, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Jia Zhihao, Zaharia M, Aiken A. Beyond data and model parallelism for deep neural networks[C/OL]//Proc of the 2nd Machine Learning and Systems Conf. 2019[2024-10-21]. https://proceedings.mlsys.org/paper_files/paper/2019/file/b422680f3db0986ddd7f8f126baaf0fa-Paper.pdf, articleTitle=null, refAbstract=null), Reference(id=1327017413666780061, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Narayanan D, Harlap A, Phanishayee A, et al. PipeDream: Generalized pipeline parallelism for DNN training[C]//Proc of the 27th ACM Symp on Operating Systems Principles. New York: ACM, 2019: 1−15, articleTitle=null, refAbstract=null), Reference(id=1327017413733888926, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Brown T, Mann B, Ryder N, et al. Language models are few-shot learners[C/OL]//Proc of the 34th Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2020[2024-10-21]. https://papers.nips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf, articleTitle=null, refAbstract=null), Reference(id=1327017413821969311, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Duan Jiangfei, Song Ziang, Miao Xupeng, et al. Parcae: Proactive, liveput-optimized DNN training on preemptible instances[C]//Proc of the 21st USENIX Symp on Networked Systems Design and Implementation. Berkeley, CA: USENIX Association, 2024: 1121−1139, articleTitle=null, refAbstract=null), Reference(id=1327017413897466784, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Thorpe J, Zhao Pengzhan, Eyolfson J, et al. Bamboo: Making preemptible instances resilient for affordable training of large DNNs[C]//Proc of the 20th USENIX Symp on Networked Systems Design and Implementation. Berkeley, CA: USENIX Association, 2023: 497−513, articleTitle=null, refAbstract=null), Reference(id=1327017413989741473, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Jiamin, Xu Hong, Zhu Yibo, et al. Lyra: Elastic scheduling for deep learning clusters[C]//Proc of the 18th European Conf on Computer Systems. New York: ACM, 2023: 835−850, articleTitle=null, refAbstract=null), Reference(id=1327017414061044642, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Narayanan D, Shoeybi M, Casper J, et al. Efficient large-scale language model training on GPU clusters using megatron-LM[C]//Proc of the 34th Int Conf for High Performance Computing, Networking, Storage and Analysis. New York: ACM, 2021: 1−15, articleTitle=null, refAbstract=null), Reference(id=1327017414132347811, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Mohan J, Phanishayee A, Chidambaram V. CheckFreq: Frequent, fine-grained DNN checkpointing[C]//Proc of the 19th USENIX Conf on File and Storage Technologies. Berkeley, CA: USENIX Association, 2021: 203−216, articleTitle=null, refAbstract=null), Reference(id=1327017414195262372, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Athlur S, Saran N, Sivathanu M, et al. Varuna: Scalable, low-cost training of massive deep learning models[C]//Proc of the 17th European Conf on Computer Systems. New York: ACM, 2022: 472−487, articleTitle=null, refAbstract=null), Reference(id=1327017414258176933, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Jang I, Yang Zhenning, Zhang Zhen, et al. Oobleck: Resilient distributed training of large models using pipeline templates[C]//Proc of the 29th Symp on Operating Systems Principles. New York: ACM, 2023: 382−395, articleTitle=null, refAbstract=null), Reference(id=1327017414329480102, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Zheng Lianmin, Li Zhuohan, Zhang Hao, et al. Alpa: Automating inter-and intra-operator parallelism for distributed deep learning[C]//Proc of the 16th USENIX Symp on Operating Systems Design and Implementation. Berkeley, CA: USENIX Association, 2022: 559−578, articleTitle=null, refAbstract=null), Reference(id=1327017414384006055, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Guodong, Miao Youshan, Lin Zhiqi, et al. Aceso: Efficient parallel DNN training through iterative bottleneck alleviation[C]//Proc of the 19th European Conf on Computer Systems. New York: ACM, 2024: 163−181, articleTitle=null, refAbstract=null), Reference(id=1327017414476280744, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Dean J, Corrado G, Monga R, et al. Large scale distributed deep networks[C/OL]//Proc of the 26th Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2012[2024-10-21]. https://papers.nips.cc/paper_files/paper/2012/file/6aca97005c68f1206823815f66102863-Paper.pdf, articleTitle=null, refAbstract=null), Reference(id=1327017414560166825, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Minjie, Huang C, Li Jinyang. Supporting very large models using automatic dataflow graph partitioning[C]//Proc of the 14th European Conf on Computer Systems. New York: ACM, 2019: 1−17, articleTitle=null, refAbstract=null), Reference(id=1327017414618887082, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Mirhoseini A, Pham H, Le Q, et al. Device placement optimization with reinforcement learning[C]//Proc of the 34th Int Conf on Machine Learning. New York: PMLR, 2017: 2430-−2439, articleTitle=null, refAbstract=null), Reference(id=1327017414685995947, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Gao Yuanxiang, Chen Li, Li Baochun. Spotlight: Optimizing device placement for training deep neural networks[C]//Proc of the 35th Int Conf on Machine Learning. New York: PMLR, 2018: 1676−1684, articleTitle=null, refAbstract=null), Reference(id=1327017414753104812, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Xu Yuanzhong, HyoukJoong L, Chen Dehao, et al. GSPMD: General and scalable parallelization for ML computation graphs[J]. arXiv preprint, arXiv: 2105.04663, articleTitle=null, refAbstract=null), Reference(id=1327017414820213677, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Yanping, Cheng Youlong, Bapna A, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism[C/OL]//Proc of the 33rd Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2019[2024-10-21]. https://proceedings.neurips.cc/paper_files/paper/2019/file/093f65e080a295f8076b1c5722a46aa2-Paper.pdf, articleTitle=null, refAbstract=null), Reference(id=1327017414874739630, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Narayanan D, Phanishayee A, Shi K, et al. Memory-efficient pipeline-parallel DNN training[C]//Proc of the 38th Int Conf on Machine Learning. New York: PMLR, 2021: 7937−7947, articleTitle=null, refAbstract=null), Reference(id=1327017414975402927, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Shigang, Torsten H. Chimera: Efficiently training large-scale neural networks with bidirectional pipelines[C]//Proc of the 34th Int Conf for High Performance Computing, Networking, Storage and Analysis. New York: ACM, 2021: 1−14, articleTitle=null, refAbstract=null), Reference(id=1327017415042511792, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Fan Shiqing, Rong Yi, Meng Chen, et al. DAPPLE: A pipelined data parallel approach for training large models[C]//Proc of the 26th ACM SIGPLAN Symp on Principles and Practice of Parallel Programming. New York: ACM, 2021: 431−445, articleTitle=null, refAbstract=null), Reference(id=1327017415109620657, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=PyTorch. Torch Distributed Elastic[EB/OL]. [2024-05-30]. https://pytorch.org/docs/stable/distributed.elastic.html, articleTitle=null, refAbstract=null), Reference(id=1327017415180923826, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=The Horovod Authors. Elastic Hovorod[EB/OL]. [2024-05-30]. https://horovod.readthedocs.io/en/stable/elastic_include.html, articleTitle=null, refAbstract=null), Reference(id=1327017415252226995, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=Wu Yidi, Ma Kaihao, Yan Xiao, et al. Elastic deep learning in multi-tenant GPU clusters[J]. IEEE Transactions on Parallel and Distributed Systems, 2021, 33(1): 144−158, articleTitle=null, refAbstract=null), Reference(id=1327017415323530164, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=26, authorNames=null, journalName=null, refType=null, unstructuredReference=Xiao Wencong, Bhardwaj R, Ramjee R, et al. Gandiva: Introspective cluster scheduling for deep learning[C]//Proc of the 13th USENIX Symp on Operating Systems Design and Implementation. Berkeley, CA: USENIX Association, 2018: 595−610, articleTitle=null, refAbstract=null), Reference(id=1327017415403221941, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=28, rfOrder=27, authorNames=null, journalName=null, refType=null, unstructuredReference=Qiao A, Choe S, Subramanya S, et al. Pollux: Co-adaptive cluster scheduling for goodput-optimized deep learning[C]//Proc of the 15th USENIX Symp on Operating Systems Design and Implementation. Berkeley, CA: USENIX Association, 2021: 1−18, articleTitle=null, refAbstract=null), Reference(id=1327017415478719414, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=29, rfOrder=28, authorNames=null, journalName=null, refType=null, unstructuredReference=Gu Diandian, Zhao Yihao, Zhong Yinmin, et al. ElasticFlow: An elastic serverless training platform for distributed deep learning[C]//Proc of the 28th ACM Int Conf on Architectural Support for Programming Languages and Operating Systems. New York: ACM, 2023: 266−280, articleTitle=null, refAbstract=null), Reference(id=1327017415604548535, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=30, rfOrder=29, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Zhuang, Jia Zhen, Zheng Shuai, et al. Gemini: Fast failure recovery in distributed training with in-memory checkpoints[C]//Proc of the 29th Symp on Operating Systems Principles. New York: ACM, 2023: 364−381, articleTitle=null, refAbstract=null), Reference(id=1327017415680046008, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=31, rfOrder=30, authorNames=null, journalName=null, refType=null, unstructuredReference=He Tao, Li Xue, Wang Zhibin, et al. Unicron: Economizing self-healing LLM training at scale[J]. arXiv preprint, arXiv: 2401.00134, 2023, articleTitle=null, refAbstract=null), Reference(id=1327017415751349177, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=32, rfOrder=31, authorNames=null, journalName=null, refType=null, unstructuredReference=Jiang Ziheng, Lin Haibin, Zhong Yinmin, et al. MegaScale: Scaling large language model training to more than <styled-content style-type="number"> 10000 </styled-content> GPUs[C]//Proc of the 21st USENIX Symp on Networked Systems Design and Implementation. Berkeley, CA: USENIX Association, 2024: 745−760, articleTitle=null, refAbstract=null), Reference(id=1327017415822652346, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=33, rfOrder=32, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Yuxin, Kang Xueze, Shi Shaohuai, et al. Reliable and efficient in-memory fault tolerance of large language model pretraining[EB/OL]. (2024-08-19)[2024-09-06]. https://arxiv.org/abs/2310.12670, articleTitle=null, refAbstract=null), Reference(id=1327017415906538427, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=34, rfOrder=33, authorNames=null, journalName=null, refType=null, unstructuredReference=Lian Xinyu, Jacobs S, Kurilenko L, et al. Universal checkpointing: Efficient and flexible checkpointing for large scale distributed training[J]. arXiv preprint, arXiv: 2406.18820, 2024, articleTitle=null, refAbstract=null), Reference(id=1327017415990424508, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=35, rfOrder=34, authorNames=null, journalName=null, refType=null, unstructuredReference=Wan Borui, Han Mingji, Sheng Yiyao, et al. ByteCheckpoint: A unified checkpointing system for LLM development[J]. arXiv preprint, arXiv: 2407.20143, 2024, articleTitle=null, refAbstract=null), Reference(id=1327017416086893501, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=36, rfOrder=35, authorNames=null, journalName=null, refType=null, unstructuredReference=Gandhi S, Zhao M, Skiadopoulos A, et al. SlipStream: Adapting pipelines for distributed training of large DNNs amid failures[J]. arXiv preprint, arXiv: 2405.14009, 2024, articleTitle=null, refAbstract=null), Reference(id=1327017416183362494, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=37, rfOrder=36, authorNames=null, journalName=null, refType=null, unstructuredReference=Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C/OL]//Proc of the 31st Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2017[2024-10-21]. https://user.phil.hhu.de/~cwurm/wp-content/uploads/2020/01/7181-attention-is-all-you-need.pdf, articleTitle=null, refAbstract=null), Reference(id=1327017416254665663, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=38, rfOrder=37, authorNames=null, journalName=null, refType=null, unstructuredReference=Navarro A, Asenjo R, Tabik S, et al. Analytical modeling of pipeline parallelism[C]//Proc of the 18th Int Conf on Parallel Architectures and Compilation Techniques. Piscataway, NJ: IEEE, 2009: 291−290, articleTitle=null, refAbstract=null), Reference(id=1327017416325968832, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=39, rfOrder=38, authorNames=null, journalName=null, refType=null, unstructuredReference=Perez T, Chen Wei, Ji R, et al. PETS: Bottleneck-aware spark tuning with parameter ensembles[C]//Proc of the 27th Int Conf on Computer Communication and Networks. Piscataway, NJ: IEEE, 2018: 1−9, articleTitle=null, refAbstract=null), Reference(id=1327017416401466305, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=40, rfOrder=39, authorNames=null, journalName=null, refType=null, unstructuredReference=Kahveci B, Gedik B. Joker: Elastic stream processing with organic adaptation[J]. Journal of Parallel and Distributed Computing, 2020, 137: 205-223, articleTitle=null, refAbstract=null), Reference(id=1327017418125325250, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=40, rfOrder=40, authorNames=null, journalName=null, refType=null, unstructuredReference=Kahveci B, Gedik B. Joker: Elastic stream processing with organic adaptation[J]. Journal of Parallel and Distributed Computing, 2020, 137: 205-223, articleTitle=null, refAbstract=null), Reference(id=1327017418238571459, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=41, rfOrder=41, authorNames=null, journalName=null, refType=null, unstructuredReference=NVIDIA. ConnectX-5: Advanced offload capabilities for the most demanding applications[EB/OL]. [2024-05-30]. https://www.nvidia.com/ en-us/networking/ethernet/connectx-5/, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1327017405328503618, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=1, ext=[AuthorCompanyExt(id=1327017405336892227, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1327017405345280836, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)]), AuthorCompany(id=1327017405404001093, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=2, ext=[AuthorCompanyExt(id=1327017405412389702, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049), AuthorCompanyExt(id=1327017405420778311, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学计算机科学与技术学院 北京 100049)]), AuthorCompany(id=1327017405504664392, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=3, ext=[AuthorCompanyExt(id=1327017405517247305, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405504664392, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3Beijing Institute of Open Source Chip, Beijing 100080), AuthorCompanyExt(id=1327017405525635914, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405504664392, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3北京开源芯片研究院 北京 100080)])], figs=[ArticleFig(id=1327017409266955124, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, label=Figure 1, caption=System overview, figureFileSmall=edI/EJ3q3g4GHyi/3F9t1A==, figureFileBig=jLHYlTcMToAbKo/OM3DfTg==, tableContent=null), ArticleFig(id=1327017409338258293, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, label=图1, caption=系统概览, figureFileSmall=edI/EJ3q3g4GHyi/3F9t1A==, figureFileBig=jLHYlTcMToAbKo/OM3DfTg==, tableContent=null), ArticleFig(id=1327017409501836150, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, label=Figure 2, caption=3D parallelism, figureFileSmall=ErDFHO+cbn2YyZFLmh490A==, figureFileBig=e0GQv5Vp6Hfvid+FUbGoLQ==, tableContent=null), ArticleFig(id=1327017409564750712, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, label=图2, caption=3D并行, figureFileSmall=ErDFHO+cbn2YyZFLmh490A==, figureFileBig=e0GQv5Vp6Hfvid+FUbGoLQ==, tableContent=null), ArticleFig(id=1327017409627665273, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, label=Figure 3, caption=Heterogenous pipeline, figureFileSmall=NzPxAokLSTYGpoah/qs6eQ==, figureFileBig=xCWAF5JlL4KwrX3VZXYi4w==, tableContent=null), ArticleFig(id=1327017409698968442, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, label=图3, caption=异构流水线, figureFileSmall=NzPxAokLSTYGpoah/qs6eQ==, figureFileBig=xCWAF5JlL4KwrX3VZXYi4w==, tableContent=null), ArticleFig(id=1327017409757688699, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, label=Figure 4, caption=Comparison of model reconstruction overhead of different fault-tolerant methods, figureFileSmall=xriYJyEkQK4zl/UGbX+1FQ==, figureFileBig=gaTr9VdIVX67L7Kr78ks5A==, tableContent=null), ArticleFig(id=1327017409812214652, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, label=图4, caption=不同容错方式的模型重构开销对比, figureFileSmall=xriYJyEkQK4zl/UGbX+1FQ==, figureFileBig=gaTr9VdIVX67L7Kr78ks5A==, tableContent=null), ArticleFig(id=1327017409866740605, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, label=Figure 5, caption=Searching for parallel schemes by using bottleneck alleviation idea, figureFileSmall=2pqxotJYB3tprKg0Uxa4Sw==, figureFileBig=+alhFog6S6ybr3v/n8HBBg==, tableContent=null), ArticleFig(id=1327017409929655166, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, label=图5, caption=使用“瓶颈消除”思想搜索并行方案, figureFileSmall=2pqxotJYB3tprKg0Uxa4Sw==, figureFileBig=+alhFog6S6ybr3v/n8HBBg==, tableContent=null), ArticleFig(id=1327017410021929855, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, label=Figure 6, caption=The first performance bottleneck in fault tolerance scenario, figureFileSmall=zVOjR5SK3O4WkbR+rV78fg==, figureFileBig=QIWRgzR03++qc8tN7NxNVA==, tableContent=null), ArticleFig(id=1327017410089038720, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, label=图6, caption=容错场景下的第1种性能瓶颈, figureFileSmall=zVOjR5SK3O4WkbR+rV78fg==, figureFileBig=QIWRgzR03++qc8tN7NxNVA==, tableContent=null), ArticleFig(id=1327017410147758977, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, label=Figure 7, caption=The second performance bottleneck in fault tolerance scenario, figureFileSmall=nYEkaerjnhjkVilZ9MNYUw==, figureFileBig=bmUY62du1UyvlTMQCN1MlA==, tableContent=null), ArticleFig(id=1327017410206479234, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, label=图7, caption=容错场景下的第2种性能瓶颈, figureFileSmall=nYEkaerjnhjkVilZ9MNYUw==, figureFileBig=bmUY62du1UyvlTMQCN1MlA==, tableContent=null), ArticleFig(id=1327017410261005187, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, label=Figure 8, caption=Workflow of bottleneck alleviation algorithm, figureFileSmall=Sn3YLyibGhe0WBk4eWCSPQ==, figureFileBig=sZNCM4NhlilL+SuQAMeMPQ==, tableContent=null), ArticleFig(id=1327017410323919748, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, label=图8, caption=“瓶颈消除”算法工作流程, figureFileSmall=Sn3YLyibGhe0WBk4eWCSPQ==, figureFileBig=sZNCM4NhlilL+SuQAMeMPQ==, tableContent=null), ArticleFig(id=1327017410399417221, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, label=Figure 9, caption=Overall throughput comparison of training system, figureFileSmall=W+DK5riCG1GCkkSnDzwFaQ==, figureFileBig=fJh0lnbkdMBebfF0Lh0NxQ==, tableContent=null), ArticleFig(id=1327017410458137478, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, label=图9, caption=训练系统总体吞吐量对比, figureFileSmall=W+DK5riCG1GCkkSnDzwFaQ==, figureFileBig=fJh0lnbkdMBebfF0Lh0NxQ==, tableContent=null), ArticleFig(id=1327017410516857735, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, label=Figure 10, caption=Comparison of training time per round, figureFileSmall=4y1aXFl/CO2YEO0Q4zc8BQ==, figureFileBig=5srYOjjpE1zeIWvrWKII1g==, tableContent=null), ArticleFig(id=1327017410567189384, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, label=图10, caption=每轮次训练时间对比, figureFileSmall=4y1aXFl/CO2YEO0Q4zc8BQ==, figureFileBig=5srYOjjpE1zeIWvrWKII1g==, tableContent=null), ArticleFig(id=1327017410621715337, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, label=Figure 11, caption=Comparison of time for fault recovery, figureFileSmall=lWzQZ+UXWOA0B5XR8bnsGw==, figureFileBig=ifWvUNGETAUx8dCQrOH4zw==, tableContent=null), ArticleFig(id=1327017410676241290, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, label=图11, caption=故障恢复时间对比, figureFileSmall=lWzQZ+UXWOA0B5XR8bnsGw==, figureFileBig=ifWvUNGETAUx8dCQrOH4zw==, tableContent=null), ArticleFig(id=1327017410734961547, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, label=Figure 12, caption=Comparison of training throughput under real spot instance trace, figureFileSmall=WXDFfSB08TWHclnCLqRrIA==, figureFileBig=CzgPOBUiMwWywA2XT1ykOQ==, tableContent=null), ArticleFig(id=1327017410785293196, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, label=图12, caption=真实弹性实例trace下的训练吞吐量对比, figureFileSmall=WXDFfSB08TWHclnCLqRrIA==, figureFileBig=CzgPOBUiMwWywA2XT1ykOQ==, tableContent=null), ArticleFig(id=1327017410848207757, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, label=Figure 13, caption=Influence of MTTF on iters, figureFileSmall=5Jc34oVkgpWfgYQ2qddClw==, figureFileBig=CRvAmRNczgm3V3iuS8iobA==, tableContent=null), ArticleFig(id=1327017410915316622, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, label=图13, caption=平均故障时间对iters的影响, figureFileSmall=5Jc34oVkgpWfgYQ2qddClw==, figureFileBig=CRvAmRNczgm3V3iuS8iobA==, tableContent=null), ArticleFig(id=1327017410990814095, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, label=Figure 14, caption=Influence of bandwidth on iters, figureFileSmall=T0PFkiiYg8ldbUo1JVQCmg==, figureFileBig=wmpnG6Y/dRBsO3Xpr+2qWg==, tableContent=null), ArticleFig(id=1327017411129226128, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, label=图14, caption=网络带宽对iters的影响, figureFileSmall=T0PFkiiYg8ldbUo1JVQCmg==, figureFileBig=wmpnG6Y/dRBsO3Xpr+2qWg==, tableContent=null), ArticleFig(id=1327017411200529297, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, label=Figure 15, caption=Influence of grace period on iters, figureFileSmall=tUOaDiineVNBwp+jRyWE2w==, figureFileBig=L9LFCtuDfJUn+OCSjxW5iA==, tableContent=null), ArticleFig(id=1327017411263443858, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, label=图15, caption=宽限期对iters的影响, figureFileSmall=tUOaDiineVNBwp+jRyWE2w==, figureFileBig=L9LFCtuDfJUn+OCSjxW5iA==, tableContent=null), ArticleFig(id=1327017411322164115, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, label=Table 1, caption=Experimental Platform Configuration, figureFileSmall=null, figureFileBig=null, tableContent=
配置参数
处理器Intel® Xeon® Gold 5218R CPU
GPUNVIDIA V100-PCIE(16 GB)
操作系统Ubuntu 18.04.5 LTS
CUDA版本11.6
PyTorch版本2.0.0
), ArticleFig(id=1327017411397661588, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, label=表1, caption=

实验平台配置

, figureFileSmall=null, figureFileBig=null, tableContent=
配置参数
处理器Intel® Xeon® Gold 5218R CPU
GPUNVIDIA V100-PCIE(16 GB)
操作系统Ubuntu 18.04.5 LTS
CUDA版本11.6
PyTorch版本2.0.0
), ArticleFig(id=1327017411464770453, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, label=Table 2, caption=DNN Model Parameters, figureFileSmall=null, figureFileBig=null, tableContent=
配置参数
参数量350×106, 1.3×109, 2.7×109, 6.7×109
精度FP16
数据批次大小1 024
序列长度2 048
), ArticleFig(id=1327017411544462230, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, label=表2, caption=

DNN模型参数

, figureFileSmall=null, figureFileBig=null, tableContent=
配置参数
参数量350×106, 1.3×109, 2.7×109, 6.7×109
精度FP16
数据批次大小1 024
序列长度2 048
), ArticleFig(id=1327017411628348311, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=EN, label=Table 3, caption=Average Throughput Comparison, figureFileSmall=null, figureFileBig=null, tableContent=
模型GPT-3 1.3BGPT-3 2.7B
trace-1trace-2trace-1trace-2
Checkpoint17.1316.388.217.90
Oobleck22.8321.1512.0711.79
AdaptDNN(本文)27.8926.9214.2214.15
), ArticleFig(id=1327017411708040089, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, language=CN, label=表3, caption=

平均吞吐量对比

, figureFileSmall=null, figureFileBig=null, tableContent=
模型GPT-3 1.3BGPT-3 2.7B
trace-1trace-2trace-1trace-2
Checkpoint17.1316.388.217.90
Oobleck22.8321.1512.0711.79
AdaptDNN(本文)27.8926.9214.2214.15
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202440560, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202440560, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202440560, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202440560, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1764518400000, fullTextJson=null, articleText=null, reference=null)
收藏切换
AdaptDNN:一个自适应可伸缩的大模型分布式训练系统
刘国栋 1, 2 , 朱家祺 1, 2 , 高梓源 1, 2 , 包云岗 1, 2, 3 , 王卅 1, 2, *
计算机研究与发展 | 分布式计算 2025,62(12): 3060-3076
收起
收藏切换
计算机研究与发展 |分布式计算 2025 , 62 (12) : 3060 -3076
AdaptDNN:一个自适应可伸缩的大模型分布式训练系统
全屏
[Author(id=1327017405626299212, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=liuguodong19z@ict.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327017405701796687, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017405626299212, language=EN, stringName=Guodong Liu, firstName=Guodong, middleName=null, lastName=Liu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049, bio={"content":"

born in 1997. PhD candidate. His main research interests include distributed systems and machine learning systems

"}, bioImg=null, bioContent=

born in 1997. PhD candidate. His main research interests include distributed systems and machine learning systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327017405785682768, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017405626299212, language=CN, stringName=刘国栋, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190
2中国科学院大学计算机科学与技术学院 北京 100049, bio={"img":"kqpEJ9QjvXTxp0ddc18OCA==","content":"

1997年生. 博士研究生. 主要研究方向为分布式系统、机器学习系统

"}, bioImg=kqpEJ9QjvXTxp0ddc18OCA==, bioContent=

1997年生. 博士研究生. 主要研究方向为分布式系统、机器学习系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327017405328503618, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=1, ext=[AuthorCompanyExt(id=1327017405336892227, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1327017405345280836, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)]), AuthorCompany(id=1327017405404001093, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=2, ext=[AuthorCompanyExt(id=1327017405412389702, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049), AuthorCompanyExt(id=1327017405420778311, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学计算机科学与技术学院 北京 100049)])]), Author(id=1327017405848597330, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=zhujiaqi22s@ict.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327017405953454933, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017405848597330, language=EN, stringName=Jiaqi Zhu, firstName=Jiaqi, middleName=null, lastName=Zhu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049, bio={"content":"

born in 1999. Master candidate. His main research interests include operating systems and machine learning systems. ()

"}, bioImg=null, bioContent=

born in 1999. Master candidate. His main research interests include operating systems and machine learning systems. ()

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327017406033146710, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017405848597330, language=CN, stringName=朱家祺, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190
2中国科学院大学计算机科学与技术学院 北京 100049, bio={"img":"Dybo/kwYC4LzvNaihwOjgw==","content":"

1999年生. 硕士研究生. 主要研究方向为操作系统、机器学习系统

"}, bioImg=Dybo/kwYC4LzvNaihwOjgw==, bioContent=

1999年生. 硕士研究生. 主要研究方向为操作系统、机器学习系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327017405328503618, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=1, ext=[AuthorCompanyExt(id=1327017405336892227, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1327017405345280836, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)]), AuthorCompany(id=1327017405404001093, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=2, ext=[AuthorCompanyExt(id=1327017405412389702, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049), AuthorCompanyExt(id=1327017405420778311, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学计算机科学与技术学院 北京 100049)])]), Author(id=1327017406133810008, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=gaoziyuan23s@ict.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327017406221890395, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017406133810008, language=EN, stringName=Ziyuan Gao, firstName=Ziyuan, middleName=null, lastName=Gao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049, bio={"content":"

born in 2001. Master candidate. His main research interests include machine learning system, virtualization, and cloud computing. ()

"}, bioImg=null, bioContent=

born in 2001. Master candidate. His main research interests include machine learning system, virtualization, and cloud computing. ()

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327017406288999260, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017406133810008, language=CN, stringName=高梓源, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190
2中国科学院大学计算机科学与技术学院 北京 100049, bio={"img":"aFfixB295d9HZRPRXvmtnA==","content":"

2001年生. 硕士研究生. 主要研究方向为机器学习系统、虚拟化、云计算

"}, bioImg=aFfixB295d9HZRPRXvmtnA==, bioContent=

2001年生. 硕士研究生. 主要研究方向为机器学习系统、虚拟化、云计算

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327017405328503618, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=1, ext=[AuthorCompanyExt(id=1327017405336892227, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1327017405345280836, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)]), AuthorCompany(id=1327017405404001093, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=2, ext=[AuthorCompanyExt(id=1327017405412389702, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049), AuthorCompanyExt(id=1327017405420778311, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学计算机科学与技术学院 北京 100049)])]), Author(id=1327017406389662558, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=baoyg@ict.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327017406557434722, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017406389662558, language=EN, stringName=Yungang Bao, firstName=Yungang, middleName=null, lastName=Bao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, 3, address=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049
3Beijing Institute of Open Source Chip, Beijing 100080, bio={"content":"

born in 1980. PhD, professor. Member of CCF. His main research interests include data center architecture, agile design methodology of processor chips, and ecosystem of open-source processor chips. ()

"}, bioImg=null, bioContent=

born in 1980. PhD, professor. Member of CCF. His main research interests include data center architecture, agile design methodology of processor chips, and ecosystem of open-source processor chips. ()

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327017406620349283, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017406389662558, language=CN, stringName=包云岗, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, 3, address=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190
2中国科学院大学计算机科学与技术学院 北京 100049
3北京开源芯片研究院 北京 100080, bio={"img":"7P6RQM1rfHT6kQbd8KJsVQ==","content":"

1980年生. 博士,研究员. CCF会员. 主要研究方向为数据中心体系结构、处理器芯片敏捷设计方法论、开源芯片生态

"}, bioImg=7P6RQM1rfHT6kQbd8KJsVQ==, bioContent=

1980年生. 博士,研究员. CCF会员. 主要研究方向为数据中心体系结构、处理器芯片敏捷设计方法论、开源芯片生态

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327017405328503618, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=1, ext=[AuthorCompanyExt(id=1327017405336892227, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1327017405345280836, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)]), AuthorCompany(id=1327017405404001093, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=2, ext=[AuthorCompanyExt(id=1327017405412389702, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049), AuthorCompanyExt(id=1327017405420778311, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学计算机科学与技术学院 北京 100049)]), AuthorCompany(id=1327017405504664392, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=3, ext=[AuthorCompanyExt(id=1327017405517247305, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405504664392, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3Beijing Institute of Open Source Chip, Beijing 100080), AuthorCompanyExt(id=1327017405525635914, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405504664392, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3北京开源芯片研究院 北京 100080)])]), Author(id=1327017406683263845, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=wangsa@ict.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1327017406771344232, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017406683263845, language=EN, stringName=Sa Wang, firstName=Sa, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, *, address=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049, bio={"content":"

born in 1986. PhD, associate professor. Member of CCF. His main research interests include cloud computing, operating systems, and system modeling and performance analysis

"}, bioImg=null, bioContent=

born in 1986. PhD, associate professor. Member of CCF. His main research interests include cloud computing, operating systems, and system modeling and performance analysis

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327017406846841705, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, authorId=1327017406683263845, language=CN, stringName=王卅, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, *, address=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190
2中国科学院大学计算机科学与技术学院 北京 100049, bio={"img":"zanVPMzIJcEekagYZAutzw==","content":"

1986年生. 博士,副研究员. CCF会员. 主要研究方向为云计算、操作系统、系统建模与性能分析

"}, bioImg=zanVPMzIJcEekagYZAutzw==, bioContent=

1986年生. 博士,副研究员. CCF会员. 主要研究方向为云计算、操作系统、系统建模与性能分析

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327017405328503618, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=1, ext=[AuthorCompanyExt(id=1327017405336892227, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1327017405345280836, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405328503618, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)]), AuthorCompany(id=1327017405404001093, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, xref=2, ext=[AuthorCompanyExt(id=1327017405412389702, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049), AuthorCompanyExt(id=1327017405420778311, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327017398063969045, companyId=1327017405404001093, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学计算机科学与技术学院 北京 100049)])])]
刘国栋1, 2 , 朱家祺1, 2 , 高梓源1, 2 , 包云岗1, 2, 3 , 王卅1, 2, *
作者信息
  • 1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190
  • 2中国科学院大学计算机科学与技术学院 北京 100049
  • 3北京开源芯片研究院 北京 100080
通讯作者:
王卅()
作者简介:

1997年生. 博士研究生. 主要研究方向为分布式系统、机器学习系统

1999年生. 硕士研究生. 主要研究方向为操作系统、机器学习系统

2001年生. 硕士研究生. 主要研究方向为机器学习系统、虚拟化、云计算

1980年生. 博士,研究员. CCF会员. 主要研究方向为数据中心体系结构、处理器芯片敏捷设计方法论、开源芯片生态

1986年生. 博士,副研究员. CCF会员. 主要研究方向为云计算、操作系统、系统建模与性能分析

AdaptDNN: An Adaptive and Scalable LLM Distributed Training System
Guodong Liu1, 2 , Jiaqi Zhu1, 2 , Ziyuan Gao1, 2 , Yungang Bao1, 2, 3 , Sa Wang1, 2, *
Affiliations
  • 1State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
  • 2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049
  • 3Beijing Institute of Open Source Chip, Beijing 100080
  • born in 1997. PhD candidate. His main research interests include distributed systems and machine learning systems

    born in 1999. Master candidate. His main research interests include operating systems and machine learning systems. ()

    born in 2001. Master candidate. His main research interests include machine learning system, virtualization, and cloud computing. ()

    born in 1980. PhD, professor. Member of CCF. His main research interests include data center architecture, agile design methodology of processor chips, and ecosystem of open-source processor chips. ()

    born in 1986. PhD, associate professor. Member of CCF. His main research interests include cloud computing, operating systems, and system modeling and performance analysis

出版时间: 2025-12-01 doi: 10.7544/issn1000-1239.202440560
文章导航
收藏切换

随着深度学习模型参数量的不断增加,训练成本也在不断上升. 为了减少训练成本,使用云服务厂商提供的弹性实例训练模型成为了一个可行的解决方案. 弹性实例的价格仅为正常实例的30%,可以有效降低训练成本. 虽然弹性实例价格低廉,但随时都有被回收的风险,对模型训练系统的稳定性提出了新的挑战. 为了解决弹性实例场景下的容错问题,现有的工作主要有2类解决方案,分别是基于存盘点的容错和基于冗余性的容错. 基于存盘点的方案开销较大,而基于冗余性的方案则对模型的并行策略有一定的限制,导致训练效率并非最优. AdaptDNN是一种自适应可伸缩的大模型分布式训练系统. 在弹性实例训练场景,AdaptDNN利用弹性实例宽限期完成训练进度的备份,降低容错开销;并利用“瓶颈消除”思想调整模型并行策略,最大化利用集群可用资源,提升训练效率. 实验结果表明,AdaptDNN既能实现低成本容错,又能保证模型训练效率,从而可以在弹性实例场景高效完成模型训练任务,降低模型训练成本.

深度学习  /  神经网络  /  分布式训练  /  弹性实例  /  容错

As the number of parameters in deep learning models continues to increase, the cost of training also keeps rising. To reduce training costs, using spot instances provided by cloud service providers has become a viable solution. Spot instances are priced at only 30% of normal instances, which can significantly lower training costs. However, despite the low cost of elastic instances, there is a risk of them being reclaimed at any time, posing new challenges to the stability of the model training system. To address the fault tolerance issue in the context of elastic instances, existing work mainly falls into two categories: checkpoint-based fault tolerance and redundancy-based fault tolerance. Checkpoint-based solutions incur substantial overhead, while redundancy-based solutions impose certain limitations on the model’s parallelism strategy, leading to suboptimal training efficiency. We propose a solution for training in the context of spot instances that leverages the grace period of spot instances to back up training progress, thereby reducing fault tolerance overhead. It also employs the bottleneck alleviation approach to adjust the parallelism strategy, maximizing the use of available cluster resources and enhancing training efficiency. Experimental results show that this solution not only achieves low-cost fault tolerance but also ensures training efficiency, allowing for efficient completion of model training tasks in the context of spot instances and reducing overall training costs.

deep learning  /  neural network  /  distributed training  /  spot instance  /  fault tolerance
刘国栋, 朱家祺, 高梓源, 包云岗, 王卅. AdaptDNN:一个自适应可伸缩的大模型分布式训练系统. 计算机研究与发展, 2025 , 62 (12) : 3060 -3076 . DOI: 10.7544/issn1000-1239.202440560
Guodong Liu, Jiaqi Zhu, Ziyuan Gao, Yungang Bao, Sa Wang. AdaptDNN: An Adaptive and Scalable LLM Distributed Training System[J]. Journal of Computer Research and Development, 2025 , 62 (12) : 3060 -3076 . DOI: 10.7544/issn1000-1239.202440560
近年来深度学习发展迅速,随着模型结构的不断更迭,以及模型参数量的不断增加,深度学习模型在越来越多的领域表现出卓越的性能,例如聊天机器人ChatGPT、视频生成工具Sora等. 随着模型规模的不断增长,训练模型所需算力也不断增加,根据OpenAI的统计数据[1],自2012年以来,深度学习模型训练所需的算力每18个月翻35倍,与单个计算设备的算力之间存在着显著的差距. 因此,大模型的训练往往需要在大规模的GPU集群上进行分布式训练,并使用各类并行方案对训练进行优化,比如数据并行[2]、张量并行[3]和流水线并行[4]等. 然而,GPU集群的购买及租用成本往往十分昂贵,例如,使用亚马逊提供的价格最低的GPU完整训练1次GPT-3175B[5]模型,都需要150万GPU小时,耗费约460万美元[6].
如何有效控制或降低大模型高昂的训练成本成为学术界和工业界都必须面对的一个关键问题. 学术界的一个思路是利用弹性实例训练,近年来亚马逊等云厂商提供了一种价格更为低廉的弹性实例(spot instance). 弹性实例提供了远低于正常实例的租用费用(仅30%),相应的代价是这些实例随时可能被回收,用户需要自行应对实例变化. 尽管不确定性高,弹性实例以其低廉的价格仍旧吸引了大量研究者的关注,如Bamboo[7]等工作已实现在弹性实例上进行低成本训练. 对于已经拥有大规模GPU集群的工业界,它们的思路是通过训推混部[8]来缓解大模型训练带来的资源压力,一些模型训练任务可能被调度到模型推理的集群,利用闲置GPU时间进行训练. 然而与弹性实例训练面临的问题相同,当推理集群负载增加,训练任务也将面临资源被随时收回的可能性.
当部分弹性实例被回收时,如果训练系统不具有自适应可伸缩性,无法根据不同的实例数量调整并行方案,训练将会一直处于停滞状态,直到新的实例再次加入训练集群,造成其他可用实例被长时间闲置. 同时,被回收实例上产生的训练进度将被丢失,进一步增加模型训练的时间. 因此,本文提出一个问题:如何在随时会被回收的弹性实例场景中,实现自适应可伸缩性的深度学习模型训练?具体而言,要解决这个问题主要面临三大挑战:
1)模型部署时. 在实例弹性变化的场景下,如何进行大模型的高效并行部署.
2)实例回收时. 如何尽可能减少训练进度的丢失,保障实例回收后的“故障”恢复.
3)实例回收后. 如何基于现有可用实例快速寻找出新的高效并行方案,调整部署继续执行训练任务.
对于第1个挑战,现有的模型分布式训练系统,在设计并行方案时往往假定集群的计算资源是固定的,不存在计算资源可变的情况. 因此,现有系统所采用的并行方案往往缺乏灵活性,无法适用于弹性实例场景. 例如,由NVIDIA设计的Megatron-LM[9]训练系统,在对GPT-3模型进行分布式训练时,会将GPT-3模型切分为多个相等大小的流水线阶段,并为每个流水线阶段分配相同数量的计算节点,从而实现负载均衡. 然而,如果我们直接将该方案应用于弹性实例场景,当某个流水线阶段的部分实例被回收后,我们可能无法充分利用剩余的所有可用实例. 例如,当我们有16个实例时,Megatron-LM可以构造长度为8的流水线,每个流水线阶段分配2个实例,当某个实例被回收时,由于Megatron-LM要求每个阶段的计算资源相同,此时每个阶段只能使用1个实例,从而造成7个可用实例被闲置.
在本文中,我们提出了一种异构流水线并行方案,在异构流水线中,每个流水线阶段都可以分配到不同数量的计算资源,相应地,每个流水线阶段的负载量和模型的并行方式都可以不同. 例如,当仅剩15个实例时,我们可以构造一条长度为5的流水线,其中每个阶段分别有4,4,4,2,1个实例,同时我们可以为每个阶段分配不同数量的神经网络层. 这样的异构流水线使得我们的系统能适应不同数量的计算资源,当部分弹性实例被回收,我们总能利用剩余可用实例数量找到可行的并行方案.
对于第2个挑战,当弹性实例被回收时,该实例的所有数据都将变为不可用的状态. 这本质上是一个容错问题,可以使用故障恢复的策略继续训练模型. 目前已有的分布式训练的故障恢复策略主要有2类:一类是基于Checkpoint (存盘点)[10-11]的故障恢复策略 [7,12],另一类是基于冗余性的故障恢复策略. 基于Checkpoint的策略会每隔一段时间保存一次模型的参数到磁盘,当故障发生时再加载最新的模型参数. 该策略会丢失自上一次保存Checkpoint以来所有的训练进度,因此减少丢失进度的方法就是提高保存Checkpoint的频率. 然而保存Checkpoint的时候,训练需要暂停以防止模型参数被修改,因此频率太高将影响训练的正常进行. 基于冗余性的故障恢复,主要是利用模型数据并行时不同实例参数的冗余性,当故障发生时,从其他实例恢复丢失的模型参数. 而模型的参数是每一个训练轮次后更新1次,因此基于冗余性的策略将丢失1个轮次的训练进度.
本文提出一种基于宽限期(grace period)的自适应故障恢复机制,在不引入冗余性的前提下,实现快速的故障恢复. 主要思想如下:云厂商在提供弹性实例时都会给出一定的宽限期,例如,在实例被回收前30 s时会提前通知. 我们可以在宽限期内完成被回收节点重要数据的传递,包括当前训练轮次产生的梯度值、上一轮次更新后的模型参数等. 通过传输梯度值,我们可以实现不丢失训练进度的故障恢复. 如果宽限期的时间不足以完成所有数据的传输,则我们将综合基于冗余性和Checkpoint的机制,在给定宽限期内选择最高效的恢复方案.
对于第3个挑战,当某个弹性实例被回收后,模型的并行训练方案需要重新调整部署,以适应新的实例数量. 目前已有的自动并行方案如Alpa[13]和Aceso[14]等都是离线搜索,只考虑并行方案的训练性能,不关注并行方案之间调整的开销,而对并行方案进行在线调整时,调整开销、故障频率等因素都会影响整体性能. 例如,当实例被频繁回收时,更好的调整策略是尽快找到可行的并行方案,并最小化模型的参数传递开销,而非寻找吞吐量最高的并行方案. 除此之外,现有工作由于必须基于一定的限制进行调整,使得并行方案的性能本身也并非最优. 例如,基于冗余性的故障恢复策略由于必须使用数据并行,当实例被回收时,剩余节点如果无法构造足够数量的数据并行组,则无法完成并行方案的调整.
本文提出了一种在线高效并行方案调整算法,该算法基于离线并行方案搜索算法Aceso[14]并对其进行优化,使其能够兼顾调整开销、实例回收频次等因素实现在线高效并行方案调整. 为达到这一目标,我们提出一个新的指标“下一次故障前的有效轮次数”,用于评估在弹性实例场景下一个并行方案的实际训练效率. 该指标综合考虑了一个并行方案的训练性能,以及调整为该方案所需的时间开销,并综合历史数据预估下一次实例回收发生的时间间隔,从而计算得到一个并行方案在下一次实例回收前可以完成的训练轮次数. 我们将“下一次故障前的有效轮次数”作为Aceso搜索算法的新目标函数,从而实现兼顾调整开销、实例回收频次的同时,在线高效搜索并行方案,提高了实例被回收后的训练效率.
我们在不同规模的GPT-3模型上进行了实验,实验结果表明,与当前最好的工作Oobleck[12]相比,AdaptDNN能显著提升弹性实例场景下的训练效率. 当可用实例数量在8~16之间弹性伸缩时,AdaptDNN将训练的总体吞吐量提升1.4倍,将并行方案的调整开销降低至25%,同时,使用同样数量的GPU,AdaptDNN可训练的模型大小是现有工作的2.5倍.
近年来,由于模型规模、训练数据和集群规模的逐渐增加,大模型的分布式训练成为了一个备受关注的课题. 各类分布式训练策略的提出,使得大模型分布式训练的性能得到了显著提升. 随着集群规模的进一步增大,训练的性能不再是唯一需要关注的指标,训练任务的可伸缩性和容错能力在集群资源可能发生变化的情况下尤为重要. 如何在使用各类高效并行方案的情况下,使系统具有弹性伸缩的能力,从故障中快速恢复,成为了一个具有挑战性的研究问题. 本节将介绍现有的一些工作在该问题上的探索,主要包括各类常见的分布式训练策略(1.1节)、无故障时的弹性伸缩(1.2节)以及大模型训练的故障恢复(1.3节).
深度学习模型的训练过程,一般采用“反向传播算法”. 该算法的过程如下:首先将训练数据输入神经网络,让网络的权重(即模型的参数值)与传入的数据进行一系列的运算,最终将输出的结果与数据的标签传入损失函数计算损失值,然后根据损失值计算梯度值,并通过链式法则,将梯度值反向传播到所有神经网络层,并更新权重值. 一般训练用的数据集会被划分为多个批次,模型会在每个批次上重复该过程,一个批次的计算往往也被称为一个训练轮次(iteration). 经过多个轮次的训练,最终即可得到一个在所有数据上表现良好的模型.
近年来深度学习模型的参数量、训练数据量都在快速增长,单个节点的内存空间有限,已经难以支持完整模型的训练,因此出现了多种分布式训练策略.
1)数据并行. 数据并行[15]是指将模型的参数复制多份,每台机器保存一份完整的模型参数,但将训练数据分成多份,每台机器只保存一部分训练数据. 所有机器分别进行运算,各自在一部分训练数据上得到梯度值,在每个批次的数据算完后多机之间同步一下梯度值. 数据并行实现简单,但最大的局限性是不能够减少内存资源的占用,每台机器拥有完整的模型参数,因此对于单机无法容纳的模型,数据并行是无法应用的. 因此出现了对模型参数进行划分的张量并行和流水线并行.
2)张量并行. 张量并行[16-19]是将某些算子的参数切分为几个部分,每台机器各保存一部分,这样每台机器算出来的结果将是不完整的,需要在每次完成计算后进行一次结果的同步. 张量并行虽然能极大程度减少每台机器内存的占用,但一般会引入比数据并行更大的通信开销,因为数据并行只需要在每一批数据计算完毕之后同步梯度值,而张量并行需要在每一个训练数据的运算过程中同步中间激活值. 因此,张量并行对通信带宽要求较高,一般只应用于带有高速通信链路的节点内部.
3)流水线并行. 为了解决张量并行引入的通信开销问题,学术界又提出了流水线并行[20-22]. 流水线也是对模型参数的分割,但并不会将每一层神经元的参数分成几份,而是在模型的层与层之间进行切分,类似CPU中的指令流水线,将整个模型的计算分为多个流水线阶段,每台机器各负责其中一个阶段的计算,只需要在机器之间传递中间激活值即可,因此流水线并行的通信开销比模型并行更小.
4)混合并行. 以上3种并行策略各有利弊,近年来很多工作尝试全部或部分地结合了这3种方案,在不同的场景下找到了更为高效的分布式训练方式. 例如FlexFlow[3]结合了数据并行和模型并行,DAPPLE[23]结合了数据并行与流水线并行,Megatron-LM[9],Alpa[13],Aceso[14]结合了数据并行、模型并行、流水线并行.
对模型分布式训练的可伸缩性研究,主要分为2类:一类工作关注单个训练任务的可伸缩性;另一类工作对深度学习集群的调度器进行研究,通过动态调节不同任务的资源,提高集群的整体利用率.
1)单个任务的可伸缩性. 现有的很多大模型训练框架已经原生支持了模型训练的弹性伸缩,例如Torch Elastic[24],Eastic Hovorod[25]等,但此类框架只支持对数据并行的弹性伸缩,即在模型使用数据并行训练时,通过修改数据并行组的大小以及每组的批处理大小,从而实现模型训练在不同资源条件下的扩容和缩容. 此外,还有一些研究工作[26]对任务伸缩的开销进行了分析,提出了更加高效的伸缩机制,但也缺乏对张量并行和流水线并行的支持.
2)集群任务的可伸缩性. 许多集群调度器的工作通过对深度学习任务特性进行分析,总结出不同的调度器优化目标,例如基于反馈调节机制的Gandiva[27]、综合考虑训练性能和模型收敛效率的Pollux[28]以及应用Serverless思想优化集群调度的ElasticFlow[29]等,此类工作更关注集群整体利用率而非单个任务的训练效率,且对单个训练任务的调节方式也是基于数据并行的可伸缩性.
此类弹性伸缩工作往往主动调整训练任务的资源和并行方案,不具有应对未知故障的容错能力,因此无法被应用到弹性实例场景.
分布式系统的故障恢复,往往需要依赖于某种形式的冗余性. 一类被广泛采用的方式是通过周期性地保存Checkpoint来维护冗余备份. 近年来有更多针对大模型分布式训练的特性的工作,提出了其他类型的冗余性,主要包括利用模型参数的冗余和引入冗余计算来实现快速的故障恢复.
1)基于Checkpoint的故障恢复. 利用Checkpoint保存和加载模型状态,是模型故障恢复的常用方法. 该方案会在模型训练时,每隔一定的训练轮次将模型的状态写入磁盘. 当故障发生时,系统会从磁盘中读取最新的模型状态. Checkpoint的写入和加载都需要进行磁盘读写,耗费的时间较长,因此保存Checkpoint过于频繁会影响模型训练的性能. 为解决该问题,一些工作提出了基于内存的Checkpoint优化方案[30-32],先将GPU中的模型状态同步写入到主机的内存中,再将主机内存的数据异步写入到磁盘中. 异步写入磁盘时不会阻塞训练任务,从而减小写入Checkpoint对性能的影响. 一些其他工作[33-35]则进一步优化了在不同并行方案之间保存和加载Checkpoint的性能. 然而,基于内存的Checkpoint方案仍有不足. 首先,将模型状态从GPU写入到主机内存的过程仍会阻塞训练任务,因此过于频繁地保存Checkpoint仍然会影响模型训练速度. 其次,当故障发生时,该方案可能会丢失训练的进度. 因为在加载模型状态时,需要使用最新的完整的Checkpoint. 如果在主机内存异步写入磁盘时发生故障,那么该Checkpoint不可用. 系统只能使用上一次保存的模型状态,这会丢失一些轮次的训练状态,造成较高的故障恢复时间.
2)基于参数冗余性的故障恢复. 为了提高故障恢复效率,避免保存和加载Checkpoint的开销,一些工作提出使用数据并行内在的冗余性进行恢复. 在数据并行中,包含一个模型完整状态的机器构成一个数据并行组,不同数据并行组之间的模型状态相同,因此当发生故障且故障节点数小于数据并行组个数时,总能从剩余数据并行组中恢复丢失的状态. Oobleck[12]利用数据并行的冗余性实现了模型训练时的故障恢复,并结合流水线并行将模型参数分割到多台机器中,实现了大模型的弹性训练. 对于每个数据并行组,Oobleck会利用分治法计算出合适的流水线并行方案来分割模型. 当出现节点故障时,Oobleck会根据剩余的节点中包含的模型状态,重新分配流水线并行方案,并根据新方案在不同节点间传递模型的状态,从而实现模型的重构. Parcae[6]也利用了数据并行的冗余性进行故障恢复,但与Oobleck不同的是Parcae会预测未来节点损坏概率并提前做出并行方案的调整. SlipStream[36]则是在故障发生后不调整并行方案,而是将发生故障的数据并行组的一部分计算任务重新路由到其余并行组,实现了故障状态下的负载均衡. 但是,利用数据并行实现冗余的方案存在一定的局限性. 首先,该方案会限制系统能训练的模型尺寸. 在总GPU内存资源有限的情况下,模型尺寸越大,能支持的数据并行组的个数越少,因此同等规模的GPU集群此类方案能够支持的模型大小会严重受限. 其次,该方案在故障恢复后会丢失一部分训练进度. 由于数据并行组仅在每轮训练结束时才进行模型状态的同步,如果节点故障发生在一轮训练的中间时刻,那么该轮训练的进度将在故障恢复后丢失. 系统只能回退到上一轮训练后的模型状态重新执行.
3)基于冗余计算的故障恢复. 除了利用参数的冗余性,也有一些其他工作通过引入计算的冗余性实现故障恢复. Bamboo[7]通过在训练流水线中填充冗余计算提高故障恢复效率. 核心思想是让每个节点在完成自身的计算任务之外,额外多做一份其他节点的计算任务,将计算结果作为冗余备份. 在故障发生时直接使用已备份数据恢复故障节点的该步计算,使得训练能够继续进行. 但Bamboo存在多方面不足,过多的冗余计算降低了训练的吞吐量,消耗的GPU显存过大使得对更大的模型训练支持性差. 根据Oobleck的实验数据,Bamboo所能支持的模型大小仅为Oobleck的1/20.
虽然已有工作可以在传统的容错场景实现高效的故障恢复,但使用弹性实例进行训练时,这些工作的效率并不高. 主要原因是从冗余备份进行恢复的做法会导致每次恢复都丢失一段时间的训练进度,而弹性实例的故障更加频繁,因此会导致大量的计算被浪费. 根据Bamboo的测试数据[7],一个64卡的弹性实例集群,在24 h内发生了127次抢占. 如果使用Checkpoint进行故障恢复,约有77%的时间被浪费在加载Checkpoint和重算丢失的训练进度. 虽然Oobleck通过利用数据并行的冗余参数进行恢复,但每次恢复后仍然会丢失一个训练轮次的进度. Bamboo通过引入冗余计算降低训练进度的丢失,但冗余计算却会导致大量的算力被浪费.
另一方面,弹性实例的频繁故障往往需要训练系统有快速调整训练方案的能力. 现有工作的并行方案大多缺乏灵活性,基于冗余计算的Bamboo选择将模型切分为均等的流水线,当集群中GPU数量难以被均等切分时,该类方案往往无法利用所有GPU进行训练,造成了大量计算资源的闲置. 基于Checkpoint的方案大多也采用了此类均等流水线的设计[11],另一部分工作[30]则不对并行方案进行调整,而是选择维护额外的资源池,用于替换故障节点,此类方案也造成了大量的资源闲置,进一步增加了训练成本. 基于参数冗余性的Oobleck通过构造多条长度不同的流水线实现了对GPU的充分利用,但该方案对数据并行有强依赖,导致训练性能可能不是最优.
与现有工作不同,本文将根据弹性实例所具有的“宽限期”特性,有针对性地提出一套故障恢复方案,通过利用“宽限期”进行故障恢复,可以实现更细粒度的状态保存,降低恢复开销. 同时,本文也提出了一套灵活可变的并行方案以及在线的并行方案调整算法,可充分利用所有硬件资源,从而在降低恢复开销的同时,保证了训练的性能.
AdaptDNN是一个自适应可伸缩的大模型分布式训练系统. AdaptDNN使用了一种灵活的异构流水线并行方案(第3节),可以适应不同的实例数量,在集群可用资源发生变化时能够充分利用所有计算资源. 同时,AdaptDNN采用基于宽限期的自适应故障恢复机制(第4节),从而避免了频繁的Checkpoint开销以及在系统中引入冗余性的开销. 最后,AdaptDNN采用了一个在线高效并行方案调整算法(第5节),从而可以快速找到资源变化时新的并行方案,实现对训练任务的快速恢复.
如图1所示,AdaptDNN在对模型进行分布式训练时(①),会在CPU端同时开始搜索集群中每个节点被回收后新的并行方案(②). 当底层的运行时系统被告知某一节点被回收时(③),AdaptDNN会按照预先找到的并行方案调整模型的部署,并在节点之间传输缺失数据,从而实现在宽限期内快速完成故障恢复(④),然后底层运行时系统按照新的并行方案继续训练(⑤).
现有的模型分布式训练相关工作都是利用混合并行来充分利用硬件资源. 但现有工作往往假定集群的计算资源是稳定的,因此采用的并行方案往往缺乏适用于计算资源可变场景的灵活性. 如图2所示,Megatron-LM提出,在Transformer[37]层内部使用张量并行缓解单个设备的内存压力;在Transformer层之间使用流水线并行缓解模型划分带来的通信压力;最后使用数据并行将模型扩展到更多设备上进行训练. 整个模型训练的配置,可以用:tp(张量并行大小)、dp(数据并行大小)和pp(流水线并行大小)3个参数来指定,只要满足tp,dp,pp的乘积等于GPU数量即可. 因此这样的并行方案也被称为“3D并行”. 然而将3D并行应用到弹性实例集群中,可能造成集群计算资源的闲置. 例如,当1个或多个实例被回收,剩余实例的数量可能无法被分解为任何tp,dp,pp的乘积(假设tp,dp,pp都大于1).
为了能让模型的并行方案适应于任意数量的GPU,我们提出了异构流水线并行方案. 异构流水线是指,流水线的不同阶段可能被分配不同数量的GPU,因此,每个阶段也可能用不同的方式来组合使用数据并行和张量并行. 如图3所示,初始流水线中包含16个GPU,如果我们用每阶段的GPU数量来表示该流水线,则可以将其表示为4-4-4-4. 在某时刻,流水线第2阶段的2个实例被回收,则我们可以将流水线调整为4-2-4-4. 如果此时流水线第3阶段的1个GPU进一步被回收,第3阶段仅剩3个GPU,考虑到大部分底层运行时系统仅支持偶数数量的模型切分,我们可以进一步将第3阶段拆分成2个阶段,则流水线可以表示为4-2-2-1-4.
流水线计算资源的异构性也意味着负载的异构性. 持有更多计算资源的阶段应该承担更多的计算任务. 然而流水线负载的划分不能简单地跟计算资源数量成正比,因为不同阶段采用的不同并行方案可能导致不同的计算效率(例如,采用张量并行时会引入通信开销,这就导致使用张量并行阶段的执行时间变长). 异构流水线使得并行方案的搜索空间进一步增大,在完整的搜索空间里寻找最优解,将引入极高的搜索开销. 受到自动并行系统Aceso[14]的启发,我们选择将异构流水线方案的搜索问题映射为一个资源调优问题,即从一个初始方案出发,通过在流水线不同阶段之间重新分配计算资源和计算任务,逐步优化其中的瓶颈阶段,直至性能无法被进一步优化为止. 关于搜索算法的更多细节将在第5节进行介绍.
云服务厂商在提供弹性实例时,往往会给出一定的宽限期,即在实例回收之前的一定时间会被提前通知. 例如,亚马逊公司给出的宽限期约为2 min,谷歌和微软云公司给出的宽限期约为30 s[10].
本文提出利用弹性实例宽限期来传输模型重构所需要的数据. 在传统容错场景中,由于不存在这样的宽限期,导致必须使用一定的冗余性来完成故障恢复,而冗余备份本身则带来了较大的开销. 在弹性实例的场景中,由于宽限期的存在,我们可以在实例回收真正发生之前有针对性地完成实例数据的传输和保存,大大减少了容错的成本,提高了系统的性能.
如图4所示,现有工作在进行故障恢复时,往往是对模型的参数进行重新部署,例如Checkpoint方案会加载上一次保存的模型参数,基于冗余性的方案Oobleck也是在设备之间传输缺失的模型参数. 但模型参数是每一个训练轮次完成后才会更新一次,这使得现有方案在完成故障恢复后,都会丢失至少1个训练轮次的进度,需要重新计算自上一次参数更新以来产生的梯度值. 因此,现有工作的故障恢复时间包含了参数读取/传输开销以及重做丢失计算的开销. Checkpoint机制和Oobleck的故障恢复时间可以分别表示为
$ {T}_{\mathrm{c}\mathrm{k}\mathrm{p}\mathrm{t}}={T}_{\mathrm{l}\mathrm{o}\mathrm{a}\mathrm{d}}+{T}_{\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{o}} \text{,} $
$ {T}_{\mathrm{o}\mathrm{o}\mathrm{b}\mathrm{l}\mathrm{e}\mathrm{c}\mathrm{k}}={T}_{\mathrm{w}}+{T}_{\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{o}} \text{,} $
其中$ {T}_{\mathrm{l}\mathrm{o}\mathrm{a}\mathrm{d}} $是加载Checkpoint的时间,$ {T}_{\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{o}} $是重做丢失的计算的时间,$ {T}_{\mathrm{w}} $是在节点间传输模型参数的时间.
由于从磁盘中读取Checkpoint的带宽较低,因此$ {T}_{\mathrm{l}\mathrm{o}\mathrm{a}\mathrm{d}} $往往比$ {T}_{\mathrm{w}} $更大. 而且,由于Checkpoint的频率较低,但Oobleck传输的模型参数每个训练轮次都会更新,因此Checkpoint方式的$ {T}_{\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{o}} $比Oobleck更大. 当模型规模较大时,一个轮次的时间也会较长,丢失的计算进度也会增多,这导致$ {T}_{\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{o}} $进一步增大.
由于弹性实例场景下宽限期的存在,我们可以在故障真实发生之前提前进行处理,因此我们提出一种更加细粒度的模型重构方案:在实例被回收之前的宽限期内,按照预先找到的新的并行方案,将被回收实例在本轮次产生的梯度值和它持有的模型参数传输给其他节点. 因此,当故障恢复完成后,其他节点持有本轮次的训练进度,模型可以从停止的地方接着训练,不会损失训练进度,也就不需要重做任何计算. 此时,重构开销可以表示为
$ {T}_{\mathrm{A}\mathrm{d}\mathrm{a}\mathrm{p}\mathrm{t}\mathrm{D}\mathrm{N}\mathrm{N}}={T}_{\mathrm{g}}+{T}_{\mathrm{w}} \text{,} $
其中$ {T}_{\mathrm{g}} $是在节点间传输梯度值的时间.
尽管我们可以利用弹性实例的宽限期进行故障恢复,但当网络带宽较低或需要传输的参数量和梯度值较多时,宽限期的时间可能不足以完成所有数据的传输. 因此,我们对需要传输的数据进行了分析,并结合基于冗余性的故障恢复和Checkpoint机制,提出能适应于不同宽限期和网络带宽的故障恢复机制. 该机制的核心思想如下3方面.
1)在宽限期内传输“无冗余性”的参数即可. 我们发现并非所有参数都具有同样的传输优先级. 在AdaptDNN采用的异构流水线并行方案中,模型每一层的冗余性可能是不同的. 当采用数据并行时,该层的模型参数天然具有冗余性;当采用流水线并行与张量并行,则不存在冗余性. 具有冗余性的参数,在宽限期结束后也能从其他设备恢复,因此当宽限期不足以完成所有的参数传输,我们只要保证在宽限期内完成 “无冗余性”的参数传输即可. 由于每轮次产生的梯度值也不具有冗余性,因此也需要在宽限期内传输.
2)宽限期较短时,回退到基于冗余性的故障恢复. 如果我们无法在宽限期内完成所有梯度值和“无冗余性”的参数传输,则无法采用细粒度的模型重构机制. 此时,我们采用基于冗余性的恢复机制:在宽限期内仅完成“无冗余性”的参数传输,在宽限期后从其他节点恢复冗余参数即可. 由于无法在宽限期内传输本轮次产生的梯度值,此时的故障恢复与基于冗余性的恢复机制相同,有额外的重计算开销.
3)宽限期较短且模型不存在冗余性,回退到Checkpoint机制. 如果我们无法在宽限期完成“无冗余性”的参数传输,则基于冗余性的恢复机制也无法使用(因为整个系统中模型的状态不再完整),此时我们回退到使用Checkpoint进行恢复.
因此,AdaptDNN的自适应重构机制的故障恢复开销可以表示为
$ {T}_{\mathrm{A}\mathrm{d}\mathrm{a}\mathrm{p}\mathrm{t}\mathrm{D}\mathrm{N}\mathrm{N}}=\left\{\begin{aligned}&{T}_{\mathrm{g}}+{T}_{\mathrm{w}},\;\;{\rm{if}}\;\;{T}_{\mathrm{g}}+{T}_{\mathrm{w}\mathrm{n}}\le {T}_{\mathrm{g}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{e}}\text{,}\\ &{T}_{\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{o}}+{T}_{\mathrm{w}},\;\;{\rm{if}}\;\;{T}_{\mathrm{w}\mathrm{n}}\le {T}_{\mathrm{g}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{e}} <{T}_{\mathrm{g}}+{T}_{\mathrm{w}\mathrm{n}}\\ &{T}_{\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{o}}+{T}_{\mathrm{l}\mathrm{o}\mathrm{a}\mathrm{d}},\;\;{\rm{if}}\;\;{T}_{\mathrm{g}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{e}} < {T}_{\mathrm{w}\mathrm{n}}\text{,}\end{aligned}\right. \text{,} $
其中$ {T}_{\mathrm{w}\mathrm{n}} $是指传输“无冗余性”参数所需时间,$ {T}_{\mathrm{g}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{e}} $是宽限期大小.
在部分实例被回收后,为了能继续模型的训练,我们需要在剩余可用资源条件下调整并行方案. 由于异构流水线引入的复杂性,使得并行方案的搜索空间进一步增大,基于动态规划的一些自动并行工作难以被应用到我们的场景. 受到自动并行工作Aceso的启发,我们发现弹性实例场景下的并行方案搜索问题可以映射为一个资源调优问题,并使用“瓶颈消除”思想来求解.
但Aceso也无法被直接应用到弹性实例的场景,因为Aceso是离线搜索,其搜索目标为性能最优的并行方案,而忽略了并行方案调整的时间开销. 而这一开销对于弹性实例场景至关重要(6.4节),因为在网络带宽较低、宽限期较小或者故障频繁的场景下,性能最优的方案可能并不是最优解. 因此,我们提出了一种高效的并行方案在线调整算法,通过使用新的搜索目标“下一次故障前有效的轮次数”,从而实现了在性能与开销之间的权衡,确保了在弹性实例场景下的训练效率.
虽然我们可以通过“瓶颈消除”的思想提高并行方案的搜索速度,但如果在实例被回收时才搜索新的并行方案,仍会占用宽限期的时间,可能导致我们无法在宽限期内完成故障恢复所需的数据传输. 我们发现,搜索过程是可以与模型的训练过程同步进行的. 因此,当训练开始时,我们在CPU端同步开始搜索集群中每个节点被回收后的最优并行方案. 当实例回收发生时,我们可以直接按照预先找到的方案调整模型的部署,从而使宽限期的时间可以完全用于模型的故障恢复.
“瓶颈消除”是计算机领域的经典思想,在流处理系统中就被用于优化系统性能[38-40]. Aceso提出使用“瓶颈消除”思想迭代优化给定并行方案的性能,直至无法找到更好的方案为止. 其核心思想如下:当给定一个初始的并行方案,通过性能模型评估其中哪一部分是性能瓶颈,并采用与之对应的策略优化该性能瓶颈. 例如,当一个流水线阶段的内存压力过大,可以通过增加该阶段的张量并行大小来降低内存占用,或者将较为空闲的流水线阶段的计算资源重新分配给该阶段使用.
图5展示了使用“瓶颈消除”思想搜索并行方案的一个例子. 一个拥有4个算子的模型,其初始并行方案为2个流水线阶段,但这样的切分会导致流水线阶段1的内存压力过大,使得阶段1成为了整个系统的性能瓶颈. 因此,可以将阶段2的2个GPU重新分配给阶段1,从而缓解阶段1的内存压力. 但此时阶段2成为了新的性能瓶颈,因为其算力减少,导致了执行时间变长,我们可以进一步选择将阶段2的一个算子重新分配给阶段1执行,由此以来,我们就实现了2个阶段的负载均衡,找到了一个比初始方案更好的并行方案.
我们发现,“瓶颈消除”的思想也能被应用于弹性实例场景,用于搜索实例被回收后新的并行方案. 与Aceso的离线搜索不同,弹性实例场景中的性能瓶颈是随着实例回收而动态产生的. 如图6所示,当某些弹性实例被回收时,我们可以假设其所在流水线阶段的其他实例承担该阶段的所有计算任务,因此该阶段将成为一个性能瓶颈(计算量不变,但算力减少了). 我们可以将此并行方案作为初始并行方案来优化该瓶颈阶段的性能,并重复这样的“发现瓶颈-消除瓶颈”过程,直至找到一个比初始方案性能更好的方案.
当某些弹性实例被回收时,也可能出现另一种性能瓶颈. 如图7所示,某一流水线阶段的所有实例可能都被回收,导致该阶段无计算资源. 此时,可以假定该阶段的计算任务被其相邻阶段的实例承担,则该相邻阶段将成为一个性能瓶颈. 类似地,我们可以应用“瓶颈消除”思想优化该瓶颈.
弹性实例场景的瓶颈往往是由于某个流水线阶段丢失了部分实例而造成的负载与算力的不均衡. 只要能实现不同流水线阶段之间的算力/负载均衡,避免某一个阶段与其他阶段的性能差异过大,就能提高流水线的整体性能. 因此,最简单的调整方式是重新选择流水线的切分位置,使得各个阶段的负载量与其算力成正比,在不调整并行方式的情况下重新实现负载均衡. 然而,直接按照不同阶段的实例数量比例切分流水线,仍然可能不是最优方案,因为不同阶段所采用的并行方式不同,相同负载量的计算时间可能有所不同(例如,张量并行会引入大量的all-reduce通信,同样的并行度下,张量并行的时间往往大于数据并行的时间[9]). 为了能更有效地消除瓶颈,我们采用了以下基于贪心的负载均衡策略.
策略1. 负载均衡. 尝试将瓶颈阶段的一部分负载分配给当前时间最短的流水线阶段,用性能模型评估重新分配后总性能是否更优,重复此过程,直至无法进一步提升性能为止.
该策略的伪代码实现如下:
算法1. 负载均衡策略.
输入:并行方案Config,瓶颈阶段BottleStage,硬件内存上限MemLimit;
输出:新的并行方案.
① CurTime ← Config. time;
② while true
③  IdleStage ← GetIdleStage();/*利用性能模型 得到当前空闲(时间最短)阶段*/
④  BottleStage.layers − −;/*减少瓶颈阶段负载 (layer)数量*/
⑤  IdleStage.layers ++;/*增加空闲阶段负载 (layer)数量*/
⑥  Time, Mem ← PerfModel();/*利用性能模型 评估Config的总时间及内存*/
⑦ if Time < CurTime and Mem < MemLimit
⑧   CurTime ← Time;
⑨ else
⑩  无法进一步提升性能,撤销对Config的上   一步修改,并返回Config;
⑪   BottleStage.layers ++;
⑫   IdleStage.layers − −;
⑬  return Config;
⑭ end if
⑮ end while
然而,使用策略1在某些场景下可能无法找到性能更优的并行方案. 例如,假如瓶颈阶段持有某个内存占用量较大的神经网络层(例如嵌入层). 如果不为该瓶颈阶段分配更多算力用于张量并行,则该瓶颈阶段将无法被分配更多的负载,造成算力的浪费.
AdaptDNN采用了与Aceso类似的“资源置换” (resource trading)思想来消除瓶颈:造成性能瓶颈的原因往往是某一类资源不足,主要包括计算、通信和内存资源. 常见的一些并行方案的调整策略则是在不同的资源之间进行“资源置换”. 例如,增加张量并行度,就是通过引入额外的通信来减少内存资源的使用;“重计算”策略则是通过“以算代存”的形式,在反向传播过程中重做前向计算,从而避免保存前向计算产生的中间结果,实现了以额外的计算来减少内存占用的效果. 我们发现,各类“资源置换”操作往往会引起多个流水线阶段的并行方式/负载量发生变化,因此我们将“资源置换”操作分为2个类别,并分别提出以下2个瓶颈消除策略.
策略2. 改变并行方式. 当内存资源成为瓶颈,减少数据并行,增加张量并行(以通信换内存);当计算资源成为瓶颈,减少张量并行,增加数据并行(以内存换计算);当通信资源成为瓶颈,增加数据并行,减少张量并行(以内存换通信). 如果瓶颈阶段的实例数量有限,并行度无法进一步增加,则将当前最空闲(时间最短)的流水线阶段的部分实例重新分配给瓶颈阶段.
策略3. 改变负载总量. 当内存资源成为瓶颈,减少训练的微批次大小(micro-batch size)或在瓶颈阶段启动“重计算”机制(以计算换内存,减少微批次大小和“重计算”都会增加计算时间,但降低内存用量);当计算资源成为瓶颈,增加训练的微批次大小或在瓶颈阶段关闭“重计算”机制(以内存换计算).
改变并行方式和负载总量后,流水线的不同阶段之间仍然可能存在算力/负载不均衡的现象,因此我们选择将策略1~3结合起来使用:每次应用策略2或策略3后,我们都会应用1次负载均衡策略,如图8所示,寻找在新的算力分布、负载总量下最优的流水线切分策略.
在搜索过程中,我们需要评估每一个并行方案在给定硬件环境上的运行时间与内存占用,从而评估哪一类资源造成了性能瓶颈. 我们使用Aceso提供的基于测量的性能模型提前测量每个算子在不同的切分方案下的内存占用和执行时间,在搜索瓶颈消除策略时直接复用测量结果,即可得到整个模型的资源用量情况.
不同的瓶颈消除策略将会带来多种不同的并行方案. 我们往往难以提前知道哪一种策略对当前的瓶颈阶段效果最好,因此我们会尝试每一种策略并从这些策略产生的所有并行方案中选择最好的方案. 已有的自动并行工作往往会选择性能最优的方案,因为这些工作都是离线搜索,以并行方案的性能为搜索目标,不考虑从旧的并行方案到新的并行方案的调整时间,但这一开销在故障恢复的场景尤其重要. 例如,当集群故障频率较高或者弹性实例回收频率较高时,如果故障恢复开销过高,则下一次故障发生前的实际可用于训练的时间将会减少,可能导致训练进度变慢,此时使用一个性能不是最优但恢复开销低的方案可能反而会带来更多的收益.
为了实现弹性实例场景下的并行方案在线调整,AdaptDNN在选择方案时会综合考虑并行方案的性能与故障恢复开销. 我们提出了一个新的指标iters(下一次故障前有效的轮次数)作为选择并行方案的依据,其计算方式为
$ iters=\frac{MTTF-{T}_{\mathrm{A}\mathrm{d}\mathrm{a}\mathrm{p}\mathrm{t}\mathrm{D}\mathrm{N}\mathrm{N}}}{{T}_{\mathrm{i}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}} \text{,} $
其中$ MTTF $(mean time to failure)是平均故障时间,基于历史数据计算得到. $ {T}_{\mathrm{A}\mathrm{d}\mathrm{a}\mathrm{p}\mathrm{t}\mathrm{D}\mathrm{N}\mathrm{N}} $是故障恢复开销,由式(4)计算得到;$ {T}_{\mathrm{i}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}} $是新方案的每轮次训练时间,由性能模型得到.
由此,AdaptDNN在调整并行策略时总会朝着下一次故障发生前有效的轮次数最大的方向搜索并行方案,从而实现在任何故障频率下,系统都能按照对训练进度最有利的方向进行快速调整.
优化某个瓶颈阶段有可能造成其他阶段的性能下降,甚至劣化并行方案的整体性能,因此AdaptDNN采用了一个迭代式的算法,即当一次瓶颈消除并未找到优于初始方案的并行方案时,会重复执行“发现瓶颈-消除瓶颈”的策略,直至达到给定的最大搜索深度.
AdaptDNN使用“瓶颈消除”思想调整并行方案的流程如图8所示,该算法的伪代码实现如下:
算法2. 瓶颈消除算法BottleResolve.
输入:初始并行方案InitConfig,初始有效轮次数InitIters, 最大搜索深度MaxHops;
输出:如果找到有效轮次数更多的方案,则输出新的方案;否则,输出None.
① if MaxHops > 0 then
②  BottleStage ← IdentifyBottlenck(InitConfig); /*利用性能模型得到当前的瓶颈阶段*/
③  C1 ←ReBalance(InitConfig);/*利用策略1(负 载均衡)得到新方案C1*/
④  C2←ReParallelize(InitConfig, BottleStage), C2 ← ReBalance(C2);/*利用策略2(改变 并行方式+负载均衡)得到新方案C2*/
⑤  C3 ← ReWorkloads(InitConfig, BottleStage), C3 ← ReBalance (C3);/*利用策略3(改变 负载总量+负载均衡)得到新方案C3*/
⑥  new_configs ← [C1, C2, C3];
⑦  new_configs. sort();/*将新方案按有效轮次数 从高到低排序*/
⑧ if new_configs[0]. iters > InitIters
⑨  return new_configs[0];
⑩ end if
⑪ for new_config in new_configs do
⑫   result = BottleResolve(new_config, InitIters, MaxHops−1);/*重复该过程,直至达到 最大搜索深度*/
⑬  if result is not None then
⑭   return result;
⑮  end if
⑯ end for
⑰ end if
⑱ return None.
使用算法2消除性能瓶颈后,新的并行方案中可能会存在新的性能瓶颈,我们可以基于新方案再次调用算法2,从而持续性改进并行方案的性能. 在弹性实例场景下,AdaptDNN会在CPU端持续性重复调用算法2,直至无法找到更好的方案或者发生实例回收. 实例回收时,AdaptDNN会使用目前已找到的最优方案进行故障恢复.
1)实验环境. 我们的实验集群包含16张V100(16 GB)GPU,由4台GPU服务器组成,其中每台服务器配备4张GPU. 节点内部GPU通过PCIE互联,节点间通过100 Gbps IB网卡[41]互联. 实验平台具体配置如表1所示.
2)实验模型. 我们选用了目前被广泛应用的GPT-3模型,并测试了GPT-3不同的规模,包括350×106,1.3×109,2.7×109,6.7×109.
3)参数设置. 本文主要关注模型训练的性能(包括每轮次训练时间、训练吞吐量等指标),不考虑将模型训练到某一准确率的总时间. 影响模型训练性能的实验参数是模型采用的精度、数据批次大小(batch size)和序列长度(sequence length),本文后续实验所选用的参数如表2所示. 对于其他不影响训练性能的参数(例如学习率等),我们按照Megatron-LM的默认配置进行设置.
4)基准系统. 我们分别与基于Checkpoint的容错机制和基于冗余性的容错机制进行了对比. 其中,我们采用了Megatron-LM实现的Checkpoint功能来测量模型保存和读取Checkpoint的时间. 因为Checkpoint机制与模型并行方案无关,为了消除AdaptDNN和Megatron-LM在模型训练上的性能差异,我们使用AdaptDNN的每轮次训练时间作为Checkpoint机制的模型训练时间. 对于基于冗余性的容错机制,我们选用了目前最先进的Oobleck系统进行对比.
5)对比方法. 我们将通过模拟实例回收,对比不同系统在同样的故障模式下的训练吞吐量以及故障恢复开销(6.2~6.3节). 进一步地,我们将使用实测的数据模拟一个弹性实例集群在一段时间内的平均训练吞吐量,用于对比各个系统在真实场景下的性能与容错能力(6.4节). 最后,我们将通过消融实验对比本文提出的并行方案在线调整算法与传统离线搜索算法在弹性实例场景下的性能(6.5节).
我们测试了不同的系统在弹性实例场景下的训练吞吐量. 我们通过主动进行GPU回收,来模拟真实场景下实例数量的弹性变化.
1)实验设计. 我们从16个GPU开始训练模型,每隔2个训练轮次后,在下一个轮次进行了一半时,结束其中一个GPU上的训练进程. 重复此过程直至仅剩8个GPU为止. 我们测量了不同系统在不同GPU数量下的每轮次训练时间,并测量从发现节点回收开始,到继续训练之间的故障恢复时间. 对于Checkpoint机制,我们每隔一个轮次保存一次模型状态.
2)对比指标. 我们将对比不同系统在实验过程中的总体训练吞吐量:
其中,总共消耗的时间包含了正常训练时间与故障恢复时间.
3)实验结果. 图9显示了实验过程中每次发生节点回收后系统可用的GPU数量与该时刻的总体训练吞吐量. 从中我们可以发现,AdaptDNN在所有场景下的总体吞吐量都优于基准系统. 与Checkpoint相比,AdaptDNN最高实现了1.87倍性能提升(GPT-3 6.7B),与Oobleck相比,AdaptDNN最高实现了1.44倍性能提升(GPT-3 1.3B). 并且,AdaptDNN实现了使用相同数量的GPU训练比Oobleck更大的模型:Oobleck在GPT-3 2.7B模型的GPU数量下降到10卡以下便无法部署,GPT-3 6.7B模型更是完全无法部署. 而AdaptDNN可以支持GPT-3 6.7B模型,并且最少支持14卡进行部署.
我们进一步对6.2节中的实验进行性能分解,从而更加详细地对比不同系统的性能差异.
1)对比指标. 我们主要对比可训练的模型大小下不同系统在不同数量的GPU下的每轮次训练时间以及实例被回收时的故障恢复时间.
2)实验结果. 实验结果如图10和图11所示.
从图中我们可以得出以下结论:
1)AdaptDNN的训练时间更短. 如图10所示,AdaptDNN在绝大多数场景下比Oobleck和Checkpoint的每轮次训练时间更短,最好性能是基于Checkpoint方式的1.7倍,是Oobleck的1.6倍. 基于Checkpoint方式由于读写磁盘的带宽较低,因此会对训练时间造成一定程度的延长,而Oobleck由于对并行方案有一定的要求(必须使用数据并行)因此并未采用最优的并行方案,导致性能不是最优.
2)AdaptDNN能支持更大的模型. 如图10所示,在现有16张V100GPU卡的实验环境下,AdaptDNN可以支持GPT-3 6.7B,而Oobleck最高支持GPT-3 2.7B;对于GPT-3 2.7B,Oobleck最少需要10个GPU,而AdaptDNN可以用8个GPU完成训练. 因为Oobleck需要使用数据并行,导致内存压力较大,而AdaptDNN对并行方案无要求,因此可以训练更大的模型.
3)AdaptDNN的故障恢复时间更小. 如图11所示,我们比较了不同系统在GPU数量发生变化时的故障恢复开销,AdaptDNN的开销最低仅为Oobleck的25%,为Checkpoint的11%. 因为Oobleck将丢失当前轮次的部分训练进度,在完成模型的重新部署后需要重新计算丢失的进度,而基于Checkpoint的容错机制从磁盘读取Checkpoint的带宽低、开销大,并且也需要重新计算丢失的进度,导致故障恢复的开销极高.
在6.2节中,我们测试了逐渐减少集群中GPU数量时,不同系统的训练吞吐量. 在本节中,进一步测试AdaptDNN和各基准系统在真实弹性实例集群中的性能.
1)实验trace. 使用了Bamboo开源的2段真实trace[7]进行模拟实验. 这2段trace分别记录了AWS的2个不同型号的32卡弹性集群在12 h内的可用实例数量. 由于本实验所用的集群只有16卡,我们对该trace进行了按比例缩放,使得实例数量的变化在8~16.
2)实验设计. 在2段不同的trace上分别模拟了GPT-3 1.3B和GPT-3 2.7B模型的训练过程. 我们测量了每个系统在每一次实例数量发生变化前后的训练吞吐量,以及对并行方案调整的开销,然后基于这些实测性能数据,利用Bamboo提供的模拟器,模拟了12 h内系统的训练吞吐量变化情况.
3)对比指标. 将对比不同系统在12 h内的平均训练吞吐量(sample/s)以及每个时刻的吞吐量. 每时刻的吞吐量通过计算一个较小时间窗口内的平均吞吐量得到.
4)实验结果. 各系统的吞吐量随着实例数量变化的趋势如图12所示,AdaptDNN在绝大多数时刻都展现出了更好的训练吞吐量. 在实例数量发生变化时,AdaptDNN吞吐量的下降幅度相比基准系统更小,体现出了对故障更强的容忍能力和快速的恢复能力.
各系统12 h内的平均吞吐量如表3所示. AdaptDNN在所有测试场景下都表现出了相比基准系统更好的性能. 其中,AdaptDNN相比Checkpoint方式,最高达到了1.79倍的训练吞吐量(GPT-3 2.7B, trace-2),相比Oobleck最高达到了1.27倍训练吞吐量(GPT-3 1.3B, trace-2). 我们发现,AdaptDNN在trace-2相比基准系统的性能提升更大,主要原因是trace-2的实例回收更加频繁,基准系统的恢复开销较高.
AdaptDNN在搜索模型的调整方案时引入了“下一次故障前有效的轮次数”指标作为优化的目标函数,而该指标与平均故障时间(mean time to failure,MTTF)和故障恢复开销有关. 进一步地,故障恢复开销又与节点间网络通信带宽和宽限期大小相关. 因此,在本节中,我们进行了一系列消融实验,分别测试MTTF、网络带宽和宽限期大小对故障恢复的影响.
1)基准系统. 我们与Oobleck和Aceso进行了对比,Oobleck和Aceso都是基于最优性能搜索模型的调整方案. Checkpoint机制可以与任意并行方案组合使用,其本身不包含对调整方案的搜索,因此不作为本次消融实验的基准系统.
2)对比指标. 比较不同系统在完成故障恢复后的下一次故障前有效的轮次数(iters).
实验1. MTTF对有效轮次数的影响. 我们保持宽限期为30 s和网络带宽为1 Gbps. 测试了4种不同的MTTF:50 s,100 s,200 s,400 s. 结果如图13所示,当MTTF较小时(例如MTTF=50 s),即故障频繁发生,此时仅以性能为搜索目标的Oobleck和Aceso无法找到可行的重构方案,或找到的方案只能产生较小有效轮次. 而AdaptDNN由于将MTTF引入搜索的目标函数,因此在各种MTTF下都能找到故障恢复的可行方案. 当MTTF较大时,则故障发生并不频繁,各系统的性能接近.
实验2. 网络带宽对有效轮次数的影响. 我们保持宽限期为30 s和MTTF为50 s,测试了3种不同的网络带宽:1 Gbps,10 Gbps,100 Gbps. 结果如图14所示,当网络带宽较低时,Oobleck和Aceso找到的调整方案无法在宽限期内完成,而AdaptDNN总能找到可行的重构方案. 当模型规模增加,模型参数量增多,故障恢复开销也随之上升,由于Oobleck搜索时不考虑故障恢复开销,因此在MTTF为50 s且模型较大时(GPT-3 2.7B),在3种带宽下都没有找到可行的调整方案.
实验3. 宽限期对有效轮次数的影响. 我们保持MTTF为50 s和带宽为1 Gbps,测试了3种不同的宽限期:30 s,60 s,120 s. 如图15所示,当宽限期较小时,Oobleck和Aceso无法找到可行解. 另外,由于此时的带宽和MTTF都比较小,导致故障恢复的时间较长,因此综合考虑故障恢复开销和并行方案性能的AdaptDNN能在所有测试下找到最优的调整方案,从而实现了更大的有效轮次数.
本文对弹性实例下的深度学习分布式训练进行了深入研究,提出了在实例回收前利用宽限期快速完成故障恢复,并利用消除瓶颈的思想快速搜索模型的调整方案,从而实现了模型训练的自适应可伸缩性. 实验表明我们的系统能更好地适应弹性实例场景,从而为大模型的低成本训练提供了可行且高效的解决方案.
作者贡献声明:刘国栋提出本文的主要思路,并完成了部分实验和撰写论文;朱家祺和高梓源完成了部分实验和撰写论文;包云岗和王卅负责对论文整体思路进行指导并提出了修改意见.

参考文献 引证文献
排序方式:
1
OpenAI. AI and compute[EB/OL]. (2018-05-16)[2024-05-30]. https://openai.com/index/ai-and-compute/
2
Li Mu, Andersen D, Park J, et al. Scaling distributed machine learning with the parameter server[C]//Proc of the 11th USENIX Symp on Operating Systems Design and Implementation. Berkeley, CA: USENIX Association, 2014: 583−598
3
Jia Zhihao, Zaharia M, Aiken A. Beyond data and model parallelism for deep neural networks[C/OL]//Proc of the 2nd Machine Learning and Systems Conf. 2019[2024-10-21]. https://proceedings.mlsys.org/paper_files/paper/2019/file/b422680f3db0986ddd7f8f126baaf0fa-Paper.pdf
4
Narayanan D, Harlap A, Phanishayee A, et al. PipeDream: Generalized pipeline parallelism for DNN training[C]//Proc of the 27th ACM Symp on Operating Systems Principles. New York: ACM, 2019: 1−15
5
Brown T, Mann B, Ryder N, et al. Language models are few-shot learners[C/OL]//Proc of the 34th Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2020[2024-10-21]. https://papers.nips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
6
Duan Jiangfei, Song Ziang, Miao Xupeng, et al. Parcae: Proactive, liveput-optimized DNN training on preemptible instances[C]//Proc of the 21st USENIX Symp on Networked Systems Design and Implementation. Berkeley, CA: USENIX Association, 2024: 1121−1139
7
Thorpe J, Zhao Pengzhan, Eyolfson J, et al. Bamboo: Making preemptible instances resilient for affordable training of large DNNs[C]//Proc of the 20th USENIX Symp on Networked Systems Design and Implementation. Berkeley, CA: USENIX Association, 2023: 497−513
8
Li Jiamin, Xu Hong, Zhu Yibo, et al. Lyra: Elastic scheduling for deep learning clusters[C]//Proc of the 18th European Conf on Computer Systems. New York: ACM, 2023: 835−850
9
Narayanan D, Shoeybi M, Casper J, et al. Efficient large-scale language model training on GPU clusters using megatron-LM[C]//Proc of the 34th Int Conf for High Performance Computing, Networking, Storage and Analysis. New York: ACM, 2021: 1−15
10
Mohan J, Phanishayee A, Chidambaram V. CheckFreq: Frequent, fine-grained DNN checkpointing[C]//Proc of the 19th USENIX Conf on File and Storage Technologies. Berkeley, CA: USENIX Association, 2021: 203−216
11
Athlur S, Saran N, Sivathanu M, et al. Varuna: Scalable, low-cost training of massive deep learning models[C]//Proc of the 17th European Conf on Computer Systems. New York: ACM, 2022: 472−487
12
Jang I, Yang Zhenning, Zhang Zhen, et al. Oobleck: Resilient distributed training of large models using pipeline templates[C]//Proc of the 29th Symp on Operating Systems Principles. New York: ACM, 2023: 382−395
13
Zheng Lianmin, Li Zhuohan, Zhang Hao, et al. Alpa: Automating inter-and intra-operator parallelism for distributed deep learning[C]//Proc of the 16th USENIX Symp on Operating Systems Design and Implementation. Berkeley, CA: USENIX Association, 2022: 559−578
14
Liu Guodong, Miao Youshan, Lin Zhiqi, et al. Aceso: Efficient parallel DNN training through iterative bottleneck alleviation[C]//Proc of the 19th European Conf on Computer Systems. New York: ACM, 2024: 163−181
15
Dean J, Corrado G, Monga R, et al. Large scale distributed deep networks[C/OL]//Proc of the 26th Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2012[2024-10-21]. https://papers.nips.cc/paper_files/paper/2012/file/6aca97005c68f1206823815f66102863-Paper.pdf
16
Wang Minjie, Huang C, Li Jinyang. Supporting very large models using automatic dataflow graph partitioning[C]//Proc of the 14th European Conf on Computer Systems. New York: ACM, 2019: 1−17
17
Mirhoseini A, Pham H, Le Q, et al. Device placement optimization with reinforcement learning[C]//Proc of the 34th Int Conf on Machine Learning. New York: PMLR, 2017: 2430-−2439
18
Gao Yuanxiang, Chen Li, Li Baochun. Spotlight: Optimizing device placement for training deep neural networks[C]//Proc of the 35th Int Conf on Machine Learning. New York: PMLR, 2018: 1676−1684
19
Xu Yuanzhong, HyoukJoong L, Chen Dehao, et al. GSPMD: General and scalable parallelization for ML computation graphs[J]. arXiv preprint, arXiv: 2105.04663
20
Huang Yanping, Cheng Youlong, Bapna A, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism[C/OL]//Proc of the 33rd Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2019[2024-10-21]. https://proceedings.neurips.cc/paper_files/paper/2019/file/093f65e080a295f8076b1c5722a46aa2-Paper.pdf
21
Narayanan D, Phanishayee A, Shi K, et al. Memory-efficient pipeline-parallel DNN training[C]//Proc of the 38th Int Conf on Machine Learning. New York: PMLR, 2021: 7937−7947
22
Li Shigang, Torsten H. Chimera: Efficiently training large-scale neural networks with bidirectional pipelines[C]//Proc of the 34th Int Conf for High Performance Computing, Networking, Storage and Analysis. New York: ACM, 2021: 1−14
23
Fan Shiqing, Rong Yi, Meng Chen, et al. DAPPLE: A pipelined data parallel approach for training large models[C]//Proc of the 26th ACM SIGPLAN Symp on Principles and Practice of Parallel Programming. New York: ACM, 2021: 431−445
24
PyTorch. Torch Distributed Elastic[EB/OL]. [2024-05-30]. https://pytorch.org/docs/stable/distributed.elastic.html
25
The Horovod Authors. Elastic Hovorod[EB/OL]. [2024-05-30]. https://horovod.readthedocs.io/en/stable/elastic_include.html
26
Wu Yidi, Ma Kaihao, Yan Xiao, et al. Elastic deep learning in multi-tenant GPU clusters[J]. IEEE Transactions on Parallel and Distributed Systems, 2021, 33(1): 144−158
27
Xiao Wencong, Bhardwaj R, Ramjee R, et al. Gandiva: Introspective cluster scheduling for deep learning[C]//Proc of the 13th USENIX Symp on Operating Systems Design and Implementation. Berkeley, CA: USENIX Association, 2018: 595−610
28
Qiao A, Choe S, Subramanya S, et al. Pollux: Co-adaptive cluster scheduling for goodput-optimized deep learning[C]//Proc of the 15th USENIX Symp on Operating Systems Design and Implementation. Berkeley, CA: USENIX Association, 2021: 1−18
29
Gu Diandian, Zhao Yihao, Zhong Yinmin, et al. ElasticFlow: An elastic serverless training platform for distributed deep learning[C]//Proc of the 28th ACM Int Conf on Architectural Support for Programming Languages and Operating Systems. New York: ACM, 2023: 266−280
30
Wang Zhuang, Jia Zhen, Zheng Shuai, et al. Gemini: Fast failure recovery in distributed training with in-memory checkpoints[C]//Proc of the 29th Symp on Operating Systems Principles. New York: ACM, 2023: 364−381
31
He Tao, Li Xue, Wang Zhibin, et al. Unicron: Economizing self-healing LLM training at scale[J]. arXiv preprint, arXiv: 2401.00134, 2023
32
Jiang Ziheng, Lin Haibin, Zhong Yinmin, et al. MegaScale: Scaling large language model training to more than <styled-content style-type="number"> 10000 </styled-content> GPUs[C]//Proc of the 21st USENIX Symp on Networked Systems Design and Implementation. Berkeley, CA: USENIX Association, 2024: 745−760
33
Wang Yuxin, Kang Xueze, Shi Shaohuai, et al. Reliable and efficient in-memory fault tolerance of large language model pretraining[EB/OL]. (2024-08-19)[2024-09-06]. https://arxiv.org/abs/2310.12670
34
Lian Xinyu, Jacobs S, Kurilenko L, et al. Universal checkpointing: Efficient and flexible checkpointing for large scale distributed training[J]. arXiv preprint, arXiv: 2406.18820, 2024
35
Wan Borui, Han Mingji, Sheng Yiyao, et al. ByteCheckpoint: A unified checkpointing system for LLM development[J]. arXiv preprint, arXiv: 2407.20143, 2024
36
Gandhi S, Zhao M, Skiadopoulos A, et al. SlipStream: Adapting pipelines for distributed training of large DNNs amid failures[J]. arXiv preprint, arXiv: 2405.14009, 2024
37
Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C/OL]//Proc of the 31st Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2017[2024-10-21]. https://user.phil.hhu.de/~cwurm/wp-content/uploads/2020/01/7181-attention-is-all-you-need.pdf
38
Navarro A, Asenjo R, Tabik S, et al. Analytical modeling of pipeline parallelism[C]//Proc of the 18th Int Conf on Parallel Architectures and Compilation Techniques. Piscataway, NJ: IEEE, 2009: 291−290
39
Perez T, Chen Wei, Ji R, et al. PETS: Bottleneck-aware spark tuning with parameter ensembles[C]//Proc of the 27th Int Conf on Computer Communication and Networks. Piscataway, NJ: IEEE, 2018: 1−9
40
Kahveci B, Gedik B. Joker: Elastic stream processing with organic adaptation[J]. Journal of Parallel and Distributed Computing, 2020, 137: 205-223
Kahveci B, Gedik B. Joker: Elastic stream processing with organic adaptation[J]. Journal of Parallel and Distributed Computing, 2020, 137: 205-223
41
NVIDIA. ConnectX-5: Advanced offload capabilities for the most demanding applications[EB/OL]. [2024-05-30]. https://www.nvidia.com/ en-us/networking/ethernet/connectx-5/
2025年第62卷第12期
87
39
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202440560
  • 接收时间:2024-06-21
  • 首发时间:2026-11-10
  • 出版时间:2025-12-01
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2024-06-21
  • 修回日期:2025-05-09
  • 录用日期:2025-06-11
基金
作者信息
    1处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190
    2中国科学院大学计算机科学与技术学院 北京 100049
    3北京开源芯片研究院 北京 100080

通讯作者:

王卅()
参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202440560
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏