Article(id=1327012676691145592, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1327012646391493336, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202550127, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=research-article, receivedDate=1740758400000, receivedDateStr=2025-03-01, revisedDate=1744473600000, revisedDateStr=2025-04-13, acceptedDate=null, acceptedDateStr=null, onlineDate=1794314076825, onlineDateStr=2026-11-10, pubDate=1748793600000, pubDateStr=2025-06-02, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1794314076825, onlineIssueDateStr=2026-11-10, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1794314076825, creator=13701087609, updateTime=1794314076825, updator=13701087609, issue=Issue{id=1327012646391493336, tenantId=1146029695717560320, journalId=1324759914146529478, year='2025', volume='62', issue='6', pageStart='1331', pageEnd='1593', issueExtLink='null', onlineDate='null', pubDate='1748793600000', pubDateStr='2025-06-02', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1794314069601, creator='13701087609', updateTime=1794376817187, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext={EN=IssueExt(id=1327275829270262152, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1327012646391493336, language=EN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=), CN=IssueExt(id=1327275829270262153, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1327012646391493336, language=CN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=)}, issueFiles=null, downloadFileDto=null}, startPage=1513, endPage=1529, ext={EN=ArticleExt(id=1327012677051855738, articleId=1327012676691145592, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=Pipe-RLHF: A Computation Mode-Aware Parallel Framework for RLHF, columnId=1327012676955386745, journalTitle=Journal of Computer Research and Development, columnName=Chip Architecture Design, runingTitle=null, highlight=null, articleAbstract=

Reinforcement learning with human feedback (RLHF) has been proven effective in aligning large language models (LLMs) with human preferences. The most costly part of RLHF is proximal policy optimization (PPO), which consists of three dependent steps. Different PPO steps in RLHF exhibit different computation modes, simply employing the same parallelization strategy to accelerate all steps that involve multiple model variants, as done in existing frameworks, will lead to poor performance in the PPO generation step due to insufficient utilization of computational resources. Thus, we introduce Pipe-RLHF, a parallelism framework for RLHF fine-tuning, which adaptively employs distinct parallelization strategies for different steps according to the computation mode. Specifically, we first investigate the characteristics of various computation modes to explore their best-fit parallelization approach. And then, we present a novel delayed inter-batch pipeline parallelization approach specifically designed for the PPO generation step, enabling the sufficient utilization of computational resources. Subsequently, based on the proposed inter-batch pipeline parallelization approach, we define a hierarchical parallel plan space for distributed RLHF fine-tuning. Finally, we present optimization algorithms to find the optimal parallelization plan from the defined hierarchical parallel plan space to minimize the overall time consumption. Implementation and evaluation across multiple LLMs demonstrates that the proposed Pipe-RLHF achieves 3.7 times speedup compared with existing methods while achieving near-linear scalability.

, authors=Ying Xu1, 2, Mengdi Wang1, Long Cheng3, Lian Liu1, 2, Shixin Zhao1, 2, Lei Zhang1, Ying Wang1, authorsList=Ying Xu, Mengdi Wang, Long Cheng, Lian Liu, Shixin Zhao, Lei Zhang, Ying Wang, authorCompany=null, correspAuthors=null, authorNote=

born in 1996. PhD candidate. Her main research interests include machine learning system and parallel deep learning framework

born in 1995. Assistant professor. Her main research interests include accelerator design and multi-core system

born in 1985. PhD, professor, PhD supervisor. His main research interest includes distributed computing and high-performance systems

born in 1999. PhD candidate. His main research interest includes LLM infrastructure、hardware-software co-design

born in 1999. PhD candidate. Her main research interest includes computer architecture

born in 1981. PhD, professor. Member of CCF. His main research interests include chip and computing system

born in 1985. PhD, professor. Senior member of CCF. His main research interests include novel EDA, and processor and memory system architecture

, correspAuthorsNote=null, copyrightStatement=Copyright © 2025 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1327012682869355437, articleId=1327012676691145592, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=Pipe-RLHF: 计算模式感知的RLHF并行加速框架, columnId=1327012677139936123, journalTitle=计算机研究与发展, columnName=芯片架构设计, runingTitle=null, highlight=null, articleAbstract=

基于人类反馈的强化学习(reinforcement learning with human feedback,RLHF)作为当前大语言模型(large language models,LLMs)对齐的主流方法,其核心优化算法——近端策略优化(proximal policy optimization,PPO)却面临着显著的效率问题. PPO由生成、推理、训练3个相互关联的阶段组成,各个阶段有着不同的计算特性. 然而,现有的RLHF并行框架采用相同并行策略顺序执行PPO的所有阶段,这导致以下2个问题:其一,生成阶段不能充分利用计算资源,进而影响整体效率;其二,阶段间严格串行执行,未能充分利用潜在并行性. 针对上述问题,提出了一个新型RLHF并行框架——Pipe-RLHF.该框架能够自适应地根据各阶段的计算特征确定最优并行策略,突破现有阶段串行范式,采用异步PPO算法发掘阶段间的并行性. 具体而言,创新性地提出了适用于PPO生成阶段的延迟批间流水线并行方法,显著提升了该阶段的计算资源利用率;再次,使用异步PPO解放阶段间的依赖关系,将阶段间并行应用到PPO的加速上;最后,针对PPO算法的整体优化,构建了分层并行策略空间,并提出了一套优化算法以实现该空间中的最优解搜索. 通过在多个大语言模型上的性能评估实验表明,相较于现有方法,Pipe-RLHF最高可实现3.7倍的加速比,充分验证了该框架的有效性和优越性.

, authors=徐颖1, 2, 王梦迪1, 程龙3, 刘炼1, 2, 赵世新1, 2, 张磊1, 王颖1, authorsList=徐颖, 王梦迪, 程龙, 刘炼, 赵世新, 张磊, 王颖, authorCompany=null, correspAuthors=null, authorNote=

1996年生. 博士研究生. 主要研究方向为机器学习系统、深度学习并行框架

1995年生. 助理研究员. 主要研究方向为加速器设计、多核系统

1985年生. 博士,教授,博士生导师. 主要研究方向为分布式计算、高性能系统

1999年生. 博士研究生. 主要研究方向为大语言模型系统架构设计、软硬件协同设计

1999年生. 博士研究生. 主要研究方向为计算机体系结构

1981年生. 博士,正高级工程师. CCF会员. 主要研究方向为芯片、计算系统

1985年生. 博士,研究员. CCF高级会员. 主要研究方向为新型EDA、处理器与存储系统体系结构

, correspAuthorsNote=null, copyrightStatement=版权所有 © 2025 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=eyqMF2eC9JdoOFRO5wUBMw==, magXml=caREVS5/O8+/AKaZPaXaMA==, pdfUrl=null, pdf=gE07r3uX3SD9rYcu5dgDHw==, pdfFileSize=1541416, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=TFUkHvRlMl9graXj4BiZNQ==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=WUc34kbl3O/mT8LREHmWaA==, mapNumber=null, fund=null)}, authors=[Author(id=1327012685042004921, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=xuying18z@ict.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012685125891004, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685042004921, language=EN, stringName=Ying Xu, firstName=Ying, middleName=null, lastName=Xu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
2University of Chinese Academy of Sciences, Beijing 100190, bio={"content":"

born in 1996. PhD candidate. Her main research interests include machine learning system and parallel deep learning framework

"}, bioImg=null, bioContent=

born in 1996. PhD candidate. Her main research interests include machine learning system and parallel deep learning framework

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012685201388477, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685042004921, language=CN, stringName=徐颖, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1中国科学院计算技术研究所 北京 100190
2中国科学院大学 北京 100190, bio={"img":"DENf+2BfcH2ih411a8jwcg==","content":"

1996年生. 博士研究生. 主要研究方向为机器学习系统、深度学习并行框架

"}, bioImg=DENf+2BfcH2ih411a8jwcg==, bioContent=

1996年生. 博士研究生. 主要研究方向为机器学习系统、深度学习并行框架

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012683150373806, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=1, ext=[AuthorCompanyExt(id=1327012683158762415, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683167151024, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所 北京 100190)]), AuthorCompany(id=1327012683255231409, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=2, ext=[AuthorCompanyExt(id=1327012683284591538, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683255231409, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2University of Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683297174451, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683255231409, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学 北京 100190)])]), Author(id=1327012685276885951, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012685356577729, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685276885951, language=EN, stringName=Mengdi Wang, firstName=Mengdi, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, bio={"content":"

born in 1995. Assistant professor. Her main research interests include accelerator design and multi-core system

"}, bioImg=null, bioContent=

born in 1995. Assistant professor. Her main research interests include accelerator design and multi-core system

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012685415297986, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685276885951, language=CN, stringName=王梦迪, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1中国科学院计算技术研究所 北京 100190, bio={"img":"XdV6mzDHK9VkDs2B56DDUA==","content":"

1995年生. 助理研究员. 主要研究方向为加速器设计、多核系统

"}, bioImg=XdV6mzDHK9VkDs2B56DDUA==, bioContent=

1995年生. 助理研究员. 主要研究方向为加速器设计、多核系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012683150373806, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=1, ext=[AuthorCompanyExt(id=1327012683158762415, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683167151024, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所 北京 100190)])]), Author(id=1327012685482406852, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012685570487238, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685482406852, language=EN, stringName=Long Cheng, firstName=Long, middleName=null, lastName=Cheng, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, address=3School of Control and Computer Engineering, North China Electric Power University, Beijing 102206, bio={"content":"

born in 1985. PhD, professor, PhD supervisor. His main research interest includes distributed computing and high-performance systems

"}, bioImg=null, bioContent=

born in 1985. PhD, professor, PhD supervisor. His main research interest includes distributed computing and high-performance systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012685683733447, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685482406852, language=CN, stringName=程龙, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, address=3华北电力大学控制与计算机工程学院 北京 102206, bio={"img":"Gqw18szoYQR2/e8+Sby8Jw==","content":"

1985年生. 博士,教授,博士生导师. 主要研究方向为分布式计算、高性能系统

"}, bioImg=Gqw18szoYQR2/e8+Sby8Jw==, bioContent=

1985年生. 博士,教授,博士生导师. 主要研究方向为分布式计算、高性能系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012684958118836, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=3, ext=[AuthorCompanyExt(id=1327012684962313141, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012684958118836, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3School of Control and Computer Engineering, North China Electric Power University, Beijing 102206), AuthorCompanyExt(id=1327012684970701750, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012684958118836, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3华北电力大学控制与计算机工程学院 北京 102206)])]), Author(id=1327012685755036617, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012685834728396, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685755036617, language=EN, stringName=Lian Liu, firstName=Lian, middleName=null, lastName=Liu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
2University of Chinese Academy of Sciences, Beijing 100190, bio={"content":"

born in 1999. PhD candidate. His main research interest includes LLM infrastructure、hardware-software co-design

"}, bioImg=null, bioContent=

born in 1999. PhD candidate. His main research interest includes LLM infrastructure、hardware-software co-design

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012685897642957, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685755036617, language=CN, stringName=刘炼, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1中国科学院计算技术研究所 北京 100190
2中国科学院大学 北京 100190, bio={"img":"AkYDGV2VDDR9H0rj0XN4Lg==","content":"

1999年生. 博士研究生. 主要研究方向为大语言模型系统架构设计、软硬件协同设计

"}, bioImg=AkYDGV2VDDR9H0rj0XN4Lg==, bioContent=

1999年生. 博士研究生. 主要研究方向为大语言模型系统架构设计、软硬件协同设计

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012683150373806, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=1, ext=[AuthorCompanyExt(id=1327012683158762415, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683167151024, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所 北京 100190)]), AuthorCompany(id=1327012683255231409, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=2, ext=[AuthorCompanyExt(id=1327012683284591538, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683255231409, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2University of Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683297174451, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683255231409, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学 北京 100190)])]), Author(id=1327012685964751823, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012686069609426, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685964751823, language=EN, stringName=Shixin Zhao, firstName=Shixin, middleName=null, lastName=Zhao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
2University of Chinese Academy of Sciences, Beijing 100190, bio={"content":"

born in 1999. PhD candidate. Her main research interest includes computer architecture

"}, bioImg=null, bioContent=

born in 1999. PhD candidate. Her main research interest includes computer architecture

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012686161884115, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685964751823, language=CN, stringName=赵世新, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1中国科学院计算技术研究所 北京 100190
2中国科学院大学 北京 100190, bio={"img":"JQwhJoDWwZGRTG1do7/HbQ==","content":"

1999年生. 博士研究生. 主要研究方向为计算机体系结构

"}, bioImg=JQwhJoDWwZGRTG1do7/HbQ==, bioContent=

1999年生. 博士研究生. 主要研究方向为计算机体系结构

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012683150373806, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=1, ext=[AuthorCompanyExt(id=1327012683158762415, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683167151024, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所 北京 100190)]), AuthorCompany(id=1327012683255231409, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=2, ext=[AuthorCompanyExt(id=1327012683284591538, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683255231409, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2University of Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683297174451, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683255231409, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学 北京 100190)])]), Author(id=1327012686224798677, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, orderNo=5, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012686312879063, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012686224798677, language=EN, stringName=Lei Zhang, firstName=Lei, middleName=null, lastName=Zhang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, bio={"content":"

born in 1981. PhD, professor. Member of CCF. His main research interests include chip and computing system

"}, bioImg=null, bioContent=

born in 1981. PhD, professor. Member of CCF. His main research interests include chip and computing system

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012686379987928, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012686224798677, language=CN, stringName=张磊, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1中国科学院计算技术研究所 北京 100190, bio={"img":"SG3c/MaOAOCQlehMUFcuog==","content":"

1981年生. 博士,正高级工程师. CCF会员. 主要研究方向为芯片、计算系统

"}, bioImg=SG3c/MaOAOCQlehMUFcuog==, bioContent=

1981年生. 博士,正高级工程师. CCF会员. 主要研究方向为芯片、计算系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012683150373806, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=1, ext=[AuthorCompanyExt(id=1327012683158762415, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683167151024, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所 北京 100190)])]), Author(id=1327012686438708186, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, orderNo=6, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012686530982876, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012686438708186, language=EN, stringName=Ying Wang, firstName=Ying, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, bio={"content":"

born in 1985. PhD, professor. Senior member of CCF. His main research interests include novel EDA, and processor and memory system architecture

"}, bioImg=null, bioContent=

born in 1985. PhD, professor. Senior member of CCF. His main research interests include novel EDA, and processor and memory system architecture

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012686627451869, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012686438708186, language=CN, stringName=王颖, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1中国科学院计算技术研究所 北京 100190, bio={"img":"oq44dw3DfE6YPKNN0iX3VQ==","content":"

1985年生. 博士,研究员. CCF高级会员. 主要研究方向为新型EDA、处理器与存储系统体系结构

"}, bioImg=oq44dw3DfE6YPKNN0iX3VQ==, bioContent=

1985年生. 博士,研究员. CCF高级会员. 主要研究方向为新型EDA、处理器与存储系统体系结构

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012683150373806, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=1, ext=[AuthorCompanyExt(id=1327012683158762415, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683167151024, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所 北京 100190)])])], keywords=[Keyword(id=1327012686786835422, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, orderNo=1, keyword=reinforcement learning with human feedback (RLHF)), Keyword(id=1327012686853944287, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, orderNo=2, keyword=proximal policy optimization (PPO)), Keyword(id=1327012686925247456, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, orderNo=3, keyword=large language models fine-tuning), Keyword(id=1327012687042687969, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, orderNo=4, keyword=distributed systems), Keyword(id=1327012687143351266, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, orderNo=5, keyword=parallel computing), Keyword(id=1327012687332094948, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, orderNo=1, keyword=基于人类反馈的强化学习), Keyword(id=1327012687411786725, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, orderNo=2, keyword=近端策略优化), Keyword(id=1327012687483089894, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, orderNo=3, keyword=大模型微调), Keyword(id=1327012687566975975, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, orderNo=4, keyword=分布式系统), Keyword(id=1327012687663444968, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, orderNo=5, keyword=并行计算)], refs=[Reference(id=1327012694005231635, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=Christiano P F, Leike J, Brown T B, et al. Deep reinforcement learning from human preferences[C]//Proc of the 31st Int Conf on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc., 2017: 4302–4310, articleTitle=null, refAbstract=null), Reference(id=1327012694068146197, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Macglashan J, Ho M K, Loftin R, et al. Interactive learning from policy-dependent human feedback[C]//Proce of the 34th Int Conf on Machine Learning - Volume 70. New York: ACM, 2017: 2285–2294, articleTitle=null, refAbstract=null), Reference(id=1327012694131060758, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Warnell G, Waytowich N, Lawhern V, et al. Deep TAMER: Interactive agent shaping in high-dimensional state spaces[C]//Proc of the 32nd AAAI Conf on Artificial Intelligence and 30th Innovative Applications of Artificial Intelligence Conf and 8th AAAI Symp on Educational Advances in Artificial Intelligence. Palo Alto, CA: AAAI, 2018: 1545−1554, articleTitle=null, refAbstract=null), Reference(id=1327012694235918359, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Brown D, Goo W, Nagarajan P, et al. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations[C]//Proc of Int Conf on Machine Learning. Cambridge, MA: JMLR, 2019: 783−792, articleTitle=null, refAbstract=null), Reference(id=1327012694370136088, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Shin D, Dragan A D, Brown D S. Benchmarks and algorithms for offline preference-based reward learning[J]. arXiv preprint, arXiv: 2301.01392, 2023, articleTitle=null, refAbstract=null), Reference(id=1327012694516936729, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Sadigh D, Dragan A, Sastry S, et al. Active preference-based learning of reward functions[C]//Proc of Robotics: Science and Systems. Cambridge, Massachusetts: Escholarship. https://www.roboticsproceedings.org/rss13/p53.html, articleTitle=null, refAbstract=null), Reference(id=1327012694600822810, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Maghakian J, Mineiro P, Panaganti K, et al. Personalized reward learning with interaction-grounded learning (IGL)[C]//Proc of the 11th Int Conf on Learning Representations. Washington, DC: OpenReview. net, 2022, articleTitle=null, refAbstract=null), Reference(id=1327012694676320283, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=矣晓沅,谢幸. 大模型道德价值观对齐问题剖析[J]. 计算机研究与发展,2023,60(9):1926−1945, articleTitle=null, refAbstract=null), Reference(id=1327012694747623452, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Yi Xiaoyuan, Xie Xing. Unpacking the ethical value alignment in big models[J]. Journal of Computer Research and Development, 2023, 60(9): 1926−1945 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1327012694810538013, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Ouyang Long, Wu J, Jiang Xu, et al. Training language models to follow instructions with human feedback[C]//Proc of Conf on Neural Information Processing Systems 2022, Red Hook, NY: Curran Associates Inc., 2022: 27730−27744, articleTitle=null, refAbstract=null), Reference(id=1327012694940561438, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Bai Y, Jones A, Ndousse K, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback[J]. arXiv preprint, arXiv: 2204.05862, 2022, articleTitle=null, refAbstract=null), Reference(id=1327012695066390559, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Achiam J, Adler S, Agarwal S, et al. GPT−4 technical report[R/OL]. OpenAI, 2023[2025-04-08]. https://arxiv.org/pdf/2303.08774, articleTitle=null, refAbstract=null), Reference(id=1327012695150276640, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Anthropic. Introducing claude[EB/OL]. 2023[2025-04-09]. https://www.anthropic.com/news/introducing-claude, articleTitle=null, refAbstract=null), Reference(id=1327012695225774113, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Glaese A, Mcaleese N, Trębacz M, et al. Improving alignment of dialogue agents via targeted human judgements[J]. arXiv preprint, arXiv: 2209.14375, 2022, articleTitle=null, refAbstract=null), Reference(id=1327012695280300066, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Pichai S, Hassabis D. Introducing Gemini: Our largest and most capable AI model[EB/OL]. 2023[2025-04-09]. https://blog.google/technology/ai/google-gemini-ai/, articleTitle=null, refAbstract=null), Reference(id=1327012695372574755, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Bi Xiao, Chen Deli, Chen Guanting, et al. DeepSeek LLM: Scaling open-source language models with longtermism[J]. arXiv preprint, arXiv: 2401.02954, 2024, articleTitle=null, refAbstract=null), Reference(id=1327012695439683620, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Stiennon N, Ouyang Long, Wu J, et al. Learning to summarize with human feedback[C]//Proc of Conf on Neural Information Processing Systems 2020. Red Hook, NY: Curran Associates Inc., 2020: 3008−3021, articleTitle=null, refAbstract=null), Reference(id=1327012695498403877, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Schulman J, Wolski F, Dhariwal P, et al. 2017. Proximal policy optimization algorithms[J]. arXiv preprint, arXiv: 1707.06347, 2017, articleTitle=null, refAbstract=null), Reference(id=1327012695569707046, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Zheng Rui, Dou Shihan, Gao Songyang, et al. Secrets of RLHF in large language models part I: PPO[J]. arXiv preprint, arXiv: 2307.04964, 2023, articleTitle=null, refAbstract=null), Reference(id=1327012695645204519, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Yao Zhewei, Aminabadi R Y, Ruwase O, et al. 2023. DeepSpeed-Chat: Easy, fast and affordable RLHF training of ChatGPT-like models at all scales[J]. arXiv preprint, arXiv: 2308.01320, 2023, articleTitle=null, refAbstract=null), Reference(id=1327012695724896296, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Werra L, Belkada Y, Tunstall L, et al. trl: Transformer reinforcement learning[EB/OL]. 2020[2025-04-09]. https://github.com/huggingface/trl, articleTitle=null, refAbstract=null), Reference(id=1327012695804588073, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Havrilla A, Zhuravinskyi M, Phung D, et al. Trlx: A framework for large scale reinforcement learning from human feedback[C]//Proc of the 2023 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2023: 8578−8595, articleTitle=null, refAbstract=null), Reference(id=1327012695884279850, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Lai Z, Li S, Tang X, et al. Merak: An efficient distributed DNN training framework with automated 3D parallelism for giant foundation models[J]. IEEE Transactions on Parallel and Distributed Systems, 2023, 34(5): 1466−1478, articleTitle=null, refAbstract=null), Reference(id=1327012695955583019, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=Song J, Yim J, Jung J, et al. Optimus-CC: Efficient large NLP model training with 3D parallelism aware communication compression[C]//Proc of the 28th ACM Int Conf on Architectural Support for Programming Languages and Operating Systems. New York: ACM, 2023: 560–573, articleTitle=null, refAbstract=null), Reference(id=1327012696018497580, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=Knox W B, Stone P. TAMER: Training an agent manually via evaluative reinforcement[C]//Proc of the 7th IEEE Int Conf on Development and Learning. Piscataway, NJ: IEEE, 2008: 292−297, articleTitle=null, refAbstract=null), Reference(id=1327012696085606445, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=Ziegler D M, Stiennon N, Wu J, et al. 2019. Fine-tuning language models from human preferences[J]. arXiv preprint, arXiv: 1909.08593, 2019, articleTitle=null, refAbstract=null), Reference(id=1327012696198852654, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=26, authorNames=null, journalName=null, refType=null, unstructuredReference=Wu J, Ouyang Long, Ziegler D M, et al. 2021. Recursively summarizing books with human feedback[J]. arXiv preprint, arXiv: 2109.10862, 2021, articleTitle=null, refAbstract=null), Reference(id=1327012696261767216, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=27, authorNames=null, journalName=null, refType=null, unstructuredReference=Santacroce M, Lu Yadong, Yu Han, et al. 2023. Efficient RLHF: Reducing the memory usage of PPO[J]. arXiv preprint, arXiv: 2309.00754, 2023, articleTitle=null, refAbstract=null), Reference(id=1327012696328876081, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=28, rfOrder=28, authorNames=null, journalName=null, refType=null, unstructuredReference=Zheng Lianmin, Li Zhouhan, Zhang Hao, et al. Alpa: Automating inter-and intra-operator parallelism for distributed deep learning[C]//Proc of the 16th USENIX Symp on Operating Systems Design and Implementation (OSDI 22). Berkeley, CA: USENIX Association, 2022: 559−578, articleTitle=null, refAbstract=null), Reference(id=1327012696400179250, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=29, rfOrder=29, authorNames=null, journalName=null, refType=null, unstructuredReference=Xiao W, Bhardwaj R, Ramjee R, et al. Gandiva: Introspective cluster scheduling for deep learning[C]//Proc of the 13th USENIX Conf on Operating Systems Design and Implementation. Berkeley, CA: USENIX Association, 2018: 595–610, articleTitle=null, refAbstract=null), Reference(id=1327012696622477363, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=30, rfOrder=30, authorNames=null, journalName=null, refType=null, unstructuredReference=Shoeybi M, Patwary M, Puri R, et al. 2019. Metatron-LM: Training multi-billion parameter language models using model parallelism[J]. arXiv preprint, arXiv: 1909.08053, 2019, articleTitle=null, refAbstract=null), Reference(id=1327012696706363444, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=31, rfOrder=31, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Yanping, Cheng Youlong, Bapna A, et al. GPipe: Efficient training of giant neural networks using pipeline parallelism[C]//Proc of the 33rd Int Conf on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc., 2019: 103−112, articleTitle=null, refAbstract=null), Reference(id=1327012696790249525, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=32, rfOrder=32, authorNames=null, journalName=null, refType=null, unstructuredReference=Narayanan D, Harlap A, Phanishayee A, et al. PipeDream: Generalized pipeline parallelism for DNN training[C]//Proc of the 27th ACM Symposium on Operating Systems Principles. New York: ACM, 2019: 1–15, articleTitle=null, refAbstract=null), Reference(id=1327012698514108470, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=33, rfOrder=33, authorNames=null, journalName=null, refType=null, unstructuredReference=Rajbhandari S, Rasley J, Ruwase O, et al. ZeRO: Memory optimizations toward training trillion parameter models[C]//Proc of SC20: Int Conf for High Performance Computing, Networking, Storage and Analysis. Piscataway, NJ: IEEE, 2020: 1−16, articleTitle=null, refAbstract=null), Reference(id=1327012698614771768, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=34, rfOrder=34, authorNames=null, journalName=null, refType=null, unstructuredReference=Hu E J, Shen Y, Wallis P, et al. Lora: Low-rank adaptation of large language models[C]//Proc of Int Conf on Learning Representations. openreview. net, 2022, articleTitle=null, refAbstract=null), Reference(id=1327012698677686329, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=35, rfOrder=35, authorNames=null, journalName=null, refType=null, unstructuredReference=Hu Jian, Wu Xibin, Zhu Zilin, et al. OpenRLHF: An easy-to-use, scalable and high-performance RLHF framework[J]. arXiv preprint, arXiv: 2405.11143, 2024, articleTitle=null, refAbstract=null), Reference(id=1327012698748989498, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=36, rfOrder=36, authorNames=null, journalName=null, refType=null, unstructuredReference=Sheng Guangming, Zhang Chi, Ye Zilingfeng, et al. HybridFlow: A flexible and efficient RLHF framework[J]. arXiv preprint, arXiv: 2409.19256, 2024, articleTitle=null, refAbstract=null), Reference(id=1327012698832875579, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=37, rfOrder=37, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Shenggui, Liu Hongxin, Bian Zhengda, et al. Colossal-AI: A unified deep learning system for large-scale parallel training[C]//Proc of the 52nd Int Conf on Parallel Processing. New York: ACM, 2023: 766–775, articleTitle=null, refAbstract=null), Reference(id=1327012698904178748, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=38, rfOrder=38, authorNames=null, journalName=null, refType=null, unstructuredReference=Zheng Yaowei, Zhang Richong, Zhang Junhao, et al. LLaMA-Factory: Unified efficient fine-tuning of 100+ language models[C]//Proc of the 62nd Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2024: 400−410, articleTitle=null, refAbstract=null), Reference(id=1327012698967093309, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=39, rfOrder=39, authorNames=null, journalName=null, refType=null, unstructuredReference=Volcengine. veRL: Volcano engine reinforcement learning for LLMs[EB/OL]. 2025[2025-04-08]. https://github.com/vocengine/verl, articleTitle=null, refAbstract=null), Reference(id=1327012699025813566, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=40, rfOrder=40, authorNames=null, journalName=null, refType=null, unstructuredReference=Qin Haotong, Ma Xudong, Zheng Xingyu, et al. Accurate LoRA-finetuning quantization of LLMs via information retention[J]. arXiv preprint, arXiv: 2402.05445, 2024, articleTitle=null, refAbstract=null), Reference(id=1327012699147448383, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=41, rfOrder=41, authorNames=null, journalName=null, refType=null, unstructuredReference=Dao T, Fu D Y, Ermon S, et al. 2022. FlashAttention: Fast and memory-efficient exact attention with IO-awareness[C]//Proc of the 36th Int Conf on Neural Information Processing Systems. New Orleans, LA: Curran Associates Inc. 1189: 16344−16359, articleTitle=null, refAbstract=null), Reference(id=1327012699231334464, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=42, rfOrder=42, authorNames=null, journalName=null, refType=null, unstructuredReference=Dao T, Haziza D, Massa F, et al. Flash-Decoding for long-context inference[EB/OL]. 2024[2025-03-01]. https://crfm.stanford.edu/2023/10/12/flashdecoding.html, articleTitle=null, refAbstract=null), Reference(id=1327012699294249025, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=43, rfOrder=43, authorNames=null, journalName=null, refType=null, unstructuredReference=葛旭冉,欧洋,王博,等. 大语言模型推理中的存储优化技术综述[J]. 计算机研究与发展,2025,62(3):545−562, articleTitle=null, refAbstract=null), Reference(id=1327012699361357890, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=43, rfOrder=44, authorNames=null, journalName=null, refType=null, unstructuredReference=Ge Xuran, Ou Yang, Wang Bo, et al. Survey of storage optimization techniques in large language model inference[J]. Journal of Computer Research and Development, 2025, 62(3): 545−562 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1327012699441049667, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=44, rfOrder=45, authorNames=null, journalName=null, refType=null, unstructuredReference=DeepSpeed. ZeRO-Inference: Democratizing massive model inference[EB/OL]. 2022[2025-04-08]. https://www.deepspeed.ai/2022/09/09/zero-inference.html, articleTitle=null, refAbstract=null), Reference(id=1327012699508158532, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=45, rfOrder=46, authorNames=null, journalName=null, refType=null, unstructuredReference=Sheng Ying, Zheng Lianmin, Yuan Binhang, et al. FlexGen: High-throughput generative inference of large language models with a single GPU[C]//Proc of Int Conf on Machine Learning. Cambridge, MA: PMLR, 2023: 31094−31116, articleTitle=null, refAbstract=null), Reference(id=1327012699587850309, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=46, rfOrder=47, authorNames=null, journalName=null, refType=null, unstructuredReference=Kwon W, Li Zhouhan, Zhuang Siyan, et al. Efficient memory management for large language model serving with pagedattention[C]//Proc of the 29th Symp on Operating Systems Principles. New York: ACM, 2023: 611−626, articleTitle=null, refAbstract=null), Reference(id=1327012699650764870, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=47, rfOrder=48, authorNames=null, journalName=null, refType=null, unstructuredReference=Nvidia. Triton inference server[EB/OL]. 2025[2025-04-09]. https://github.com/triton-inference-server/server, articleTitle=null, refAbstract=null), Reference(id=1327012699722068039, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=48, rfOrder=49, authorNames=null, journalName=null, refType=null, unstructuredReference=Yu G I, Jeong J S, Kim G W, et al. Orca: A distributed serving system for Transformer-Based generative models[C]//Proc of the 16th USENIX Symp on Operating Systems Design and Implementation (OSDI 22). Berkeley, CA: USENIX Association, 2022: 521−538, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1327012683150373806, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=1, ext=[AuthorCompanyExt(id=1327012683158762415, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683167151024, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所 北京 100190)]), AuthorCompany(id=1327012683255231409, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=2, ext=[AuthorCompanyExt(id=1327012683284591538, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683255231409, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2University of Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683297174451, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683255231409, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学 北京 100190)]), AuthorCompany(id=1327012684958118836, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=3, ext=[AuthorCompanyExt(id=1327012684962313141, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012684958118836, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3School of Control and Computer Engineering, North China Electric Power University, Beijing 102206), AuthorCompanyExt(id=1327012684970701750, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012684958118836, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3华北电力大学控制与计算机工程学院 北京 102206)])], figs=[ArticleFig(id=1327012687785079785, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, label=Figure 1, caption=Process of fine-tuning LLM with PPO, figureFileSmall=lZ026fZFWNGOLXiXPcq5nA==, figureFileBig=TFUkHvRlMl9graXj4BiZNQ==, tableContent=null), ArticleFig(id=1327012689450218475, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, label=图1, caption=使用PPO微调大语言模型的流程, figureFileSmall=lZ026fZFWNGOLXiXPcq5nA==, figureFileBig=TFUkHvRlMl9graXj4BiZNQ==, tableContent=null), ArticleFig(id=1327012689651545069, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, label=Figure 2, caption=Three computation modes for LLMs, figureFileSmall=Co81khKO5EWAKkTlJZjGpQ==, figureFileBig=zliUO0gE9KB1aIiofxw2SQ==, tableContent=null), ArticleFig(id=1327012689718653934, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, label=图2, caption=大语言模型的3种计算模式, figureFileSmall=Co81khKO5EWAKkTlJZjGpQ==, figureFileBig=zliUO0gE9KB1aIiofxw2SQ==, tableContent=null), ArticleFig(id=1327012689831900143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, label=Figure 3, caption=Time-consuming analysis of speeding up all PPO steps using 3D parallelism strategy, figureFileSmall=TUBu93FxxysKClYC32L0LQ==, figureFileBig=ROU6oj5/5pRMdasrlfhRcg==, tableContent=null), ArticleFig(id=1327012689919980528, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, label=图3, caption=使用三维并行策略加速所有PPO阶段的耗时分析, figureFileSmall=TUBu93FxxysKClYC32L0LQ==, figureFileBig=ROU6oj5/5pRMdasrlfhRcg==, tableContent=null), ArticleFig(id=1327012690003866609, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, label=Figure 4, caption=Acceleration generation step using 3D parallelism strategy, figureFileSmall=1E58PsRixR516B+GOO6VOA==, figureFileBig=wrrl8+xcoF2QXEjqWbknJA==, tableContent=null), ArticleFig(id=1327012690070975474, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, label=图4, caption=使用三维并行策略加速生成阶段, figureFileSmall=1E58PsRixR516B+GOO6VOA==, figureFileBig=wrrl8+xcoF2QXEjqWbknJA==, tableContent=null), ArticleFig(id=1327012690159055859, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, label=Figure 5, caption=Inter-stage dependent relationship analysis in different PPO algorithms, figureFileSmall=p2AjCGSHWNI7C2M69UbcWQ==, figureFileBig=V3028UY2SXOgVI0/bnm8nA==, tableContent=null), ArticleFig(id=1327012690230359028, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, label=图5, caption=不同PPO算法中阶段间依赖关系分析, figureFileSmall=p2AjCGSHWNI7C2M69UbcWQ==, figureFileBig=V3028UY2SXOgVI0/bnm8nA==, tableContent=null), ArticleFig(id=1327012690297467893, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, label=Figure 6, caption=Timing diagram of operation of generation stage under diverse bubble filling approaches, figureFileSmall=5F/Rgt7/q/jk37WbMfpeHA==, figureFileBig=GDF22XWIgF3EAuurm/Hx+w==, tableContent=null), ArticleFig(id=1327012690351993846, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, label=图6, caption=不同气泡填充方法下生成阶段的运算时序图, figureFileSmall=5F/Rgt7/q/jk37WbMfpeHA==, figureFileBig=GDF22XWIgF3EAuurm/Hx+w==, tableContent=null), ArticleFig(id=1327012690414908407, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, label=Figure 7, caption=An example of delayed inter-batch pipeline, figureFileSmall=uGzLXk0emHQtW5SSmWDLDg==, figureFileBig=qM0crw+IKaOD/eSIbYoh8Q==, tableContent=null), ArticleFig(id=1327012690482017272, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, label=图7, caption=延迟批间流水线实例, figureFileSmall=uGzLXk0emHQtW5SSmWDLDg==, figureFileBig=qM0crw+IKaOD/eSIbYoh8Q==, tableContent=null), ArticleFig(id=1327012690544931833, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, label=Figure 8, caption=Version delay of delayed inter-batch pipeline parallelism, figureFileSmall=MG+mdSwUNe3vbWlzAeBsVg==, figureFileBig=wqRqfSQfJZabIQNZQkYZxQ==, tableContent=null), ArticleFig(id=1327012690607846394, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, label=图8, caption=延迟批间流水线并行的版本延迟, figureFileSmall=MG+mdSwUNe3vbWlzAeBsVg==, figureFileBig=wqRqfSQfJZabIQNZQkYZxQ==, tableContent=null), ArticleFig(id=1327012690674955259, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, label=Figure 9, caption=Comparison of the model evaluation reward after using PPO and asynchronous PPO for fine-tuning, figureFileSmall=Y7xTAWon6xMhatXEBPecSw==, figureFileBig=VXKVOOdjM+kAPhU3E6rn3Q==, tableContent=null), ArticleFig(id=1327012690737869820, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, label=图9, caption=使用PPO和异步PPO进行微调后的模型评估奖励对比, figureFileSmall=Y7xTAWon6xMhatXEBPecSw==, figureFileBig=VXKVOOdjM+kAPhU3E6rn3Q==, tableContent=null), ArticleFig(id=1327012690834338813, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, label=Figure 10, caption=Comparision of throughput, figureFileSmall=Tnl3PYO87MqtwwiyiyeaBA==, figureFileBig=iNw1VXfO/06zsMBIhV5pZg==, tableContent=null), ArticleFig(id=1327012690905641982, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, label=图10, caption=吞吐量对比, figureFileSmall=Tnl3PYO87MqtwwiyiyeaBA==, figureFileBig=iNw1VXfO/06zsMBIhV5pZg==, tableContent=null), ArticleFig(id=1327012690976945151, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, label=Figure 11, caption=Comparison of scalability and time cost analysis, figureFileSmall=/T2pTg1Put00HKTGdqVRvA==, figureFileBig=dRKFWyKLXVt4KQJYjw23zA==, tableContent=null), ArticleFig(id=1327012691052441600, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, label=图11, caption=可扩展性对比和耗时分析, figureFileSmall=/T2pTg1Put00HKTGdqVRvA==, figureFileBig=dRKFWyKLXVt4KQJYjw23zA==, tableContent=null), ArticleFig(id=1327012691136327682, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, label=Figure 12, caption=Comparison between 3D parallelism and delayed inter-batch pipeline parallelism in generation efficiency, figureFileSmall=Rw1l9PSpsSKTvWYYAJfOhg==, figureFileBig=h8gUm3ySWqfU9LXpOlP5EQ==, tableContent=null), ArticleFig(id=1327012691211825155, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, label=图12, caption=三维并行与延迟批间流水线并行的生成效率对比, figureFileSmall=Rw1l9PSpsSKTvWYYAJfOhg==, figureFileBig=h8gUm3ySWqfU9LXpOlP5EQ==, tableContent=null), ArticleFig(id=1327012691278934020, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, label=Figure 13, caption=Time-consuming of GPT-6B for the low computational density task group and the high computational density task group, figureFileSmall=JGgFG/JQwYFYaMT+3BRK0Q==, figureFileBig=Ihm+yc95Wq0PD7BdwaduDg==, tableContent=null), ArticleFig(id=1327012691354431493, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, label=图13, caption=GPT-6B的低计算密度任务组耗时和高计算密度任务组耗时, figureFileSmall=JGgFG/JQwYFYaMT+3BRK0Q==, figureFileBig=Ihm+yc95Wq0PD7BdwaduDg==, tableContent=null), ArticleFig(id=1327012691429928966, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, label=Figure 14, caption=Fine-tuning time cost of three models obtained by combining Pipe-RLHF with DeepSpeed inference engine, figureFileSmall=KVAVeTmibdAmrU3iiK+Ofw==, figureFileBig=mM7l1u29X6hpnXbMV4HWMA==, tableContent=null), ArticleFig(id=1327012691492843527, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, label=图14, caption=对3种模型结合Pipe-RLHF和DeepSpeed推理引擎取得的微调耗时, figureFileSmall=KVAVeTmibdAmrU3iiK+Ofw==, figureFileBig=mM7l1u29X6hpnXbMV4HWMA==, tableContent=null), ArticleFig(id=1327012691559952392, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, label=Figure 15, caption=Training curves of synchronous PPO and asynchronous PPO, figureFileSmall=cj66X+0wxR5WYdbubkShIw==, figureFileBig=wnCtEJbQXU8bQp4/LhcF4A==, tableContent=null), ArticleFig(id=1327012691627061257, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, label=图15, caption=同步PPO和异步PPO的训练曲线, figureFileSmall=cj66X+0wxR5WYdbubkShIw==, figureFileBig=wnCtEJbQXU8bQp4/LhcF4A==, tableContent=null), ArticleFig(id=1327012691706753034, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, label=Figure 16, caption=Throughput of models for fine-tuning using Pipe-RLHF with different Pmax, figureFileSmall=o4IhuesSblnAqbYrcf+YJw==, figureFileBig=fWWqy/FAC0Vyx2KtjNxD/Q==, tableContent=null), ArticleFig(id=1327012691782250507, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, label=图16, caption=不同Pmax下使用Pipe-RLHF进行模型微调的吞吐量, figureFileSmall=o4IhuesSblnAqbYrcf+YJw==, figureFileBig=fWWqy/FAC0Vyx2KtjNxD/Q==, tableContent=null), ArticleFig(id=1327012691874525196, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, label=Table 1, caption=Computation Mode of Models in Each PPO Stage, figureFileSmall=null, figureFileBig=null, tableContent=
阶段名称模型种类计算模式
生成策略模型生成
推理策略模型、参考模型、奖励模型推理
训练策略模型训练
), ArticleFig(id=1327012691958411277, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, label=表1, caption=

PPO各阶段中模型的计算模式

, figureFileSmall=null, figureFileBig=null, tableContent=
阶段名称模型种类计算模式
生成策略模型生成
推理策略模型、参考模型、奖励模型推理
训练策略模型训练
), ArticleFig(id=1327012692046491662, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, label=Table 2, caption=Model Architectures in Experiments, figureFileSmall=null, figureFileBig=null, tableContent=
模型隐藏层维度注意力头数目模型层数
GPT-6B4 0963232
GPT-20B6 1446444
GPT-66B9 2167264
GPT-175B12 2889696
), ArticleFig(id=1327012692147154959, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, label=表2, caption=

实验模型结构

, figureFileSmall=null, figureFileBig=null, tableContent=
模型隐藏层维度注意力头数目模型层数
GPT-6B4 0963232
GPT-20B6 1446444
GPT-66B9 2167264
GPT-175B12 2889696
), ArticleFig(id=1327012692226846737, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=EN, label=Table 3, caption=Comparison of Computing Resource Utilization %, figureFileSmall=null, figureFileBig=null, tableContent=
模型三维并行
计算资源利用率
Pipe-RLHF
计算资源利用率
GPT-6B23.768.7
GPT-20B38.374.4
GPT-66B55.676.3
GPT-175B64.277.5
), ArticleFig(id=1327012692310732818, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, language=CN, label=表3, caption=

计算资源利用率对比

, figureFileSmall=null, figureFileBig=null, tableContent=
模型三维并行
计算资源利用率
Pipe-RLHF
计算资源利用率
GPT-6B23.768.7
GPT-20B38.374.4
GPT-66B55.676.3
GPT-175B64.277.5
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202550127, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202550127, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202550127, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202550127, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1748793600000, fullTextJson=null, articleText=null, reference=null)
收藏切换
Pipe-RLHF: 计算模式感知的RLHF并行加速框架
徐颖 1, 2 , 王梦迪 1 , 程龙 3 , 刘炼 1, 2 , 赵世新 1, 2 , 张磊 1 , 王颖 1
计算机研究与发展 | 芯片架构设计 2025,62(6): 1513-1529
收起
收藏切换
计算机研究与发展 |芯片架构设计 2025 , 62 (6) : 1513 -1529
Pipe-RLHF: 计算模式感知的RLHF并行加速框架
全屏
[Author(id=1327012685042004921, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=xuying18z@ict.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012685125891004, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685042004921, language=EN, stringName=Ying Xu, firstName=Ying, middleName=null, lastName=Xu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
2University of Chinese Academy of Sciences, Beijing 100190, bio={"content":"

born in 1996. PhD candidate. Her main research interests include machine learning system and parallel deep learning framework

"}, bioImg=null, bioContent=

born in 1996. PhD candidate. Her main research interests include machine learning system and parallel deep learning framework

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012685201388477, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685042004921, language=CN, stringName=徐颖, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1中国科学院计算技术研究所 北京 100190
2中国科学院大学 北京 100190, bio={"img":"DENf+2BfcH2ih411a8jwcg==","content":"

1996年生. 博士研究生. 主要研究方向为机器学习系统、深度学习并行框架

"}, bioImg=DENf+2BfcH2ih411a8jwcg==, bioContent=

1996年生. 博士研究生. 主要研究方向为机器学习系统、深度学习并行框架

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012683150373806, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=1, ext=[AuthorCompanyExt(id=1327012683158762415, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683167151024, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所 北京 100190)]), AuthorCompany(id=1327012683255231409, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=2, ext=[AuthorCompanyExt(id=1327012683284591538, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683255231409, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2University of Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683297174451, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683255231409, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学 北京 100190)])]), Author(id=1327012685276885951, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012685356577729, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685276885951, language=EN, stringName=Mengdi Wang, firstName=Mengdi, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, bio={"content":"

born in 1995. Assistant professor. Her main research interests include accelerator design and multi-core system

"}, bioImg=null, bioContent=

born in 1995. Assistant professor. Her main research interests include accelerator design and multi-core system

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012685415297986, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685276885951, language=CN, stringName=王梦迪, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1中国科学院计算技术研究所 北京 100190, bio={"img":"XdV6mzDHK9VkDs2B56DDUA==","content":"

1995年生. 助理研究员. 主要研究方向为加速器设计、多核系统

"}, bioImg=XdV6mzDHK9VkDs2B56DDUA==, bioContent=

1995年生. 助理研究员. 主要研究方向为加速器设计、多核系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012683150373806, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=1, ext=[AuthorCompanyExt(id=1327012683158762415, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683167151024, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所 北京 100190)])]), Author(id=1327012685482406852, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012685570487238, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685482406852, language=EN, stringName=Long Cheng, firstName=Long, middleName=null, lastName=Cheng, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, address=3School of Control and Computer Engineering, North China Electric Power University, Beijing 102206, bio={"content":"

born in 1985. PhD, professor, PhD supervisor. His main research interest includes distributed computing and high-performance systems

"}, bioImg=null, bioContent=

born in 1985. PhD, professor, PhD supervisor. His main research interest includes distributed computing and high-performance systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012685683733447, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685482406852, language=CN, stringName=程龙, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, address=3华北电力大学控制与计算机工程学院 北京 102206, bio={"img":"Gqw18szoYQR2/e8+Sby8Jw==","content":"

1985年生. 博士,教授,博士生导师. 主要研究方向为分布式计算、高性能系统

"}, bioImg=Gqw18szoYQR2/e8+Sby8Jw==, bioContent=

1985年生. 博士,教授,博士生导师. 主要研究方向为分布式计算、高性能系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012684958118836, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=3, ext=[AuthorCompanyExt(id=1327012684962313141, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012684958118836, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3School of Control and Computer Engineering, North China Electric Power University, Beijing 102206), AuthorCompanyExt(id=1327012684970701750, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012684958118836, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3华北电力大学控制与计算机工程学院 北京 102206)])]), Author(id=1327012685755036617, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012685834728396, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685755036617, language=EN, stringName=Lian Liu, firstName=Lian, middleName=null, lastName=Liu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
2University of Chinese Academy of Sciences, Beijing 100190, bio={"content":"

born in 1999. PhD candidate. His main research interest includes LLM infrastructure、hardware-software co-design

"}, bioImg=null, bioContent=

born in 1999. PhD candidate. His main research interest includes LLM infrastructure、hardware-software co-design

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012685897642957, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685755036617, language=CN, stringName=刘炼, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1中国科学院计算技术研究所 北京 100190
2中国科学院大学 北京 100190, bio={"img":"AkYDGV2VDDR9H0rj0XN4Lg==","content":"

1999年生. 博士研究生. 主要研究方向为大语言模型系统架构设计、软硬件协同设计

"}, bioImg=AkYDGV2VDDR9H0rj0XN4Lg==, bioContent=

1999年生. 博士研究生. 主要研究方向为大语言模型系统架构设计、软硬件协同设计

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012683150373806, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=1, ext=[AuthorCompanyExt(id=1327012683158762415, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683167151024, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所 北京 100190)]), AuthorCompany(id=1327012683255231409, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=2, ext=[AuthorCompanyExt(id=1327012683284591538, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683255231409, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2University of Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683297174451, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683255231409, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学 北京 100190)])]), Author(id=1327012685964751823, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012686069609426, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685964751823, language=EN, stringName=Shixin Zhao, firstName=Shixin, middleName=null, lastName=Zhao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
2University of Chinese Academy of Sciences, Beijing 100190, bio={"content":"

born in 1999. PhD candidate. Her main research interest includes computer architecture

"}, bioImg=null, bioContent=

born in 1999. PhD candidate. Her main research interest includes computer architecture

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012686161884115, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012685964751823, language=CN, stringName=赵世新, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1中国科学院计算技术研究所 北京 100190
2中国科学院大学 北京 100190, bio={"img":"JQwhJoDWwZGRTG1do7/HbQ==","content":"

1999年生. 博士研究生. 主要研究方向为计算机体系结构

"}, bioImg=JQwhJoDWwZGRTG1do7/HbQ==, bioContent=

1999年生. 博士研究生. 主要研究方向为计算机体系结构

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012683150373806, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=1, ext=[AuthorCompanyExt(id=1327012683158762415, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683167151024, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所 北京 100190)]), AuthorCompany(id=1327012683255231409, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=2, ext=[AuthorCompanyExt(id=1327012683284591538, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683255231409, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2University of Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683297174451, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683255231409, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2中国科学院大学 北京 100190)])]), Author(id=1327012686224798677, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, orderNo=5, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012686312879063, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012686224798677, language=EN, stringName=Lei Zhang, firstName=Lei, middleName=null, lastName=Zhang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, bio={"content":"

born in 1981. PhD, professor. Member of CCF. His main research interests include chip and computing system

"}, bioImg=null, bioContent=

born in 1981. PhD, professor. Member of CCF. His main research interests include chip and computing system

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012686379987928, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012686224798677, language=CN, stringName=张磊, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1中国科学院计算技术研究所 北京 100190, bio={"img":"SG3c/MaOAOCQlehMUFcuog==","content":"

1981年生. 博士,正高级工程师. CCF会员. 主要研究方向为芯片、计算系统

"}, bioImg=SG3c/MaOAOCQlehMUFcuog==, bioContent=

1981年生. 博士,正高级工程师. CCF会员. 主要研究方向为芯片、计算系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012683150373806, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=1, ext=[AuthorCompanyExt(id=1327012683158762415, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683167151024, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所 北京 100190)])]), Author(id=1327012686438708186, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, orderNo=6, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327012686530982876, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012686438708186, language=EN, stringName=Ying Wang, firstName=Ying, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, bio={"content":"

born in 1985. PhD, professor. Senior member of CCF. His main research interests include novel EDA, and processor and memory system architecture

"}, bioImg=null, bioContent=

born in 1985. PhD, professor. Senior member of CCF. His main research interests include novel EDA, and processor and memory system architecture

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327012686627451869, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, authorId=1327012686438708186, language=CN, stringName=王颖, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1中国科学院计算技术研究所 北京 100190, bio={"img":"oq44dw3DfE6YPKNN0iX3VQ==","content":"

1985年生. 博士,研究员. CCF高级会员. 主要研究方向为新型EDA、处理器与存储系统体系结构

"}, bioImg=oq44dw3DfE6YPKNN0iX3VQ==, bioContent=

1985年生. 博士,研究员. CCF高级会员. 主要研究方向为新型EDA、处理器与存储系统体系结构

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327012683150373806, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, xref=1, ext=[AuthorCompanyExt(id=1327012683158762415, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1327012683167151024, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327012676691145592, companyId=1327012683150373806, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所 北京 100190)])])]
徐颖1, 2 , 王梦迪1, 程龙3, 刘炼1, 2, 赵世新1, 2, 张磊1, 王颖1
作者信息
  • 1中国科学院计算技术研究所 北京 100190
  • 2中国科学院大学 北京 100190
  • 3华北电力大学控制与计算机工程学院 北京 102206
作者简介:

1996年生. 博士研究生. 主要研究方向为机器学习系统、深度学习并行框架

1995年生. 助理研究员. 主要研究方向为加速器设计、多核系统

1985年生. 博士,教授,博士生导师. 主要研究方向为分布式计算、高性能系统

1999年生. 博士研究生. 主要研究方向为大语言模型系统架构设计、软硬件协同设计

1999年生. 博士研究生. 主要研究方向为计算机体系结构

1981年生. 博士,正高级工程师. CCF会员. 主要研究方向为芯片、计算系统

1985年生. 博士,研究员. CCF高级会员. 主要研究方向为新型EDA、处理器与存储系统体系结构

Pipe-RLHF: A Computation Mode-Aware Parallel Framework for RLHF
Ying Xu1, 2 , Mengdi Wang1, Long Cheng3, Lian Liu1, 2, Shixin Zhao1, 2, Lei Zhang1, Ying Wang1
Affiliations
  • 1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
  • 2University of Chinese Academy of Sciences, Beijing 100190
  • 3School of Control and Computer Engineering, North China Electric Power University, Beijing 102206
  • born in 1996. PhD candidate. Her main research interests include machine learning system and parallel deep learning framework

    born in 1995. Assistant professor. Her main research interests include accelerator design and multi-core system

    born in 1985. PhD, professor, PhD supervisor. His main research interest includes distributed computing and high-performance systems

    born in 1999. PhD candidate. His main research interest includes LLM infrastructure、hardware-software co-design

    born in 1999. PhD candidate. Her main research interest includes computer architecture

    born in 1981. PhD, professor. Member of CCF. His main research interests include chip and computing system

    born in 1985. PhD, professor. Senior member of CCF. His main research interests include novel EDA, and processor and memory system architecture

出版时间: 2025-06-02 doi: 10.7544/issn1000-1239.202550127
文章导航
收藏切换

基于人类反馈的强化学习(reinforcement learning with human feedback,RLHF)作为当前大语言模型(large language models,LLMs)对齐的主流方法,其核心优化算法——近端策略优化(proximal policy optimization,PPO)却面临着显著的效率问题. PPO由生成、推理、训练3个相互关联的阶段组成,各个阶段有着不同的计算特性. 然而,现有的RLHF并行框架采用相同并行策略顺序执行PPO的所有阶段,这导致以下2个问题:其一,生成阶段不能充分利用计算资源,进而影响整体效率;其二,阶段间严格串行执行,未能充分利用潜在并行性. 针对上述问题,提出了一个新型RLHF并行框架——Pipe-RLHF.该框架能够自适应地根据各阶段的计算特征确定最优并行策略,突破现有阶段串行范式,采用异步PPO算法发掘阶段间的并行性. 具体而言,创新性地提出了适用于PPO生成阶段的延迟批间流水线并行方法,显著提升了该阶段的计算资源利用率;再次,使用异步PPO解放阶段间的依赖关系,将阶段间并行应用到PPO的加速上;最后,针对PPO算法的整体优化,构建了分层并行策略空间,并提出了一套优化算法以实现该空间中的最优解搜索. 通过在多个大语言模型上的性能评估实验表明,相较于现有方法,Pipe-RLHF最高可实现3.7倍的加速比,充分验证了该框架的有效性和优越性.

基于人类反馈的强化学习  /  近端策略优化  /  大模型微调  /  分布式系统  /  并行计算

Reinforcement learning with human feedback (RLHF) has been proven effective in aligning large language models (LLMs) with human preferences. The most costly part of RLHF is proximal policy optimization (PPO), which consists of three dependent steps. Different PPO steps in RLHF exhibit different computation modes, simply employing the same parallelization strategy to accelerate all steps that involve multiple model variants, as done in existing frameworks, will lead to poor performance in the PPO generation step due to insufficient utilization of computational resources. Thus, we introduce Pipe-RLHF, a parallelism framework for RLHF fine-tuning, which adaptively employs distinct parallelization strategies for different steps according to the computation mode. Specifically, we first investigate the characteristics of various computation modes to explore their best-fit parallelization approach. And then, we present a novel delayed inter-batch pipeline parallelization approach specifically designed for the PPO generation step, enabling the sufficient utilization of computational resources. Subsequently, based on the proposed inter-batch pipeline parallelization approach, we define a hierarchical parallel plan space for distributed RLHF fine-tuning. Finally, we present optimization algorithms to find the optimal parallelization plan from the defined hierarchical parallel plan space to minimize the overall time consumption. Implementation and evaluation across multiple LLMs demonstrates that the proposed Pipe-RLHF achieves 3.7 times speedup compared with existing methods while achieving near-linear scalability.

reinforcement learning with human feedback (RLHF)  /  proximal policy optimization (PPO)  /  large language models fine-tuning  /  distributed systems  /  parallel computing
徐颖, 王梦迪, 程龙, 刘炼, 赵世新, 张磊, 王颖. Pipe-RLHF: 计算模式感知的RLHF并行加速框架. 计算机研究与发展, 2025 , 62 (6) : 1513 -1529 . DOI: 10.7544/issn1000-1239.202550127
Ying Xu, Mengdi Wang, Long Cheng, Lian Liu, Shixin Zhao, Lei Zhang, Ying Wang. Pipe-RLHF: A Computation Mode-Aware Parallel Framework for RLHF[J]. Journal of Computer Research and Development, 2025 , 62 (6) : 1513 -1529 . DOI: 10.7544/issn1000-1239.202550127
基于人类反馈的强化学习(reinforcement learning with human feedback,RLHF)作为一种经典的强化学习算法,已在游戏操作[1-3]、机器人控制[4-6]及推荐系统[7]等多个领域取得了显著成效. 近期研究表明,RLHF在指导大语言模型(large language models,LLMs)学习符合人类偏好和道德基准的行为方面展现出卓越性能,从而成为大语言模型价值观对齐的主流方法[8-10]. 目前,RLHF已被广泛应用于多个先进大语言模型的微调过程中,包括Open-AI的GPT-4[11]、Anthropic的Claud[12]、DeepMind 的Sparrow[13]、Google的 Gemini[14]、以及DeepSeek[15]等. 这充分体现了RLHF在大语言模型对齐领域的重要地位和实际价值.
使用RLHF对大模型进行微调主要包含3个关键环节:监督微调(supervised fine-tuning,SFT)、奖励模型训练和近端策略优化(proximal policy optimization,PPO)[9,16]. PPO作为RLHF的核心环节[17-18],因其高复杂度和高计算开销成为了整个RLHF流程的性能瓶颈[19]. PPO的每个训练迭代过程包含3个紧密关联的阶段,分别对应大语言模型中生成、推理和训练3种不同计算模式. 首先,在生成阶段,被微调的大模型(即策略模型)根据输入的提示词(prompt)生成一批输出文本. 随后,在推理阶段,这些输出文本将被同时输入到奖励模型、SFT模型以及策略模型本身,用于计算奖励值(reward)等关键统计量. 最后,在训练阶段,基于前2个阶段产生的文本和统计量,对策略模型进行参数更新和优化. 值得注意的是,PPO的各阶段之间存在着密切的依赖关系,包括数据传递和参数同步等. 在分布式计算环境下,如何对这些阶段涉及的模型和计算任务进行切分与调度,将直接影响大语言模型微调的效率和收敛性能. 因此,本文研究致力于解决一个关键问题:如何在不破坏阶段间的数据关联的前提下,将RLHF这一涉及多模型协同的复杂负载高效地映射到分布式系统上,从而实现最优的执行效率.
当前,针对RLHF微调流程,已有多个分布式框架被提出,包括Trl[20],Trlx[21],DeepSpeed-Chat[19]等,这些框架普遍采用三维并行(3D parallelism)方法来加速PPO的所有阶段,并对所有阶段实施相同的并行策略,这种统一的并行方案存在显著的优化空间. 尽管三维并行方法在大模型训练和推理阶段表现出良好的加速效果[22-23],但其在生成阶段的性能却不尽如人意,这主要源于以下原因:大模型在文本生成阶段本身的计算密度较低,而三维并行方法进一步对算子进行切分,导致计算密度进一步下降,从而造成计算资源利用率的严重不足. 因此,采用统一的三维并行策略来加速PPO的所有阶段是一种次优方案,其导致的生成阶段耗时增加将显著影响整个RLHF流程的效率. 此外,现有框架普遍采用顺序执行PPO各阶段的方式,因此仅能关注到单个阶段内部的并行优化机会. 这种设计思路未能从PPO阶段的全局视角出发,探索更丰富的并行优化可能性,例如阶段间的并行执行机会. 这一局限性进一步限制了现有框架的性能提升空间.
为了解决上述问题,本文研究提出了一个高效的RLHF大模型微调并行框架—Pipe-RLHF. 不同于现有工作仅关注于单个PPO阶段内部并行优化,Pipe-RLHF从PPO的整体行为出发,统筹考虑所有PPO阶段的特性与关联性,充分挖掘PPO中的并行机会. 本文研究的主要创新和贡献有3点:
1) 针对PPO各阶段计算模式的差异性,深入分析各阶段特性. 基于分析结果,创新性地提出了适用于PPO生成阶段的延迟批间流水线并行方法,显著提升了生成阶段的计算资源利用率.
2) 验证了异步PPO算法在大模型微调中的有效性,打破了现在大模型微调中PPO的3个阶段必须依次执行的桎梏,使得阶段间并行成为可能.
3) 构建了一套基于PPO优化过程全局的分层并行策略空间,并设计了一套高效的优化算法,能够在该空间中快速求解最优并行策略. 这一创新为RLHF框架的并行优化提供了系统性的解决方案.
为验证所提出并行框架Pipe-RLHF的有效性,本文研究在不同规模的大语言模型上开展了系统性实验评估. 实验结果表明,与现有主流框架相比,Pipe-RLHF在性能优化方面取得了显著提升,最高可实现3.7倍的加速比. 这一结果充分证明了本文所提出方法的优越性和实用价值.
RLHF是一种融合了强化学习与人类主观判断的训练技术. 该技术通过将人类偏好转化为奖励信号来指导模型优化,使模型能够习得那些难以通过传统奖励函数定义的复杂行为模式[13,24]. 近期研究表明,RLHF在大语言模型微调领域展现出显著的应用价值,特别是在实现大语言模型与人类偏好对齐方面具有巨大潜力[16,25-26]. 具体而言,将RLHF用于大模型微调包含3个环节[9,16]:
1) 监督微调环节. 首先由专业人员构建人工标注的示例数据集,即针对一组提示词,人工生成相应的回复,形成问答对集合. 随后,利用该数据集对预训练模型进行初步微调,使其具备基本任务执行能力. 该环节输出的模型称为SFT模型,作为后续优化的基础.
2) 奖励模型训练环节. 我们首先收集人类偏好数据集. 具体做法为:根据给定提示词生成多个回复,并由人类进行排序,这些提示词、回复、排序共同组成了偏好数据集里的一条数据. 随后基于偏好数据集训练奖励模型,使其评分能够准确反映人类偏好特征. 一般而言,奖励模型从SFT初始化,添加一个线性层来获取表示分值的标量.
3) PPO环节,又称为强化学习环节. 此环节通过强化学习算法优化大语言模型,使其生成满足人类偏好的高质量输出. 在此阶段,大模型以SFT模型为起点,以最大化奖励模型评分为目标,应用PPO算法进行模型微调. PPO是一种强健有效的强化学习算法,在RLHF中起到了重要作用,是RLHF算法的核心,同时也是RLHF算法中最复杂且耗时的环节.
PPO作为OpenAI提出的经典强化学习算法,凭借其卓越的性能和良好的易用性,在自然语言处理等多个领域得到了广泛应用[18]. 本节将详细介绍使用PPO对大语言模型进行微调的过程.
典型的PPO大语言模型微调系统通常包含多个协同工作的模型组件,主要包括:策略模型(actor model),即待优化的主体模型;参考模型(reference model),即RLHF第1阶段获得的SFT模型;奖励模型(reward model),基于人类偏好数据训练的打分模型;评论家模型,通常通过在策略模型主体上添加线性层实现,与策略模型共享主体参数[20-21,27]. PPO的完整训练流程如图1所示,包含生成、推理、训练3个关键阶段.
1)生成阶段. 策略模型采用自回归生成方式,基于给定的提示词集合逐token地生成相应的输出文本.
2)推理阶段. 将生成阶段的提示词及其对应的输出文本拼接成完整的文本,然后输入到参考模型、奖励模型、策略模型中,分别获得参考概率分布、评分、价值和初始概率分布等统计量.
3)训练阶段. 策略模型首先计算完整文本的当前概率分布. 随后,将当前概率分布与推理阶段获得的统计量(包括参考概率分布、奖励、价值估计等)相结合,共同计算得到损失函数. 基于该损失函数,系统执行反向传播算法获取梯度,进而对策略模型的参数进行迭代更新. 本阶段通常进行多次模型更新. 当仅执行单次更新时,当前概率分布可直接复用初始概率分布计算结果.
如图2所示,大语言模型主要存在3种计算模式,每种模式具有独特的计算特征:
推理模式作为最基础的计算模式,仅包含1个前向传播过程. 推理模式下的输入通常是一批文本,张量尺寸为[b, s, h],其中b为批次大小,s为文本长度,h为隐藏层维度.
训练模式由前向传播与反向传播共同构成,其输入张量尺寸为[b, s, h]. 具体而言,模型首先通过前向传播计算损失函数,随后执行反向传播获取参数梯度,最终利用这些梯度更新模型参数.
生成模式采用自回归机制,通过多次前向传播迭代完成. 每个前向传播以一批token作为输入生成新的token,且新生成的token会作为下次前向传播的输入,构成迭代处理链路. 值得注意的是,生成模式的输入数据尺寸为[b, 1,h],与推理模式和训练模式存在显著差异.
目前主流的针对单模型的并行方法主要有三维并行和ZeRO.
三维并行作为当前大语言模型并行计算的标杆性方法,结合了数据并行、张量并行和流水线并行3种策略,显著提升了模型的训练效率[28-29]. 其中,数据并行将输入数据均匀切分成多个小批次,并将它们分配到不同的GPU上,每个GPU都持有模型的全部参数,各个GPU并行单独计算本地的参数梯度,而后通过all-reduce聚合梯度并更新参数. 不同于数据并行,张量并行是将模型参数进行划分,每个GPU持有全部的输入数据,但仅存储每层的部分参数,每个GPU只计算每层的部分输出或中间结果,然后通过all-reduce或者all-gather等通信原语重构完整结果,每个GPU只负责本地参数的更新[30]. 流水线并行以层为单位将模型划分为多段,每段包含多层[31-32]. 在基础流水线并行方案中,当前段需要等待之前段结束之后才能开始执行,这使得计算资源因等待充斥着大量的空闲时间,我们称之为“气泡”. 为最小化气泡影响,通常将输入数据切分为多个微批次(micro-batches),通过流水线执行策略提高资源利用率[31],本文将这种方法称为“批内流水线并行”.
ZeRO[33]由微软公司的DeepSpeed团队提出,是数据并行的变体. 不同于传统数据并行中每个GPU都存储完整模型副本的做法,ZeRO通过切分参数、梯度、优化器状态来消除存储冗余. 根据切分范围,ZeRO分为3个递进的阶段:ZeRO-1,ZeRO-2和ZeRO-3. ZeRO-1是最基本的ZeRO阶段,仅切分优化器状态,即每个GPU存储完整的参数、梯度和部分优化器状态. 在每块GPU计算得到本地梯度后,先使用ReduceScatter获取本地优化器状态对应的全局梯度,然后更新本地参数,最后通过AllGather获取全部的最新参数. ZeRO-1的通信开销及内存节省与模型可学习参数的数量成正比. ZeRO-2在ZeRO-1的基础上,进一步对梯度进行了切分,在使用ReduceScatter获取本地优化器状态对应的全局梯度后,释放优化器状态不在本地的梯度所占用的存储空间. ZeRO-2的通信开销与ZeRO-1相同,但进一步节省了存储空间. ZeRO-3对参数、梯度、优化器状态全部进行切分. 因其每个GPU仅存储部分参数,故而在每个算子进行前向传播、反向传播前,均需要进行AllGather从其他GPU获取完整参数,因此其通信开销及节省内存与模型总参数量成正比.
在描述我们的研究动机之前,有必要明确以下关键术语的定义:
1)并行方法. 指特定的并行范式,通常由任务的可切分维度定义. 例如,数据并行、模型并行以及三维并行等都属于并行方法的范畴.
2)并行策略. 指在特定并行方法下的具体任务的切分方案. 例如,当我们使用三维并行在64个GPU上对大模型进行并行训练时,[2, 4, 8]就是一个可行的三维并行策略,含义为在数据维度的切分为2,张量维度的切分为4,流水线维度的切分为8.
如表1所示,PPO中的3个阶段分别对应3种不同的计算模式. 然而,现有RLHF框架忽略了这一重要特性,采用统一的三维并行策略对所有阶段进行加速. 我们对这一方法进行了复现并量化分析了该方法在各阶段的耗时分布. 复现实验中SFT模型采用GPT-J-6B架构,且所有模型均从SFT模型初始化. 遵循现有框架的做法,对于每一个集群配置,我们通过网格搜索获取训练阶段的最优三维并行策略,并将其应用于推理阶段和生成阶段.
如图3所示,相同并行策略在不同计算模式下呈现出显著的性能差异. 在三维并行策略下,PPO的训练阶段和推理阶段均能稳定获得良好的加速效果,而增加GPU给生成阶段带来的加速效益随GPU数量的增加趋于无. 换言之,在训练阶段表现优秀的三维并行策略并不适用于生成阶段的加速,现有框架对所有阶段统一采用相同三维并行策略的做法,最终导致生成阶段成为PPO算法的性能瓶颈,严重制约了整体效率的提升. 已知训练阶段使用的三维并行策略不适用于生成阶段,我们进一步探讨了为生成阶段单独优化三维并行策略的可行性. 图4展示了通过网格搜索为生成阶段专门求取三维并行策略的性能表现. 实验结果表明,现有的由数据并行、张量并行、批内流水线并行组成的三维并行方法在生成阶段的加速效果随GPU数量的增加而显著降低,甚至在GPU数量超过一定数值后彻底失效. 这种现象的根源在于,生成阶段本身具有较低的计算密度,而现有的三维并行方法对算子进行切分,导致计算密度进一步降低,最终导致计算负载不足以充分利用GPU的计算资源,三维并行失效. 因此,现有的三维并行方法并不适用于加速GPU数目较多时的生成阶段,其大规模GPU集群的生成阶段加速场景中存在根本性局限.
综上所述,现有RLHF并行框架和方法在生成阶段面临计算密度过小和计算资源利用率不足的关键挑战. 为了解决这个问题,我们提出以下2个优化方向:其一为优化资源配置,通过减少参与生成阶段的GPU数量,降低算子切分程度,从而提高计算密度;其二为开发新型并行方法,避免或最小化算子层面的切分. 以上述2点为出发点,我们开始设计新的RLHF并行框架.
本节将详细介绍Pipe-RLHF的框架设计,主要包括阶段间并行、延迟批间流水线并行、策略空间形式化和并行策略优化算法.
我们根据第2节测量的PPO各个阶段的并行特征,将PPO的3个阶段划分为2个任务组:
1) 高计算密度任务组,包括推理阶段和训练阶段.
2)低计算密度任务组,包括生成阶段.
我们的目标是减少低计算密度任务组的GPU数目,在GPU总数目不变的前提下,一个可行的思路是将总GPU资源划分为2个小的资源组,分别分配给高计算密度任务组和低计算密度任务组. 然而在现有的PPO算法中,各个阶段之间都存在依赖关系导致2个任务组之间无法同时计算,必须相互等待,这会造成资源的浪费,如图5(a)所示.
为了解决这个问题,我们使用异步PPO算法,该算法的核心思想是允许生成文本的策略模型和当前训练的策略模型不同,即我们可以利用先前策略模型进行文本生成. 异步强化学习已被广泛应用于各种分布式系统中,其有效性已得到充分证明[32-33].
异步PPO算法的阶段间依赖关系如图5(b)所示,该算法通过允许使用历史参数版本的策略模型生成的数据来训练当前策略模型,从而缓解了生成阶段与训练阶段间的参数版本依赖,使得阶段间并行成为可能. 为简化表示,本文研究将3个阶段重新组织为2个任务组,任务组间的依赖关系如图5(c)所示,可以看到,参数版本相同的任务组间不存在依赖关系,可以并行执行.
因此,基于异步PPO算法,本文将总GPU资源划分为2组,分别分配给高计算密度任务组和低计算密度任务组,实现2组任务并行处理. 如图5(d)所示,因为任务组间的参数版本依赖关系的约束(图5(c)中虚线箭头),系统整体的执行呈现流水线模式.
本节旨在探索一种适用于PPO生成阶段的并行计算方法. 基于第2节的分析结果,生成阶段应避免对算子内部进行切分,以防止计算密度的进一步降低. 现有的数据并行和张量并行方法均涉及算子层面的切分,因此无法满足需求;而原始的流水线并行方法以层为单位对模型进行切分,未涉及算子内部的切分,符合本文研究的优化目标.
以流水线并行为基础,本文进一步探讨生成阶段的并行优化问题. 生成阶段是一个自回归的过程,由多个首尾相接的前向传播组成,简单地进行流水线并行会导致大量的“气泡”. 如图6(a)所示,当模型以层为单位被切分为2段时,在执行单个前向传播过程中,第2段计算必须等待第1段计算完成后才能启动. 此外,由于自回归特性,下一个前向传播必须等待前一个前向传播完成后才能开始. 这种相互等待机制使得“气泡”比例高达50%,且随着切分段数的增加,“气泡”比例会进一步上升. 为解决这一问题,必须引入有效的“气泡”填充机制.
为了解决“气泡”填充问题,本文使用其他批数据的计算来进行“气泡”填充,而非现在常见的将输入数据切分为多个微批的填充方式. 因为该做法对输入数据进行了切分,不符合我们避免对算子内部进行切分的需求. 如图6(b)所示,本文研究采用的方法是在第一批输入数据D1进行第2段计算的同时,启动第2批输入数据D2在第1段的计算. 通过这种设计,2批数据的计算能够相互填充,从而有效消除“气泡”.
生成阶段会执行多次前向传播,即输入数据D1会经历很多次第2段计算,我们要考虑在D1的哪一个第2段计算时插入D2. 如图6(b)所示,最直观的做法是在D1的首个第2段计算就插入D2,但这种做法破坏了生成阶段的有效批大小(batch size),进而损坏模型的训练质量. 例如,当D1的有效批大小为B,切分段数为PN时,图6(b)中的办法会突发式地集中生成B×PN个文本,而这B×PN个文本会被应用到接下来的训练阶段,造成PPO有效批大小变动为B×PN. 有效批大小是一个高度敏感的超参数,贸然改动可能会导致不可预料的后果,在有些情况下会严重影响模型训练质量. 例如,在情感分析任务中,将有效批大小从32增加到128,会导致微调后的策略模型的评分严重下降(从2.4降到2.126).
为了在填充气泡的同时保持模型的训练质量,本文研究引入了延迟调度策略. 具体方法如图6(c)所示,假设一批数据的首个第1段计算开始到最后1个第1段计算结束的时间间隔为T,模型切分段数为PN,则我们在$ T/ PN$时插入下一批数据. 为了更加清晰地说明所使用的延迟调度方法,我们给出如下实例:
假设模型被切分为2段,每段的计算耗时为1秒,生成阶段新生成的token数目为128,即每批数据要经历128次前向传播,则延迟调度下生成阶段的执行情况如图7所示. D1分别在1,3,5,…,255秒时执行第1段计算,在2,4,…,256秒时执行第2段计算,在256秒时输出由D1生成的文本. 则T=255,PN=2,255/2=127.5,最接近该时刻的下一段空闲时间为第128秒,我们在该空闲插入D2,则D2在128,130,…,382秒执行第1段计算,在129,131,…,383秒执行第2段计算,最终在第383秒时输出文本. 同理,D3将在257秒时插入,在512秒时输出;D4将在384秒时插入,在639秒时输出等.
通过上述实例可以看到延迟调度可以有效解决“气泡”问题,除了D2插入之前的启动阶段存在“气泡”,当D2插入后“气泡”被完全填充.
接下来延迟调度是否会破坏PPO的有效批大小. 从整体来看,每批数据的总计算时长为T,且每过T/NP的时间开启一批新的数据计算,其执行时间如图8中的红色方块所示. 我们可以将其等效为一系列单批处理时常为T/PN的任务块,该任务块每过T/PN的时间产生一个输出. 为使整体PPO框架性能最大化,与该任务块进行流水线并行的高计算密度度任务组的单批耗时也应约等于P/PN. 系统整体的执行时间如图8所示,与图5(d)相比,唯一的差别为2个任务组间的参数版本延迟由1变成了PN,即第i版模型进行训练时,使用的是i-PN版模型生成的数据,其他方面与异步PPO完全相同. 为了进一步验证异步调度的有效性,本文选取文本总结、情感分析2个任务场景,分别使用原始PPO算法和延迟版本数目为4的异步PPO算法对GPT-1.3B进行微调,其结果如图9所示,经过异步PPO微调的大模型质量与原始PPO相差无几,甚至在文本总结任务上优于原始PPO的微调结果,该实验进一步验证了异步PPO不会伤害模型的微调质量.
我们将本节中提出的批间流水线与延迟调度结合的并行方法称为延迟批间流水线并行. 该方法专为PPO生成阶段设计,可以有效避免对算子内部进行切分,同时通过延迟调度,在保证模型微调质量的前提下加速生成阶段.
Pipe-RLHF并行框架采用2级并行设计:第1级并行为阶段间并行,即低计算密度任务组和高计算密度任务组以流水线的形式并行计算;第2级为2个任务组内的并行. 具体来说,对于高计算密度任务组,即推理阶段和训练阶段,我们使用传统的由数据并行、张量并行、批内流水线并行组成的三维并行方法,而对于低计算密度任务组,即生成阶段,我们使用由数据并行、张量并行和批间流水线并行组成的并行方法,因为该并行方法除了引入数据切分并行、张量切分并行、层间切分并行之外,还引入了批间并行,我们将其称之为3.5维并行方法.
相应地,两级并行的策略空间可以分层形式化为:对于阶段间并行,其对应的并行策略空间可以形式化为$({N_{\rm l}},{N_{\rm h}})$,其中${N_{\rm l}}$为分配给低密度组的GPU数目,${N_{\rm h}}$为分配给高密度组的GPU数目. 对于任务组内并行,其策略空间可以形式化为$(dp,tp,pp)$,其中,$pp$在表示高计算密度任务组的并行策略时,其含义为批内流水线并行度,而表示低计算密度任务组的并行策略时,其含义为延迟批间流水线并行度.
对上述策略空间求解可以形式化为如下问题:
$ {\rm{obj}}:\mathop {\min }\limits_{({N_{\rm l}},{N_{\rm h}},d{p_{_{\rm l}}},t{p_{_{\rm l}}},p{p_{\rm l}},d{p_{\rm h}},t{p_{\rm h}},p{p_{\rm h}})} {T_{\rm PPO}} \text{,}$
$ {\rm{sbj}}:{T_{\rm PPO}} = \max ({T_{\rm l}}({N_{\rm l}}),{T_{\rm h}}({N_{\rm h}}))+T_{\rm comm} $
$ {T_{\rm l}}({N_{\rm l}}) = {T^{3.5\rm D}}({\pi ^{\rm a}},d{p_{\rm l}},t{p_{\rm l}},p{p_{\rm l}})\text{,} $
$ {T_{\rm h}}({N_{\rm h}}) = {T^{\rm 3 D}}\,\,({\pi ^{\rm a}},{\pi ^{\rm r}},{\pi ^{\rm ref}},d{p_{\rm h}},t{p_{\rm h}},p{p_{\rm h}}) \text{,}$
$ {N_{\rm l}} + {N_{\rm h}} \le N \text{,}$
$ d{p_{\rm l}} \times t{p_{\rm l}} \times p{p_{\rm l}} \le {N_{\rm l}}\text{,} $
$ d{p_{\rm h}} \times t{p_{\rm h}} \times p{p_{\rm h}} \le {N_{\rm h}}\text{,} $
其中,${\pi ^{\rm a}}$,${\pi ^{\rm r}}$,${\pi ^{\rm ref}} $分别为策略模型、奖励模型和参考模型. Tcomm为2个任务组间进行参数同步的耗时,由2个任务组内策略模型的参数分布决定.
本文设计了一套启发式的算法来求解上述问题,如算法1所示. 遍历所有的合法配置$({N_{\rm l}},{N_{\rm h}})$,根据式(2)求取各个合法配置下的${T_{\rm PPO}}$ ,取结果最小的配置. ${S_{\rm l}}$ 和${S_{\rm h}}$ 分别表示2个任务组的内部并行策略,其格式为$S = (dp,tp,pp)$. 本文研究采用了一种算法优化技巧:按照${N_{\rm l}}$从小到大的顺序进行遍历,当出现局部极小点,则停止遍历,该局部极小点就为所求最优点,该优化技巧的有效性证明如下:
已知低密度任务组耗时$ {T_{\rm l}}({N_{\rm l}}) $ 随着${N_{\rm l}}$的增大单调递减,高密度任务组耗时$ {T'_{\rm h}}({N_{\rm l}}) = {T_{\rm h}}(N - {N_{\rm l}}) $,其随着${N_{\rm l}}$的增大单调递增,则所求目标函数$f({N_{\rm l}}) = \max ({T_{\rm l}}({N_{\rm l}}), T{'_{\rm h}}({N_{\rm l}}))$在$ {T_{\rm l}} $和$ T{'_{\rm h}} $的交点k处取得最小值. 具体分析为:
当${N_{\rm l}} = k$时,$ {T_{\rm l}}(k) = {T'_{\rm h}}(k) = f(k) $,
当${N_{\rm l}} < k$时,$f({N_{\rm l}}) = {T_{_{\rm l}}}({N_{\rm l}}) > {T_{\rm l}}(k) = f(k)$,
当${N_{\rm l}} > k$时,$f({N_{\rm l}}) = T'{}_{\rm h}({N_{\rm l}}) > T'_{\rm h}(k) = f(k)$.
因此$k$是目标函数$f({T_{\rm l}}) = \max ({T_{\rm l}},{T_{\rm h}})$的局部极小点,同时也是全局最小点.
算法1. PPO的并行优化.
① procedure AutoParallelPPO(N, $\text{π}^{\rm a} $, $\text{π}^{\rm r} $, $\text{π}^{\rm ref} $);
②  Buff = $\varnothing $;
③ for Nl $\in $ {1, 2, …, N − 1} do
④   Nh = N − Nl;
⑤   Tl (Nl), Sl = AutoParallel3.5D($\text{π}^{\rm a} $, Nl);
⑥   Th (Nh), Sh = AutoParallel3D($\text{π}^{\rm a} $, $\text{π}^{\rm r} $, $\text{π}^{\rm ref} $, Nh);
⑦  使用通信模拟器获取Tcomm (Sl, Sh);
⑧  get TPPO according to 式(2);
⑨  if出现局部极小值
⑩   return Nl, Nh, Sl, Sh;
⑪  else
⑫   add {TPPO, Nl, Nh, Sl, Sh} to Buff;
⑬  end if
⑭ end for
⑮ end procedure
此外, ${T_{\rm comm}}$ 相对$\max ({T_{\rm l}},{T_{\rm h}})$来说往往较小,式(2)中由$\max ({T_{\rm l}},{T_{\rm h}})$占主导地位,因此$\max ({T_{\rm l}},{T_{\rm h}})$的最小值即为${T_{\rm PPO}}$的最小值.
阶段内并行策略的求取采用网格搜索法,详见算法2和算法3. 本文使用多进程并行评估网格上所有的合法点,以加速算法的执行速度. 对于生成阶段,为了保证异步PPO算法的有效性,参数版本延迟不宜过大,故而对延迟批间流水线并行度进行限制,规定其不可大于阈值Pmax(算法3行④).
算法2. 训练阶段和推理阶段的并行优化.
① procedure AutoParallel3D(Nh, $\text{π}^{\rm a} $, $\text{π}^{\rm r} $, $\text{π}^{\rm ref} $);
②  Buff = $\varnothing $;
③ for tp $\in $ {1, 2, …, Nh} do
④   $ {\rm{for}}\text{ }pp\text{ }\in \left\{1,2,…,\dfrac{{N}_{\rm h}}{tp}\right\}{\rm {do}} $
⑤    $dp \leftarrow \dfrac{{{N_{\rm h}}}}{{tp \times pp}}$;
⑥   if simu_mem_3D$(\text{π}^{\rm a}, \text{π}^{\rm r}, \text{π}^{\rm ref}, dp, tp, pp) \leqslant $     memory_limit then
⑦     t ← simu_time_3D$(\text{π}^{\rm a}, \text{π}^{\rm r}, \text{π}^{\rm ref}, dp, tp, pp) $;
⑧    add {t, dp, tp, pp} to Buff;
⑨   end if
⑩  end for
⑪ end for
⑫  topt = $ \mathop{\min}\limits_t(Buf f ) $;
⑬  dpopt, tpopt, ppopt=$\mathop {{\rm arg}\min}\limits_t(Buf f ) $;
⑭ return topt, (dpopt, tpopt, ppopt);
⑮ end procedure
算法3. 生成阶段的并行优化.
① procedure AutoParallel3.5D(Nl, $\text{π}^{\rm a} $);
②  Buff = $\varnothing $;
③ for tp $\in $ {1, 2, …, Nl} do
④   ${\rm{ for}}\text{ }pp \in \left\{ {1,2},…,\min \left(\dfrac{{N}_{\rm h}}{tp},P_{\max}\right)\right\} {\rm{do}} $
⑤    $dp \leftarrow \dfrac{{{N_{\rm l}}}}{{tp \times pp}}$;
⑥   if simu_mem_3.5D$(\text{π}^{\rm a}, dp, tp, pp) \leqslant $     memory_limit then
⑦     t ← simu_time_3.5D$(\text{π}^{\rm a}, dp, tp, pp) $;
⑧    add {t, dp, tp, pp} to Buff;
⑨   end if
⑩  end for
⑪ end for
⑫  topt = $\mathop {\min}\limits_t(Buf f ) $;
⑬  dpopt, tpopt, ppopt= $\mathop {{\rm arg}\min}\limits_t(Buf f ) $;
⑭ return topt, (dpopt, tpopt, ppopt);
⑮ end procedure
我们针对高计算密度任务组和低计算密度任务组分别设计了耗时开销模拟器和存储开销模拟器,以求取${T^{3.5\rm D}}({\pi ^{\rm a}},{S_{\rm l}})$,${T^{3\rm D}}({\pi ^{\rm a}},{\pi ^{\rm r}},{\pi ^{\rm ref}},{S_{\rm h}})$. 对于${T_{{\text{comm}}}}$,我们使用multi2multi的传输方式,各个节点的通信耗时通过$ WS/BW $进行估算,其中$WS$为其收发的数据量,$BW$为其节点传输带宽,然后求取所有节点的最大通信时长来作为${T_{{\text{comm}}}}$.
1)集群配置. 所有实验均在一个A100集群上进行. 该集群的每个节点配备4块A100(40 GB)GPU,同一节点内的GPU通过NVLink v3进行互联,节点内通信带宽高达600 GB/s. 不同的节点通过InfiniBand网络相连,节点间带宽为400 Gb/s.
2)大语言模型. 在GPT系列模型上测试Pipe-RLHF的性能,参与实验的模型包括GPT-6B,GPT-20B,GPT-66B和GPT-175B. 这些模型结构参数如表2所示,这4个模型的隐藏层维度分别为4 096, 6 144, 9 216和12 288;注意力头数目分别为32,64,72和96;分别由32层、44层、64层和96 Transformer block组成.
3)微调细节. 我们使用包括Anthropic’s Helpful Harmless dataset[10]在内的多个据集对上述模型进行微调,并使用文献[18]中提出的数据预处理方法来统一不同数据集的格式. 对于所有的模型,我们将有效批大小设置为256,PPO每个迭代中训练阶段的参数更新次数设置为1,输入的提示词长度和最大新生成文本长度均设置为256. 此外,由于全参数微调的内存开销过大,我们使用LoRA算法来冻结大部分参数来降低内存占用,以便测试Pipe-RLHF在更大尺寸模型上的表现[34]. 对于上述4个模型,参数版本延迟阈值$P_{\max} $设置为16.
4)对比方法. 在吞吐量实验中,我们设置了如下方法作为对比组.
①全局三维并行策略. PPO中所有阶段都使用相同的三维并行策略,现有RLHF微调框架Trl,Trlx,DeepSpeed均采用此种方法.
②阶段独立三维并行策略. 对PPO中每个阶段单独求取其最优三维并行策略,且3个阶段顺序执行.
③延迟批间流水线并行. 该方法中,PPO各个阶段顺序执行,但是PPO的生成阶段使用包含批间流水线并行的3.5维并行.
④OpenRLHF[35]. OpenRLHF将计算资源划分为若干份,分别分配给奖励模型、参考模型、策略模型的生成阶段和策略模型的训练阶段. 对于策略模型的生成阶段使用数据并行和张量并行,对于其他模型则使用数据并行和ZeRO-3.
⑤HybridFlow[36]. HybridFlow首先将RLHF中的模型进行分组,每组中有1个或多个模型,之后以组为单位进行计算资源分配. HybridFlow对所有模型都使用三维并行.
我们使用全局三维并行策略、阶段独立三维并行策略、延迟批间流水线并行、OpenRLHF、HybridFlow及Pipe-RLHF在8个节点(32块GPU)上对表2中4个模型进行微调,测量其每秒处理的文本数目,实验结果如图10所示. Pipe-RLHF在所有情况下均优于现有并行框架使用的方法. 在4个模型的微调中,Pipe-RLHF对于全局三维并行策略和阶段独立三维并行策略分别取得了3.34倍、2.17倍、2.31倍、1.12倍的加速和3.3倍、2.18倍、1.66倍、1.1倍的加速. 阶段独立三维并行策略较优于全局三维并行策略,但是因为三维并行方法本身不适用于生成阶段,因此对生成阶段单独求取三维并行策略效果并不明显. OpenRLHF在本文实验中吞吐量很低,多数情况下低于全局三维并行,导致这种现象的一个原因是OpenRLHF的资源分配方法在同步PPO下会导致大量的资源空闲,比如分配给奖励模型、参考模型和策论模型生成阶段的GPU在生成阶段会空闲等待. 另一个原因则是它使用ZeRO-3加速单个大模型,导致大量的全局数据传输,当节点间带宽受限时,通信开销会非常庞大. OpenRLHF在微调GPT-66B和GPT-175B时出现了内存不足的问题. 在本实验中,HybridFlow往往选择将所有模型分为一组来避免资源等待空闲,在这种情况下,HybridFlow等效于阶段独立三维并行策略. 通过对实验结果进行进一步分析可以发现,在计算资源固定的情况下,Pipe-RLHF的相对加速效果随着模型的增大而减小. 导致这种现象的一个原因是模型越大,生成阶段的计算密度越高,计算资源利用率不足的程度越轻微. 另一个原因是越大尺寸的模型部署所需的最小GPU数目越多,有效策略集$({N_{\rm l}},{N_{\rm h}})$ 较小,难以找到可以平衡2个任务组耗时的GPU分配. 相对于延迟批间流水线并行,Pipe-RLHF执行4个模型微调的加速比分别为1.26、1.15、1.01、0.76,意味着随着模型的增大,阶段间流水线的收益逐渐降低,甚至产生负收益. 这一方面是因为随着计算密度增大,单独使用延迟批间流水线就可以解决计算资源利用率不足的问题;另一方面是因为阶段间流水线并行效策略集$({N_{\rm l}},{N_{\rm h}})$ 的缩小会导致阶段间耗时差异过大.
在之后的实验中,我们将单独使用延迟批间流水线并行看作Pipe-RLHF禁用阶段间并行的一个特殊情况,不再对其单独列出. 此外,因为全局三维并行策略、阶段独立三维并行策略和HybridFlow这3种方法的性能相似,我们之后仅对阶段独立三维并行策略进行探讨.
为了分析Pipe-RLHF的可扩展性,我们在8~32个节点上分别使用Pipe-RLHF、OpenRLHF和阶段独立三维并行策略对4个大模型进行微调,分别测量各种情况下的吞吐量及耗时组成. 测量结果如图11所示,Pipe-RLHF的可扩展性显著优于阶段独立三维并行策略和OpenRLHF. 阶段独立三维并行策略在模型较小时的可扩展性接近于无,随着模型的增大可扩展性有略微提升,但依旧很差. 这是因为在三维并行方法下,随着GPU的增加,生成阶段成为了总体微调耗时的瓶颈,且单纯扩展GPU数目无法进一步减小生成阶段的耗时. OpenRLHF将不同的模型放到不同的GPU上,在同步PPO下,这种放置策略导致了巨大的资源空闲,即图中Idle的部分;其次,这种分开放置的方法下,各模型的共享参数要被存储多份,增大了存储压力,减小了可以微调的模型规模的上限,比如在微调GPT-175B时,OpenRLHF出现了内存不足的问题. 此外,当模型超出单个GPU显存时,OpenRLHF使用ZeRO-3并行方法切分模型,这会带来巨大的通信开销,显著提高总体耗时. 与之相反,Pipe-RLHF展现出接近线性的可扩展性,这是因为阶段间并行和延迟批间流水线并行可以有效保持生成阶段的计算密度,且异步PPO使得不同的阶段可以同时执行,避免了资源等待空闲.
本节探讨延迟批间流水线对生成阶段的加速效果. 我们分别测试了在使用三维并行和包含批间流水线并行的3.5维并行时,GPT-6B在1~32个GPU上生成文本的速度. 结果如图12所示,当GPU数目为4时,三维并行开始出现计算资源利用率不足的问题,加速效果降低. 而3.5维并行依靠延迟批间流水线并行完美规避了这个问题,呈现出近乎线性的可扩展性.
为了更直观地说明Pipe-RLHF的加速原理,我们使用NVDIA Nsight Compute统计了使用Pipe-RLHF和三维并行微调大模型时的全局计算资源利用率. 全局计算资源利用率数值上等于每一个GPU的资源利用率的平均值,而每一个GPU的资源利用率可以通过对算子利用率加权求和获得,权值为该算子的执行时长占比. 全局资源利用率的统计结果如表3所示. 不出意外我们观察到模型越小计算资源利用率越低,这不仅是因为算子切片太小不足以完全利用所有计算单元,还因为算子切片的GPU的执行时间太短,不能覆盖CPU对算子的调度时间,造成了GPU的空闲等待. 而Pipe-RLHF一直保持着较高的全局计算资源利用率,这得益于它避免了对算子的过度切分,保证了每个算子切片的计算负载,从而阻止了上述问题的发生.
在本节中,我们基于一个实例对4.3节中的并行策略求取算法的有效性进行分析. 当可用GPU数目为128,微调模型为GPT-6B时,通过算法1求取的最优阶段间GPU分配为Nl=32,Nh=96. 图13展示了该分配策略下低计算密度任务组和高计算密度任务组的平均单批耗时,由图13可得2个阶段的耗时非常接近,说明算法1求得了符合需求的分配策略. 对于生成阶段,算法3求得的并行策略为dp=2, tp=1, pp=16,最大化地减少了对算子的切分.
目前,针对生成阶段资源利用率低下这一问题,有工作设计了专用于生成阶段的推理引擎. 推理引擎优化了小尺寸算子在单个GPU上的运算,与本文方法是正交的关系,推理引擎和本文方法可以结合使用,同时从系统层面和任务切分层面对生成阶段进行优化.
DeepSpeed-Chat是当前顶级的RLHF微调框架之一,它的核心技术是专为生成阶段设计的推理引擎,而在并行方法上则简单使用ZeRO(训练阶段、推理阶段)和张量并行(生成阶段). 我们将DeepSpeed-Chat的推理引擎应用到Pipe-RLHF中,探索Pipe-RLHF和生成专用推理引擎结合后的效果. 作为对比,我们还测量了原始DeepSpeed-Chat的性能以及DeepSpeed-Chat卸除专用引擎后的性能,测量结果如图14所示. 可以看到,不论是否使用专用推理引擎,Pipe-RLHF的性能都显著优于使用简单并行方法的DeepSpeed-Chat,虽然DeepSpeed-Chat的专用引擎一定程度上减缓了资源利用率不足的问题,但效果有限. 此外,相对于原始Pipe-RLHF,装备了专用引擎的Pipe-RLHF取得了1.18倍、1.15倍、1.06倍的加速比,这说明Pipe-RLHF可以有效与生成专用的推理引擎相结合,进一步提升性能,并且提升效果随着模型的减小而增加.
本节探索了Pipe-RLHF中,超参数Pmax对微调的影响. Pmax表示Pipe-RLHF的延迟版本阈值,控制着生成阶段中延迟批间流水线并行的最大并行度,也控制着生成阶段和训练阶段间的参数延迟版本上限. 我们分别探索了Pmax对收敛曲线和吞吐量的影响.
图15中展示了不同延迟版本下GPT-2在情感分析任务上的微调曲线. 由图可得,延迟版本数的增加不会影响模型最终的收敛质量,但是会影响模型的收敛速率,即延迟版本数越大,模型收敛所需要的训练步数越多. 此外,收敛速率下降的速度随着延迟版本数的增大而增大,比如延迟版本数从0增加到16,收敛步数从90增加到130,增加了40;而延迟版本数从16增加到32,收敛步数从130增加到了210,增加了80. 图16中展示了Pmax下,Pipe-RLHF在128块GPU上进行的微调吞吐量. 总体来说,Pmax与微调吞吐量成正比, Pmax越大,吞吐量越大. 对于较小的模型,Pmax的影响比较大,这是因为模型越小,计算密度越低,所需的延迟批间流水线并行度越大,以减小算子切分. 此时若设置较小的Pmax,则会导致算子切分过多,最终影响吞吐量. 此外,在资源固定的情况下,随着Pmax越大其影响力越小,这是因为当算子内切分小于一定程度时,计算密度已经可以充分利用GPU计算资源,故而进一步减小算子内切程序不会带来更显著的收益.
综上,Pmax太大会显著增加模型的收敛步数,而Pmax太小又会影响微调的吞吐量,我们需要选择合适的Pmax来减小模型的整体耗时.
目前已经有多个工作进行了RLHF框架设计方面的创新和努力,包括Trl[20],Trlx[21],ColossalChat[37],DeepSpeed-Chat[19],HybridFlow[36],OpenRLHF[35]和LLaMA-Factory[38]等.
其中,Trl是由Huggingface推出的全栈库,它提供了一组使用强化学习对大语言模型进行微调的工具,涵盖了从监督微调(SFT)步骤、奖励建模(RM)步骤到PPO步骤的全流程. Trlx是由CarperAI提出的一个分布式大模型微调框架,除了奖励模以外,还支持用户使用自定义的奖励函数对大模型进行微调. Trl和Trlx均采用Hugging Face的Accelerate库来减少RLHF的时间消耗. ColossalChat是一个开源项目,它提供了完整的RLHF流水线,包括数据集收集、有监督微调、奖励模型训练、强化学习微调和模型部署.
DeepSpeed-Chat是一个端到端的RLHF框架,它利用DeepSpeed推理引擎加速生成步骤,该引擎具备量化、推理定制内核等功能,可高效实现大语言模型的生成. 上述框架均使用同一并行策略来加速所有的PPO阶段.
HybridFlow是字节跳动公司与香港大学联合开发的RLHF微调框架,作为开源大模型强化学习引擎veRL[39]的底层加速框架,它采用混合控制器架构和分层API设计,同时通过3D-HybridEngine优化参数分配. OpenRLHF是由OpenLLMAI等联合开发的开源RLHF微调框架,利用Ray,DeepSpeed和vLLM优化资源利用率和训练性能,此外该框架还将PPO中的多个模型部署至不同GPU上,显著提升内存利用率与训练吞吐量. HybridFlow和OpenLLMAI均关注于PPO单阶段内的优化,没有探索PPO阶段间的并行性.
LLaMA-Factory是一个集成了多种高效微调方法的大模型训练与微调平台,支持DPO,PPO等多种微调技术,并通过图形界面简化了大模型微调的复杂性. 框架旨在降低大模型微调的门槛,为大众提供一个简单易用的RLHF框架,并未对RLHF的高效分布式实现进行探索.
除了本文提到的并行以外,还有许多技术被应用于优化大模型的生成开销,它们大致可以分为4类:
1)低精度量化. 这类方法通过对权重、激活值进行低精度量化来缓解大语言模型生成时的访存瓶颈. 在存储、传输时,数据被量化为低精度,减少数据存储和加载开销;加载完成后,数据被反量化到正常精度进行计算,以保持计算准确性[40].
2)计算核心优化. 这类方法通过优化大模型生成的注意力计算等算子的底层CUDA核心代码来加速大模型的生成. 比如FlashAttention[41]利用算子平铺和重计算等技术,将注意力计算在批大小和查询长度上切分成多个可以放到SRAM中的小块分别计算,有效减少了对GPU显存的访问. Flash-Decoding[42]在FlashAttention的基础上添加了key/value序列长度这一切分维度,从而提高了算子在GPU内核上的并行度,进而加速了注意力算子的计算.
3)kv-cache优化. 大模型生成过程中往往需要存储大量的历史key/value,花费了大量的显存[43]. ZeRO-Inference[44]、FlexGen[45]将key/value卸载到SSD等其他资源上,并制定相应策略预先加载需要的数据,在减少kv-cache显存占用的基础上保证吞吐量. PagedAttention[46]不再一次性分配所有的kv-cache,而是以块为单位动态分配和回收kv-cache.
4)批处理技术. 通常用在大模型服务部署中,将多个用户的请求合并为一个批次统一处理,可以有效提高GPU利用率和吞吐量. 常用的批处理方法有静态批处理、动态批处理和连续批处理. 静态批处理使用固定的批次大小,在批次内所有请求完成计算后统一返回响应. 而动态批处理则根据当前的请求量实时调整批次的大小,但依然是在一个批次内所有计算完成后统一返回响应[47]. 作为改进,连续批处理不再等待批次内所有请求都完成,而是当某一请求完成时立刻返回该请求结果,并插入新的请求填充其位置[48].
上述方法中的低精度量化、计算核心优化、kv-cache优化等方法均属于本地GPU上的优化,旨在加速单块GPU上的任务执行,可以与本文提出的Pipe-RLHF结合使用. 而批处理多适用于大模型服务部署等动态输入的场景,RLHF的生成阶段的输入是静态且规律的,不存在使用批处理的动态、不可预测的先决条件,故而批处理技术无法用于加速RLHF的生成阶段.
本文提出了一个灵活高效的RLHF分布式的微调框架Pipe-RLHF. 具体而言,我们提出了阶段间并行和延迟批间流水线并行性方法,该方法有效提升了生成阶段的计算资源利用率,显著提高了PPO的执行效率. 另外,我们构建了面向PPO算法全流程的分层并行策略空间,并设计了一套优化算法用于获得最优并行策略. 我们对Pipe-RLHF进行了系统实现和全面评估. 实验结果表明,Pipe-RLHF相比现有方案表现出显著的性能提升.
作者贡献声明:徐颖负责方案整体设计和实验分析并撰写论文;王梦迪、程龙、王颖提出论文思路并负责理论指导和论文修改;刘炼、赵世新负责论文的修改和润色;张磊负责论文的审核.

参考文献 引证文献
排序方式:
1
Christiano P F, Leike J, Brown T B, et al. Deep reinforcement learning from human preferences[C]//Proc of the 31st Int Conf on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc., 2017: 4302–4310
2
Macglashan J, Ho M K, Loftin R, et al. Interactive learning from policy-dependent human feedback[C]//Proce of the 34th Int Conf on Machine Learning - Volume 70. New York: ACM, 2017: 2285–2294
3
Warnell G, Waytowich N, Lawhern V, et al. Deep TAMER: Interactive agent shaping in high-dimensional state spaces[C]//Proc of the 32nd AAAI Conf on Artificial Intelligence and 30th Innovative Applications of Artificial Intelligence Conf and 8th AAAI Symp on Educational Advances in Artificial Intelligence. Palo Alto, CA: AAAI, 2018: 1545−1554
4
Brown D, Goo W, Nagarajan P, et al. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations[C]//Proc of Int Conf on Machine Learning. Cambridge, MA: JMLR, 2019: 783−792
5
Shin D, Dragan A D, Brown D S. Benchmarks and algorithms for offline preference-based reward learning[J]. arXiv preprint, arXiv: 2301.01392, 2023
6
Sadigh D, Dragan A, Sastry S, et al. Active preference-based learning of reward functions[C]//Proc of Robotics: Science and Systems. Cambridge, Massachusetts: Escholarship. https://www.roboticsproceedings.org/rss13/p53.html
7
Maghakian J, Mineiro P, Panaganti K, et al. Personalized reward learning with interaction-grounded learning (IGL)[C]//Proc of the 11th Int Conf on Learning Representations. Washington, DC: OpenReview. net, 2022
8
矣晓沅,谢幸. 大模型道德价值观对齐问题剖析[J]. 计算机研究与发展,2023,60(9):1926−1945
Yi Xiaoyuan, Xie Xing. Unpacking the ethical value alignment in big models[J]. Journal of Computer Research and Development, 2023, 60(9): 1926−1945 (in Chinese)
9
Ouyang Long, Wu J, Jiang Xu, et al. Training language models to follow instructions with human feedback[C]//Proc of Conf on Neural Information Processing Systems 2022, Red Hook, NY: Curran Associates Inc., 2022: 27730−27744
10
Bai Y, Jones A, Ndousse K, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback[J]. arXiv preprint, arXiv: 2204.05862, 2022
11
Achiam J, Adler S, Agarwal S, et al. GPT−4 technical report[R/OL]. OpenAI, 2023[2025-04-08]. https://arxiv.org/pdf/2303.08774
12
Anthropic. Introducing claude[EB/OL]. 2023[2025-04-09]. https://www.anthropic.com/news/introducing-claude
13
Glaese A, Mcaleese N, Trębacz M, et al. Improving alignment of dialogue agents via targeted human judgements[J]. arXiv preprint, arXiv: 2209.14375, 2022
14
Pichai S, Hassabis D. Introducing Gemini: Our largest and most capable AI model[EB/OL]. 2023[2025-04-09]. https://blog.google/technology/ai/google-gemini-ai/
15
Bi Xiao, Chen Deli, Chen Guanting, et al. DeepSeek LLM: Scaling open-source language models with longtermism[J]. arXiv preprint, arXiv: 2401.02954, 2024
16
Stiennon N, Ouyang Long, Wu J, et al. Learning to summarize with human feedback[C]//Proc of Conf on Neural Information Processing Systems 2020. Red Hook, NY: Curran Associates Inc., 2020: 3008−3021
17
Schulman J, Wolski F, Dhariwal P, et al. 2017. Proximal policy optimization algorithms[J]. arXiv preprint, arXiv: 1707.06347, 2017
18
Zheng Rui, Dou Shihan, Gao Songyang, et al. Secrets of RLHF in large language models part I: PPO[J]. arXiv preprint, arXiv: 2307.04964, 2023
19
Yao Zhewei, Aminabadi R Y, Ruwase O, et al. 2023. DeepSpeed-Chat: Easy, fast and affordable RLHF training of ChatGPT-like models at all scales[J]. arXiv preprint, arXiv: 2308.01320, 2023
20
Werra L, Belkada Y, Tunstall L, et al. trl: Transformer reinforcement learning[EB/OL]. 2020[2025-04-09]. https://github.com/huggingface/trl
21
Havrilla A, Zhuravinskyi M, Phung D, et al. Trlx: A framework for large scale reinforcement learning from human feedback[C]//Proc of the 2023 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2023: 8578−8595
22
Lai Z, Li S, Tang X, et al. Merak: An efficient distributed DNN training framework with automated 3D parallelism for giant foundation models[J]. IEEE Transactions on Parallel and Distributed Systems, 2023, 34(5): 1466−1478
23
Song J, Yim J, Jung J, et al. Optimus-CC: Efficient large NLP model training with 3D parallelism aware communication compression[C]//Proc of the 28th ACM Int Conf on Architectural Support for Programming Languages and Operating Systems. New York: ACM, 2023: 560–573
24
Knox W B, Stone P. TAMER: Training an agent manually via evaluative reinforcement[C]//Proc of the 7th IEEE Int Conf on Development and Learning. Piscataway, NJ: IEEE, 2008: 292−297
25
Ziegler D M, Stiennon N, Wu J, et al. 2019. Fine-tuning language models from human preferences[J]. arXiv preprint, arXiv: 1909.08593, 2019
26
Wu J, Ouyang Long, Ziegler D M, et al. 2021. Recursively summarizing books with human feedback[J]. arXiv preprint, arXiv: 2109.10862, 2021
27
Santacroce M, Lu Yadong, Yu Han, et al. 2023. Efficient RLHF: Reducing the memory usage of PPO[J]. arXiv preprint, arXiv: 2309.00754, 2023
28
Zheng Lianmin, Li Zhouhan, Zhang Hao, et al. Alpa: Automating inter-and intra-operator parallelism for distributed deep learning[C]//Proc of the 16th USENIX Symp on Operating Systems Design and Implementation (OSDI 22). Berkeley, CA: USENIX Association, 2022: 559−578
29
Xiao W, Bhardwaj R, Ramjee R, et al. Gandiva: Introspective cluster scheduling for deep learning[C]//Proc of the 13th USENIX Conf on Operating Systems Design and Implementation. Berkeley, CA: USENIX Association, 2018: 595–610
30
Shoeybi M, Patwary M, Puri R, et al. 2019. Metatron-LM: Training multi-billion parameter language models using model parallelism[J]. arXiv preprint, arXiv: 1909.08053, 2019
31
Huang Yanping, Cheng Youlong, Bapna A, et al. GPipe: Efficient training of giant neural networks using pipeline parallelism[C]//Proc of the 33rd Int Conf on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc., 2019: 103−112
32
Narayanan D, Harlap A, Phanishayee A, et al. PipeDream: Generalized pipeline parallelism for DNN training[C]//Proc of the 27th ACM Symposium on Operating Systems Principles. New York: ACM, 2019: 1–15
33
Rajbhandari S, Rasley J, Ruwase O, et al. ZeRO: Memory optimizations toward training trillion parameter models[C]//Proc of SC20: Int Conf for High Performance Computing, Networking, Storage and Analysis. Piscataway, NJ: IEEE, 2020: 1−16
34
Hu E J, Shen Y, Wallis P, et al. Lora: Low-rank adaptation of large language models[C]//Proc of Int Conf on Learning Representations. openreview. net, 2022
35
Hu Jian, Wu Xibin, Zhu Zilin, et al. OpenRLHF: An easy-to-use, scalable and high-performance RLHF framework[J]. arXiv preprint, arXiv: 2405.11143, 2024
36
Sheng Guangming, Zhang Chi, Ye Zilingfeng, et al. HybridFlow: A flexible and efficient RLHF framework[J]. arXiv preprint, arXiv: 2409.19256, 2024
37
Li Shenggui, Liu Hongxin, Bian Zhengda, et al. Colossal-AI: A unified deep learning system for large-scale parallel training[C]//Proc of the 52nd Int Conf on Parallel Processing. New York: ACM, 2023: 766–775
38
Zheng Yaowei, Zhang Richong, Zhang Junhao, et al. LLaMA-Factory: Unified efficient fine-tuning of 100+ language models[C]//Proc of the 62nd Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2024: 400−410
39
Volcengine. veRL: Volcano engine reinforcement learning for LLMs[EB/OL]. 2025[2025-04-08]. https://github.com/vocengine/verl
40
Qin Haotong, Ma Xudong, Zheng Xingyu, et al. Accurate LoRA-finetuning quantization of LLMs via information retention[J]. arXiv preprint, arXiv: 2402.05445, 2024
41
Dao T, Fu D Y, Ermon S, et al. 2022. FlashAttention: Fast and memory-efficient exact attention with IO-awareness[C]//Proc of the 36th Int Conf on Neural Information Processing Systems. New Orleans, LA: Curran Associates Inc. 1189: 16344−16359
42
Dao T, Haziza D, Massa F, et al. Flash-Decoding for long-context inference[EB/OL]. 2024[2025-03-01]. https://crfm.stanford.edu/2023/10/12/flashdecoding.html
43
葛旭冉,欧洋,王博,等. 大语言模型推理中的存储优化技术综述[J]. 计算机研究与发展,2025,62(3):545−562
Ge Xuran, Ou Yang, Wang Bo, et al. Survey of storage optimization techniques in large language model inference[J]. Journal of Computer Research and Development, 2025, 62(3): 545−562 (in Chinese)
44
DeepSpeed. ZeRO-Inference: Democratizing massive model inference[EB/OL]. 2022[2025-04-08]. https://www.deepspeed.ai/2022/09/09/zero-inference.html
45
Sheng Ying, Zheng Lianmin, Yuan Binhang, et al. FlexGen: High-throughput generative inference of large language models with a single GPU[C]//Proc of Int Conf on Machine Learning. Cambridge, MA: PMLR, 2023: 31094−31116
46
Kwon W, Li Zhouhan, Zhuang Siyan, et al. Efficient memory management for large language model serving with pagedattention[C]//Proc of the 29th Symp on Operating Systems Principles. New York: ACM, 2023: 611−626
47
Nvidia. Triton inference server[EB/OL]. 2025[2025-04-09]. https://github.com/triton-inference-server/server
48
Yu G I, Jeong J S, Kim G W, et al. Orca: A distributed serving system for Transformer-Based generative models[C]//Proc of the 16th USENIX Symp on Operating Systems Design and Implementation (OSDI 22). Berkeley, CA: USENIX Association, 2022: 521−538
2025年第62卷第6期
92
38
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202550127
  • 接收时间:2025-03-01
  • 首发时间:2026-11-10
  • 出版时间:2025-06-02
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2025-03-01
  • 修回日期:2025-04-13
基金
作者信息
    1中国科学院计算技术研究所 北京 100190
    2中国科学院大学 北京 100190
    3华北电力大学控制与计算机工程学院 北京 102206
参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202550127
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏