收藏切换
Pipe-RLHF: A Computation Mode-Aware Parallel Framework for RLHF
收藏切换
Ying Xu1, 2, Mengdi Wang1, Long Cheng3, Lian Liu1, 2, Shixin Zhao1, 2, Lei Zhang1, Ying Wang1
Journal of Computer Research and Development | 2025, 62(6) : 1513 - 1529
Less
收藏切换
Journal of Computer Research and Development | 2025, 62(6): 1513-1529
• Chip Architecture Design •
Pipe-RLHF: A Computation Mode-Aware Parallel Framework for RLHF
Full
Ying Xu1, 2, Mengdi Wang1, Long Cheng3, Lian Liu1, 2, Shixin Zhao1, 2, Lei Zhang1, Ying Wang1
Affiliations
  • 1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
  • 2University of Chinese Academy of Sciences, Beijing 100190
  • 3School of Control and Computer Engineering, North China Electric Power University, Beijing 102206
  • born in 1996. PhD candidate. Her main research interests include machine learning system and parallel deep learning framework

    born in 1995. Assistant professor. Her main research interests include accelerator design and multi-core system

    born in 1985. PhD, professor, PhD supervisor. His main research interest includes distributed computing and high-performance systems

    born in 1999. PhD candidate. His main research interest includes LLM infrastructure、hardware-software co-design

    born in 1999. PhD candidate. Her main research interest includes computer architecture

    born in 1981. PhD, professor. Member of CCF. His main research interests include chip and computing system

    born in 1985. PhD, professor. Senior member of CCF. His main research interests include novel EDA, and processor and memory system architecture

About Author:

born in 1996. PhD candidate. Her main research interests include machine learning system and parallel deep learning framework

born in 1995. Assistant professor. Her main research interests include accelerator design and multi-core system

born in 1985. PhD, professor, PhD supervisor. His main research interest includes distributed computing and high-performance systems

born in 1999. PhD candidate. His main research interest includes LLM infrastructure、hardware-software co-design

born in 1999. PhD candidate. Her main research interest includes computer architecture

born in 1981. PhD, professor. Member of CCF. His main research interests include chip and computing system

born in 1985. PhD, professor. Senior member of CCF. His main research interests include novel EDA, and processor and memory system architecture

Published: 2025-06-02 doi: 10.7544/issn1000-1239.202550127
Outline
收藏切换

Reinforcement learning with human feedback (RLHF) has been proven effective in aligning large language models (LLMs) with human preferences. The most costly part of RLHF is proximal policy optimization (PPO), which consists of three dependent steps. Different PPO steps in RLHF exhibit different computation modes, simply employing the same parallelization strategy to accelerate all steps that involve multiple model variants, as done in existing frameworks, will lead to poor performance in the PPO generation step due to insufficient utilization of computational resources. Thus, we introduce Pipe-RLHF, a parallelism framework for RLHF fine-tuning, which adaptively employs distinct parallelization strategies for different steps according to the computation mode. Specifically, we first investigate the characteristics of various computation modes to explore their best-fit parallelization approach. And then, we present a novel delayed inter-batch pipeline parallelization approach specifically designed for the PPO generation step, enabling the sufficient utilization of computational resources. Subsequently, based on the proposed inter-batch pipeline parallelization approach, we define a hierarchical parallel plan space for distributed RLHF fine-tuning. Finally, we present optimization algorithms to find the optimal parallelization plan from the defined hierarchical parallel plan space to minimize the overall time consumption. Implementation and evaluation across multiple LLMs demonstrates that the proposed Pipe-RLHF achieves 3.7 times speedup compared with existing methods while achieving near-linear scalability.

reinforcement learning with human feedback (RLHF)  /  proximal policy optimization (PPO)  /  large language models fine-tuning  /  distributed systems  /  parallel computing
徐颖, 王梦迪, 程龙, 刘炼, 赵世新, 张磊, 王颖. Pipe-RLHF: 计算模式感知的RLHF并行加速框架. 计算机研究与发展, 2025 , 62 (6) : 1513 -1529 . DOI: 10.7544/issn1000-1239.202550127
Ying Xu, Mengdi Wang, Long Cheng, Lian Liu, Shixin Zhao, Lei Zhang, Ying Wang. Pipe-RLHF: A Computation Mode-Aware Parallel Framework for RLHF[J]. Journal of Computer Research and Development, 2025 , 62 (6) : 1513 -1529 . DOI: 10.7544/issn1000-1239.202550127
Year 2025 volume 62 Issue 6
91
38
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202550127
  • Receive Date:2025-03-01
  • Online Date:2026-11-10
  • Published:2025-06-02
Article Data
Affiliations
History
  • Received:2025-03-01
  • Revised:2025-04-13
Affiliations
    1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
    2University of Chinese Academy of Sciences, Beijing 100190
    3School of Control and Computer Engineering, North China Electric Power University, Beijing 102206
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202550127
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT