收藏切换
HiTrain: Heterogeneous Memory Offloading and I/O Optimization for Large Language Model Training
收藏切换
Zhihang Tang1, 2, Hao Lan3, *, Zhengguo Liu4, Qing Wang3, Laiping Zhao1, Keqiu Li1
Journal of Computer Research and Development | 2026, 63(3) : 627 - 639
Less
收藏切换
Journal of Computer Research and Development | 2026, 63(3): 627-639
• Special Issue on Data Storage and Processing for New Quality Productive Forces •
HiTrain: Heterogeneous Memory Offloading and I/O Optimization for Large Language Model Training
Full
Zhihang Tang1, 2, Hao Lan3, *, Zhengguo Liu4, Qing Wang3, Laiping Zhao1, Keqiu Li1
Affiliations
  • 1College of Intelligence and Computing, Tianjin University, Tianjin 300350
  • 2Research Center for High Efficiency Computing Infrastructure, Zhejiang Lab, Hangzhou 311121
  • 3Department of Computer Science and Technology, Tsinghua University, Beijing 100084
  • 4Department of Information and Communication Engineering, Xiamen University, Xiamen, Fujian 361005
  • born in 1993. PhD candidate, assistant researcher. Member of CCF. His main research interests include distributed computing and large model system

    born in 1993. PhD, assistant researcher. Member of CCF. His main research interest includes AI systems and algorithms

    born in 1999. Master candidate. Student member of CCF. His main research interests include AI systems and storage systems

    born in 1997. PhD, assistant researcher. Member of CCF. His main research interests include storage systems and memory systems

    born in 1985. PhD, professor, PhD supervisor. Member of CCF. His main research interest includes cloud computing system

    born in 1971. PhD, professor, PhD supervisor. Fellow of CCF. His main research interests include computer networks and cloud computing

About Author:

born in 1993. PhD candidate, assistant researcher. Member of CCF. His main research interests include distributed computing and large model system

born in 1993. PhD, assistant researcher. Member of CCF. His main research interest includes AI systems and algorithms

born in 1999. Master candidate. Student member of CCF. His main research interests include AI systems and storage systems

born in 1997. PhD, assistant researcher. Member of CCF. His main research interests include storage systems and memory systems

born in 1985. PhD, professor, PhD supervisor. Member of CCF. His main research interest includes cloud computing system

born in 1971. PhD, professor, PhD supervisor. Fellow of CCF. His main research interests include computer networks and cloud computing

Published: 2026-03-01 doi: 10.7544/issn1000-1239.202550478
Outline
收藏切换

With the continuous growth of the parameter scale of large language models (LLMs), fine-tuning models with hundreds of billions of parameters pose extremely high requirements for computing and storage resources. Traditional distributed training schemes often rely on a large number of high-end graphics processing units (GPUs) and high-speed interconnect networks, resulting in extremely high training costs. Although the existing single-GPU training schemes relieve the pressure on GPU memory through tensor offloading, they still face problems such as low I/O transmission efficiency and insufficient device utilization. Traditional kernel-space I/O operations introduce frequent system calls and context switching during large-scale tensor transfers, becoming a key bottleneck that restricts performance. Meanwhile, the optimizer computations fail to fully leverage the parallel capabilities of multi-core CPUs, making it difficult to effectively overlap with GPU computations, further limiting system performance. In response to the above problems, we propose HiTrain, a heterogeneous memory offloading and I/O optimization scheme for LLM training, with a focus on the design and implementation of two key technologies: First, it constructs a high-performance tensor storage module based on the storage performance development kit (SPDK), which manages tensor data in user space, thereby avoiding the overhead of the kernel I/O stack and improving the concurrency and throughput of tensor offloading. Second, it designs and implements an asynchronous-optimizer based storage-compute pipeline scheduling module, which reorders optimizer execution to reduce GPU idle time and thus enhance overall training efficiency. The experimental results show that on the server equipped with a single GPU and a non-volatile memory express solid state drive (NVMe SSD), the scheme proposed can make full use of the computing and storage resources in the system, improve the efficiency of tensor unloading and loading in the process of model training by 32.7%, and increase the overall training throughput to 1.49 times of the existing scheme, offering a practical and cost-effective path for LLM training.

large model training  /  heterogeneous memory  /  tensor offload  /  user-space high-performance I/O  /  storage-computing pipeline
汤志航, 兰颢, 刘政国, 汪庆, 赵来平, 李克秋. HiTrain:面向大模型训练的异构内存卸载与I/O优化. 计算机研究与发展, 2026 , 63 (3) : 627 -639 . DOI: 10.7544/issn1000-1239.202550478
Zhihang Tang, Hao Lan, Zhengguo Liu, Qing Wang, Laiping Zhao, Keqiu Li. HiTrain: Heterogeneous Memory Offloading and I/O Optimization for Large Language Model Training[J]. Journal of Computer Research and Development, 2026 , 63 (3) : 627 -639 . DOI: 10.7544/issn1000-1239.202550478
Year 2026 volume 63 Issue 3
202
94
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202550478
  • Receive Date:2025-06-17
  • Online Date:2026-11-06
  • Published:2026-03-01
Article Data
Affiliations
History
  • Received:2025-06-17
  • Revised:2025-10-16
  • Accepted:2025-10-31
Affiliations
    1College of Intelligence and Computing, Tianjin University, Tianjin 300350
    2Research Center for High Efficiency Computing Infrastructure, Zhejiang Lab, Hangzhou 311121
    3Department of Computer Science and Technology, Tsinghua University, Beijing 100084
    4Department of Information and Communication Engineering, Xiamen University, Xiamen, Fujian 361005
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202550478
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT