收藏切换
From BERT to ChatGPT: Challenges and Technical Development of Storage Systems for Large Model Training
收藏切换
Yangyang Feng, Qing Wang, Minhui Xie, Jiwu Shu*
Journal of Computer Research and Development | 2024, 61(4) : 809 - 823
Less
收藏切换
Journal of Computer Research and Development | 2024, 61(4): 809-823
• Architecture •
From BERT to ChatGPT: Challenges and Technical Development of Storage Systems for Large Model Training
Full
Yangyang Feng, Qing Wang, Minhui Xie, Jiwu Shu*
Affiliations
  • 1Department of Computer Science and Technology, Tsinghua University, Beijing 100084
  • born in 1998. PhD candidate. His main research interests include storage systems and machine learning systems

    born in 1997. PhD. His main research interests include storage systems and memory systems

    born in 1997. PhD candidate. Student member of CCF. His main research interests include storage systems and machine learning systems

    born in 1968. PhD, professor, PhD supervisor. Fellow of CCF. His main research interests include intelligent storage systems, non-volatile memory storage systems and technologies, storage security and reliability, and parallel and distributed computing

About Author:

born in 1998. PhD candidate. His main research interests include storage systems and machine learning systems

born in 1997. PhD. His main research interests include storage systems and memory systems

born in 1997. PhD candidate. Student member of CCF. His main research interests include storage systems and machine learning systems

born in 1968. PhD, professor, PhD supervisor. Fellow of CCF. His main research interests include intelligent storage systems, non-volatile memory storage systems and technologies, storage security and reliability, and parallel and distributed computing

Published: 2024-04-01 doi: 10.7544/issn1000-1239.202330554
Outline
收藏切换

The large models represented by ChatGPT have attracted a lot of attention from industry and academia for their excellent performance on text generation and semantic understanding tasks. The number of large model parameters has increased tens of thousands of times in three years and is still growing, which brings new challenges to storage systems. First, we analyze the storage challenges of large model training, pointing out that large model training has unique computation patterns, storage access patterns, and data characteristics, which makes traditional storage techniques inefficient in handling large model training tasks. Then, we describe three types of storage acceleration techniques and two types of fault-tolerant techniques. The storage acceleration techniques for large model training include: 1) distributed storage technique based on large model computation patterns designs the partitioning, storage, and transferring strategies of model data in distributed clusters based on the partitioning of large model computation tasks and the dependencies between computation tasks; 2) heterogeneous storage access pattern-aware technique for large model training develops data prefetching and transferring strategies among heterogeneous devices with the predictability of storage access patterns in large model training; 3) large model data reduction technique reduces the data size in the model training process according to the characteristics of large model data. The storage fault-tolerant techniques for large model training include: 1) parameter checkpointing technique stores the large model parameters to persistent storage devices; 2) redundant computation technique computes the same version of parameters repeatedly in multiple GPUs. Finally, we give the summary and suggestions for future research.

ChatGPT  /  large model  /  storage system  /  fault-tolerant  /  large model training system
冯杨洋, 汪庆, 谢旻晖, 舒继武. 从BERT到ChatGPT:大模型训练中的存储系统挑战与技术发展. 计算机研究与发展, 2024 , 61 (4) : 809 -823 . DOI: 10.7544/issn1000-1239.202330554
Yangyang Feng, Qing Wang, Minhui Xie, Jiwu Shu. From BERT to ChatGPT: Challenges and Technical Development of Storage Systems for Large Model Training[J]. Journal of Computer Research and Development, 2024 , 61 (4) : 809 -823 . DOI: 10.7544/issn1000-1239.202330554
Year 2024 volume 61 Issue 4
78
37
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202330554
  • Receive Date:2023-07-03
  • Online Date:2026-11-17
  • Published:2024-04-01
Article Data
Affiliations
History
  • Received:2023-07-03
  • Revised:2023-11-27
Affiliations
    1Department of Computer Science and Technology, Tsinghua University, Beijing 100084
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202330554
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT