收藏切换
ScaleFS: High Performance and Scalable Metadata Design for Large Language Models
收藏切换
Biyun Shang1, Yinjun Han2, Rong Xiao2, Zhenghua Chen2, Yaofeng Tu2, Zhenjiang Dong1, *
Journal of Computer Research and Development | 2025, 62(3) : 589 - 604
Less
收藏切换
Journal of Computer Research and Development | 2025, 62(3): 589-604
• Special Issue on Storage Management and Data Analysis in the Era of Large Language Models •
ScaleFS: High Performance and Scalable Metadata Design for Large Language Models
Full
Biyun Shang1, Yinjun Han2, Rong Xiao2, Zhenghua Chen2, Yaofeng Tu2, Zhenjiang Dong1, *
Affiliations
  • 1School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing 210023
  • 2ZTE Corporation, Nanjing 210012
  • born in 2004. Undergraduate. Student member of CCF. Her main research interests include big data and machine learning

    born in 1977. Master, senior engineer. Senior member of CCF. His main research interests include storage systems and data management systems

    born in 1983. Master, senior engineer. Her main research interests include storage systems and distributed systems

    born in 1983. Senior engineer. Member of CCF. His main research interests include storage systems and distributed systems

    born in 1972. PhD, professor. Distinguished member of CCF. His main research interests include distributed systems, big data and machine learning

    born in 1970. PhD, professor and PhD supervisor. Distinguished member of CCF. His main research interests include artificial intelligence, big data, and information security

About Author:

born in 2004. Undergraduate. Student member of CCF. Her main research interests include big data and machine learning

born in 1977. Master, senior engineer. Senior member of CCF. His main research interests include storage systems and data management systems

born in 1983. Master, senior engineer. Her main research interests include storage systems and distributed systems

born in 1983. Senior engineer. Member of CCF. His main research interests include storage systems and distributed systems

born in 1972. PhD, professor. Distinguished member of CCF. His main research interests include distributed systems, big data and machine learning

born in 1970. PhD, professor and PhD supervisor. Distinguished member of CCF. His main research interests include artificial intelligence, big data, and information security

Published: 2025-03-01 doi: 10.7544/issn1000-1239.202440373
Outline
收藏切换

In recent years, large language models (LLMs) represented by ChatGPT have developed rapidly. As the scale of model parameters continues to grow, building and deploying LLMs puts forward higher requirement for data scale and storage access efficiency, which poses significant challenges to traditional storage systems. This study first analyzes the storage access characteristics across the three critical stages of LLM workflows: data preparation, model training, and inference. It also explores in depth the major issues and bottlenecks faced by traditional storage systems in LLM scenarios. To address these challenges, the study proposes and implements ScaleFS, a high-performance and scalable distributed metadata design. ScaleFS adopts a decoupled design for directory tree metadata and attribute metadata, and combines with a hierarchical partitioning strategy that balances depth and breadth in the directory tree. This design enables efficient path resolution, load balancing, and system scalability, thereby making it capable of effectively managing hundreds of billions of files. Additionally, ScaleFS introduces fine-grained metadata structures, optimizes metadata access patterns, and develops a metadata key-value store tailored for file semantics. These innovations significantly improve metadata access efficiency while reducing disk I/O operations. The experimental results demonstrate that ScaleFS achieves operations per second (OPS) rates 1.04 to 7.12 times higher than HDFS, with latency reduced to only 12.67% to 99.55% of HDFS. Furthermore, at a scale of hundreds of billions of files, ScaleFS outperforms HDFS in most operations, even when HDFS operates at a billion-file scale. This demonstrates its superior scalability and access efficiency. ScaleFS is thus well-suited to meet the demands of LLM scenarios for managing and efficiently accessing massive file datasets.

large language model  /  large model training  /  storage system  /  metadata management  /  scalability
尚碧筠, 韩银俊, 肖蓉, 陈正华, 屠要峰, 董振江. ScaleFS:面向大语言模型的高性能可扩展元数据设计. 计算机研究与发展, 2025 , 62 (3) : 589 -604 . DOI: 10.7544/issn1000-1239.202440373
Biyun Shang, Yinjun Han, Rong Xiao, Zhenghua Chen, Yaofeng Tu, Zhenjiang Dong. ScaleFS: High Performance and Scalable Metadata Design for Large Language Models[J]. Journal of Computer Research and Development, 2025 , 62 (3) : 589 -604 . DOI: 10.7544/issn1000-1239.202440373
Year 2025 volume 62 Issue 3
85
38
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202440373
  • Receive Date:2024-05-31
  • Online Date:2026-11-10
  • Published:2025-03-01
Article Data
Affiliations
History
  • Received:2024-05-31
  • Revised:2024-11-08
Affiliations
    1School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing 210023
    2ZTE Corporation, Nanjing 210012
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202440373
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT