收藏切换
Towards Network Infrastructure Research for the Era of Large Language Models: Challenges, Practices, and Prospects
收藏切换
Ennan Zhai, Jiamin Cao, Kun Qian, Yu Guan
Journal of Computer Research and Development | 2024, 61(11) : 2664 - 2677
Less
收藏切换
Journal of Computer Research and Development | 2024, 61(11): 2664-2677
• Research Highlights •
Towards Network Infrastructure Research for the Era of Large Language Models: Challenges, Practices, and Prospects
Full
Ennan Zhai, Jiamin Cao, Kun Qian, Yu Guan
Affiliations
  • Alibaba Cloud Computing Ltd., Hangzhou 310000
  • born in 1984. PhD. Member of CCF. His main research interests include computer network and distributed systems

    born in 1997. PhD. Member of CCF. Her main research interests include AI infrastructure and programmable networks

    born in 1993. PhD. Member of CCF. His main research interests include AI infrastructure and storage

    born in 1993. PhD. His main research interests include AI infrastructure and video streaming

About Author:

born in 1984. PhD. Member of CCF. His main research interests include computer network and distributed systems

born in 1997. PhD. Member of CCF. Her main research interests include AI infrastructure and programmable networks

born in 1993. PhD. Member of CCF. His main research interests include AI infrastructure and storage

born in 1993. PhD. His main research interests include AI infrastructure and video streaming

Published: 2024-11-01 doi: 10.7544/issn1000-1239.202440576
Outline
收藏切换

Large language models (LLMs) with hundreds of billions of parameters have brought significant technological and business transformations to today’s AI and cloud services. However, there exists a fundamental difference in network pattern between LLM training and general cloud computing (e.g., Amazon EC2 Elastic compute service), leading to a variety of new challenges. These challenges mainly include load balancing difficulties due to the traffic pattern difference (Challenge 1), the impact of multi-job communication contention on GPU utilization (Challenge 2), and high sensitivity to network failures (Challenge 3). Therefore, data center network technologies designed for general cloud computing (e.g., network architecture, routing, communication scheduling, and reliability) are no longer suitable for LLM training today. This necessitates the development of new data center networks and accompanying technical solutions specifically for LLM training. We introduce Alibaba Cloud’s high-performance network (HPN) and the multi-job communication scheduling approach Crux, designed to address the aforementioned challenges. HPN introduces a two-layer, dual-plane network architecture, which not only achieves high-speed interconnectivity for 15 000 GPUs within a Pod but also ensures precise routing suitable for LLM training (addressing Challenge 1). Furthermore, HPN proposes a novel dual-top-of-rack (ToR) design, replacing the traditional single ToR switch connection in data center networks and fundamentally avoiding single-point failure reliability risks (partially addressing Challenge 3). To tackle Challenge 2, Crux reduces the NP-complete problem of optimizing GPU utilization by modeling it as a communication scheduling issue related to GPU computational intensity. Crux then proposes an algorithm that prioritizes the flows of job with higher GPU computational intensity, significantly reducing multi-job communication contention and improving GPU utilization. Compared with the state-of-the-art efforts, Crux increases GPU utilization by up to 23%. Both HPN and Crux have been deployed and used in Alibaba Cloud production for over eight months and will continue to evolve and iterate. Building on this, we further envision possible research directions in the field of LLM training and inference, providing guidance for subsequent work.

AI infrastructure  /  large language model (LLMs)  /  large models  /  model training  /  data center networks  /  collective communication  /  communication scheduling
翟恩南, 操佳敏, 钱坤, 关宇. 面向大模型时代的网络基础设施研究:挑战、阶段成果与展望. 计算机研究与发展, 2024 , 61 (11) : 2664 -2677 . DOI: 10.7544/issn1000-1239.202440576
Ennan Zhai, Jiamin Cao, Kun Qian, Yu Guan. Towards Network Infrastructure Research for the Era of Large Language Models: Challenges, Practices, and Prospects[J]. Journal of Computer Research and Development, 2024 , 61 (11) : 2664 -2677 . DOI: 10.7544/issn1000-1239.202440576
Year 2024 volume 61 Issue 11
71
34
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202440576
  • Receive Date:2024-06-25
  • Online Date:2026-11-17
  • Published:2024-11-01
Article Data
Affiliations
History
  • Received:2024-06-25
  • Revised:2024-09-20
Affiliations
    Alibaba Cloud Computing Ltd., Hangzhou 310000
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202440576
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT