收藏切换
Survey on System Optimization for Mixture of Experts in the Era of Large Models
收藏切换
Hongzhi Shi, Jian Zhao*, Yaqian Zhao, Ruyang Li, Hui Wei, Kekun Hu, Dongchao Wen, Liang Jin
Journal of Computer Research and Development | 2025, 62(5) : 1164 - 1189
Less
收藏切换
Journal of Computer Research and Development | 2025, 62(5): 1164-1189
• Artificial Intelligence •
Survey on System Optimization for Mixture of Experts in the Era of Large Models
Full
Hongzhi Shi, Jian Zhao*, Yaqian Zhao, Ruyang Li, Hui Wei, Kekun Hu, Dongchao Wen, Liang Jin
Affiliations
  • IEIT SYSTEMS Co., Ltd., Jinan 250101
  • Inspur (Beijing) Electronic Information Industry Co., Ltd., Beijing 100095
  • born in 1988. Master. Member of CCF. His main research interests include computer architecture and deep learning

    born in 1987. Master. His main research interests include computer architecture, and deep learning

    born in 1981. PhD, senior engineer. Senior member of CCF. Her main research interests include computer architecture and artificial intelligence

    born in 1990. PhD, senior engineer. Senior member of CCF. Her main research interests include deep reinforcement learning, autonomous driving, and scenario-oriented AI acceleration

    born in 1987. PhD, senior engineer. Member of CCF. His main research interests include virtual reality and 3D vision

    born in 1987. PhD, senior engineer. Member of CCF. His main research interests include graph computing and graph deep learning

    born in 1979. Master, professor. Member of IEEE and CCF. His main research interests include computer vision and trustworthy deep learning

    born in 1986. Master. His main research interests include computer vision and multimodalities

About Author:

born in 1988. Master. Member of CCF. His main research interests include computer architecture and deep learning

born in 1987. Master. His main research interests include computer architecture, and deep learning

born in 1981. PhD, senior engineer. Senior member of CCF. Her main research interests include computer architecture and artificial intelligence

born in 1990. PhD, senior engineer. Senior member of CCF. Her main research interests include deep reinforcement learning, autonomous driving, and scenario-oriented AI acceleration

born in 1987. PhD, senior engineer. Member of CCF. His main research interests include virtual reality and 3D vision

born in 1987. PhD, senior engineer. Member of CCF. His main research interests include graph computing and graph deep learning

born in 1979. Master, professor. Member of IEEE and CCF. His main research interests include computer vision and trustworthy deep learning

born in 1986. Master. His main research interests include computer vision and multimodalities

Published: 2025-05-01 doi: 10.7544/issn1000-1239.202440016
Outline
收藏切换

In recent years, large models have made unprecedented progresses in variety of domains, such as natural language processing and machine vision. Mixture of experts (MoE) has emerged as one of the most popular architectures for large models due to its distinct advantages in model parameter scalability, computational cost control and complex task processing. However, with the continuous increase of the parameter scale, the execution efficiency and scalability of the system are becoming increasingly challenging to meet the demand, and must be addressed urgently. The system optimization approach is an effective solution to solve this problem, which has become a hot research area. In light of this, we review the present research status of MoE system optimization techniques in the era of large model in this paper. To begin, we describe the present development state of work for MoE large model, and analyze the performance bottlenecks it faces on the system side. Then, we comprehensively sort out and deeply analyze the most recent research progress from four system core dimensions, ranging from memory occupation, communication latency, computational efficiency to parallel scaling, and compare and elaborate on the key technologies, application scenarios and optimization directions; finally, we summarize the current research state of MoE system optimization and outline some future research directions as well.

large model  /  mixture of experts  /  memory offloading  /  hierarchical communication  /  expert placement  /  expert activation prediction  /  adaptive parallelism
史宏志, 赵健, 赵雅倩, 李茹杨, 魏辉, 胡克坤, 温东超, 金良. 大模型时代的混合专家系统优化综述. 计算机研究与发展, 2025 , 62 (5) : 1164 -1189 . DOI: 10.7544/issn1000-1239.202440016
Hongzhi Shi, Jian Zhao, Yaqian Zhao, Ruyang Li, Hui Wei, Kekun Hu, Dongchao Wen, Liang Jin. Survey on System Optimization for Mixture of Experts in the Era of Large Models[J]. Journal of Computer Research and Development, 2025 , 62 (5) : 1164 -1189 . DOI: 10.7544/issn1000-1239.202440016
Year 2025 volume 62 Issue 5
91
35
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202440016
  • Receive Date:2024-01-12
  • Online Date:2026-11-10
  • Published:2025-05-01
Article Data
Affiliations
History
  • Received:2024-01-12
  • Revised:2024-09-18
  • Accepted:2024-10-15
Affiliations
    IEIT SYSTEMS Co., Ltd., Jinan 250101
    Inspur (Beijing) Electronic Information Industry Co., Ltd., Beijing 100095
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202440016
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT