收藏切换
MC2 Energy Consumption Model for Massively Distributed Data Parallel Training of Deep Neural Network
收藏切换
Jia Wei, Xingjun Zhang*, Longxiang Wang, Mingqiang Zhao, Xiaoshe Dong
Journal of Computer Research and Development | 2024, 61(12) : 2985 - 3004
Less
收藏切换
Journal of Computer Research and Development | 2024, 61(12): 2985-3004
• Computer Architecture •
MC2 Energy Consumption Model for Massively Distributed Data Parallel Training of Deep Neural Network
Full
Jia Wei, Xingjun Zhang*, Longxiang Wang, Mingqiang Zhao, Xiaoshe Dong
Affiliations
  • School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710127
  • born in 1997. PhD candidate. His research interests include computer architecture, high performance computing, and deep learning

    born in 1969. PhD, professor, PhD, supervisor. Senior member of CCF. His main research interests include computer architect­ure, high performance computing, big data storage system, and computer networks

    born in 1988. PhD. Member of CCF. His main research interests include data deduplication, big data storage system, and artificial intelligence

    born in 1998. Master candidate. His main research interests include job scheduling, high performance computing, and reinforcement learning

    born in 1963. PhD, professor, PhD supervisor. Member of CCF. His main research interests include high performance computer system, storage system, and cloud computing

About Author:

born in 1997. PhD candidate. His research interests include computer architecture, high performance computing, and deep learning

born in 1969. PhD, professor, PhD, supervisor. Senior member of CCF. His main research interests include computer architect­ure, high performance computing, big data storage system, and computer networks

born in 1988. PhD. Member of CCF. His main research interests include data deduplication, big data storage system, and artificial intelligence

born in 1998. Master candidate. His main research interests include job scheduling, high performance computing, and reinforcement learning

born in 1963. PhD, professor, PhD supervisor. Member of CCF. His main research interests include high performance computer system, storage system, and cloud computing

Published: 2024-12-01 doi: 10.7544/issn1000-1239.202330164
Outline
收藏切换

Deep neural network (DNN) have achieved state-of-the-art accuracy in many modern artificial intelligence (AI) tasks. In recent years, it has become increasingly popular to use high performance computing platforms for massively distributed parallel training of DNN. Energy consumption models have been crucial in designing and optimizing DNN for massively parallel training and restraining excessive energy consumption on HPC (high performance computing) platforms. Currently, most energy consumption models model the energy consumption of a single device or a cluster of multiple devices from a hardware perspective. From an energy consumption perspective, the need for disaggregated analysis of distributed parallel DNN applications has resulted in a dearth of energy consumption models that model the characteristics of distributed DNN applications. In this paper, we propose the “materials preprocessing-computing-communicating” three-stage MC2 model from the perspective of the essential features of DNN model training for the most commonly used DNN distributed data parallel training model. The model is validated by training the classical VGG16, ResNet50 networks and the latest Vision Transformer network using up to 128 MT nodes and 32 FT nodes on the domestic E-class prototype Tianhe-3. The experimental results show that the difference between MC2 and the actual energy measurements is only 2.84%. Compared with the four linear proportional energy models and the AR, SES, and ARIMA time prediction models, the accuracy of the model proposed is improved by 69.12%, 69.50%, 34.58%, 13.47%, 5.23%, 22.13%, and 10.53%, respectively. By using the models proposed in this paper, the energy consumption of DNN models at each stage and the overall energy consumption can be obtained on a supercomputer platform, which provides a basis for evaluating the efficiency of DNN energy-aware massively distributed parallel training and inference, as well as optimizing the strategies of task scheduling, job scheduling, model partitioning, and model pruning.

deep neural network (DNN)  /  energy consumption model  /  massively distributed training  /  data parallel  /  supercomputer
魏嘉, 张兴军, 王龙翔, 赵明强, 董小社. 面向深度神经网络大规模分布式数据并行训练的MC2能耗模型. 计算机研究与发展, 2024 , 61 (12) : 2985 -3004 . DOI: 10.7544/issn1000-1239.202330164
Jia Wei, Xingjun Zhang, Longxiang Wang, Mingqiang Zhao, Xiaoshe Dong. MC2 Energy Consumption Model for Massively Distributed Data Parallel Training of Deep Neural Network[J]. Journal of Computer Research and Development, 2024 , 61 (12) : 2985 -3004 . DOI: 10.7544/issn1000-1239.202330164
Year 2024 volume 61 Issue 12
99
47
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202330164
  • Receive Date:2023-03-17
  • Online Date:2026-11-17
  • Published:2024-12-01
Article Data
Affiliations
History
  • Received:2023-03-17
  • Revised:2023-08-15
Affiliations
    School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710127
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202330164
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT