Article(id=1327013038424699357, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1327012646391493336, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202550143, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=research-article, receivedDate=1740758400000, receivedDateStr=2025-03-01, revisedDate=1744041600000, revisedDateStr=2025-04-08, acceptedDate=null, acceptedDateStr=null, onlineDate=1794314163069, onlineDateStr=2026-11-10, pubDate=1748793600000, pubDateStr=2025-06-02, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1794314163069, onlineIssueDateStr=2026-11-10, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1794314163069, creator=13701087609, updateTime=1794314163069, updator=13701087609, issue=Issue{id=1327012646391493336, tenantId=1146029695717560320, journalId=1324759914146529478, year='2025', volume='62', issue='6', pageStart='1331', pageEnd='1593', issueExtLink='null', onlineDate='null', pubDate='1748793600000', pubDateStr='2025-06-02', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1794314069601, creator='13701087609', updateTime=1794376817187, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext={EN=IssueExt(id=1327275829270262152, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1327012646391493336, language=EN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=), CN=IssueExt(id=1327275829270262153, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1327012646391493336, language=CN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=)}, issueFiles=null, downloadFileDto=null}, startPage=1333, endPage=1346, ext={EN=ArticleExt(id=1327013038667968990, articleId=1327013038424699357, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=SW-IntraCC: A Collective Communication Mechanism for the Internals of Sunway AI Acceleration, columnId=1327012740067078573, journalTitle=Journal of Computer Research and Development, columnName=High Performance Computing System, runingTitle=null, highlight=null, articleAbstract=

The number of large-scale language model parameters is growing exponentially, which puts forward higher requirements on the arithmetic density and communication efficiency of the acceleration card, and promotes the rapid development of new architectures, such as single-card multi-core, multi-chip and multi-communication entities. Sunway AI acceleration card adopts four-core group on-chip Ring bus architecture, but in the large model training, the data communication volume is large, and the traditional Ring collection communication method faces the core bottlenecks such as the double limitation of single-core group memory capacity and transmission bandwidth, low collection communication efficiency, and the inability to overlap the communication and computation. In this paper, the optimization framework of SW-IntraCC (Sunway-intra collective communication) is proposed by adopting the concept of software-hardware collaborative design to break through the above limitations through the three-tier storage architecture. First, the three-tier storage architecture is constructed based on on-chip high-speed Ring network, which expands the memory capacity of a single core group by up to four times and increases the host-accelerator card transmission bandwidth by 2.5 times; Second, an intra-chip cross shared communication (CSC) algorithm is designed with interleaved memory access patterns, implementing CSC-AG (CSC-AllGather) and CSC-RS (CSC-ReduceScatter) operators optimized for large model training. Benchmark results demonstrate that CSC achieves 2.15 times higher communication efficiency compared with conventional collective primitives. Finally, a bidirectional operator fusion strategy is proposed to enable communication-computation overlap, yielding a 59% improvement in communication performance after optimization.

, authors=Yulong Zhao1, Yanqing Gu1, Songtao Tian1, Chunzhi Wu2, Lingtao Tang1, Lufei Zhang1, Xiaojun Qin1, Xin Liu3, Zuoning Chen3, authorsList=Yulong Zhao, Yanqing Gu, Songtao Tian, Chunzhi Wu, Lingtao Tang, Lufei Zhang, Xiaojun Qin, Xin Liu, Zuoning Chen, authorCompany=null, correspAuthors=null, authorNote=

born in 1984. PhD candidate. His main research interest includes artificial intelligence infrastructure software

born in 1992. Master. His main research interests include high-performance computing and operating systems

born in 1983. Bachelor. His main research interests include high-performance computing and operating systems

born in 1991. PhD candidate. His main research interests include space launch equipment assurance and deep learning algorithms

born in 1994. PhD. His main research interests include machine learning, privacy protection, and intelligent operating systems

born in 1986. PhD, senior engineer. His main research interests include high-performance computing, operating systems, and machine learning

born in 1975. PhD. Full senior engineer. His main research interests include system security, software vulnerability analysis, and artificial intelligence

born in 1979. PhD, professor. Her main research interest includes parallel algorithms and applications

born in 1957. PhD, PhD supervisor, Academician of Chinese Academy of Engineering. Her main research interests include software theory, operating systems, and information security

, correspAuthorsNote=null, copyrightStatement=Copyright © 2025 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1327013043843740177, articleId=1327013038424699357, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=SW-IntraCC:一种面向申威智能加速卡内部的集合通信机制, columnId=1327012740230656431, journalTitle=计算机研究与发展, columnName=高性能计算系统, runingTitle=null, highlight=null, articleAbstract=

大规模语言模型参数量呈指数级增长趋势,对加速卡算力密度与通信效率提出更高要求,推动单卡多芯粒、多芯片及多通信实体等新型架构的快速发展. 申威智能加速卡采用4个核组片上环网架构,但在大模型训练中,数据通信量大和卡内传统Ring集合通信方式面临单核组显存容量与传输带宽双重限制、卡内集合通信效率低、通信计算无法重叠等核心瓶颈. 采用软硬协同设计理念提出SW-IntraCC(Sunway-intra collective communication)的优化框架,通过三级存储架构突破上述限制. 首先,基于片上高速环网构建三级存储架构,单核组显存容量最高扩大至4倍,主机-加速卡传输带宽提升2.5倍;其次,设计采用交叉共享访存的片内高效CSC(cross shared communication)通信算法,实现面向大模型训练的典型通信算子CSC-AG(CSC-AllGather)和CSC-RS(CSC-ReduceScatter),通信效率是传统方式的2.15倍;最后,提出双向算子融合的通信计算重叠方法,实现通信与计算重叠,优化后通信性能提升59%.

, authors=赵玉龙1, 顾燕卿1, 田松涛1, 吴春志2, 汤凌韬1, 张鲁飞1, 秦晓军1, 刘鑫3, 陈左宁3, authorsList=赵玉龙, 顾燕卿, 田松涛, 吴春志, 汤凌韬, 张鲁飞, 秦晓军, 刘鑫, 陈左宁, authorCompany=null, correspAuthors=null, authorNote=

1984年生. 博士研究生. 主要研究方向为人工智能基础软件

1992年生. 硕士. 主要研究方向为高性能计算、操作系统

1983年生. 学士. 主要研究方向为高性能计算、操作系统

1991年生. 博士研究生. 主要研究方向为航天发射装备保障、深度学习算法

1994年生. 博士. 主要研究方向为机器学习、隐私保护、智能操作系统

1986年生. 博士,高级工程师. 主要研究方向为高性能计算、操作系统、机器学习

1975年生. 博士. 正高级工程师. 主要研究方向为系统安全、软件脆弱性分析、人工智能

1979年生. 博士,研究员. 主要研究方向为并行算法及其应用

1957年生. 博士,博士生导师,中国工程院院士. 主要研究方向为软件理论、操作系统、信息安全

, correspAuthorsNote=null, copyrightStatement=版权所有 © 2025 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=oubZPAHWJBXUlgAA67kStA==, magXml=lWKFsGqWnHXJi8SzBM7FcQ==, pdfUrl=null, pdf=2JWNIUGpoaAEeDTFjD3y6w==, pdfFileSize=1546621, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=k6WW8QxACjrpjM4ybGyzmw==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=2VpktlDwDA612FP+RmOjzg==, mapNumber=null, fund=null)}, authors=[Author(id=1327013044397388317, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=yyylx@263.net, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327013044498051615, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013044397388317, language=EN, stringName=Yulong Zhao, firstName=Yulong, middleName=null, lastName=Zhao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1984. PhD candidate. His main research interest includes artificial intelligence infrastructure software

"}, bioImg=null, bioContent=

born in 1984. PhD candidate. His main research interest includes artificial intelligence infrastructure software

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327013044569354784, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013044397388317, language=CN, stringName=赵玉龙, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"Qg28vE6ZmdgpT+odMtp7qA==","content":"

1984年生. 博士研究生. 主要研究方向为人工智能基础软件

"}, bioImg=Qg28vE6ZmdgpT+odMtp7qA==, bioContent=

1984年生. 博士研究生. 主要研究方向为人工智能基础软件

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327013044061843987, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=1, ext=[AuthorCompanyExt(id=1327013044074426900, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1327013044082815509, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1327013044636463650, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327013044724544036, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013044636463650, language=EN, stringName=Yanqing Gu, firstName=Yanqing, middleName=null, lastName=Gu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1992. Master. His main research interests include high-performance computing and operating systems

"}, bioImg=null, bioContent=

born in 1992. Master. His main research interests include high-performance computing and operating systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327013044795847205, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013044636463650, language=CN, stringName=顾燕卿, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"7UTDfhjxGsPyZ0GxsU2AyA==","content":"

1992年生. 硕士. 主要研究方向为高性能计算、操作系统

"}, bioImg=7UTDfhjxGsPyZ0GxsU2AyA==, bioContent=

1992年生. 硕士. 主要研究方向为高性能计算、操作系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327013044061843987, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=1, ext=[AuthorCompanyExt(id=1327013044074426900, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1327013044082815509, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1327013044871344679, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327013044946842153, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013044871344679, language=EN, stringName=Songtao Tian, firstName=Songtao, middleName=null, lastName=Tian, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1983. Bachelor. His main research interests include high-performance computing and operating systems

"}, bioImg=null, bioContent=

born in 1983. Bachelor. His main research interests include high-performance computing and operating systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327013045110420011, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013044871344679, language=CN, stringName=田松涛, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"elbCuLSKXszZVS/G5/25ww==","content":"

1983年生. 学士. 主要研究方向为高性能计算、操作系统

"}, bioImg=elbCuLSKXszZVS/G5/25ww==, bioContent=

1983年生. 学士. 主要研究方向为高性能计算、操作系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327013044061843987, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=1, ext=[AuthorCompanyExt(id=1327013044074426900, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1327013044082815509, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1327013045194306093, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327013045286580783, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013045194306093, language=EN, stringName=Chunzhi Wu, firstName=Chunzhi, middleName=null, lastName=Wu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2Space Engineering University, Beijing 100004, bio={"content":"

born in 1991. PhD candidate. His main research interests include space launch equipment assurance and deep learning algorithms

"}, bioImg=null, bioContent=

born in 1991. PhD candidate. His main research interests include space launch equipment assurance and deep learning algorithms

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327013045362078256, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013045194306093, language=CN, stringName=吴春志, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2航天工程大学 北京 100004, bio={"img":"1Te0sLEsMBry79868RpDvQ==","content":"

1991年生. 博士研究生. 主要研究方向为航天发射装备保障、深度学习算法

"}, bioImg=1Te0sLEsMBry79868RpDvQ==, bioContent=

1991年生. 博士研究生. 主要研究方向为航天发射装备保障、深度学习算法

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327013044162507286, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=2, ext=[AuthorCompanyExt(id=1327013044217033239, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044162507286, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Space Engineering University, Beijing 100004), AuthorCompanyExt(id=1327013044221227544, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044162507286, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2航天工程大学 北京 100004)])]), Author(id=1327013045462741554, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327013045546627636, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013045462741554, language=EN, stringName=Lingtao Tang, firstName=Lingtao, middleName=null, lastName=Tang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1994. PhD. His main research interests include machine learning, privacy protection, and intelligent operating systems

"}, bioImg=null, bioContent=

born in 1994. PhD. His main research interests include machine learning, privacy protection, and intelligent operating systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327013045630513717, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013045462741554, language=CN, stringName=汤凌韬, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"+xPurftt/LW1nXWcBx5TSg==","content":"

1994年生. 博士. 主要研究方向为机器学习、隐私保护、智能操作系统

"}, bioImg=+xPurftt/LW1nXWcBx5TSg==, bioContent=

1994年生. 博士. 主要研究方向为机器学习、隐私保护、智能操作系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327013044061843987, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=1, ext=[AuthorCompanyExt(id=1327013044074426900, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1327013044082815509, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1327013045714399799, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, orderNo=5, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327013045815063097, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013045714399799, language=EN, stringName=Lufei Zhang, firstName=Lufei, middleName=null, lastName=Zhang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1986. PhD, senior engineer. His main research interests include high-performance computing, operating systems, and machine learning

"}, bioImg=null, bioContent=

born in 1986. PhD, senior engineer. His main research interests include high-performance computing, operating systems, and machine learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327013045886366266, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013045714399799, language=CN, stringName=张鲁飞, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"2ptlfac30AVN5QYR6a658g==","content":"

1986年生. 博士,高级工程师. 主要研究方向为高性能计算、操作系统、机器学习

"}, bioImg=2ptlfac30AVN5QYR6a658g==, bioContent=

1986年生. 博士,高级工程师. 主要研究方向为高性能计算、操作系统、机器学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327013044061843987, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=1, ext=[AuthorCompanyExt(id=1327013044074426900, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1327013044082815509, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1327013045970252348, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, orderNo=6, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327013046104470078, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013045970252348, language=EN, stringName=Xiaojun Qin, firstName=Xiaojun, middleName=null, lastName=Qin, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1975. PhD. Full senior engineer. His main research interests include system security, software vulnerability analysis, and artificial intelligence

"}, bioImg=null, bioContent=

born in 1975. PhD. Full senior engineer. His main research interests include system security, software vulnerability analysis, and artificial intelligence

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327013046179967551, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013045970252348, language=CN, stringName=秦晓军, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"KxUTqyS021esBcF0zSuIyA==","content":"

1975年生. 博士. 正高级工程师. 主要研究方向为系统安全、软件脆弱性分析、人工智能

"}, bioImg=KxUTqyS021esBcF0zSuIyA==, bioContent=

1975年生. 博士. 正高级工程师. 主要研究方向为系统安全、软件脆弱性分析、人工智能

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327013044061843987, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=1, ext=[AuthorCompanyExt(id=1327013044074426900, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1327013044082815509, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1327013046255465025, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, orderNo=7, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327013046347739715, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013046255465025, language=EN, stringName=Xin Liu, firstName=Xin, middleName=null, lastName=Liu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, address=3National Parallel Computer Engineering Technology Research Center, Beijing 100083, bio={"content":"

born in 1979. PhD, professor. Her main research interest includes parallel algorithms and applications

"}, bioImg=null, bioContent=

born in 1979. PhD, professor. Her main research interest includes parallel algorithms and applications

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327013046515511877, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013046255465025, language=CN, stringName=刘鑫, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, address=3国家并行计算机工程技术研究中心 北京 100083, bio={"img":"bTkJDDnmHuPvxqALYoj94w==","content":"

1979年生. 博士,研究员. 主要研究方向为并行算法及其应用

"}, bioImg=bTkJDDnmHuPvxqALYoj94w==, bioContent=

1979年生. 博士,研究员. 主要研究方向为并行算法及其应用

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327013044300919321, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=3, ext=[AuthorCompanyExt(id=1327013044309307930, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044300919321, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3National Parallel Computer Engineering Technology Research Center, Beijing 100083), AuthorCompanyExt(id=1327013044321890843, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044300919321, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3国家并行计算机工程技术研究中心 北京 100083)])]), Author(id=1327013046607786567, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, orderNo=8, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327013046691672649, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013046607786567, language=EN, stringName=Zuoning Chen, firstName=Zuoning, middleName=null, lastName=Chen, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, address=3National Parallel Computer Engineering Technology Research Center, Beijing 100083, bio={"content":"

born in 1957. PhD, PhD supervisor, Academician of Chinese Academy of Engineering. Her main research interests include software theory, operating systems, and information security

"}, bioImg=null, bioContent=

born in 1957. PhD, PhD supervisor, Academician of Chinese Academy of Engineering. Her main research interests include software theory, operating systems, and information security

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327013046775558730, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013046607786567, language=CN, stringName=陈左宁, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, address=3国家并行计算机工程技术研究中心 北京 100083, bio={"img":"F50nax0ZtLx+/Fzfny6huw==","content":"

1957年生. 博士,博士生导师,中国工程院院士. 主要研究方向为软件理论、操作系统、信息安全

"}, bioImg=F50nax0ZtLx+/Fzfny6huw==, bioContent=

1957年生. 博士,博士生导师,中国工程院院士. 主要研究方向为软件理论、操作系统、信息安全

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327013044300919321, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=3, ext=[AuthorCompanyExt(id=1327013044309307930, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044300919321, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3National Parallel Computer Engineering Technology Research Center, Beijing 100083), AuthorCompanyExt(id=1327013044321890843, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044300919321, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3国家并行计算机工程技术研究中心 北京 100083)])])], keywords=[Keyword(id=1327013046905582155, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, orderNo=1, keyword=collective communications), Keyword(id=1327013047002051148, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, orderNo=2, keyword=Sunway AI acceleration card), Keyword(id=1327013047081742925, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, orderNo=3, keyword=SW-IntraCC), Keyword(id=1327013047161434702, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, orderNo=4, keyword=communications optimization), Keyword(id=1327013047287263823, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, orderNo=5, keyword=ring network), Keyword(id=1327013047379538512, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, orderNo=1, keyword=集合通信), Keyword(id=1327013047492784721, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, orderNo=2, keyword=申威智能加速卡), Keyword(id=1327013047597642322, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, orderNo=3, keyword=SW-IntraCC), Keyword(id=1327013047681528403, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, orderNo=4, keyword=通信优化), Keyword(id=1327013047765414484, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, orderNo=5, keyword=环形网络)], refs=[Reference(id=1327013051515122302, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=王睿,张留洋,高志涌,等. 面向边缘智能的大模型研究进展[J/OL]. 计算机研究与发展,[2025-02-07]. http://kns.cnki.net/kcms/detail/11.1777.TP.20250127.0921.006.html, articleTitle=null, refAbstract=null), Reference(id=1327013051615785599, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Rui, Zhang Liuyang, Gao Zhiyong, et al. Progress in research on large models for edge-oriented intelligence[J/OL]. Computer Research and Development, [2025-02-07]. http://kns.cnki.net/kcms/detail/11.1777.TP.20250127.0921.006.html (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1327013051695477376, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning[J]. arXiv preprint, arXiv: 2501.12948, 2025. DOI: 10.48550/arXiv.2501.12948, articleTitle=null, refAbstract=null), Reference(id=1327013051787752065, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Ouyang S, Dong D Z, Xu Y M, et al. Communication optimization strategies for distributed deep neural network training: A survey[J]. Journal of Parallel and Distributed Computing, 2021, 149: 52–65. DOI: 10.1016/j.jpdc.2020.11.005, articleTitle=null, refAbstract=null), Reference(id=1327013051947135618, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Weingram A, Li Y, Qi H, et al. xCCL: A survey of industry-led collective communication libraries for deep learning[J]. Journal of Computer Science and Technology, 2023, 38: 166−195, articleTitle=null, refAbstract=null), Reference(id=1327013052022633091, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=NVIDIA. NVIDIA DGX−1 System Architecture White Paper 2017[R/OL].[2024-12-30]. https://images.nvidia.cn/content/pdf/dgx1-system-architecture-whitepaper1.pdf, articleTitle=null, refAbstract=null), Reference(id=1327013052085547652, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Tipparaju V, Nieplocha J, Panda D K. Fast collective operations using shared and remote memory access protocols on clusters[C]//Proc of Int Parallel and Distributed Processing Symposium. Washington, DC: IEEE Computer Society, 2003: 10, articleTitle=null, refAbstract=null), Reference(id=1327013052156850821, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Pabst S, Koch A, Straßer W. Fast and scalable CPU/GPU collision detection for rigid and deformable surfaces[J]. Computer Graphics Forum, 2010, 29(7): 2145−2154, articleTitle=null, refAbstract=null), Reference(id=1327013052236542598, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Xu Q, Jeon H, Annavaram M. Graph processing on GPUs: Where are the bottlenecks?[C]//Proc of 2014 IEEE Int Symp on Workload Characterization (IISWC). Piscataway, NJ: IEEE, 2014: 140−149, articleTitle=null, refAbstract=null), Reference(id=1327013052349788808, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Next Platform. The system bottleneck shifts to PCI-express[R/OL] [2017-07-14]. https://www.nextplatform.com/2017/07/14/system-bottleneck-shifts-pci-express/, articleTitle=null, refAbstract=null), Reference(id=1327013052416897673, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=黄彤彤, 陈昊, 武辰飞, 等. Concurrent multi-die optimization物理实现方案的应用[J]. 电子技术应用,2023,49(8):30−35, articleTitle=null, refAbstract=null), Reference(id=1327013052492395146, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Tongtong, Chen Hao, Wu Chenfei, et al. Application of physical implementation scheme for Concurrent multi-die optimization[J]. Electronics Applications, 2023, 49(8): 30−35 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1327013052630807179, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=曾嵘浩. 基于海上的的多目标跟踪与行为分析研究[D]. 西安:西安电子科技大学,2023. DOI:10.27389/d.cnki.gxadu.2023.001428, articleTitle=null, refAbstract=null), Reference(id=1327013052710498956, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Zeng Ronghao. Research on multi-target tracking and behavior analysis based on maritime[D]. Xi’an: Xi’an Electronic Science and Technology University, 2023. DOI:10.27389/d.cnki.gxadu.2023.001428 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1327013052836328077, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=俞铭辉. 基于国产智能芯片的模型移植和优化的研究[D]. 杭州:杭州电子科技大学,2024. DOI:10.27075/d.cnki.ghzdc.2024.000968, articleTitle=null, refAbstract=null), Reference(id=1327013052962157199, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Yu Minghui. Research on model transplantation and optimization based on domestic smart chip[D]. Hangzhou: Hangzhou University of Electronic Science and Technology, 2024. DOI:10.27075/d.cnki.ghzdc.2024.000968 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1327013053079597712, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=赵玉龙, 张鲁飞, 许国春, 等. SDAA:面向申威智能加速卡的运行时系统[J]. 软件学报,2024,35(12):5710−5724, articleTitle=null, refAbstract=null), Reference(id=1327013053176066705, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Yulong, Zhang Lufei, Xu Guochun, et al. SDAA: Runtime system for Sunway intelligent acceleration card[J]. Journal of Software, 2024, 35(12): 5710−5724 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1327013053297701522, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Wikipedia. RDMA over converged Ethernet[EB/OL]. Wikipedia, the free encyclopedia. 2017[2024-12-01]. https://en.wikipedia.org/w/index.php?title=RDMA_over_Converged_ Ethernet&oldid=782744462, articleTitle=null, refAbstract=null), Reference(id=1327013053373198995, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proc of the 31st Conf on Neural Information Processing Systems (NIPS 2017). Long Beach, CA: Curran Associates Inc, 2017: 5998−6008, articleTitle=null, refAbstract=null), Reference(id=1327013053436113556, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Sergeev A, Balso M D. Horovod: Fast and easy distributed deep learning in TensorFlow[J]. arXiv preprint, arXiv: 1802.05799, 2018, articleTitle=null, refAbstract=null), Reference(id=1327013053536776853, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Shoeybi M, Patwary M, Puri R, et al. Megatron-LM: Training multi-billion parameter language models using model parallelism[J]. arXiv preprint, arXiv 1909.08053, 2019, articleTitle=null, refAbstract=null), Reference(id=1327013053624857238, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Y, Cheng Y, Bapna A, et al. GPipe: Efficient training of giant neural networks using pipeline parallelism[C]//Advances in the 32nd Conf on Neural Information Processing Systems (NeurIPS 2019). Vancouver, CA: Curran Associates Inc 2019: 8−14, articleTitle=null, refAbstract=null), Reference(id=1327013053738103447, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=Narayanan D, Harlap A, Phanishayee A, et al. PipeDream: Generalized pipeline parallelism for DNN training[C/OL]//Proc of the 27th ACM Symp on Operating Systems Principles. New York: ACM, [2025-01-15]. https://doi.org/10.48550/arXiv.1806.03377, 2019, articleTitle=null, refAbstract=null), Reference(id=1327013053847155352, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=Li S, Xue F, Li Y, et al. Sequence parallelism: Long sequence training from system perspective[C]//Proc of the 61st Annual Meeting of the Association for Computational Linguistics. Toronto, Canada: 2021: 2391−2404, articleTitle=null, refAbstract=null), Reference(id=1327013053922652825, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=高亦沁,罗智宇,王一超,等. 面向国产超算的操作系统评测与优化[J/OL]. 计算机科学,[2025-02-11]. http://kns.cnki.net/kcms/detail/50.1075.tp.20240925.1330.007.html, articleTitle=null, refAbstract=null), Reference(id=1327013053989761690, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=26, authorNames=null, journalName=null, refType=null, unstructuredReference=Gao Yiqin, Luo Zhiyu, Wang Yichao, et al. Evaluation and optimization of operating system for domestic supercomputing[J/OL]. Computer Science, [2025-02-11]. http://kns.cnki.net/kcms/detail/50.1075.tp.20240925.1330.007.html (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1327013054073647771, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=27, authorNames=null, journalName=null, refType=null, unstructuredReference=Li M, Andersen D G, Park J W, et al. Scaling distributed machine learning with the parameter server[C]//Proc of the 11th USENIX Symp on Operating Systems Design and Implementation. Berkeley, CA: USENIX Association, 2014: 583−598, articleTitle=null, refAbstract=null), Reference(id=1327013054157533852, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=28, authorNames=null, journalName=null, refType=null, unstructuredReference=Träff J L. Optimal, non-pipelined Reduce-scatter and Allreduce algorithms[J]. arXiv preprint, arXiv: 2410.14234, 2024, articleTitle=null, refAbstract=null), Reference(id=1327013054295945885, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=29, authorNames=null, journalName=null, refType=null, unstructuredReference=Wasi-ur-Rahman M, Lu X, Islam N S, et al. HOMR: A hybrid approach to exploit maximum overlapping in MapReduce over high performance interconnects[C]//Proc of the Int Conf on Supercomputing (ICS). New York: ACM, 2014: 33−42. DOI: 10.1145/2597652.2597684, articleTitle=null, refAbstract=null), Reference(id=1327013054392414878, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=30, authorNames=null, journalName=null, refType=null, unstructuredReference=Ba J, Kiros J R, Hinton G E. Layer normalization[J]. arXiv preprint, arXiv: 1607.06450, 2016, articleTitle=null, refAbstract=null), Reference(id=1327013054476300959, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=31, authorNames=null, journalName=null, refType=null, unstructuredReference=Black S, Biderman S, Hallahan E, et al. GPT-NeoX−20B: An open-source autoregressive language model[J]. arXiv preprint, arXiv: 2204.06745, 2022, articleTitle=null, refAbstract=null), Reference(id=1327013054572769952, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=32, authorNames=null, journalName=null, refType=null, unstructuredReference=Touvron H, Martin L, Stone K, et al. Llama 2: Open foundation and fine-tuned chat models[J]. arXiv preprint, arXiv: 2307.09288, 2023, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1327013044061843987, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=1, ext=[AuthorCompanyExt(id=1327013044074426900, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1327013044082815509, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)]), AuthorCompany(id=1327013044162507286, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=2, ext=[AuthorCompanyExt(id=1327013044217033239, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044162507286, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Space Engineering University, Beijing 100004), AuthorCompanyExt(id=1327013044221227544, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044162507286, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2航天工程大学 北京 100004)]), AuthorCompany(id=1327013044300919321, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=3, ext=[AuthorCompanyExt(id=1327013044309307930, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044300919321, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3National Parallel Computer Engineering Technology Research Center, Beijing 100083), AuthorCompanyExt(id=1327013044321890843, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044300919321, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3国家并行计算机工程技术研究中心 北京 100083)])], figs=[ArticleFig(id=1327013047916409429, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Figure 1, caption=Structure of Huawei Ascend 310 accelerator card, figureFileSmall=mIuym0q6LtUh9CPFKdrybw==, figureFileBig=k6WW8QxACjrpjM4ybGyzmw==, tableContent=null), ArticleFig(id=1327013048004489814, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=图1, caption=华为昇腾310加速卡结构, figureFileSmall=mIuym0q6LtUh9CPFKdrybw==, figureFileBig=k6WW8QxACjrpjM4ybGyzmw==, tableContent=null), ArticleFig(id=1327013048247759447, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Figure 2, caption=Sunway AI processor architecture, figureFileSmall=W6nmwOryl05hr07P0BmfXw==, figureFileBig=k6JCftjalqpQ8OypOLnEmQ==, tableContent=null), ArticleFig(id=1327013048348422744, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=图2, caption=申威AI处理器架构, figureFileSmall=W6nmwOryl05hr07P0BmfXw==, figureFileBig=k6JCftjalqpQ8OypOLnEmQ==, tableContent=null), ArticleFig(id=1327013048419725913, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Figure 3, caption=TP+SP parallel scheme in the accelerating card, figureFileSmall=ULrgMki6/RMjR8dyGHPSBg==, figureFileBig=BddHiQpzhSkctfg8TIl6ig==, tableContent=null), ArticleFig(id=1327013048503611994, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=图3, caption=加速卡内TP+SP并行方案, figureFileSmall=ULrgMki6/RMjR8dyGHPSBg==, figureFileBig=BddHiQpzhSkctfg8TIl6ig==, tableContent=null), ArticleFig(id=1327013048570720859, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Figure 4, caption=Illustration of communication of Ring-AG algorithm, figureFileSmall=X4QLG3rirYoQrRDcNQULlQ==, figureFileBig=v+oi0SOWxBzVfCt0xeVGIg==, tableContent=null), ArticleFig(id=1327013048683967068, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=图4, caption=Ring-AG算法的通信示意图, figureFileSmall=X4QLG3rirYoQrRDcNQULlQ==, figureFileBig=v+oi0SOWxBzVfCt0xeVGIg==, tableContent=null), ArticleFig(id=1327013048767853149, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Figure 5, caption=Architecture of two-stier storage, figureFileSmall=RoflQaCSB1cCeDXPtuanoQ==, figureFileBig=iBDdNVC/CJVdDEWdEhwwpA==, tableContent=null), ArticleFig(id=1327013048851739230, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=图5, caption=两级存储架构, figureFileSmall=RoflQaCSB1cCeDXPtuanoQ==, figureFileBig=iBDdNVC/CJVdDEWdEhwwpA==, tableContent=null), ArticleFig(id=1327013048918848095, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Figure 6, caption=Three-tier storage architecture based on ring-on-chip, figureFileSmall=8HjOCnogLJFxzCdIPCvvLA==, figureFileBig=OBpQ6Rx/dwAkvruRAVgraw==, tableContent=null), ArticleFig(id=1327013048994345568, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=图6, caption=基于片上环网的三级存储架构, figureFileSmall=8HjOCnogLJFxzCdIPCvvLA==, figureFileBig=OBpQ6Rx/dwAkvruRAVgraw==, tableContent=null), ArticleFig(id=1327013049082425953, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Figure 7, caption=SP+TP optimization based on CSC algorithm, figureFileSmall=LCZO0kCBiUFcC3vvOJ/NaA==, figureFileBig=ArsyFYiO7RgttNcG5Z3GcA==, tableContent=null), ArticleFig(id=1327013049162117730, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=图7, caption=基于CSC算法的SP+TP优化, figureFileSmall=LCZO0kCBiUFcC3vvOJ/NaA==, figureFileBig=ArsyFYiO7RgttNcG5Z3GcA==, tableContent=null), ArticleFig(id=1327013049254392419, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Figure 8, caption=Mechanism of memory allocation, figureFileSmall=FkkHOS24J+LcpgJBx5SsyQ==, figureFileBig=Im2DOUvJuXh/8Yk7ksw5hA==, tableContent=null), ArticleFig(id=1327013049334084196, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=图8, caption=内存分配机制, figureFileSmall=FkkHOS24J+LcpgJBx5SsyQ==, figureFileBig=Im2DOUvJuXh/8Yk7ksw5hA==, tableContent=null), ArticleFig(id=1327013049405387365, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Figure 9, caption=CSC-AG algorithm in the four-core group, figureFileSmall=OyyZpks84VR4uIObyUMisA==, figureFileBig=tHbgoc/kbiN34C7rgMSVTA==, tableContent=null), ArticleFig(id=1327013049480884838, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=图9, caption=四核组CSC-AG算法, figureFileSmall=OyyZpks84VR4uIObyUMisA==, figureFileBig=tHbgoc/kbiN34C7rgMSVTA==, tableContent=null), ArticleFig(id=1327013049568965223, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Figure 10, caption=Bidirectional operator fusion for overlapping methods in communication computing, figureFileSmall=q1S/5ZO3egU86egZYPLOcQ==, figureFileBig=3B7cxiEj4DwP0jsGcPBuYQ==, tableContent=null), ArticleFig(id=1327013049657045608, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=图10, caption=双向算子融合的通信计算重叠方法, figureFileSmall=q1S/5ZO3egU86egZYPLOcQ==, figureFileBig=3B7cxiEj4DwP0jsGcPBuYQ==, tableContent=null), ArticleFig(id=1327013049753514601, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Figure 11, caption=Communication calculation overlapping of the slave core internal, figureFileSmall=GgP+/V13hIsiuTHxjl1E4g==, figureFileBig=0itd8gyuuc/stTQpPQD59A==, tableContent=null), ArticleFig(id=1327013049833206378, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=图11, caption=从核内部通信计算重叠, figureFileSmall=GgP+/V13hIsiuTHxjl1E4g==, figureFileBig=0itd8gyuuc/stTQpPQD59A==, tableContent=null), ArticleFig(id=1327013049908703851, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Figure 12, caption=Host-accelerator card transfer performance under different storage architectures, figureFileSmall=pICdeXCwzrYW3wsn8ZiHQA==, figureFileBig=5lGm0/IilLwhCfMikqB8yg==, tableContent=null), ArticleFig(id=1327013049984201324, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=图12, caption=不同存储架构下主机-加速卡传输性能, figureFileSmall=pICdeXCwzrYW3wsn8ZiHQA==, figureFileBig=5lGm0/IilLwhCfMikqB8yg==, tableContent=null), ArticleFig(id=1327013050055504493, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Figure 13, caption=Bandwidth utilization of CSC and Ring algorithms, figureFileSmall=CajQPGM2DFrrP45ntAGdFg==, figureFileBig=W2yRkCaXYQiBvijbQ2zj/Q==, tableContent=null), ArticleFig(id=1327013050139390574, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=图13, caption=CSC算法和Ring算法的带宽利用率, figureFileSmall=CajQPGM2DFrrP45ntAGdFg==, figureFileBig=W2yRkCaXYQiBvijbQ2zj/Q==, tableContent=null), ArticleFig(id=1327013050206499439, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Figure 14, caption=Bandwidth comparison of CSC-AG and Ring-AG algorithms, figureFileSmall=Lp3Lgr+P7zG1jg6Z+JzYlQ==, figureFileBig=FuRkqpc5l3OrH/oR+zvODQ==, tableContent=null), ArticleFig(id=1327013050294579824, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=图14, caption=CSC-AG and Ring-AG算法的带宽对比, figureFileSmall=Lp3Lgr+P7zG1jg6Z+JzYlQ==, figureFileBig=FuRkqpc5l3OrH/oR+zvODQ==, tableContent=null), ArticleFig(id=1327013050382660209, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Figure 15, caption=Bandwidth comparison of Ring-RS和CSC-RS algorithms, figureFileSmall=85e3sIGlaDplgAczN4X/YQ==, figureFileBig=Y2woW13/YDwHq7qHLY7Dgw==, tableContent=null), ArticleFig(id=1327013050453963378, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=图15, caption=Ring-RS和CSC-RS算法的带宽对比, figureFileSmall=85e3sIGlaDplgAczN4X/YQ==, figureFileBig=Y2woW13/YDwHq7qHLY7Dgw==, tableContent=null), ArticleFig(id=1327013050521072243, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Table 1, caption=Runtime Library Memory Management Interface, figureFileSmall=null, figureFileBig=null, tableContent=
SDAA接口功能描述备注
sdaaMalloc在设备端Local-HBM
分配内存
已有接口,最大15 GB
sdaaMallocCross在设备端Global-HBM
分配内存
新增接口,最大分配60 GB
sdaaFree释放Local-HBM已
分配内存
已有接口
sdaaFreeCross释放Global-HBM已
分配内存
新增接口
), ArticleFig(id=1327013050613346932, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=表1, caption=

运行时库内存管理接口

, figureFileSmall=null, figureFileBig=null, tableContent=
SDAA接口功能描述备注
sdaaMalloc在设备端Local-HBM
分配内存
已有接口,最大15 GB
sdaaMallocCross在设备端Global-HBM
分配内存
新增接口,最大分配60 GB
sdaaFree释放Local-HBM已
分配内存
已有接口
sdaaFreeCross释放Global-HBM已
分配内存
新增接口
), ArticleFig(id=1327013050705621621, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Table 2, caption=Testing Environment, figureFileSmall=null, figureFileBig=null, tableContent=
服务器配置操作系统加速卡配置驱动运行时
系统
CPU Silver4214 2.20 GHz
内存 DDR4 128 GB
CentOS
7.8
申威智能加速卡AI-driver1.0-TSDAA
1.0-T
), ArticleFig(id=1327013050793702006, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=表2, caption=

测试环境

, figureFileSmall=null, figureFileBig=null, tableContent=
服务器配置操作系统加速卡配置驱动运行时
系统
CPU Silver4214 2.20 GHz
内存 DDR4 128 GB
CentOS
7.8
申威智能加速卡AI-driver1.0-TSDAA
1.0-T
), ArticleFig(id=1327013050873393783, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Table 3, caption=Experimental Data of CSC-AG for Communi-cation Calculation Overlapping, figureFileSmall=null, figureFileBig=null, tableContent=
通信数据量/KB通讯时间/μs重叠收益/μs收益百分比/%
2048321959
4096552952
81922004654
163841958443
3276835618150
), ArticleFig(id=1327013050965668472, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=表3, caption=

CSC-AG通信计算重叠实验数据

, figureFileSmall=null, figureFileBig=null, tableContent=
通信数据量/KB通讯时间/μs重叠收益/μs收益百分比/%
2048321959
4096552952
81922004654
163841958443
3276835618150
), ArticleFig(id=1327013051057943162, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Table 4, caption=Experimental Data of CSC-RS for Communication Calculation Overlapping, figureFileSmall=null, figureFileBig=null, tableContent=
通信数据量/KB通讯时间/μs重叠收益/μs收益百分比/%
2048541935
4096983233
81921845832
1638434711633
3276863120933
), ArticleFig(id=1327013051150217851, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=表4, caption=

CSC-RS通信计算重叠实验数据

, figureFileSmall=null, figureFileBig=null, tableContent=
通信数据量/KB通讯时间/μs重叠收益/μs收益百分比/%
2048541935
4096983233
81921845832
1638434711633
3276863120933
), ArticleFig(id=1327013051242492540, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=EN, label=Table 5, caption=Expected Performance of the Intra Card Comm-unication Training, figureFileSmall=null, figureFileBig=null, tableContent=
模型通信方式单层时间/s每轮迭代/s预训练/d
GPT-NeoX 20BRing-TP1.283.526.1
双向融合后CSC-TP0.802.203.8
LLaMa 70BRing-TP1.155.7534.9
双向融合后CSC-TP0.713.5621.6
), ArticleFig(id=1327013051364127357, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, language=CN, label=表5, caption=

卡内通信训练的预期性能

, figureFileSmall=null, figureFileBig=null, tableContent=
模型通信方式单层时间/s每轮迭代/s预训练/d
GPT-NeoX 20BRing-TP1.283.526.1
双向融合后CSC-TP0.802.203.8
LLaMa 70BRing-TP1.155.7534.9
双向融合后CSC-TP0.713.5621.6
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202550143, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202550143, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202550143, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202550143, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1748793600000, fullTextJson=null, articleText=null, reference=null)
收藏切换
SW-IntraCC:一种面向申威智能加速卡内部的集合通信机制
赵玉龙 1 , 顾燕卿 1 , 田松涛 1 , 吴春志 2 , 汤凌韬 1 , 张鲁飞 1 , 秦晓军 1 , 刘鑫 3 , 陈左宁 3
计算机研究与发展 | 高性能计算系统 2025,62(6): 1333-1346
收起
收藏切换
计算机研究与发展 |高性能计算系统 2025 , 62 (6) : 1333 -1346
SW-IntraCC:一种面向申威智能加速卡内部的集合通信机制
全屏
[Author(id=1327013044397388317, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=yyylx@263.net, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327013044498051615, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013044397388317, language=EN, stringName=Yulong Zhao, firstName=Yulong, middleName=null, lastName=Zhao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1984. PhD candidate. His main research interest includes artificial intelligence infrastructure software

"}, bioImg=null, bioContent=

born in 1984. PhD candidate. His main research interest includes artificial intelligence infrastructure software

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327013044569354784, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013044397388317, language=CN, stringName=赵玉龙, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"Qg28vE6ZmdgpT+odMtp7qA==","content":"

1984年生. 博士研究生. 主要研究方向为人工智能基础软件

"}, bioImg=Qg28vE6ZmdgpT+odMtp7qA==, bioContent=

1984年生. 博士研究生. 主要研究方向为人工智能基础软件

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327013044061843987, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=1, ext=[AuthorCompanyExt(id=1327013044074426900, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1327013044082815509, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1327013044636463650, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327013044724544036, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013044636463650, language=EN, stringName=Yanqing Gu, firstName=Yanqing, middleName=null, lastName=Gu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1992. Master. His main research interests include high-performance computing and operating systems

"}, bioImg=null, bioContent=

born in 1992. Master. His main research interests include high-performance computing and operating systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327013044795847205, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013044636463650, language=CN, stringName=顾燕卿, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"7UTDfhjxGsPyZ0GxsU2AyA==","content":"

1992年生. 硕士. 主要研究方向为高性能计算、操作系统

"}, bioImg=7UTDfhjxGsPyZ0GxsU2AyA==, bioContent=

1992年生. 硕士. 主要研究方向为高性能计算、操作系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327013044061843987, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=1, ext=[AuthorCompanyExt(id=1327013044074426900, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1327013044082815509, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1327013044871344679, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327013044946842153, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013044871344679, language=EN, stringName=Songtao Tian, firstName=Songtao, middleName=null, lastName=Tian, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1983. Bachelor. His main research interests include high-performance computing and operating systems

"}, bioImg=null, bioContent=

born in 1983. Bachelor. His main research interests include high-performance computing and operating systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327013045110420011, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013044871344679, language=CN, stringName=田松涛, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"elbCuLSKXszZVS/G5/25ww==","content":"

1983年生. 学士. 主要研究方向为高性能计算、操作系统

"}, bioImg=elbCuLSKXszZVS/G5/25ww==, bioContent=

1983年生. 学士. 主要研究方向为高性能计算、操作系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327013044061843987, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=1, ext=[AuthorCompanyExt(id=1327013044074426900, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1327013044082815509, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1327013045194306093, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327013045286580783, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013045194306093, language=EN, stringName=Chunzhi Wu, firstName=Chunzhi, middleName=null, lastName=Wu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2Space Engineering University, Beijing 100004, bio={"content":"

born in 1991. PhD candidate. His main research interests include space launch equipment assurance and deep learning algorithms

"}, bioImg=null, bioContent=

born in 1991. PhD candidate. His main research interests include space launch equipment assurance and deep learning algorithms

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327013045362078256, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013045194306093, language=CN, stringName=吴春志, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2航天工程大学 北京 100004, bio={"img":"1Te0sLEsMBry79868RpDvQ==","content":"

1991年生. 博士研究生. 主要研究方向为航天发射装备保障、深度学习算法

"}, bioImg=1Te0sLEsMBry79868RpDvQ==, bioContent=

1991年生. 博士研究生. 主要研究方向为航天发射装备保障、深度学习算法

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327013044162507286, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=2, ext=[AuthorCompanyExt(id=1327013044217033239, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044162507286, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Space Engineering University, Beijing 100004), AuthorCompanyExt(id=1327013044221227544, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044162507286, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2航天工程大学 北京 100004)])]), Author(id=1327013045462741554, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327013045546627636, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013045462741554, language=EN, stringName=Lingtao Tang, firstName=Lingtao, middleName=null, lastName=Tang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1994. PhD. His main research interests include machine learning, privacy protection, and intelligent operating systems

"}, bioImg=null, bioContent=

born in 1994. PhD. His main research interests include machine learning, privacy protection, and intelligent operating systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327013045630513717, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013045462741554, language=CN, stringName=汤凌韬, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"+xPurftt/LW1nXWcBx5TSg==","content":"

1994年生. 博士. 主要研究方向为机器学习、隐私保护、智能操作系统

"}, bioImg=+xPurftt/LW1nXWcBx5TSg==, bioContent=

1994年生. 博士. 主要研究方向为机器学习、隐私保护、智能操作系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327013044061843987, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=1, ext=[AuthorCompanyExt(id=1327013044074426900, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1327013044082815509, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1327013045714399799, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, orderNo=5, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327013045815063097, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013045714399799, language=EN, stringName=Lufei Zhang, firstName=Lufei, middleName=null, lastName=Zhang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1986. PhD, senior engineer. His main research interests include high-performance computing, operating systems, and machine learning

"}, bioImg=null, bioContent=

born in 1986. PhD, senior engineer. His main research interests include high-performance computing, operating systems, and machine learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327013045886366266, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013045714399799, language=CN, stringName=张鲁飞, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"2ptlfac30AVN5QYR6a658g==","content":"

1986年生. 博士,高级工程师. 主要研究方向为高性能计算、操作系统、机器学习

"}, bioImg=2ptlfac30AVN5QYR6a658g==, bioContent=

1986年生. 博士,高级工程师. 主要研究方向为高性能计算、操作系统、机器学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327013044061843987, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=1, ext=[AuthorCompanyExt(id=1327013044074426900, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1327013044082815509, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1327013045970252348, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, orderNo=6, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327013046104470078, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013045970252348, language=EN, stringName=Xiaojun Qin, firstName=Xiaojun, middleName=null, lastName=Qin, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1975. PhD. Full senior engineer. His main research interests include system security, software vulnerability analysis, and artificial intelligence

"}, bioImg=null, bioContent=

born in 1975. PhD. Full senior engineer. His main research interests include system security, software vulnerability analysis, and artificial intelligence

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327013046179967551, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013045970252348, language=CN, stringName=秦晓军, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"KxUTqyS021esBcF0zSuIyA==","content":"

1975年生. 博士. 正高级工程师. 主要研究方向为系统安全、软件脆弱性分析、人工智能

"}, bioImg=KxUTqyS021esBcF0zSuIyA==, bioContent=

1975年生. 博士. 正高级工程师. 主要研究方向为系统安全、软件脆弱性分析、人工智能

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327013044061843987, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=1, ext=[AuthorCompanyExt(id=1327013044074426900, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1327013044082815509, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044061843987, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1327013046255465025, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, orderNo=7, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327013046347739715, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013046255465025, language=EN, stringName=Xin Liu, firstName=Xin, middleName=null, lastName=Liu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, address=3National Parallel Computer Engineering Technology Research Center, Beijing 100083, bio={"content":"

born in 1979. PhD, professor. Her main research interest includes parallel algorithms and applications

"}, bioImg=null, bioContent=

born in 1979. PhD, professor. Her main research interest includes parallel algorithms and applications

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327013046515511877, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013046255465025, language=CN, stringName=刘鑫, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, address=3国家并行计算机工程技术研究中心 北京 100083, bio={"img":"bTkJDDnmHuPvxqALYoj94w==","content":"

1979年生. 博士,研究员. 主要研究方向为并行算法及其应用

"}, bioImg=bTkJDDnmHuPvxqALYoj94w==, bioContent=

1979年生. 博士,研究员. 主要研究方向为并行算法及其应用

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327013044300919321, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=3, ext=[AuthorCompanyExt(id=1327013044309307930, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044300919321, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3National Parallel Computer Engineering Technology Research Center, Beijing 100083), AuthorCompanyExt(id=1327013044321890843, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044300919321, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3国家并行计算机工程技术研究中心 北京 100083)])]), Author(id=1327013046607786567, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, orderNo=8, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327013046691672649, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013046607786567, language=EN, stringName=Zuoning Chen, firstName=Zuoning, middleName=null, lastName=Chen, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, address=3National Parallel Computer Engineering Technology Research Center, Beijing 100083, bio={"content":"

born in 1957. PhD, PhD supervisor, Academician of Chinese Academy of Engineering. Her main research interests include software theory, operating systems, and information security

"}, bioImg=null, bioContent=

born in 1957. PhD, PhD supervisor, Academician of Chinese Academy of Engineering. Her main research interests include software theory, operating systems, and information security

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327013046775558730, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, authorId=1327013046607786567, language=CN, stringName=陈左宁, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, address=3国家并行计算机工程技术研究中心 北京 100083, bio={"img":"F50nax0ZtLx+/Fzfny6huw==","content":"

1957年生. 博士,博士生导师,中国工程院院士. 主要研究方向为软件理论、操作系统、信息安全

"}, bioImg=F50nax0ZtLx+/Fzfny6huw==, bioContent=

1957年生. 博士,博士生导师,中国工程院院士. 主要研究方向为软件理论、操作系统、信息安全

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327013044300919321, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, xref=3, ext=[AuthorCompanyExt(id=1327013044309307930, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044300919321, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3National Parallel Computer Engineering Technology Research Center, Beijing 100083), AuthorCompanyExt(id=1327013044321890843, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327013038424699357, companyId=1327013044300919321, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3国家并行计算机工程技术研究中心 北京 100083)])])]
赵玉龙1 , 顾燕卿1, 田松涛1, 吴春志2, 汤凌韬1, 张鲁飞1, 秦晓军1, 刘鑫3, 陈左宁3
作者信息
  • 1数学工程与先进计算国家重点实验室 江苏无锡 214125
  • 2航天工程大学 北京 100004
  • 3国家并行计算机工程技术研究中心 北京 100083
作者简介:

1984年生. 博士研究生. 主要研究方向为人工智能基础软件

1992年生. 硕士. 主要研究方向为高性能计算、操作系统

1983年生. 学士. 主要研究方向为高性能计算、操作系统

1991年生. 博士研究生. 主要研究方向为航天发射装备保障、深度学习算法

1994年生. 博士. 主要研究方向为机器学习、隐私保护、智能操作系统

1986年生. 博士,高级工程师. 主要研究方向为高性能计算、操作系统、机器学习

1975年生. 博士. 正高级工程师. 主要研究方向为系统安全、软件脆弱性分析、人工智能

1979年生. 博士,研究员. 主要研究方向为并行算法及其应用

1957年生. 博士,博士生导师,中国工程院院士. 主要研究方向为软件理论、操作系统、信息安全

SW-IntraCC: A Collective Communication Mechanism for the Internals of Sunway AI Acceleration
Yulong Zhao1 , Yanqing Gu1, Songtao Tian1, Chunzhi Wu2, Lingtao Tang1, Lufei Zhang1, Xiaojun Qin1, Xin Liu3, Zuoning Chen3
Affiliations
  • 1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125
  • 2Space Engineering University, Beijing 100004
  • 3National Parallel Computer Engineering Technology Research Center, Beijing 100083
  • born in 1984. PhD candidate. His main research interest includes artificial intelligence infrastructure software

    born in 1992. Master. His main research interests include high-performance computing and operating systems

    born in 1983. Bachelor. His main research interests include high-performance computing and operating systems

    born in 1991. PhD candidate. His main research interests include space launch equipment assurance and deep learning algorithms

    born in 1994. PhD. His main research interests include machine learning, privacy protection, and intelligent operating systems

    born in 1986. PhD, senior engineer. His main research interests include high-performance computing, operating systems, and machine learning

    born in 1975. PhD. Full senior engineer. His main research interests include system security, software vulnerability analysis, and artificial intelligence

    born in 1979. PhD, professor. Her main research interest includes parallel algorithms and applications

    born in 1957. PhD, PhD supervisor, Academician of Chinese Academy of Engineering. Her main research interests include software theory, operating systems, and information security

出版时间: 2025-06-02 doi: 10.7544/issn1000-1239.202550143
文章导航
收藏切换

大规模语言模型参数量呈指数级增长趋势,对加速卡算力密度与通信效率提出更高要求,推动单卡多芯粒、多芯片及多通信实体等新型架构的快速发展. 申威智能加速卡采用4个核组片上环网架构,但在大模型训练中,数据通信量大和卡内传统Ring集合通信方式面临单核组显存容量与传输带宽双重限制、卡内集合通信效率低、通信计算无法重叠等核心瓶颈. 采用软硬协同设计理念提出SW-IntraCC(Sunway-intra collective communication)的优化框架,通过三级存储架构突破上述限制. 首先,基于片上高速环网构建三级存储架构,单核组显存容量最高扩大至4倍,主机-加速卡传输带宽提升2.5倍;其次,设计采用交叉共享访存的片内高效CSC(cross shared communication)通信算法,实现面向大模型训练的典型通信算子CSC-AG(CSC-AllGather)和CSC-RS(CSC-ReduceScatter),通信效率是传统方式的2.15倍;最后,提出双向算子融合的通信计算重叠方法,实现通信与计算重叠,优化后通信性能提升59%.

集合通信  /  申威智能加速卡  /  SW-IntraCC  /  通信优化  /  环形网络

The number of large-scale language model parameters is growing exponentially, which puts forward higher requirements on the arithmetic density and communication efficiency of the acceleration card, and promotes the rapid development of new architectures, such as single-card multi-core, multi-chip and multi-communication entities. Sunway AI acceleration card adopts four-core group on-chip Ring bus architecture, but in the large model training, the data communication volume is large, and the traditional Ring collection communication method faces the core bottlenecks such as the double limitation of single-core group memory capacity and transmission bandwidth, low collection communication efficiency, and the inability to overlap the communication and computation. In this paper, the optimization framework of SW-IntraCC (Sunway-intra collective communication) is proposed by adopting the concept of software-hardware collaborative design to break through the above limitations through the three-tier storage architecture. First, the three-tier storage architecture is constructed based on on-chip high-speed Ring network, which expands the memory capacity of a single core group by up to four times and increases the host-accelerator card transmission bandwidth by 2.5 times; Second, an intra-chip cross shared communication (CSC) algorithm is designed with interleaved memory access patterns, implementing CSC-AG (CSC-AllGather) and CSC-RS (CSC-ReduceScatter) operators optimized for large model training. Benchmark results demonstrate that CSC achieves 2.15 times higher communication efficiency compared with conventional collective primitives. Finally, a bidirectional operator fusion strategy is proposed to enable communication-computation overlap, yielding a 59% improvement in communication performance after optimization.

collective communications  /  Sunway AI acceleration card  /  SW-IntraCC  /  communications optimization  /  ring network
赵玉龙, 顾燕卿, 田松涛, 吴春志, 汤凌韬, 张鲁飞, 秦晓军, 刘鑫, 陈左宁. SW-IntraCC:一种面向申威智能加速卡内部的集合通信机制. 计算机研究与发展, 2025 , 62 (6) : 1333 -1346 . DOI: 10.7544/issn1000-1239.202550143
Yulong Zhao, Yanqing Gu, Songtao Tian, Chunzhi Wu, Lingtao Tang, Lufei Zhang, Xiaojun Qin, Xin Liu, Zuoning Chen. SW-IntraCC: A Collective Communication Mechanism for the Internals of Sunway AI Acceleration[J]. Journal of Computer Research and Development, 2025 , 62 (6) : 1333 -1346 . DOI: 10.7544/issn1000-1239.202550143
人工智能技术的快速发展推动大规模预训练模型参数量突破千亿级,其训练推理过程对计算资源、存储带宽及通信系统构成三重挑战[1]. 以DeepSeek为代表的大模型[2]在分布式训练中面临显存容量、实时性及多卡协同效率等关键约束,促使分布式多机多卡集群训练架构[3]成为主流解决方案. 集合通信技术[4]虽有效支撑跨卡数据交互,但在大规模扩展时面临传输效率衰减与通信开销非线性增长等瓶颈问题[5],催生出高密度集成加速卡等技术路线,单卡多芯粒、多芯片等集成技术成为研究热点.
然而,此类架构在提升并行度的同时,导致通信拓扑复杂度显著增加:1)传统Ring通信依赖PCIe总线绕经主机中转,引入额外延迟[6];2)片内多实体通信受限于PCIe带宽及通道竞争,通信效率低下[7-9];3)显存碎片化问题使单个通信实体可用容量受限,制约十亿级参数模型部署[10]. 当前研究主要聚焦跨节点通信协议优化[11-12],而节点内多通信实体间(Die内/芯片间/卡内多个逻辑或物理通信单元)的高效通信机制仍缺乏系统性探索,成为制约新型加速卡效能释放的关键瓶颈.
申威智能(SW-AI)加速卡采用单卡四核组环网架构,内部直接内存访问带宽高、存储层次丰富、端到端延迟低,为卡内通信优化提供硬件基础. 然而,在LLaMa 70B模型训练中,Ring集合通信[3]将4个核组作为4个通信实体,暴露以下问题:1)卡内4个核组间的AllGather和ReduceScatter操作实际带宽为环网带宽的30%左右,均远低于环网的设计带宽;2)计算与通信串行化导致计算资源闲置率不小于50%;3)单核组显存上限15 GB,需频繁触发主机-加速卡数据传输. 上述问题使128卡集群训练中通信开销占比超20%,严重限制系统有效算力. 针对上述挑战,本文提出基于申威智能加速卡软件生态的SW-IntraCC通信优化框架,其主要贡献有3点:
1)基于片上环网构建L0(局部高速存储)-L1(Local-HBM)-L2(Global-HBM)层次化三级存储架构,在SDAA[13]运行时系统中实现了三级存储共享显存管理机制;单核组可访问的最大显存容量是原来的4倍,从15 GB提升至60 GB;主机-加速卡之间的数据传输带宽实现3倍的提升.
2)基于三级存储架构,本文提出采用交叉共享访存的片内通信算法,设计实现CSC-AG、CSC-RS等集合通信算子,该算法充分利用了环网总线直接交叉段内存访问的特性,减少通信开销,提高数据传输的并行性,性能是Ring集合通信方式的2.15倍.
3)针对通信计算无法重叠问题,将通信算子与前后计算算子深度融合,利用局部高速存储空间绕过主存中转环节,实现双向算子融合的通信计算重叠方法,完成了计算与通信之间的高效重叠,提高了实体间的通信效率;通过通信与计算的重叠,性能较重叠前提升了59%.
在接下来的章节中,将详细介绍本文的研究背景与研究动机,SW-IntraCC机制的设计原理、实现方法以及性能评估结果,以展示其在提升申威智能加速卡内部通信效率方面的有效性和优势.
现代芯片设计已进入“后摩尔定律”时代,设计技术正朝着多芯粒[11]方向发展. 硬件厂商通过将多个独立的通用计算Die、AI加速芯粒等集成到一块芯片中,并借助高速互连技术实现协同工作,单芯片的计算能力得以显著提升. 在此基础上,多个相同功能的芯片还可集成于单张加速卡,从而进一步提高单卡的计算能力.
基于现代芯片设计发展趋势,华为、寒武纪等公司相继推出集成多个人工智能芯片的加速卡. 此类加速卡采用独特的设计,对用户呈现出多个独立的加速设备(通信实体). 如图1所示,华为 Atlas 300I[12]推理卡配备了4个Ascend 310芯片,芯片之间通过PCIe总线实现数据通信和任务协作. 为了实现4个加速实体间的高效通信,华为提供了基于Ring和mesh+Ring等算法的集合通信库HCCL(Huawei collective communication library). 寒武纪 MLU370-X8[13]加速卡采用了双芯片多Die设计方法,芯片间通过 MLU-Link,PCIe等多种高速总线互联,为支持卡内2个加速实体之间的高效通信,寒武纪提供了专用的集合通信库CNCL(Cambricon communications library),该库支持基于PCIe,MLU-Link,RoCE[14]和IB等多种互联技术的通信传输.
申威智能加速卡通过PCIE-X16与主机连接,内置新一代申威异构AI处理器[13],该处理器采用片上多核组、通专结合计算核心、脉动阵列、乘加运算单元相结合的多级瓦片架构,并基于申威研发的自主SW64指令集系统,支持32 b定长指令和64 b字长,能满足多种计算需求. 处理器逻辑结构如图2所示,通过运行时系统向用户空间抽象出4个逻辑独立的加速设备(通信实体),采用多通信实体架构设计,集成1个管理控制核心MPE(management processing element),又称主核;4个从核阵列(Core Group),又称核组、通信实体;以及片上环网总线(Ring Bus). 其中,MPE作为主控单元负责系统级调度与I/O资源管理;从核阵列作为计算单元,每个阵列包含32个运算核心CPE(computing process element),又称从核,以4×8二维阵列拓扑结构实现并行加速,并配备局部数据存储器(local data memory,LDM)与高带宽显存(high bandwidth memory,HBM);环网总线实现MPE和核组间的高速互连,支持全局交叉段共享,又称交叉段、大共享段机制. 存储系统采用多级层次化架构:LDM与CPE紧耦合,容量为KB级,带宽TB级;本地HBM为阵列内部共享,容量15 GB左右;全局交叉段HBM通过环网实现跨核组共享,容量可扩展至60 GB. CPE可以通过ld/st访存指令直接访问主存空间,还可以通过DMA(direct memory access)实现LDM和高速显存之间的批量数据传输,DMA传输的效率与传输的数据量、DMA命令数量、数据在主存中的连续性以及DMA传输方式等密切相关. 此外,运算核心阵列还支持LDM共享和RMA(remote memory access)两种机制以实现阵列上LDM间的数据交互. 该架构通过硬件级计算-通信解耦设计,为大规模并行任务提供高吞吐、低延迟的数据通路支撑.
在Transformer[15]架构的大模型训练过程中,随着模型规模和训练数据量的急剧增长,单机单卡训练模式已无法满足算力需求,分布式训练成为主流解决方案. 当前主流的分布式并行策略包括数据并行(data parallelism,DP)[16]、张量并行(tensor parallelism,TP)[17]、流水并行(pipeline parallelism,PP)[18-19]、序列并行(sequence parallelism,SP)[20]等.
以国产申威智能加速卡分布式训练场景为例,在LLaMa 70B大模型训练场景中,硬件配置采用16台服务器组成的集群,每台服务器搭载8张申威智能加速卡,共计128张加速卡. 为优化计算效率,采用混合并行策略组合:集群层面实施流水并行,并行度pp=16;节点层面采用数据并行,并行度dp=8;单卡层面4个高带宽通信实体组成张量并行组,并行度tp=4,同时开启序列并行[20],加速卡内TP+SP并行方案如图3所示.
为了保证单卡内核组的高效通信和同步,通常采用2种集合通信操作Ring-AG (Ring AllGather)和Ring-RS(Ring ReduceScatter). AllGather[21]是实现跨节点数据全局同步的集合通信操作. 各计算节点通过互连网络交换本地数据,最终所有节点持有多节点数据的完整副本. 在分布式训练中,AllGather通常用于汇聚各节点的梯度信息,确保所有节点使用相同的模型参数[22]. ReduceScatter[23]是一种将全体计算节点的数据分散并减少的操作. 每个节点计算部分结果并发送给其他节点,最终节点会汇总并得到一个聚合值. 在深度学习中,ReduceScatter 主要用于将大规模数据集进行分散计算以减少单一节点的计算负担. 这2个操作为共轭操作,如图3所示,在前向传播中,g对应Ring-AG操作,而在反向传播中对应 Ring-RS操作;相反,$\bar {\rm g} $在前向传播中对应Ring-RS操作,而在反向传播中对应Ring-AG操作.
加速卡内部采用TP+SP并行模式后通信量的计算如式(1)(2)所示:
$ {V}_{\rm TP-AG}=6\times {fp}_{\rm SIZE}\times \dfrac{bs}{dp}\times s\times h\times \dfrac{l}{pp}=120\;\rm GB\text{,} $
$ {V}_{\rm TP-RS}=6\times {fp}_{\rm SIZE}\times \dfrac{bs}{dp}\times s\times h\times \dfrac{l}{pp}=120\;\rm GB. $
在式(1)中,$ {V}_{\rm TP-AG} $表示在一次全激活重计算训练中,4个通信实体进行Ring-AG的通信量. 其中,“6”表示每层正向和反向传播总计有6次通信;$ {fp}_{\rm SIZE} $为单个浮点数占用字节数(2 个字节);总批大小bs=2048;dp是数据并行粒度;隐藏维度h=8192,模型层数l=80. 而式(2)中的 $ {V}_{\rm TP-RS} $表示Ring-RS的通信量.
上述计算结果表明,加速卡内部通信实体需要频繁进行Ring-AG和Ring-RS操作,该过程涉及大规模数据传输. 然而,实际使用中,数据量为 8 MB 时,Ring-AG的算法带宽为34.1 GB/s,Ring-RS的平均数据量为32 MB,算法带宽为54.9 GB/s,均远低于环网设计带宽,整体训练中通信开销占比超20%,加速卡内的集合通信耗时长,已成为训练过程的瓶颈,极大限制了基于申威智能加速卡的大规模模型训练性能.
本节以图4所示的基于Ring算法实现的AllGather传输模型为例,系统分析申威智能加速卡内部集合通信面临单核组显存容量与传输带宽双重限制、卡内集合通信效率低及通信计算无法重叠等核心瓶颈三大问题.
首先,卡内集合通信效率低问题. 如图4所示,在AllGather操作中,主机侧首先发起通信请求launch1,其时间开销t_launch1包含指令下发与参数配置延迟;随后各核组通过从核DMA执行数据拷贝操作;完成数据迁移后需通过PCIe总线进行全局同步,时间开销为t_sync. 采用环网拓扑的AllGather集合通信需要迭代执行3次上述流程,最终实现全量数据同步,如图4中a、b、c、d数据聚合过程所示.
$ \begin{split} & BW_{\rm{A}G}= \\ & \dfrac{V_{\rm{A}G}}{3\times \left(t_{\rm{L}AUNCH}+t_{\rm{S}YNC}+\dfrac{V_{\rm{A}G}}{4\times BW_{\rm{R}ing}}+t_{\rm{O}THERS}\right)}=41\; \rm{G}B/s\end{split}. $
根据式(3)Ring-AG算法的带宽上限模型,当单次AllGather传输量$ {V}_{\rm AG} $为8 MB和环网物理带宽$ {BW}_{\rm Ring} $为100 GB/s时,结合四核组并行启动特性,其理论最大算法带宽可估算为41 GB/s. 该值显著低于理论物理带宽,主要源于启动延迟$ {t}_{\rm LAUNCH} $约为25 μs,同步开销$ {t}_{\rm SYNC} $为15 μs,其他时间开销$ {t}_{\rm OTHERS} $为5 μs的累计影响.
其次,通信计算无法重叠问题. 在分布式计算架构中,计算与通信的流水线重叠(overlap)[24]是性能优化的核心策略. 然而申威智能加速卡的从核DMA引擎需要同时服务于计算算子(将数据从HBM加载到LDM)和集合通信算子(核组间数据传输),导致硬件资源竞争. 这种资源共享机制使得计算与通信操作无法并行执行,通信时延无法被计算时间掩盖. 这种分离的计算与通信模式不仅增加了通信延迟,而且降低了整体运行效率.
最后,单核组显存容量与传输带宽双重限制问题.1)显存容量约束问题. 单核组可用显存容量上限为15 GB. 虽能满足中小规模计算需求,但在大模型推理场景下,该容量限制迫使系统频繁执行多核组分布式通信,引发额外的内存交换开销,导致计算延迟增加和推理效率降低. 2)主机-加速卡带宽竞争问题. 4个核组共享PCIe总线与主机通信,在传输模型参数等关键数据时,各核组实际可用带宽仅为总带宽的1/4. 这种带宽分配机制在大规模并行计算场景下,尤其是高密度数据交换任务中,严重制约了加速卡的性能表现.
综上,申威智能加速卡的四核组架构虽提升了本地显存带宽利用率,但在处理大模型推理等高内存需求任务时,仍面临主机-设备传输带宽及显存容量约束、通信带宽受限、通信计算无法重叠三大关键挑战. 这些问题导致计算延迟增加和整体效率下降,限制了其在AI领域的应用潜力. 为此,本文提出基于三级存储架构的新型卡内集合通信机制,旨在优化申威智能加速卡内多通信实体间的数据传输效率.
针对申威智能加速卡内部Ring集合通信存在的通信效率低、计算与通信无法重叠执行、本地显存容量及主机-加速卡数据传输受限等问题. 本文运用软硬协同设计的理念,提出 SW-IntraCC的设计方案. 该方案基于芯片环网特性设计出三级存储架构,提升单核组可用显存空间,有效缓解了主机-加速卡数据传输瓶颈;在三级存储架构上采用交叉共享访存的片内通信算法和双向算子融合的通信计算重叠方法,显著降低加速卡内通信开销,提升集合通信效率,并实现通信与计算高效重叠.
SW-AI处理器采用异构多核组架构设计,其硬件包含4个核组及对应的HBM控制器. 传统二级存储架构(LDM+HBM)中,每个核组独立使用,从存储空间视图来看,包含 LDM 和HBM两个存储层次,如图5所示. 该架构在4个核组之间无交互时能够最大化利用本地 HBM 的性能,能满足早期的应用需求,但在面对大模型推理和训练任务时,单核组显存容量不足、通信效率低以及数据传输存在瓶颈等问题逐渐凸显,限制了申威智能加速卡在高性能计算和人工智能领域的进一步应用.
为充分利用片上环网总线的高级硬件特性,本文基于片上环网构建L0(LDM)-L1(Local-HBM)-L2(Global-HBM)层次化三级存储架构. 具体而言,三级架构将HBM主存从物理和逻辑上划分为2个区域:一块作为全局内存(交叉段),可以通过环网直接进行读写操作,并由4个核组共享;另一块作为本地内存(连续区),仅供相应核组本地访问. 如图6所示,Local-HBM对应环网配置的连续段,作为本地内存;Global-HBM对应环网配置的交叉段,作为全局共享内存;LDM性能最高,是每个从核的私有空间. 三级存储架构通过引入全局内存,提供了一个共享的数据交换平台,它允许核组之间通过全局内存进行高效的数据交换,核组仅需将交互的数据写入全局内存,显著降低了程序设计的复杂度,简化了数据交换的流程,减少了开发者在编写和调试程序时的工作量,提高了程序的可读性和可维护性,降低了因频繁通信带来的性能开销.
三级存储架构能够显著提升单核组的显存容量. 在传统的二级架构中,每个核组的显存容量固定为15 GB. 三级存储架构利用环网总线的交叉段特性,通过硬件的配置开关灵活地设置每个核组的本地Local-HBM和交叉共享Global-HBM空间的大小. Local-HBM设置为15 GB时,Global-HBM大小为0;Local-HBM设置为0时,Global-HBM为60 GB. 这种灵活性使得单核组可分配使用的最大显存容量达到传统方式的4倍,能够支持更大的数据集和批处理大小.
采用三级存储架构能够显著提高主机与加速卡之间的数据传输效率. 在传统的二级存储架构中,由于4个核组的本地内存Local-HBM是离散分布的,当主机与加速卡之间进行数据传输时,必须针对每个核组分别执行一次数据传输操作,如图6所示. 这种分散的数据传输方式不仅增加了传输次数,还可能导致PCIe传输带宽的浪费,进而降低了整体的数据传输效率. 相比之下,三级存储架构通过引入一块共享的全局显存来优化这一过程. 该架构下,训练推理任务中的模型参数仅需通过1次sdaaMemcpy操作就从主机拷贝数据到加速卡的全局显存中. 随后,4个核组可通过高速环网共同复用这块全局显存中的模型参数进行计算,如图7所示. 由于减少了低带宽PCIe通路上不必要的重复传输,充分利用了片上环网高速传输,由4次PCIe传输变为1次PCIe传输加上4次环网传输,从而降低整体传输时间,提高了主机与加速卡之间的数据传输效率. 这种优化不仅提升了数据传输的速度,还减少了因频繁数据传输带来的系统开销,使得整个计算过程更加高效.
为了实现对三级存储的管理,结合SDAA运行时系统中原有内存管理算法,增加了对全局HBM的统一管理,三级架构内存分配原理如图8所示. 首先使用全局位图(bitmap)进行粗粒度分配,8 KB粒度的空间支持主核页式访问,1 MB大小的块是从核的段式访问空间,从核组-Local空间对应三级存储中的L1部分,从核组-Global空间为L2. 在具体实现过程中结合SW-AI硬件特点,将位图存储区按照缓存行进行对齐,使用向量优化方法,提高了位图检索以及处理的速度. 然后使用free_list链表串起来的细粒度空间管理算法,内存块的头部存放该块内存的元数据(meta data). 分配时,从链表上分配一个对象出去;释放时,把对象插入到链表. 链表指针直接分配在待分配内存中,不需要额外的内存开销. 该方法能够支持本地HBM段、交叉共享HBM段,主核页内存统一管理,具有分配回收速度快、数据块合并效率高、空间利用率高等优势. SDAA运行时库中三级内存管理接口如表1所示.
在申威智能加速卡的四核组架构中,传统Ring算法的集合通信实现存在开销大、通信带宽低的问题. 在处理大模型训练推理等高内存需求任务时,其通信效率不足严重制约系统整体性能表现.
为了解决上述问题,本文基于申威智能加速卡的环网总线直接访问交叉共享内存的硬件特性与三级存储架构,提出采用交叉共享访存的片内通信算法,并设计实现卡内TP+SP通信最常用的通信算子CSC-AG(cross share communication AllGather)和CSC-RS(cross share communication ReduceScatter). 相较于传统Ring通信方式依赖Send/Recv操作的跨核组数据传输,CSC算法的核心思想是利用片上环网总线支持直接访问交叉共享内存的硬件特性,以访存操作访问全局内存Global-HBM的方式实现通信功能.
申威智能加速卡内四核组CSC-AG算法的实现与传统Ring通信算法有显著不同,如图9所示,CSC-AG传输模型相较于图4的Ring-AG算法具有三大优势:1)主机launch次数从12降至4,同步次数由3降为1,降低67%的核函数启动和同步开销;2)采用全并行读写架构,突破Ring算法必需的3轮串行迭代限制,实现端到端时延优化;3)引入动态内存块调整机制,通过3倍连续内存块读取策略提升带宽利用率,有效缓解Ring算法固定块传输导致的带宽受限问题.
以单个核组上实现的CSC算法为例,基于CSC算法实现的All Gather如算法1所示.
算法1. CSC-AG.
输入:每个核组输入数据input,核组号rank,总核组数N=4;
输出:每个核组包含来自所有核组的完整数据.
① CSC_AG(input, rank, N=4);
②  size = len(input);
③  hbm_global = hbm_local = rank × size;
④  DMA_GET(ldm, hbm_local);
⑤  DMA_PUT(ldm, hbm_global);
⑥  SYNC();
⑦ FOR i = 1 TO N−1;
⑧   hbm_global = hbm_local = ((i + rank)MOD    N) × size;
⑨   DMA_GET(ldm, hbm_local);
⑩   DMA_PUT(ldm, hbm_global);
⑪ END FOR
⑫ END
算法1具体实现流程如下:输入参数包括各核组输入数据input、核组号rank及核组总数N,输出为包含全量数据的本地副本. 行①~⑥完成数据共享阶段,各核组将待共享数据写入三级存储的交叉段后,执行全局同步确保写入完成;行⑦~⑪执行数据聚合阶段,各核组并行读取交叉段中其他核组的数据.
算法2. CSC-RS.
输入:每个核组输入数据input,核组号rank,总核组数N=4;
输出:每个核组包含来自所有核组的1/N的Reduce数据.
① CSC_AG(size, rank, N=4);
②  size = len(input);
③  hbm_global = hbm_local = rank × size;
④  DMA_GET(ldm, hbm_local);
⑤  DMA_PUT(ldm, hbm_global);
⑥  SYNC();
⑦ FOR i = 1 TO N−1;
⑧   DMA_GET(ldm, hbm_global);
⑨   temp = Reduce(ldm);
⑩   DMA_PUT(ldm, hbm_global);
⑪ END FOR
⑫ END
算法2展示了CSC-RS的实现框架,其工作流程与CSC-AG保持结构一致性,核心差异在于数据读取阶段需执行规约操作后写入本地.
以AllGather操作为例,CSC-AG算法的带宽上限模型计算方法如式(4)所示.
$ {BW}_{\rm AG}=\dfrac{{V}_{\rm AG}}{{t}_{\rm LAUNCH}+{t}_{\rm SYNC}+3\times \left(\dfrac{{V}_{\rm AG}}{4\times {BW}_{\rm GLOBAL}}+{t}_{\rm OTHERS}\right)}. $
带宽上限模型计算方法$ {BW}_{\rm AG} $是总数据量除以总耗时,其中总耗时包括主机启动开销$ {t}_{\rm LAUNCH} $、主机同步开销$ {t}_{\rm SYNC} $和内核执行开销等.
总数据量$ {V}_{\rm AG} $较小时(小于1 MB),$ \dfrac{{V}_{\rm AG}}{{BW}_{\rm GLOBAL}}\ll {t}_{\rm LAUNCH}+{t}_{\rm SYNC}+{t}_{\rm OTHERS} $,式(3)(4)可以简化为:
$ {BW}_{\rm AG-Ring}\approx \dfrac{{V}_{\rm AG}}{3\times \left({t}_{\rm LAUNCH}+{t}_{\rm SYNC}+{t}_{\rm OTHERS}\right)} \text{,}$
$ {BW}_{\rm AG-CSC}\approx \dfrac{{V}_{\rm AG}}{{t}_{\rm LAUNCH}+{t}_{\rm SYNC}+3\times {t}_{\rm OTHERS}}. $
将式(5)(6)对比可知,在数据量较小时,CSC-AG算法理论带宽为传统Ring算法的3倍左右.
数据量$ {V}_{\rm AG} $较大时(大于256 MB),$ \dfrac{{V}_{\rm AG}}{{BW}_{\rm GLOBAL}}\gg {t}_{\rm LAUNCH}+{t}_{\rm SYNC}+{t}_{\rm OTHERS} $,式(3)(4)可以简化为:
$ {BW}_{\rm AG-Ring}\approx \dfrac{{V}_{\rm AG}}{3\times \left(\dfrac{{V}_{\rm AG}}{4\times {BW}_{\rm GLOBAL}}+{t}_{\rm OTHERS}\right)}\text{,} $
$ {BW}_{\rm AG-CSC}\approx \dfrac{{V}_{\rm AG}}{3\times \left(\dfrac{{V}_{\rm AG}}{4\times {BW}_{\rm GLOBAL}}+{t}_{\rm OTHERS}\right)}. $
将式(7)(8)对比可知,数据量较大时,CSC-AG算法理论性能与传统Ring算法相当. ReduceScatter算法性能与AllGather算法性能分析过程类似,不再赘述.
与式(3)基于Ring-AG算法的带宽上限模型相比,当单次AllGather传输量$ {V}_{\rm AG} $为8 MB时,结合四核组并行启动特性,其理论最大算法带宽可估算为69.6 GB/s,在相同数据规模下明显高于Ring算法41 GB/s的理论带宽.
在使用方式上,CSC算法采用非侵入式设计,可实现与Ring-AG、Ring-RS的无缝兼容替换,无需修改现有应用架构即可直接部署. 将CSC算法用于优化图3所示的Transformer模型结构的序列并行和张量并行,如图7所示. 在序列并行场景中,各核组将输出数据$ [\boldsymbol{Y}_0,\boldsymbol{Y}_1,\boldsymbol{Y}_2,\boldsymbol{Y}_3] $写入Local-HBM. CSC-AG操作执行时,首先完成Global-HBM写入及全局同步,实现本地数据向交叉段分区的映射;随后通过Global-HBM读取操作,将增量数据并行加载至Local-HBM,最终达成全量数据同步. 每个核组获得全局数据$ \left[\boldsymbol{Y}\right] $.
访存驱动的高效CSC-AG,CSC-RS通信算法充分利用了环网的硬件特性,大大提升了带宽利用率. 但是CSC-AG,CSC-RS需要单独发起并等待完成,通信操作与计算操作独立运行,仍旧无法重叠. 这种串行化的执行方式导致通信延迟无法被有效掩盖,限制了整体性能的提升.
CUDA(compute unified device architecture)编程模型利用 CUDA 流(stream) 实现通信与计算重叠,CUDA流是一个操作序列,其中的操作按照提交顺序串行执行. 如果将通信操作任务分配到一个 CUDA 流,将计算任务放入另一个独立的 CUDA 流中,GPU中数据传输和计算依赖的部件相互独立,通信和计算任务可在2条不同的流中并行执行,这样就实现通信操作和计算操作的并行化,掩盖了通信的延迟.
在申威智能加速卡上,难以直接通过流的方式实现通信与计算的重叠. 该加速卡的通信和计算都依赖从核发起DMA指令进行访存操作,两者共用从核及DMA引擎部件,即使将两者分配到不同的流上也无法真正并行. 从硬件资源角度来看,实际上通信操作主要占用环网带宽,并不占用从核的计算资源. 因此,在等待环网传输数据期间,从核的计算单元仍可以并行其他计算任务,从而实现硬件资源的优化利用.
为实现申威智能加速卡在SP和TP分布式训练场景中的高效通信与计算重叠,本文借鉴算子融合思想,结合三级存储架构,提出了一种双向算子融合的通信计算重叠方法. 图10展示了通信计算重叠优化后的MLP层次结构. 图中每一行表示一个核组的执行过程,标注的①②③分别对应Layernorm[25],GEMM(FFN路径包含多个GEMM算子)以及DROPOUT算子的3个执行阶段:①从Local-HBM读取数据至局部高速存储区LDM;②在LDM中执行计算操作,并将中间结果暂存于LDM;③将最终结果写回Local-HBM. 图中箭头表示数据流动方向,其中绿色箭头代表计算型算子,红色箭头代表CSC通信型算子,带虚线的白色箭头则表示被完全掩盖的操作.
双向算子融合的通信计算重叠方法的核心思想是通过将通信算子与前序和后序计算算子融合,利用LDM绕过全局内存的中转环节,避免数据的重复存储与加载,从而实现通信与计算的高效重叠. 具体而言,该方法将CSC-AG与前序算子(LayerNorm)和后序算子(GEMM)融合,同时将CSC-RS与前序算子(GEMM)和后序算子(Dropout)融合. 通过这种方式,上一个算子的计算结果可直接从LDM传递至下一个算子,无需写入全局内存,显著减少了数据搬运开销.
以0号核组的CSC-AG为例,在SP阶段,LayerNorm算子的执行分为3步:①从Local-HBM读取输入向量$ \left[{\boldsymbol X}_{n}\right] $到LDM;②在LDM上完成归一化计算;③将结果$ \left[{\boldsymbol Y}_{n}\right] $同时写入Global-HBM和Local-HBM(由于Global-HBM和Local-HBM位于不同路径,写操作可并行). 这一过程融合了CSC-AG算子从Local-HBM搬运数据并聚合至Global-HBM的操作,节省了时间开销. 在TP阶段,数据$ \left[\boldsymbol Y\right] $已经完整聚合至Global-HBM,GEMM算子第①步从Global-HBM加载数据$ \left[\boldsymbol Y\right] $至LDM,第②步在LDM上执行矩阵运算,同时并行执行CSC-AG算子将数据$ \left[\boldsymbol Y\right] $写回Local-HBM,从而实现通信与计算的重叠. 类似地,0号核组的CSC-RS算子也通过相同机制实现了传输与计算的并行. 通过双向算子融合的操作,算子性能模型发生了变化,以下是算子在融合前和融合后数据传输的耗时对比:
$ \begin{split} {T}_{1}=&\dfrac{{V}_{\rm AG}}{{BW}_{\rm LOCAL}}+\dfrac{{V}_{\rm AG}}{{BW}_{\rm GLOBAL}}+\dfrac{3\times {V}_{\rm AG}}{{BW}_{\rm GLOBAL}}+\\&\dfrac{4\times {V}_{\rm AG}}{{BW}_{\rm LOCAL}}+3\times {t}_{\rm LAUNCH} \text{,} \end{split}$
$ {T}_{2}=\dfrac{{V}_{\rm AG}}{{BW}_{\rm GLOBAL}}+\dfrac{3\times {V}_{\rm AG}}{{BW}_{\rm GLOBAL}}+\dfrac{{V}_{\rm AG}}{{BW}_{\rm LOCAL}}+2\times {t}_{\rm LAUNCH}\text{,} $
$ p=\dfrac{\dfrac{4\times {V}_{\rm AG}}{{BW}_{\rm LOCAL}}}{\dfrac{{V}_{\rm AG}}{{BW}_{\rm LOCAL}}+\dfrac{{V}_{\rm AG}}{{BW}_{\rm GLOBAL}}+\dfrac{3\times {V}_{\rm AG}}{{BW}_{\rm GLOBAL}}+\dfrac{4\times {V}_{\rm AG}}{{BW}_{\rm LOCAL}}}\text{.} $
在式(9)~(11)中,$ {T}_{1} $是融合前的耗时,$ {T}_{2} $是融合后的耗时,p是融合后的收益百分比. 其中$ {t}_{\rm LAUNCH} $表示主机启动开销,$ {V}_{\rm AG} $表示每个核组进行Layernorm运算的数据量,占AllGather通信数据的1/4;$ {BW}_{\rm LOCAL} $表示本地Local-HBM的带宽;$ {BW}_{\rm GLOBAL} $表示经环网访问Global-HBM的带宽;$ {T}_{1} $和$ {T}_{2} $统计的时间由前序算子把结果写入内存、通信算子传输数据、后序算子从内存获取到数据3段构成. 经过计算,在AllGather通信数据量为8 MB的情况下,融合的收益大概为50%.
图11从微观层次上展示了单个从核内的通信计算重叠的原理. LayerNorm与CSC-AG交错运行,CSC-AG与GEMM交错运行,前序算子LayerNorm在执行归一化运算的同时发起CSC-AG操作,将运算结果Yn写入到环网Global-HBM;CSC-AG在Global-HBM读取数据Yn的同时,后序算子GEMM并行执行. 由于LayerNorm和GEMM算子更大,因此可以很好地掩盖掉CSC-AG的通信开销. 通过将通信操作与计算过程融合,通信的延迟可以被计算任务掩盖,实现高效的通信与计算重叠. 这种方式可以有效提高核组的使用率和整体性能,尤其是在分布式训练中,提升大规模分布式训练的效率,从而确保大规模语言模型 LLM 训练的高效进行.
实验评估主要针对基于三级存储架构设计的访存替代通信算法CSC-AG和CSC-RS,以及经过双向算子融合优化后的算法. 评估的目的是对比这些新算法与传统基于环网Ring方式实现的AllGather和ReduceScatter算法的性能差异.
测试硬件环境为一台型号为NF5468M5的浪潮英信服务器,在其PCIe插槽上插入一块申威智能加速卡,确保主机侧环境独占使用. 运行时测试环境如表2所示,AI-driver1.0版本的环境对应的是传统二级存储架构下的软件,AI-driver1.0-T版本的软件是使用三级存储架构下的运行时及驱动.
本节对二级存储架构与三级存储架构下主机与加速卡之间通过PCIe传输数据的性能进行了对比分析. 实验通过选取不同规模的模型参数,在主机与加速卡之间进行数据传输,并测量其有效带宽. 在二级存储架构中,测试方法为分别向4个Local-HBM各执行1次独立的PCIe传输;而在三级存储架构中,测试方法为先向Global-HBM执行1次PCIe传输,随后通过4次环网传输将数据分发至各Local-HBM. 图12展示了不同数据规模下2种存储架构的传输性能对比结果. 实验结果表明,随着传输数据量的增加,三级存储架构在传输性能上展现出显著优势,其性能较二级存储架构提升了2.5倍. 由于大模型训练场景下数据传输以大规模数据为主,三级存储架构能够更好地适应此类需求.
本节对基于二级存储架构设计的传统集合通信算法Ring-AG和Ring-RS,以及基于三级存储架构设计的交叉共享访存的片内通信算法CSC-AG和CSC-RS进行了性能评估. 图13展示了Ring和CSC算法的带宽利用率,计算方法为数据传输时间占总时间的比值. 由于Ring算法在主机启动和同步过程中存在较高的开销,尤其是在小数据量情况下,数据传输时间占总时间的比值仅为20%,带宽利用率较低. 相较之下,CSC算法通过减少主机启动次数和同步次数,提高了数据传输时间的占比,从而实现了更高的带宽利用率. 随着数据量增大,CSC和Ring算法的总体时间逐渐接近,带宽利用率也趋于相似.
图14展示了CSC-AG与Ring-AG算法的带宽对比结果. 实验表明,在小数据量场景下,CSC-AG的带宽显著高于Ring-AG,尤其在4096 KB数据粒度时,CSC-AG的性能达到Ring-AG的2.15倍. 这一性能优势主要源于CSC-AG优化了Ring-AG算法中因主机启动时间和同步时间占比较高而引入的开销. 随着数据量的增加,CSC-AG与Ring-AG的带宽差距逐渐缩小,两者性能趋于接近.
图15展示了Ring-RS和CSC-RS算法的带宽对比结果. 实验表明,在小数据量场景下,CSC-RS的带宽显著高于Ring-RS,在32 768 KB的数据粒度时,CSC-RS的性能达到Ring-RS的1.2倍.
为验证双向算子融合的通信计算重叠方法的有效性,本节以通信和计算算子的执行时间为核心指标来设计对照实验. 实验以CSC-AG、CSC-RS算子为测试对象,选取不同的通信数据量进行性能测试. 表3和表4分别展示了CSC-AG和CSC-RS在应用双向算子融合的通信计算重叠方法后的性能收益. 在经过优化之前,算子间互相独立,中间结果需要通过内存中转;在优化之后,通信操作与计算操作融合,中间结果直接通过LDM存取,避免了冗余的内存访问开销. 实验结果表明,在应用双向算子融合的通信计算重叠方法后,通信算子的性能得到显著提升,CSC-AG取得最大59%的性能收益,CSC-RS取得最大35%的性能收益.
本文研究目前已完成单卡原型验证,为评估GPT-NeoX 20B,LLaMa 70B模型在多卡训练场景下的优化潜力,基于实测数据构建性能预估模型如表5所示.
表5中的GPT-NeoX 20B模型的训练数据为0.4 T tokens[26],LLaMa 70B模型的训练数据是1.2节的场景下的训练数据2 T tokens[27]. 表5中第1行表示使用传统Ring算法下卡内TP集合通信AG和RS对应消耗的总时间,第2行表示使用双向融合后的CSC算法进行卡内通信对应的时间消耗. 从表5可以看到,如果使用本文提出的双向融合CSC算法,完成整个GPT-NeoX 20B模型训练预计总时间可以节约2.3天;整个LLaMa 70B模型训练预计总时间可以节约13.3天.
本文针对申威智能加速卡(SW-AI)在大模型推理任务中面临的显存容量及主机-设备带宽双重限制、卡内集合通信开销大、通信计算无法重叠等核心问题,提出SW-IntraCC优化框架. 该框架通过构建基于片上环网的三级存储架构,显著提升了单核组可用的显存空间,并有效缓解了主机-加速卡数据传输竞争;此外,该框架设计并实现了一种采用交叉共享访存的片内通信算法,该算法通过减少通信开销,提升了CSC-AG与CSC-RS的带宽. 同时,双向算子融合的通信计算重叠方法通过并行执行计算与通信任务,降低了时间开销,从而提升整体效率.
实验结果表明,SW-IntraCC在申威智能加速卡上的集合通信性能是传统Ring算法的2.15倍,单核组显存容量最高扩大4倍,主机-加速卡传输带宽提升2.5倍;重叠优化后性能进一步提升59%. 这一成果不仅为申威智能加速卡在大模型计算环境下的应用提供了更高效的通信解决方案,也为其他类似架构的加速卡提供了参考. 未来,该研究将进一步丰富和完善SW-IntraCC接口,增加在不同场景下的适应性和通用性,以期早日把该研究工作尽快融入到SDAA生态中.
作者贡献声明:赵玉龙、顾燕卿负责方案的整体设计并撰写论文;田松涛、吴春志负责部分算法思路和实验方法;汤凌韬、张鲁飞、秦晓军、刘鑫、陈左宁提出指导意见并修改论文.

参考文献 引证文献
排序方式:
1
王睿,张留洋,高志涌,等. 面向边缘智能的大模型研究进展[J/OL]. 计算机研究与发展,[2025-02-07]. http://kns.cnki.net/kcms/detail/11.1777.TP.20250127.0921.006.html
Wang Rui, Zhang Liuyang, Gao Zhiyong, et al. Progress in research on large models for edge-oriented intelligence[J/OL]. Computer Research and Development, [2025-02-07]. http://kns.cnki.net/kcms/detail/11.1777.TP.20250127.0921.006.html (in Chinese)
2
DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning[J]. arXiv preprint, arXiv: 2501.12948, 2025. DOI: 10.48550/arXiv.2501.12948
3
Ouyang S, Dong D Z, Xu Y M, et al. Communication optimization strategies for distributed deep neural network training: A survey[J]. Journal of Parallel and Distributed Computing, 2021, 149: 52–65. DOI: 10.1016/j.jpdc.2020.11.005
4
Weingram A, Li Y, Qi H, et al. xCCL: A survey of industry-led collective communication libraries for deep learning[J]. Journal of Computer Science and Technology, 2023, 38: 166−195
5
NVIDIA. NVIDIA DGX−1 System Architecture White Paper 2017[R/OL].[2024-12-30]. https://images.nvidia.cn/content/pdf/dgx1-system-architecture-whitepaper1.pdf
6
Tipparaju V, Nieplocha J, Panda D K. Fast collective operations using shared and remote memory access protocols on clusters[C]//Proc of Int Parallel and Distributed Processing Symposium. Washington, DC: IEEE Computer Society, 2003: 10
7
Pabst S, Koch A, Straßer W. Fast and scalable CPU/GPU collision detection for rigid and deformable surfaces[J]. Computer Graphics Forum, 2010, 29(7): 2145−2154
8
Xu Q, Jeon H, Annavaram M. Graph processing on GPUs: Where are the bottlenecks?[C]//Proc of 2014 IEEE Int Symp on Workload Characterization (IISWC). Piscataway, NJ: IEEE, 2014: 140−149
9
Next Platform. The system bottleneck shifts to PCI-express[R/OL] [2017-07-14]. https://www.nextplatform.com/2017/07/14/system-bottleneck-shifts-pci-express/
10
黄彤彤, 陈昊, 武辰飞, 等. Concurrent multi-die optimization物理实现方案的应用[J]. 电子技术应用,2023,49(8):30−35
Huang Tongtong, Chen Hao, Wu Chenfei, et al. Application of physical implementation scheme for Concurrent multi-die optimization[J]. Electronics Applications, 2023, 49(8): 30−35 (in Chinese)
11
曾嵘浩. 基于海上的的多目标跟踪与行为分析研究[D]. 西安:西安电子科技大学,2023. DOI:10.27389/d.cnki.gxadu.2023.001428
Zeng Ronghao. Research on multi-target tracking and behavior analysis based on maritime[D]. Xi’an: Xi’an Electronic Science and Technology University, 2023. DOI:10.27389/d.cnki.gxadu.2023.001428 (in Chinese)
12
俞铭辉. 基于国产智能芯片的模型移植和优化的研究[D]. 杭州:杭州电子科技大学,2024. DOI:10.27075/d.cnki.ghzdc.2024.000968
Yu Minghui. Research on model transplantation and optimization based on domestic smart chip[D]. Hangzhou: Hangzhou University of Electronic Science and Technology, 2024. DOI:10.27075/d.cnki.ghzdc.2024.000968 (in Chinese)
13
赵玉龙, 张鲁飞, 许国春, 等. SDAA:面向申威智能加速卡的运行时系统[J]. 软件学报,2024,35(12):5710−5724
Zhao Yulong, Zhang Lufei, Xu Guochun, et al. SDAA: Runtime system for Sunway intelligent acceleration card[J]. Journal of Software, 2024, 35(12): 5710−5724 (in Chinese)
14
Wikipedia. RDMA over converged Ethernet[EB/OL]. Wikipedia, the free encyclopedia. 2017[2024-12-01]. https://en.wikipedia.org/w/index.php?title=RDMA_over_Converged_ Ethernet&oldid=782744462
15
Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proc of the 31st Conf on Neural Information Processing Systems (NIPS 2017). Long Beach, CA: Curran Associates Inc, 2017: 5998−6008
16
Sergeev A, Balso M D. Horovod: Fast and easy distributed deep learning in TensorFlow[J]. arXiv preprint, arXiv: 1802.05799, 2018
17
Shoeybi M, Patwary M, Puri R, et al. Megatron-LM: Training multi-billion parameter language models using model parallelism[J]. arXiv preprint, arXiv 1909.08053, 2019
18
Huang Y, Cheng Y, Bapna A, et al. GPipe: Efficient training of giant neural networks using pipeline parallelism[C]//Advances in the 32nd Conf on Neural Information Processing Systems (NeurIPS 2019). Vancouver, CA: Curran Associates Inc 2019: 8−14
19
Narayanan D, Harlap A, Phanishayee A, et al. PipeDream: Generalized pipeline parallelism for DNN training[C/OL]//Proc of the 27th ACM Symp on Operating Systems Principles. New York: ACM, [2025-01-15]. https://doi.org/10.48550/arXiv.1806.03377, 2019
20
Li S, Xue F, Li Y, et al. Sequence parallelism: Long sequence training from system perspective[C]//Proc of the 61st Annual Meeting of the Association for Computational Linguistics. Toronto, Canada: 2021: 2391−2404
21
高亦沁,罗智宇,王一超,等. 面向国产超算的操作系统评测与优化[J/OL]. 计算机科学,[2025-02-11]. http://kns.cnki.net/kcms/detail/50.1075.tp.20240925.1330.007.html
Gao Yiqin, Luo Zhiyu, Wang Yichao, et al. Evaluation and optimization of operating system for domestic supercomputing[J/OL]. Computer Science, [2025-02-11]. http://kns.cnki.net/kcms/detail/50.1075.tp.20240925.1330.007.html (in Chinese)
22
Li M, Andersen D G, Park J W, et al. Scaling distributed machine learning with the parameter server[C]//Proc of the 11th USENIX Symp on Operating Systems Design and Implementation. Berkeley, CA: USENIX Association, 2014: 583−598
23
Träff J L. Optimal, non-pipelined Reduce-scatter and Allreduce algorithms[J]. arXiv preprint, arXiv: 2410.14234, 2024
24
Wasi-ur-Rahman M, Lu X, Islam N S, et al. HOMR: A hybrid approach to exploit maximum overlapping in MapReduce over high performance interconnects[C]//Proc of the Int Conf on Supercomputing (ICS). New York: ACM, 2014: 33−42. DOI: 10.1145/2597652.2597684
25
Ba J, Kiros J R, Hinton G E. Layer normalization[J]. arXiv preprint, arXiv: 1607.06450, 2016
26
Black S, Biderman S, Hallahan E, et al. GPT-NeoX−20B: An open-source autoregressive language model[J]. arXiv preprint, arXiv: 2204.06745, 2022
27
Touvron H, Martin L, Stone K, et al. Llama 2: Open foundation and fine-tuned chat models[J]. arXiv preprint, arXiv: 2307.09288, 2023
2025年第62卷第6期
113
53
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202550143
  • 接收时间:2025-03-01
  • 首发时间:2026-11-10
  • 出版时间:2025-06-02
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2025-03-01
  • 修回日期:2025-04-08
基金
作者信息
    1数学工程与先进计算国家重点实验室 江苏无锡 214125
    2航天工程大学 北京 100004
    3国家并行计算机工程技术研究中心 北京 100083
参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202550143
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏