Article(id=1329456772353471440, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1329456575464452444, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202440055, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=null, receivedDate=1706457600000, receivedDateStr=2024-01-29, revisedDate=1711468800000, revisedDateStr=2024-03-27, acceptedDate=null, acceptedDateStr=null, onlineDate=1794896794641, onlineDateStr=2026-11-17, pubDate=1717171200000, pubDateStr=2024-06-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1794896794641, onlineIssueDateStr=2026-11-17, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1794896794641, creator=13701087609, updateTime=1794896794641, updator=13701087609, issue=Issue{id=1329456575464452444, tenantId=1146029695717560320, journalId=1324759914146529478, year='2024', volume='61', issue='6', pageStart='1349', pageEnd='1607', issueExtLink='null', onlineDate='null', pubDate='1717171200000', pubDateStr='2024-06-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1794896747699, creator='13701087609', updateTime=1794896747699, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext=null, issueFiles=null, downloadFileDto=null}, startPage=1388, endPage=1400, ext={EN=ArticleExt(id=1329456774077330385, articleId=1329456772353471440, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=Direct xPU: A Novel Distributed Heterogeneous Computing Architecture Optimized for Inter-node Communication Optimization, columnId=1329456577611936094, journalTitle=Journal of Computer Research and Development, columnName=Special lssue on Cross-layer Optimization of Processor Chips, runingTitle=null, highlight=null, articleAbstract=

The explosive growth of the application of large-scale artificial intelligence models has made it difficult to achieve the scale deployment of applications relying on a single node or a single type of computing architecture. Distributed heterogeneous computing has become the mainstream choice, and inter-node communication has become one of the main bottlenecks in the training or inference process of large models. Currently, there are still some deficiencies in the inter-node communicating solutions dominated by leading chip manufacturers. On the one hand, some architectures choose to use a simple but less scalable point-to-point transmission scheme in order to pursue the ultimate inter-node communication performance. On the other hand, traditional heterogeneous computing engines (such as GPUs) are independent of CPUs in terms of computing resources such as memory and computing cores, but they lack dedicated communicating network devices in terms of communication resources and need to rely entirely or partially on CPUs to handle transmission between heterogeneous computing engines and the shared communicating network device through physical links such as PCIe. The proposed Direct xPU distributed heterogeneous computing architecture in this article enables heterogeneous computing engines to have independent and dedicated devices in both computing and communication resources, achieving zero-copy data and further eliminating the energy consumption and latency associated with cross-chip data transfer during inter-node communication. Evaluations show that Direct xPU achieves communication latency comparable to computing architectures pursuing ultimate inter-node communication performance, with bandwidth close to the physical limit.

, authors=Rengang Li1, 2, Yanwei Wang2, Rui Hao2, Linge Xiao2, Le Yang3, *, Guangwen Yang1, Hongwei Kan3, authorsList=Rengang Li, Yanwei Wang, Rui Hao, Linge Xiao, Le Yang, Guangwen Yang, Hongwei Kan, authorCompany=null, correspAuthors=Le Yang, authorNote=

born in 1980. PhD, Senior engineer. Member of CCF. His main research interest includes heterogeneous computing

born in 1985. PhD, associate researcher. Member of CCF. His main research interest includes heterogeneous computing

born in 1983. Master, Senior engineer. Member of CCF. His main research interest includes heterogeneous computing

born in 1994. PhD. His main research interest includes heterogeneous computing

born in 1990. Master. Her main research interest includes heterogeneous computing

born in 1963. PhD, professor, PhD supervisor. Member of CCF. His main research interest includes high performance computing

born in 1975. Master, professor-Senior engineer. Senior member of CCF. His main research interest includes heterogeneous computing

, correspAuthorsNote=null, copyrightStatement=Copyright © 2024 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1329456779324403713, articleId=1329456772353471440, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=Direct xPU:一种新型节点间通信优化的分布式异构计算架构, columnId=1329456577762931040, journalTitle=计算机研究与发展, columnName=处理器芯片跨层优化专题, runingTitle=null, highlight=null, articleAbstract=

人工智能大模型应用的爆发式增长,使得难以依靠单一节点、单一类型的算力实现应用的规模部署,分布式异构计算成为主流选择,而节点间通信成为大模型训练或推理过程中的主要瓶颈之一. 目前,主要由GPU,FPGA等头部芯片厂商所主导的各种计算架构的节点间通信方案还存在一些问题. 一方面,为了追求极致的节点间通信性能,一部分架构选择使用协议简单而可扩展性差的点对点传输方案. 另一方面,传统的异构计算引擎(例如GPU)虽然在内存、计算管线等算力要素方面独立于CPU,但在通信要素方面却缺少专属的网络通信设备,需要完全或部分借助于CPU通过PCIe等物理链路来处理异构计算引擎与共享网络通信设备之间的通信. 所实现的Direct xPU分布式异构计算架构,使得异构计算引擎在算力要素和通信要素两方面均具有独立的、专属的设备,实现了数据的零拷贝,并进一步消除了节点间通信过程中处理跨芯片传输数据所带来的能耗和延迟. 测试结果表明,Direct xPU取得了与追求极致的节点间通信性能的计算架构相当的通信延迟,带宽接近物理通信带宽的上限.

, authors=李仁刚1, 2, 王彦伟2, 郝锐2, 肖麟阁2, 杨乐3, *, 杨广文1, 阚宏伟3, authorsList=李仁刚, 王彦伟, 郝锐, 肖麟阁, 杨乐, 杨广文, 阚宏伟, authorCompany=null, correspAuthors=杨乐, authorNote=

1980年生. 博士,正高级工程师. CCF会员. 主要研究方向为异构计算

1985年生. 博士,副研究员. CCF会员. 主要研究方向为异构计算

1983年生. 硕士,高级工程师. CCF会员. 主要研究方向为异构计算

1994年生. 博士. 主要研究方向为异构计算

1990年生. 硕士. 主要研究方向为异构计算

1963年生. 博士,教授,博士生导师. CCF会员. 主要研究方向为高性能计算

1975年生. 硕士,教授-正高级工程师. CCF高级会员. 主要研究方向为异构计算

, correspAuthorsNote=
肖麟阁 ()
, copyrightStatement=版权所有 © 2024 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=EjiOaLoaiCIzndYg1myTbg==, magXml=qyuMf68q90hg825vJThBfQ==, pdfUrl=null, pdf=6PWIDqD2v/mqVjEOEPkbjw==, pdfFileSize=2404693, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=nP5aqf0m9Rm/eBOwlZYTZA==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=nY5EniNuuDlhaQNGnPaSgQ==, mapNumber=null, fund=null)}, authors=[Author(id=1329456779819331596, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=lirg@ieisystem.com, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329456779886440463, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456779819331596, language=EN, stringName=Rengang Li, firstName=Rengang, middleName=null, lastName=Li, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1Department of Computer Science and Technology, Tsinghua University, Beijing 100084
2Inspur (Beijing) Electronic Information Industry Co., Ltd, Beijing 100085, bio={"content":"

born in 1980. PhD, Senior engineer. Member of CCF. His main research interest includes heterogeneous computing

"}, bioImg=null, bioContent=

born in 1980. PhD, Senior engineer. Member of CCF. His main research interest includes heterogeneous computing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329456779940966416, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456779819331596, language=CN, stringName=李仁刚, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1清华大学计算机科学与技术系 北京 100084
2浪潮(北京)电子信息产业有限公司 北京 100085, bio={"img":"6V/SiWKVnpW/WYFikr2Bcw==","content":"

1980年生. 博士,正高级工程师. CCF会员. 主要研究方向为异构计算

"}, bioImg=6V/SiWKVnpW/WYFikr2Bcw==, bioContent=

1980年生. 博士,正高级工程师. CCF会员. 主要研究方向为异构计算

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329456779546701826, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, xref=1, ext=[AuthorCompanyExt(id=1329456779559284739, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779546701826, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Department of Computer Science and Technology, Tsinghua University, Beijing 100084), AuthorCompanyExt(id=1329456779563479044, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779546701826, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1清华大学计算机科学与技术系 北京 100084)]), AuthorCompany(id=1329456779634782213, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, xref=2, ext=[AuthorCompanyExt(id=1329456779643170822, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779634782213, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Inspur (Beijing) Electronic Information Industry Co., Ltd, Beijing 100085), AuthorCompanyExt(id=1329456779651559431, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779634782213, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2浪潮(北京)电子信息产业有限公司 北京 100085)])]), Author(id=1329456779999686674, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329456780100349972, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456779999686674, language=EN, stringName=Yanwei Wang, firstName=Yanwei, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2Inspur (Beijing) Electronic Information Industry Co., Ltd, Beijing 100085, bio={"content":"

born in 1985. PhD, associate researcher. Member of CCF. His main research interest includes heterogeneous computing

"}, bioImg=null, bioContent=

born in 1985. PhD, associate researcher. Member of CCF. His main research interest includes heterogeneous computing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329456780171653141, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456779999686674, language=CN, stringName=王彦伟, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2浪潮(北京)电子信息产业有限公司 北京 100085, bio={"img":"QjyspQeHI8at3AAmeQWjBQ==","content":"

1985年生. 博士,副研究员. CCF会员. 主要研究方向为异构计算

"}, bioImg=QjyspQeHI8at3AAmeQWjBQ==, bioContent=

1985年生. 博士,副研究员. CCF会员. 主要研究方向为异构计算

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329456779634782213, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, xref=2, ext=[AuthorCompanyExt(id=1329456779643170822, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779634782213, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Inspur (Beijing) Electronic Information Industry Co., Ltd, Beijing 100085), AuthorCompanyExt(id=1329456779651559431, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779634782213, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2浪潮(北京)电子信息产业有限公司 北京 100085)])]), Author(id=1329456780234567703, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329456780301676569, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456780234567703, language=EN, stringName=Rui Hao, firstName=Rui, middleName=null, lastName=Hao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2Inspur (Beijing) Electronic Information Industry Co., Ltd, Beijing 100085, bio={"content":"

born in 1983. Master, Senior engineer. Member of CCF. His main research interest includes heterogeneous computing

"}, bioImg=null, bioContent=

born in 1983. Master, Senior engineer. Member of CCF. His main research interest includes heterogeneous computing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329456780364591130, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456780234567703, language=CN, stringName=郝锐, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2浪潮(北京)电子信息产业有限公司 北京 100085, bio={"img":"DhHTBr/Ax4tSq5g+Q8n57w==","content":"

1983年生. 硕士,高级工程师. CCF会员. 主要研究方向为异构计算

"}, bioImg=DhHTBr/Ax4tSq5g+Q8n57w==, bioContent=

1983年生. 硕士,高级工程师. CCF会员. 主要研究方向为异构计算

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329456779634782213, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, xref=2, ext=[AuthorCompanyExt(id=1329456779643170822, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779634782213, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Inspur (Beijing) Electronic Information Industry Co., Ltd, Beijing 100085), AuthorCompanyExt(id=1329456779651559431, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779634782213, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2浪潮(北京)电子信息产业有限公司 北京 100085)])]), Author(id=1329456780431699996, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329456780503003166, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456780431699996, language=EN, stringName=Linge Xiao, firstName=Linge, middleName=null, lastName=Xiao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2Inspur (Beijing) Electronic Information Industry Co., Ltd, Beijing 100085, bio={"content":"

born in 1994. PhD. His main research interest includes heterogeneous computing

"}, bioImg=null, bioContent=

born in 1994. PhD. His main research interest includes heterogeneous computing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329456780599472159, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456780431699996, language=CN, stringName=肖麟阁, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2浪潮(北京)电子信息产业有限公司 北京 100085, bio={"img":"y3RrTv3/z2ysMQEd5yRkpQ==","content":"

1994年生. 博士. 主要研究方向为异构计算

"}, bioImg=y3RrTv3/z2ysMQEd5yRkpQ==, bioContent=

1994年生. 博士. 主要研究方向为异构计算

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329456779634782213, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, xref=2, ext=[AuthorCompanyExt(id=1329456779643170822, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779634782213, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Inspur (Beijing) Electronic Information Industry Co., Ltd, Beijing 100085), AuthorCompanyExt(id=1329456779651559431, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779634782213, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2浪潮(北京)电子信息产业有限公司 北京 100085)])]), Author(id=1329456780670775329, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=yangle01@ieisystem.com, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329456780750467107, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456780670775329, language=EN, stringName=Le Yang, firstName=Le, middleName=null, lastName=Yang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, *, address=3Guangdong Inspur Intelligent Computing Technology Co., Ltd, Guangzhou 510623, bio={"content":"

born in 1990. Master. Her main research interest includes heterogeneous computing

"}, bioImg=null, bioContent=

born in 1990. Master. Her main research interest includes heterogeneous computing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329456780809187364, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456780670775329, language=CN, stringName=杨乐, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, *, address=3广东浪潮智慧计算技术有限公司 广州 510623, bio={"img":"KLkWUxvjR+bDMr/Yrvvk/A==","content":"

1990年生. 硕士. 主要研究方向为异构计算

"}, bioImg=KLkWUxvjR+bDMr/Yrvvk/A==, bioContent=

1990年生. 硕士. 主要研究方向为异构计算

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329456779743834120, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, xref=3, ext=[AuthorCompanyExt(id=1329456779748028425, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779743834120, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3Guangdong Inspur Intelligent Computing Technology Co., Ltd, Guangzhou 510623), AuthorCompanyExt(id=1329456779756417034, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779743834120, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3广东浪潮智慧计算技术有限公司 广州 510623)])]), Author(id=1329456780872101926, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, orderNo=5, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329456780955988008, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456780872101926, language=EN, stringName=Guangwen Yang, firstName=Guangwen, middleName=null, lastName=Yang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1Department of Computer Science and Technology, Tsinghua University, Beijing 100084, bio={"content":"

born in 1963. PhD, professor, PhD supervisor. Member of CCF. His main research interest includes high performance computing

"}, bioImg=null, bioContent=

born in 1963. PhD, professor, PhD supervisor. Member of CCF. His main research interest includes high performance computing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329456781031485481, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456780872101926, language=CN, stringName=杨广文, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1清华大学计算机科学与技术系 北京 100084, bio={"img":"2fyXPhE8r9p1no8L0V4NXg==","content":"

1963年生. 博士,教授,博士生导师. CCF会员. 主要研究方向为高性能计算

"}, bioImg=2fyXPhE8r9p1no8L0V4NXg==, bioContent=

1963年生. 博士,教授,博士生导师. CCF会员. 主要研究方向为高性能计算

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329456779546701826, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, xref=1, ext=[AuthorCompanyExt(id=1329456779559284739, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779546701826, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Department of Computer Science and Technology, Tsinghua University, Beijing 100084), AuthorCompanyExt(id=1329456779563479044, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779546701826, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1清华大学计算机科学与技术系 北京 100084)])]), Author(id=1329456781106982955, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, orderNo=6, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329456781186674733, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456781106982955, language=EN, stringName=Hongwei Kan, firstName=Hongwei, middleName=null, lastName=Kan, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, address=3Guangdong Inspur Intelligent Computing Technology Co., Ltd, Guangzhou 510623, bio={"content":"

born in 1975. Master, professor-Senior engineer. Senior member of CCF. His main research interest includes heterogeneous computing

"}, bioImg=null, bioContent=

born in 1975. Master, professor-Senior engineer. Senior member of CCF. His main research interest includes heterogeneous computing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329456781266366510, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456781106982955, language=CN, stringName=阚宏伟, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, address=3广东浪潮智慧计算技术有限公司 广州 510623, bio={"img":"y3uiBIRgHYLc05CmqpN6lA==","content":"

1975年生. 硕士,教授-正高级工程师. CCF高级会员. 主要研究方向为异构计算

"}, bioImg=y3uiBIRgHYLc05CmqpN6lA==, bioContent=

1975年生. 硕士,教授-正高级工程师. CCF高级会员. 主要研究方向为异构计算

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329456779743834120, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, xref=3, ext=[AuthorCompanyExt(id=1329456779748028425, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779743834120, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3Guangdong Inspur Intelligent Computing Technology Co., Ltd, Guangzhou 510623), AuthorCompanyExt(id=1329456779756417034, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779743834120, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3广东浪潮智慧计算技术有限公司 广州 510623)])])], keywords=[Keyword(id=1329456782977642543, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, orderNo=1, keyword=inter-node communication), Keyword(id=1329456783048945712, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, orderNo=2, keyword=FPGA), Keyword(id=1329456783120248881, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, orderNo=3, keyword=GPU), Keyword(id=1329456783183163442, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, orderNo=4, keyword=RDMA), Keyword(id=1329456783262855219, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, orderNo=5, keyword=zero copy), Keyword(id=1329456783355129908, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, orderNo=1, keyword=节点间通信), Keyword(id=1329456783430627381, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, orderNo=2, keyword=FPGA), Keyword(id=1329456783590010934, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, orderNo=3, keyword=GPU), Keyword(id=1329456783657119799, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, orderNo=4, keyword=RDMA), Keyword(id=1329456783728422968, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, orderNo=5, keyword=零拷贝)], refs=[Reference(id=1329456788946137188, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=Fröning H, Nüssle M, Litz H, et al. A case for FPGA based accelerated communication[C]//Proc of the 9th Int Conf on Networks. Piscataway, NJ: IEEE, 2010: 28−33, articleTitle=null, refAbstract=null), Reference(id=1329456789013246053, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Shainer G, Ayoub A, Lui P, et al. The development of Mellanox/NVIDIA GPUDirect over InfiniBand—a new model for GPU to GPU communications[J]. Computer Science Research and Development, 2011, 26: 267−273, articleTitle=null, refAbstract=null), Reference(id=1329456789143269478, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Ammendola R, Bernaschi M, Biagioni A, et al. GPU peer-to-peer techniques applied to a cluster interconnect[C]//Proc of 2013 IEEE Int Symp on Parallel & Distributed Processing, Workshops and PhD Forum. Piscataway, NJ: IEEE, 2013: 806−815, articleTitle=null, refAbstract=null), Reference(id=1329456789201989735, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Agostini E, Rossetti D, Potluri S. Offloading communication control logic in GPU accelerated applications[C]//Proc of 17th IEEE/ACM Int Symp on Cluster, Cloud and Grid Computing (CCGRID). Piscataway, NJ: IEEE, 2017: 248−257, articleTitle=null, refAbstract=null), Reference(id=1329456789269098600, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Agostini E, Rossetti D, Potluri S. GPUDirect Async: Exploring GPU synchronous communication techniques for InfiniBand clusters[J]. Journal of Parallel and Distributed Computing, 2018, 114: 28−45, articleTitle=null, refAbstract=null), Reference(id=1329456789340401769, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Oden L, Fröning H. GGAS: Global GPU address spaces for efficient communication in heterogeneous clusters[C]//Proc of 2013 IEEE Int Conf on Cluster Computing (CLUSTER). Piscataway, NJ: IEEE, 2013: 1−8, articleTitle=null, refAbstract=null), Reference(id=1329456789403316330, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Oden L, Fröning H. InfiniBand Verbs on GPU: A case study of controlling an InfiniBand network device from the GPU[J]. The International Journal of High Performance Computing Applications, 2017, 31(4): 274−284, articleTitle=null, refAbstract=null), Reference(id=1329456789470425195, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Daoud F, Watad A, Silberstein M. GPUrdma: GPU-side library for high performance networking from GPU kernels[C]//Procof the 6th Int Workshop on Runtime and Operating Systems for Supercomputers. New York: ACM, 2016: 1−8, articleTitle=null, refAbstract=null), Reference(id=1329456789650780268, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Silberstein M, Kim S, Huh S, et al. GPUnet: Networking abstractions for GPU programs[J]. ACM Transactions on Computer Systems, 2016, 34(3): 1−31, articleTitle=null, refAbstract=null), Reference(id=1329456789734666349, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Balle S M, Tetreault M, Dicecco R. Inter-kernel links for direct inter-FPGA communication[DB/OL]. [2022-10-19]. https://cdrdv2-public.intel.com/650535/wp-01305-inter-kernel-links-for-direct-inter-fpga-communication.pdf., articleTitle=null, refAbstract=null), Reference(id=1329456789822746734, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Fujita N, Kobayashi R, Yamaguchi Y, et al. Parallel processing on FPGA combining computation and communication in OpenCL programming[C]/Proc of 2019 IEEE Int Parallel and Distributed Processing Symp Workshops (IPDPSW). Piscataway, NJ: IEEE, 2019: 479−488, articleTitle=null, refAbstract=null), Reference(id=1329456789894049903, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Fujita N, Kobayashi R, Yamaguchi Y, et al. Performance evaluation of pipelined communication combined with computation in OpenCL programming on FPGA[C]//Proc of 2020 IEEE Int Parallel and Distributed Processing Symp Workshops (IPDPSW). Piscataway, NJ: IEEE, 2020: 450−459, articleTitle=null, refAbstract=null), Reference(id=1329456789969547376, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Kobayashi R, Fujita N, Yamaguchi Y, et al. GPU-FPGA heterogeneous computing with OpenCL-enabled direct memory access[C]//Proc of 2019 IEEE Int Parallel and Distributed Processing Symp Workshops (IPDPSW). Piscataway, NJ: IEEE, 2019: 489−498, articleTitle=null, refAbstract=null), Reference(id=1329456790053433457, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Kobayashi R, Fujita N, Yamaguchi Y, et al. OpenCL-enabled GPU-FPGA accelerated computing with inter-FPGA communication[C]//Proc of the Int Conf on High Performance Computing in Asia-Pacific Region Workshops. New York: ACM 2020: 17−20, articleTitle=null, refAbstract=null), Reference(id=1329456790137319538, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Burstein I. Nvidia data center processing unit (DPU) architecture[C]//Proc of 2021 IEEE Hot Chips 33 Symp (HCS). Piscataway, NJ: IEEE, 2021: 1−20, articleTitle=null, refAbstract=null), Reference(id=1329456791752126579, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Sundar N, Burres B, Li Y, et al. An in-depth look at the Intel IPU E2000[C]//Proc of 2023 IEEE Int Solid-State Circuits Conf (ISSCC). Piscataway, NJ: IEEE, 2023: 162−164, articleTitle=null, refAbstract=null), Reference(id=1329456791877955700, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Lant J, Navaridas J, Luján M, et al. Toward FPGA-based HPC: Advancing interconnect technologies[J]. IEEE Micro, 2019, 40(1): 25−34, articleTitle=null, refAbstract=null), Reference(id=1329456791945064565, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Mittal R, Shpiner A, Panda A, et al. Revisiting network support for RDMA[C]//Proc of the 2018 Conf of the ACM Special Interest Group on Data Communication. New York: ACM, 2018: 313−326, articleTitle=null, refAbstract=null), Reference(id=1329456792024756342, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Wadekar M. Handbook of Fiber Optic Data Communication[M]//Cambridge, MA: Academic Press, 2013: 267−287, articleTitle=null, refAbstract=null), Reference(id=1329456792091865207, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Jääskeläinen P, de La Lama C S, Schnetter E, et al. PoCL: A performance-portable OpenCL implementation[J]. International Journal of Parallel Programming, 2015, 43: 752−785, articleTitle=null, refAbstract=null), Reference(id=1329456792163168376, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Tine B, Yalamarthy K P, Elsabbagh F, et al. Vortex: Extending the RISC-V ISA for GPGPU and 3D-graphics[C]//Proc of the 54th Annual IEEE/ACM Int Symp on Microarchitecture (MICRO-54). New York: ACM, 2021: 754−766, articleTitle=null, refAbstract=null), Reference(id=1329456792226082937, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Elsabbagh F, Tine B, Roshan P, et al. Vortex: OpenCL Compatible RISC-V GPGPU[J]. arXiv preprint, arXiv: 2002.12151, 2020, articleTitle=null, refAbstract=null), Reference(id=1329456792297386106, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Intel OPAE. Open programmable acceleration engine (OPAE) C API programming guide [DB/OL]. [2021-11-08] https://cdrdv2.intel.com/v1/dl/getContent/686262?explicitVersion=true&wapkw=opae., articleTitle=null, refAbstract=null), Reference(id=1329456792360300667, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=曾高雄,胡水海,张骏雪,等. 数据中心网络传输协议综述[J]. 计算机研究与发展, 2020, 57(1): 74−84, articleTitle=null, refAbstract=null), Reference(id=1329456792456769660, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=Zeng Gaoxiong, Hu Shuihai, Zhang Junxue, et al. Transport protocols for data center networks: A survey[J]. Journal of Computer Research and Development, 2020, 57(1): 74−84 (in Chinese), articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1329456779546701826, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, xref=1, ext=[AuthorCompanyExt(id=1329456779559284739, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779546701826, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Department of Computer Science and Technology, Tsinghua University, Beijing 100084), AuthorCompanyExt(id=1329456779563479044, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779546701826, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1清华大学计算机科学与技术系 北京 100084)]), AuthorCompany(id=1329456779634782213, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, xref=2, ext=[AuthorCompanyExt(id=1329456779643170822, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779634782213, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Inspur (Beijing) Electronic Information Industry Co., Ltd, Beijing 100085), AuthorCompanyExt(id=1329456779651559431, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779634782213, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2浪潮(北京)电子信息产业有限公司 北京 100085)]), AuthorCompany(id=1329456779743834120, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, xref=3, ext=[AuthorCompanyExt(id=1329456779748028425, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779743834120, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3Guangdong Inspur Intelligent Computing Technology Co., Ltd, Guangzhou 510623), AuthorCompanyExt(id=1329456779756417034, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779743834120, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3广东浪潮智慧计算技术有限公司 广州 510623)])], figs=[ArticleFig(id=1329456783879417913, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Figure 1, caption=Development of NVIDIA GPUDirect, figureFileSmall=EPYIeK01BKR+9Wwk8dNKkg==, figureFileBig=nP5aqf0m9Rm/eBOwlZYTZA==, tableContent=null), ArticleFig(id=1329456783967498298, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=图1, caption=NVIDIA GPUDirect的发展, figureFileSmall=EPYIeK01BKR+9Wwk8dNKkg==, figureFileBig=nP5aqf0m9Rm/eBOwlZYTZA==, tableContent=null), ArticleFig(id=1329456784189796411, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Figure 2, caption=Illustration of Intel IKL, figureFileSmall=PnWlmrUmqkbvGoiVIESE8w==, figureFileBig=LSGdli7ouo5h8EiW5HkyMA==, tableContent=null), ArticleFig(id=1329456784307236924, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=图2, caption=Intel IKL示意图, figureFileSmall=PnWlmrUmqkbvGoiVIESE8w==, figureFileBig=LSGdli7ouo5h8EiW5HkyMA==, tableContent=null), ArticleFig(id=1329456784391123005, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Figure 3, caption=Illustrations of systematic organization architecture based on the traditional GPU and Direct xPU, figureFileSmall=2mJVyP1F+ghlFuvan+qeYg==, figureFileBig=ix1ZHVRK3PzbMFfA0lpTdQ==, tableContent=null), ArticleFig(id=1329456784483397694, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=图3, caption=基于传统GPU和Direct xPU的系统组织架构示意图, figureFileSmall=2mJVyP1F+ghlFuvan+qeYg==, figureFileBig=ix1ZHVRK3PzbMFfA0lpTdQ==, tableContent=null), ArticleFig(id=1329456784579866687, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Figure 4, caption=Illustrations of architecture of Direct xPU hardware, figureFileSmall=GVr1xQ5dtaYkN1JTV9v8iA==, figureFileBig=gZp3STqHC1BV8T8Dp6DaUQ==, tableContent=null), ArticleFig(id=1329456784680529984, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=图4, caption=Direct xPU的硬件架构示意图, figureFileSmall=GVr1xQ5dtaYkN1JTV9v8iA==, figureFileBig=gZp3STqHC1BV8T8Dp6DaUQ==, tableContent=null), ArticleFig(id=1329456784781193282, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Figure 5, caption=Illustration of RLTP protocol, figureFileSmall=wuyDv0O199FawQJFbQMXIw==, figureFileBig=yBed567qe/4kYMVYmf+ZHg==, tableContent=null), ArticleFig(id=1329456784848302147, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=图5, caption=RLTP协议的示意图, figureFileSmall=wuyDv0O199FawQJFbQMXIw==, figureFileBig=yBed567qe/4kYMVYmf+ZHg==, tableContent=null), ArticleFig(id=1329456784932188228, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Figure 6, caption=Go-back-N strategy and the selective repeat transmission strategy, figureFileSmall=xEwwedkVLHHOs30qLJZiCg==, figureFileBig=7PEVeykrTULDMR8Me+gdAA==, tableContent=null), ArticleFig(id=1329456785028657221, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=图6, caption=Go-back-N策略和选择性重传策略, figureFileSmall=xEwwedkVLHHOs30qLJZiCg==, figureFileBig=7PEVeykrTULDMR8Me+gdAA==, tableContent=null), ArticleFig(id=1329456785116737606, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Figure 7, caption=The software architecture of Direct xPU, figureFileSmall=/b2CoirL/yRCBkOfd/RaqA==, figureFileBig=XUwPjX5IwzP8ySuDNRg0pg==, tableContent=null), ArticleFig(id=1329456785188040775, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=图7, caption=Direct xPU的软件架构, figureFileSmall=/b2CoirL/yRCBkOfd/RaqA==, figureFileBig=XUwPjX5IwzP8ySuDNRg0pg==, tableContent=null), ArticleFig(id=1329456785263538248, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Figure 8, caption=Details of GPU MMD library, figureFileSmall=PVbx6KSFZB3/0WoV7MIxTw==, figureFileBig=A93s/gAsgEdJqyGC/Hj+wA==, tableContent=null), ArticleFig(id=1329456785360007241, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=图8, caption=GPU MMD库的详细信息, figureFileSmall=PVbx6KSFZB3/0WoV7MIxTw==, figureFileBig=A93s/gAsgEdJqyGC/Hj+wA==, tableContent=null), ArticleFig(id=1329456785435504714, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Figure 9, caption=Generation process of GPU binary program, figureFileSmall=N3iwi4FZ0UGYwu+NxBu8pA==, figureFileBig=3yNiea9LIDLr1MnJlgj2ew==, tableContent=null), ArticleFig(id=1329456785519390795, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=图9, caption=GPU二进制程序的生成过程, figureFileSmall=N3iwi4FZ0UGYwu+NxBu8pA==, figureFileBig=3yNiea9LIDLr1MnJlgj2ew==, tableContent=null), ArticleFig(id=1329456785582305356, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Figure 10, caption=xPU Box equipped with 16 PCIe slots, figureFileSmall=SaN+MuQYc6yP9SDslrxGLA==, figureFileBig=t58Y7nWiG97la5qOIYCm5g==, tableContent=null), ArticleFig(id=1329456785670385741, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=图10, caption=配备了16个PCIe插槽的xPU Box, figureFileSmall=SaN+MuQYc6yP9SDslrxGLA==, figureFileBig=t58Y7nWiG97la5qOIYCm5g==, tableContent=null), ArticleFig(id=1329456787327135822, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Figure 11, caption=Bandwidth tests with different network cards at three packet loss rate, figureFileSmall=dRK2jWwXT32AYpMps6EjqQ==, figureFileBig=NwX3CMAKirH4PUFiR0+hmQ==, tableContent=null), ArticleFig(id=1329456787415216207, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=图11, caption=不同网卡在3种丢包率下的带宽测试, figureFileSmall=dRK2jWwXT32AYpMps6EjqQ==, figureFileBig=NwX3CMAKirH4PUFiR0+hmQ==, tableContent=null), ArticleFig(id=1329456787478130768, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Figure 12, caption=The complete process of communication between GPU nodes from the perspective of sender, figureFileSmall=djGxONRjoGpjyWyV2gw6bw==, figureFileBig=ahLwMBnnsZtDRss8TQQG9A==, tableContent=null), ArticleFig(id=1329456787541045329, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=图12, caption=发送方视角下GPU节点间通信的完整过程, figureFileSmall=djGxONRjoGpjyWyV2gw6bw==, figureFileBig=ahLwMBnnsZtDRss8TQQG9A==, tableContent=null), ArticleFig(id=1329456787608154194, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Figure 13, caption=Latency testing setup between GPU nodes on $t_3-t'_2 $, figureFileSmall=4pOu9c3Z9TGfXQiQNvbwIg==, figureFileBig=EDxC/SrhYOlFE8RWChM8fA==, tableContent=null), ArticleFig(id=1329456787671068755, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=图13, caption=GPU节点之间的$t_3-t'_2 $延迟测试设置, figureFileSmall=4pOu9c3Z9TGfXQiQNvbwIg==, figureFileBig=EDxC/SrhYOlFE8RWChM8fA==, tableContent=null), ArticleFig(id=1329456787750760532, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Figure 14, caption=Latency testing results between GPU nodes communication on $ t_3-t'_2 $, figureFileSmall=rBI5YRZXaqGmQ6D2ACCHSw==, figureFileBig=gun/17SRFCqgroUgsZD0Aw==, tableContent=null), ArticleFig(id=1329456787809480789, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=图14, caption=GPU节点间通信在$ t_3-t'_2 $的延迟测试结果, figureFileSmall=rBI5YRZXaqGmQ6D2ACCHSw==, figureFileBig=gun/17SRFCqgroUgsZD0Aw==, tableContent=null), ArticleFig(id=1329456787889172566, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Figure 15, caption=Latency of different number of registers when the host node writes to the GPU node, figureFileSmall=H2haym72v0Rx7W0MZpUgjw==, figureFileBig=1h4FvCyFNXDoZOz1CwdwQw==, tableContent=null), ArticleFig(id=1329456787981447255, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=图15, caption=主机节点写入GPU节点的不同数量寄存器的延迟, figureFileSmall=H2haym72v0Rx7W0MZpUgjw==, figureFileBig=1h4FvCyFNXDoZOz1CwdwQw==, tableContent=null), ArticleFig(id=1329456788048556120, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Figure 16, caption=Bandwidth performance testing result between GPU nodes when the amount of data is 1 Mb and 1 Gb respectively, figureFileSmall=L6sv9vMkIamesyDdLTv/1A==, figureFileBig=CM9SHWocT3g2+G5BZTMCvg==, tableContent=null), ArticleFig(id=1329456788128247897, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=图16, caption=数据量大小分别为1Mb和1Gb时GPU节点间的带宽性能测试结果, figureFileSmall=L6sv9vMkIamesyDdLTv/1A==, figureFileBig=CM9SHWocT3g2+G5BZTMCvg==, tableContent=null), ArticleFig(id=1329456788195356762, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Figure 17, caption=Application of a prototype system based on Direct xPU distributed computing platform, figureFileSmall=aQ0Mo7ux+P/bA+7wB01TJw==, figureFileBig=Fg7yt2TG+uWtBfP03HAtbQ==, tableContent=null), ArticleFig(id=1329456788258271323, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=图17, caption=基于Direct xPU分布式计算平台的原型系统的应用, figureFileSmall=aQ0Mo7ux+P/bA+7wB01TJw==, figureFileBig=Fg7yt2TG+uWtBfP03HAtbQ==, tableContent=null), ArticleFig(id=1329456788325380188, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Table 1, caption=

Features Comparison of Three Types of RDMA Implementations

, figureFileSmall=null, figureFileBig=null, tableContent=
特性InfiniBandiWARPRoCE
性能最好稍差与InfiniBand相当
成本高中低
稳定性好差较好
交换机需求IB交换机以太网交换机以太网交换机
), ArticleFig(id=1329456788396683357, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=表1, caption=

3种RDMA实现的特性对比

, figureFileSmall=null, figureFileBig=null, tableContent=
特性InfiniBandiWARPRoCE
性能最好稍差与InfiniBand相当
成本高中低
稳定性好差较好
交换机需求IB交换机以太网交换机以太网交换机
), ArticleFig(id=1329456788472180830, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Table 2, caption=

Features of Three Types of RDMA Implementations in Terms of Retransmission

, figureFileSmall=null, figureFileBig=null, tableContent=
网络类型传输层重传协议复杂度可靠性
iWARPTCPSR高高
RoCEv2UDPGo-back-N低低
iRDMAUDPSR中等与iWARP相当
), ArticleFig(id=1329456788543483999, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=表2, caption=

3种RDMA实现在重传方面的特性

, figureFileSmall=null, figureFileBig=null, tableContent=
网络类型传输层重传协议复杂度可靠性
iWARPTCPSR高高
RoCEv2UDPGo-back-N低低
iRDMAUDPSR中等与iWARP相当
), ArticleFig(id=1329456788627370080, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Table 3, caption=

Specification of Main Systematic Components

, figureFileSmall=null, figureFileBig=null, tableContent=
组件规格
CPUIntel Xeon E5-2640 V4
CPU内存DDR4,2133 MHz,256 GB
主机网卡Intel x520-sr2,10 Gbps
主机操作系统CentOS 7.8
FPGA板卡Inspur F10A(Intel Arria 10)
FPGA内存DDR4,2133 MHz,16 GB
交换机锐捷S6220
FPGA编译工具Quartus 19.1
Vortex GPU版本0.2.3,32 Core
), ArticleFig(id=1329456788690284641, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=表3, caption=

主要系统组件的规格

, figureFileSmall=null, figureFileBig=null, tableContent=
组件规格
CPUIntel Xeon E5-2640 V4
CPU内存DDR4,2133 MHz,256 GB
主机网卡Intel x520-sr2,10 Gbps
主机操作系统CentOS 7.8
FPGA板卡Inspur F10A(Intel Arria 10)
FPGA内存DDR4,2133 MHz,16 GB
交换机锐捷S6220
FPGA编译工具Quartus 19.1
Vortex GPU版本0.2.3,32 Core
), ArticleFig(id=1329456788749004898, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=EN, label=Table 4, caption=

Comparison of Inter-Node Communicating Latency of Four FPGA-Based Distributed Computing Architectures

, figureFileSmall=null, figureFileBig=null, tableContent=
架构FPGA时钟频率/MHz延迟/µs
CoE200~2500.95(4 B包大小)
CIRCUS295.80.5~1.87(根据包大小和
设备的间隔距离变化)
AiS222.55.04(4 B包大小)
Direct xPU2002.69(4 B包大小)
), ArticleFig(id=1329456788832890979, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, language=CN, label=表4, caption=

4种基于FPGA的分布式计算架构的节点间通信延迟对比

, figureFileSmall=null, figureFileBig=null, tableContent=
架构FPGA时钟频率/MHz延迟/µs
CoE200~2500.95(4 B包大小)
CIRCUS295.80.5~1.87(根据包大小和
设备的间隔距离变化)
AiS222.55.04(4 B包大小)
Direct xPU2002.69(4 B包大小)
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202440055, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202440055, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202440055, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202440055, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1717171200000, fullTextJson=null, articleText=null, reference=null)
收藏切换
Direct xPU:一种新型节点间通信优化的分布式异构计算架构
李仁刚 1, 2 , 王彦伟 2 , 郝锐 2 , 肖麟阁 2 , 杨乐 3, * , 杨广文 1 , 阚宏伟 3
计算机研究与发展 | 处理器芯片跨层优化专题 2024,61(6): 1388-1400
收起
收藏切换
计算机研究与发展 |处理器芯片跨层优化专题 2024 , 61 (6) : 1388 -1400
Direct xPU:一种新型节点间通信优化的分布式异构计算架构
全屏
[Author(id=1329456779819331596, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=lirg@ieisystem.com, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329456779886440463, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456779819331596, language=EN, stringName=Rengang Li, firstName=Rengang, middleName=null, lastName=Li, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1Department of Computer Science and Technology, Tsinghua University, Beijing 100084
2Inspur (Beijing) Electronic Information Industry Co., Ltd, Beijing 100085, bio={"content":"

born in 1980. PhD, Senior engineer. Member of CCF. His main research interest includes heterogeneous computing

"}, bioImg=null, bioContent=

born in 1980. PhD, Senior engineer. Member of CCF. His main research interest includes heterogeneous computing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329456779940966416, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456779819331596, language=CN, stringName=李仁刚, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1清华大学计算机科学与技术系 北京 100084
2浪潮(北京)电子信息产业有限公司 北京 100085, bio={"img":"6V/SiWKVnpW/WYFikr2Bcw==","content":"

1980年生. 博士,正高级工程师. CCF会员. 主要研究方向为异构计算

"}, bioImg=6V/SiWKVnpW/WYFikr2Bcw==, bioContent=

1980年生. 博士,正高级工程师. CCF会员. 主要研究方向为异构计算

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329456779546701826, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, xref=1, ext=[AuthorCompanyExt(id=1329456779559284739, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779546701826, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Department of Computer Science and Technology, Tsinghua University, Beijing 100084), AuthorCompanyExt(id=1329456779563479044, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779546701826, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1清华大学计算机科学与技术系 北京 100084)]), AuthorCompany(id=1329456779634782213, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, xref=2, ext=[AuthorCompanyExt(id=1329456779643170822, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779634782213, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Inspur (Beijing) Electronic Information Industry Co., Ltd, Beijing 100085), AuthorCompanyExt(id=1329456779651559431, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779634782213, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2浪潮(北京)电子信息产业有限公司 北京 100085)])]), Author(id=1329456779999686674, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329456780100349972, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456779999686674, language=EN, stringName=Yanwei Wang, firstName=Yanwei, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2Inspur (Beijing) Electronic Information Industry Co., Ltd, Beijing 100085, bio={"content":"

born in 1985. PhD, associate researcher. Member of CCF. His main research interest includes heterogeneous computing

"}, bioImg=null, bioContent=

born in 1985. PhD, associate researcher. Member of CCF. His main research interest includes heterogeneous computing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329456780171653141, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456779999686674, language=CN, stringName=王彦伟, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2浪潮(北京)电子信息产业有限公司 北京 100085, bio={"img":"QjyspQeHI8at3AAmeQWjBQ==","content":"

1985年生. 博士,副研究员. CCF会员. 主要研究方向为异构计算

"}, bioImg=QjyspQeHI8at3AAmeQWjBQ==, bioContent=

1985年生. 博士,副研究员. CCF会员. 主要研究方向为异构计算

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329456779634782213, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, xref=2, ext=[AuthorCompanyExt(id=1329456779643170822, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779634782213, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Inspur (Beijing) Electronic Information Industry Co., Ltd, Beijing 100085), AuthorCompanyExt(id=1329456779651559431, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779634782213, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2浪潮(北京)电子信息产业有限公司 北京 100085)])]), Author(id=1329456780234567703, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329456780301676569, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456780234567703, language=EN, stringName=Rui Hao, firstName=Rui, middleName=null, lastName=Hao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2Inspur (Beijing) Electronic Information Industry Co., Ltd, Beijing 100085, bio={"content":"

born in 1983. Master, Senior engineer. Member of CCF. His main research interest includes heterogeneous computing

"}, bioImg=null, bioContent=

born in 1983. Master, Senior engineer. Member of CCF. His main research interest includes heterogeneous computing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329456780364591130, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456780234567703, language=CN, stringName=郝锐, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2浪潮(北京)电子信息产业有限公司 北京 100085, bio={"img":"DhHTBr/Ax4tSq5g+Q8n57w==","content":"

1983年生. 硕士,高级工程师. CCF会员. 主要研究方向为异构计算

"}, bioImg=DhHTBr/Ax4tSq5g+Q8n57w==, bioContent=

1983年生. 硕士,高级工程师. CCF会员. 主要研究方向为异构计算

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329456779634782213, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, xref=2, ext=[AuthorCompanyExt(id=1329456779643170822, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779634782213, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Inspur (Beijing) Electronic Information Industry Co., Ltd, Beijing 100085), AuthorCompanyExt(id=1329456779651559431, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779634782213, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2浪潮(北京)电子信息产业有限公司 北京 100085)])]), Author(id=1329456780431699996, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329456780503003166, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456780431699996, language=EN, stringName=Linge Xiao, firstName=Linge, middleName=null, lastName=Xiao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2Inspur (Beijing) Electronic Information Industry Co., Ltd, Beijing 100085, bio={"content":"

born in 1994. PhD. His main research interest includes heterogeneous computing

"}, bioImg=null, bioContent=

born in 1994. PhD. His main research interest includes heterogeneous computing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329456780599472159, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456780431699996, language=CN, stringName=肖麟阁, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2浪潮(北京)电子信息产业有限公司 北京 100085, bio={"img":"y3RrTv3/z2ysMQEd5yRkpQ==","content":"

1994年生. 博士. 主要研究方向为异构计算

"}, bioImg=y3RrTv3/z2ysMQEd5yRkpQ==, bioContent=

1994年生. 博士. 主要研究方向为异构计算

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329456779634782213, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, xref=2, ext=[AuthorCompanyExt(id=1329456779643170822, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779634782213, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Inspur (Beijing) Electronic Information Industry Co., Ltd, Beijing 100085), AuthorCompanyExt(id=1329456779651559431, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779634782213, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2浪潮(北京)电子信息产业有限公司 北京 100085)])]), Author(id=1329456780670775329, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=yangle01@ieisystem.com, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329456780750467107, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456780670775329, language=EN, stringName=Le Yang, firstName=Le, middleName=null, lastName=Yang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, *, address=3Guangdong Inspur Intelligent Computing Technology Co., Ltd, Guangzhou 510623, bio={"content":"

born in 1990. Master. Her main research interest includes heterogeneous computing

"}, bioImg=null, bioContent=

born in 1990. Master. Her main research interest includes heterogeneous computing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329456780809187364, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456780670775329, language=CN, stringName=杨乐, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, *, address=3广东浪潮智慧计算技术有限公司 广州 510623, bio={"img":"KLkWUxvjR+bDMr/Yrvvk/A==","content":"

1990年生. 硕士. 主要研究方向为异构计算

"}, bioImg=KLkWUxvjR+bDMr/Yrvvk/A==, bioContent=

1990年生. 硕士. 主要研究方向为异构计算

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329456779743834120, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, xref=3, ext=[AuthorCompanyExt(id=1329456779748028425, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779743834120, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3Guangdong Inspur Intelligent Computing Technology Co., Ltd, Guangzhou 510623), AuthorCompanyExt(id=1329456779756417034, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779743834120, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3广东浪潮智慧计算技术有限公司 广州 510623)])]), Author(id=1329456780872101926, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, orderNo=5, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329456780955988008, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456780872101926, language=EN, stringName=Guangwen Yang, firstName=Guangwen, middleName=null, lastName=Yang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1Department of Computer Science and Technology, Tsinghua University, Beijing 100084, bio={"content":"

born in 1963. PhD, professor, PhD supervisor. Member of CCF. His main research interest includes high performance computing

"}, bioImg=null, bioContent=

born in 1963. PhD, professor, PhD supervisor. Member of CCF. His main research interest includes high performance computing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329456781031485481, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456780872101926, language=CN, stringName=杨广文, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1清华大学计算机科学与技术系 北京 100084, bio={"img":"2fyXPhE8r9p1no8L0V4NXg==","content":"

1963年生. 博士,教授,博士生导师. CCF会员. 主要研究方向为高性能计算

"}, bioImg=2fyXPhE8r9p1no8L0V4NXg==, bioContent=

1963年生. 博士,教授,博士生导师. CCF会员. 主要研究方向为高性能计算

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329456779546701826, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, xref=1, ext=[AuthorCompanyExt(id=1329456779559284739, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779546701826, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Department of Computer Science and Technology, Tsinghua University, Beijing 100084), AuthorCompanyExt(id=1329456779563479044, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779546701826, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1清华大学计算机科学与技术系 北京 100084)])]), Author(id=1329456781106982955, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, orderNo=6, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329456781186674733, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456781106982955, language=EN, stringName=Hongwei Kan, firstName=Hongwei, middleName=null, lastName=Kan, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, address=3Guangdong Inspur Intelligent Computing Technology Co., Ltd, Guangzhou 510623, bio={"content":"

born in 1975. Master, professor-Senior engineer. Senior member of CCF. His main research interest includes heterogeneous computing

"}, bioImg=null, bioContent=

born in 1975. Master, professor-Senior engineer. Senior member of CCF. His main research interest includes heterogeneous computing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329456781266366510, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, authorId=1329456781106982955, language=CN, stringName=阚宏伟, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, address=3广东浪潮智慧计算技术有限公司 广州 510623, bio={"img":"y3uiBIRgHYLc05CmqpN6lA==","content":"

1975年生. 硕士,教授-正高级工程师. CCF高级会员. 主要研究方向为异构计算

"}, bioImg=y3uiBIRgHYLc05CmqpN6lA==, bioContent=

1975年生. 硕士,教授-正高级工程师. CCF高级会员. 主要研究方向为异构计算

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329456779743834120, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, xref=3, ext=[AuthorCompanyExt(id=1329456779748028425, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779743834120, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3Guangdong Inspur Intelligent Computing Technology Co., Ltd, Guangzhou 510623), AuthorCompanyExt(id=1329456779756417034, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329456772353471440, companyId=1329456779743834120, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3广东浪潮智慧计算技术有限公司 广州 510623)])])]
李仁刚1, 2 , 王彦伟2, 郝锐2, 肖麟阁2, 杨乐3, * , 杨广文1, 阚宏伟3
作者信息
  • 1清华大学计算机科学与技术系 北京 100084
  • 2浪潮(北京)电子信息产业有限公司 北京 100085
  • 3广东浪潮智慧计算技术有限公司 广州 510623
通讯作者:
作者简介:

1980年生. 博士,正高级工程师. CCF会员. 主要研究方向为异构计算

1985年生. 博士,副研究员. CCF会员. 主要研究方向为异构计算

1983年生. 硕士,高级工程师. CCF会员. 主要研究方向为异构计算

1994年生. 博士. 主要研究方向为异构计算

1990年生. 硕士. 主要研究方向为异构计算

1963年生. 博士,教授,博士生导师. CCF会员. 主要研究方向为高性能计算

1975年生. 硕士,教授-正高级工程师. CCF高级会员. 主要研究方向为异构计算

Direct xPU: A Novel Distributed Heterogeneous Computing Architecture Optimized for Inter-node Communication Optimization
Rengang Li1, 2 , Yanwei Wang2, Rui Hao2, Linge Xiao2, Le Yang3, * , Guangwen Yang1, Hongwei Kan3
Affiliations
  • 1Department of Computer Science and Technology, Tsinghua University, Beijing 100084
  • 2Inspur (Beijing) Electronic Information Industry Co., Ltd, Beijing 100085
  • 3Guangdong Inspur Intelligent Computing Technology Co., Ltd, Guangzhou 510623
  • born in 1980. PhD, Senior engineer. Member of CCF. His main research interest includes heterogeneous computing

    born in 1985. PhD, associate researcher. Member of CCF. His main research interest includes heterogeneous computing

    born in 1983. Master, Senior engineer. Member of CCF. His main research interest includes heterogeneous computing

    born in 1994. PhD. His main research interest includes heterogeneous computing

    born in 1990. Master. Her main research interest includes heterogeneous computing

    born in 1963. PhD, professor, PhD supervisor. Member of CCF. His main research interest includes high performance computing

    born in 1975. Master, professor-Senior engineer. Senior member of CCF. His main research interest includes heterogeneous computing

出版时间: 2024-06-01 doi: 10.7544/issn1000-1239.202440055
文章导航
收藏切换

人工智能大模型应用的爆发式增长,使得难以依靠单一节点、单一类型的算力实现应用的规模部署,分布式异构计算成为主流选择,而节点间通信成为大模型训练或推理过程中的主要瓶颈之一. 目前,主要由GPU,FPGA等头部芯片厂商所主导的各种计算架构的节点间通信方案还存在一些问题. 一方面,为了追求极致的节点间通信性能,一部分架构选择使用协议简单而可扩展性差的点对点传输方案. 另一方面,传统的异构计算引擎(例如GPU)虽然在内存、计算管线等算力要素方面独立于CPU,但在通信要素方面却缺少专属的网络通信设备,需要完全或部分借助于CPU通过PCIe等物理链路来处理异构计算引擎与共享网络通信设备之间的通信. 所实现的Direct xPU分布式异构计算架构,使得异构计算引擎在算力要素和通信要素两方面均具有独立的、专属的设备,实现了数据的零拷贝,并进一步消除了节点间通信过程中处理跨芯片传输数据所带来的能耗和延迟. 测试结果表明,Direct xPU取得了与追求极致的节点间通信性能的计算架构相当的通信延迟,带宽接近物理通信带宽的上限.

节点间通信  /  FPGA  /  GPU  /  RDMA  /  零拷贝

The explosive growth of the application of large-scale artificial intelligence models has made it difficult to achieve the scale deployment of applications relying on a single node or a single type of computing architecture. Distributed heterogeneous computing has become the mainstream choice, and inter-node communication has become one of the main bottlenecks in the training or inference process of large models. Currently, there are still some deficiencies in the inter-node communicating solutions dominated by leading chip manufacturers. On the one hand, some architectures choose to use a simple but less scalable point-to-point transmission scheme in order to pursue the ultimate inter-node communication performance. On the other hand, traditional heterogeneous computing engines (such as GPUs) are independent of CPUs in terms of computing resources such as memory and computing cores, but they lack dedicated communicating network devices in terms of communication resources and need to rely entirely or partially on CPUs to handle transmission between heterogeneous computing engines and the shared communicating network device through physical links such as PCIe. The proposed Direct xPU distributed heterogeneous computing architecture in this article enables heterogeneous computing engines to have independent and dedicated devices in both computing and communication resources, achieving zero-copy data and further eliminating the energy consumption and latency associated with cross-chip data transfer during inter-node communication. Evaluations show that Direct xPU achieves communication latency comparable to computing architectures pursuing ultimate inter-node communication performance, with bandwidth close to the physical limit.

inter-node communication  /  FPGA  /  GPU  /  RDMA  /  zero copy
李仁刚, 王彦伟, 郝锐, 肖麟阁, 杨乐, 杨广文, 阚宏伟. Direct xPU:一种新型节点间通信优化的分布式异构计算架构. 计算机研究与发展, 2024 , 61 (6) : 1388 -1400 . DOI: 10.7544/issn1000-1239.202440055
Rengang Li, Yanwei Wang, Rui Hao, Linge Xiao, Le Yang, Guangwen Yang, Hongwei Kan. Direct xPU: A Novel Distributed Heterogeneous Computing Architecture Optimized for Inter-node Communication Optimization[J]. Journal of Computer Research and Development, 2024 , 61 (6) : 1388 -1400 . DOI: 10.7544/issn1000-1239.202440055
随着GPT等大模型的大量涌现和部署应用,同时具有计算密集型和I/O密集型特点的巨型神经网络使芯片算力和芯片间互联通信承压. 这一方面促进了数据中心和超级计算中心的融合;另一方面,也激发了底层架构的快速发展. 分布式异构计算是其中的研究重点之一. 高效的分布式计算不仅要求保持传统体系结构下高性能计算的优点,还要求提供与之匹配的高带宽、低延迟的节点间通信. 为了实现这一目标,学术界和工业界投入了大量的资金和精力,基于几种互连通信技术,如PCIe、RDMA等开发了各种编程范式和相应的硬件架构.
然而,现有的分布式异构计算体系结构似乎使得高性能、高带宽和低延迟成为一个不可能的三角[1]. 在这些分布式异构计算系统中,每个节点通过使用最先进的计算设备来实现高性能,而在节点之间,设备通过可选的几种不同的通信方式进行互联. 一般来说,高性能计算伴随着大量数据的生成和移动,带来数据重复拷贝开销、上下文切换、PCIe读写速度不均衡等一系列问题. 一个根本原因是在当前的分布式异构计算系统中,除了CPU之外的计算引擎在系统中并不是一个能完全独立实现计算和通信的角色. 例如,GPU之间的通信需要CPU的协助,或者GPU与CPU由于共享同一个网卡而被PCIe等物理链路限制带宽和拓扑结构.
我们的第1个想法是使所有计算引擎在分布式异构计算系统中具有完全的独立自主通信的能力,从而消除PCIe等物理链路的带宽和拓扑限制以及传统架构中由PCIe带来的读写速度不对称的问题. 我们的第2个想法是构建一套软硬件系统,以提供端到端的零拷贝能力. 此外,新型的计算架构与主流的编程范式兼容也是十分重要的一环.
本文的主要贡献包括3个方面:
1)提出了一种新型的分布式异构计算架构Direct xPU. 通过将分布式计算的2个关键要素——通信引擎层面和计算引擎芯片层面进行重新排布,实现更高效、更灵活的分布式异构计算系统.
2)基于FPGA构建了一个Direct xPU的原型系统,并实现了完整的软硬件基础设施,在有限的硬件条件下实现了GPU节点之间的类RDMA通信,延迟性能与之前的研究结果相当.
3)开发了xPU Box以及与之配套的一系列软硬件模块,以匹配数据中心的真实部署环境,充分挖掘基于Direct xPU新架构的新型系统拓扑结构,并基于原型系统进行了应用展示.
登纳德缩放定律和摩尔定律的失效使得开发最先进芯片的边际效应愈发明显,引发了研究分布式异构计算系统的热潮. 在一个分布式异构系统中存在多个节点,每个节点中可能存在不同类型的计算引擎,通常包括CPU,GPU或FPGA. 如何实现跨节点通信以获得一条低延迟、高带宽的数据通路是分布式异构系统的一个关键问题. 因此,关于分布式异构计算系统中的节点间通信问题的研究主要集中在这3类计算引擎上.
NVIDIA作为GPU行业的领导者,在GPU节点间互联通信方面开发了GPUDirect系列技术,包括GPUDirect Shared Memory,GPUDirect P2P,GPUDirect RDMA和GPUDirect Async. 这些技术之间的主要区别如图1所示. 早期的GPUDirect Shared Memory将GPU内存中的原始数据复制到主机的页锁定内存中,主机CPU控制网卡从页锁定内存中读取数据并发送给远端的节点的NIC(network interface controller)[2]. 很明显,该过程包括多次数据移动,并且需要CPU的深度参与,从而导致数据传输延迟的增加,并占用不必要的CPU资源. GPUDirect P2P实现了数据的零拷贝,但仍然需要CPU的参与,并且由于PCIe物理链路的限制,GPUDirect P2P只支持同一节点内GPU之间的通信[3]. GPUDirect RDMA通过使能同一节点的GPU和NIC之间的PCIe通信,支持不同节点的GPU之间的通信,但它也需要CPU触发NIC从GPU内存中读取数据,数据从GPU内存传输至通信端口需要跨越多个芯片[4]. GPUDirect Async则进一步实现了GPU自主地触发本节点中的NIC[5]. 总之,在GPUDirect系列技术中,总是由单个节点中的多个计算引擎共享同一个网络通信设备,这就意味着这些计算引擎与通信设备之间必定存在某些物理链路以提供直接或间接的触发路径,在GPUDirect系列技术中,由PCIe链路提供这样的路径. 此外,数据从GPU内存至数据发送起点的路径中存在多次跨芯片数据传输.
基于NVIDIA提供的软硬件,Oden等人[6]提出了一种名为GGAS(global GPU address space)的通信模型,该方法通过开发自定义的支持RDMA的FPGA网卡,实现了GPU之间的零拷贝和无GPU协助的数据传输. 但同一节点内CPU和GPU仍共享一个通信设备,其性能受限于PCIe链路. 此外,由于当时普通芯片组的PCIe系统对P2P通信的支持较差,导致实际带宽不到理论带宽的1/3. 此外,Oden等人[7]还指出,由于GPU的单线程性能与CPU有较大差距以及PCIe的开销,使得GPU自主发送和接收网络流量的效率较低. Daoud等人[8]提出了GPU端RDMA通信库——GPUrdma,实现了GPU kernel间的高效通信. 然而该方法需要使用专用的PCIe Switch以避免读写速度的显著差异,并且由于NVIDIA GPU的硬件限制和闭源性,需要花费大量精力修改相应的GPU和网卡驱动程序. Silberstein等人[9]为GPU的kernel程序提出了一种名为GPUnet的网络抽象模型,该抽象模型提供了一个类socket的API,以旁路CPU实现GPU的自主通信,然而NVIDIA GPU的局限性使得其不可能完全旁路CPU,GPUnet实际上仍需要CPU的协助来处理网络数据包,例如GPU和CPU之间的流控同步等. 总之,由于PCIe链路和共享同一个网卡的限制,现有的方案难以实现有效的GPU独立自主通信.
FPGA厂商和研究人员也纷纷涌入分布式异构领域. Intel发布了用于分布式FPGA间直接通信的IKL(inter-kernel links)[10],如图2所示. IKL使得FPGA在无需PCIe链路和CPU参与的情况下能够进行直接通信,因为该类型的FPGA具有非共享的网络通信模块. 然而,这种基于信用的流量控制和基于信道的P2P通信模型的复杂性极大地削弱了系统的可扩展性和互联性. IKL的另一个问题是该技术仅适用于FPGA之间的通信,而不支持CPU与FPGA之间的数据传输.
Fujita等人[11]实现了一项与IKL类似工作,名为CoE(channel over Ethernet),该方案中使用第三方的以太网控制器模块来实现IKL IP的功能,随后他们进一步提出了一种通信集成可重构计算系统(communication integrated reconfigurable computing system,CIRCUS) [12],该系统基于P2P拓扑和SeriaLite III协议代替交换机拓扑和以太网协议,以提高带宽和降低延迟. CoE和CIRCUS之间的权衡实际上是效率和兼容性之间的竞争. Kobayashi等人[13-14]提出了AiS (Accelerator-in-Switch),以实现GPU和FPGA之间或FPGA之间的直接通信,但是在该方案中,GPU之间的数据传输仍需要进行多次数据拷贝,并且PCIe链路所带来的带宽和系统拓扑限制仍然是一个问题. 更进一步,DPU作为具有独立组网功能的领域专用加速器的代表,越来越受到NVIDIA,Intel等公司[15-16]的重视,为了实现RDMA网络数据包处理、NVMe存储访问加速、AES/RSA加解密加速等复杂功能,这些DPU通常不仅包含了FPGA以及DRAM等外部设备,还包含了ARM/X86 CPU等通用处理器,为上述复杂功能提供支撑,其本质上是以另一种形式构建了一个完整的通用系统,不可避免地引入了高昂的额外成本. Lant等人[17]针对FPGA类型的异构加速器提出了与本文类似的观点,即FPGA加速器应该是分布式异构系统中的完全对等的个体,他们开发了自定义互连解决方案,该方案将CPU、FPGA异构加速器和网络协议栈集成到单个FPGA芯片中,使CPU和FPGA计算引擎均可以完全独立控制网络设备. 尽管该方案没有解决CPU与其他计算引擎位于不同芯片的情况下各计算引擎如何独立控制网络设备的问题,但该方案中所展示出来的改进仍具有一定的参考意义.
综上所述,对于不具有独立通信组件的GPU的分布式节点间通信,由于需要通过PCIe与CPU共享同一个网络设备,使得GPU不可能完全旁路CPU实现独立自主通信,带来了额外的延迟,也限制了GPU设备的网络拓扑灵活度. 对于通过FPGA/ASIC实现的类DPU设备,一方面,其追求极致的节点间通信性能,极大地牺牲了系统的可扩展性;另一方面,通过在DPU中集成额外的CPU追求完全的通用性和功能性带来了额外的成本.
RDMA在高速网络通信领域发挥着关键作用,并逐渐成为分布式计算的底层标准硬件设施之一[18-19]. NVIDIA首先在GPUDirect系列技术中引入了RDMA技术,即GPUDirect RDMA,用于减少设备之间移动数据时的拷贝数量. 一个典型的场景是,网卡可以通过GPUDirect RDMA直接访问GPU的内存,而不需要将数据复制到主机内存中,尽管需要CPU通过PCIe向网卡发送命令进行控制或同步. 目前已有多种RDMA实现,包括InfiniBand,iWARP,RoCE等,如表1所示. 由于RoCE协议的兼容性较好,RoCE RDMA实现成为主流解决方案之一,因此本文的RDMA通信也采用了类似的实现方案.
OpenCL是一个受大部分加速设备类型支持的开源编程框架,已成为异构编程的常见标准之一. OpenCL规范中的API通常分为主机API(通常在CPU上执行)和设备API(通常在GPU或其他加速器上执行). OpenCL使程序员可以完全控制加速器的存储层次和工作负载粒度. 为了提高多种类型支持OpenCL的设备的互操作性,Jääskeläinen等人[20]提出了可移植的OpenCL——PoCL,即OpenCL规范的一个开源实现. PoCL使用Clang作为OpenCL的编译器前端、LLVM作为编译器后端. 在本文分布式异构计算系统中,我们采用OpenCL作为编程语言,实现了从其他平台的无缝迁移. 此外我们为本系统开发并集成了一套类OpenCL的API,实现主机启动任意GPU节点之间的类RDMA通信的功能.
Vortex是一个基于RISC-V的开源GPGPU,它通过扩展RISC-V的指令集实现了GPU的计算功能. 并且支持OpenCL和OpenGL规范[21-22]. 在本文系统中,xPU可以是任意类型的加速器. 由于GPU的普遍适用性以及RISC-V的开源特性,本文采用了Vortex GPU作为xPU的一个实例,后续表述将使用Vortex GPU作为xPU的代表性案例.
基于Direct xPU的系统组织架构如图3(b)所示. 在主机节点上,CPU和网卡通过PCIe连接. 在GPU节点上,位于同一芯片中的GPU和网卡通过片上总线连接. 与图3(a)所示的传统组织架构相比,我们在新体系结构中实现了一些明显的优势:1)系统中的任何节点都有一个专属的、服务于该节点内的单个计算引擎的网卡,使得每个计算引擎成为算力网络中完全独立自主的个体. 2)GPU节点还具有降低通信延迟、减少功耗的天然优势,即通过计算设备GPU与通信设备网卡紧密结合,消除了PCIe链路绑定,并为计算引擎和网络设备之间高效协作提供了可能. 3)通过为各网卡实现RDMA功能,我们可以自然地获得低延迟、高带宽的数据通路. GPU节点间的通信步骤为:
1)CPU准备通信任务,并通过主机端的网卡向GPU发送通信任务.
2)GPU根据通信任务的信息,控制GPU节点的网卡直接从GPU内存中读取数据.
3)GPU节点的网卡向远端GPU节点发送数据.
由于面向通用性服务的主机节点的硬件架构目前已经趋于成熟,因此关于主机节点,本文的工作主要集中在软件系统方面,而硬件架构主要描述了GPU节点的设计和改进. Direct xPU的硬件架构如图4所示,其使用了FPGA作为原型系统的开发平台.
节点间通信时,实现了可靠的轻量级传输协议的iRDMA模块负责解析来自于MAC的数据包,根据数据包的功能,将处理后的数据包流转至内部的DHCP客户端模块(主要负责节点在系统中的初始化)、QP模块(主要负责节点与其他节点间的数据交换)或者CFG模块(主要负责查询和控制GPU的状态). DMA控制器根据QP模块的信号,与DDR内存进行数据交换. CFG控制器根据iRDMA中CFG模块的信号,实现了对与GPU相连的CFG中的各类寄存器的访存,以获取和控制GPU的状态,此外,GPU通过执行特定的程序指令,也可以访存与之相连的CFG中的各类寄存器,实现与DMA控制器的交互.
为了加强Direct xPU与当前普通网络的兼容性,我们需要为Direct xPU实现与RoCEv2 RDMA类似的功能. 此外,为了建立一个可靠的链接,通常的做法是使用TCP协议,然而,与UDP协议相比,高度复杂的TCP协议将会消耗大量的FPGA硬件资源,并在数据路径中引入更多的延迟. 为此,在网络协议栈的传输层,我们定义了一种基于UDP的可靠的轻量级传输协议(Reliable Lightweight Transport Protocol,RLTP),如图5所示. RLTP提供超时/乱序重传、流量控制和拥塞管理等基础功能. 由于RLTP依赖于UDP,不需要像TCP那样为每条链路的接收和发送创建专用的缓冲区,大大降低了对FPGA资源的需求和设计的复杂性. 基于RLTP的网络数据包能实现多种功能,包括节点间的正常数据交换,以及传递控制或管理网络上的各类设备的消息. 利用RLTP协议中的“帧类型”字段区分数据包的功能,使不同功能的数据包被交付给配置(Config,CFG)模块或QP(Queue Pair)模块进一步处理. “目的ID”或“源ID”字段对应于网络上每个设备的唯一ID,以便快速根据节点表查找该ID对应的节点在网络中的IP地址. 通过RLTP over UDP方法,可以在不涉及CPU的情况下,将设备之间直接通信所需的基本网络协议栈完全卸载到GPU端,该方法实际上是在有限的硬件资源条件、链路的可靠性和协议的轻量化之间的一种权衡.
系统在启动或运行过程中需要提供一些不可缺少的状态信息,并提供更改这些状态信息的方式. 为了简化这一过程,我们将这些零散信息和控制方式组织成一个统一的CFG模块. CFG由一些控制寄存器和状态寄存器组成,可以通过GPU或iRDMA模块访问. CFG模块提供了一种获取整个系统关键信息以及将GPU的通信操作传输到DMA控制器的途径.
原生的Vortex GPU通过PCIe与主机进行通信,软件方面则依赖于Intel OPAE接口[23],这不利于GPU在通信网络上作为完全独立的对等体的实现. 因此,我们只使用其计算核心,该计算核心通过DDR控制器访问GPU内存. 此外,通过在CFG模块中添加一系列可通过网络数据包访问的控制状态寄存器,实现网络对GPU的使用和控制. 使用Vortex GPU的主要目的不是获得一个可与商用GPU相媲美的高性能GPU计算引擎,而是实现和验证所提出的理论框架的可行性,因此GPU性能不是本文的主要关注点.
iRDMA负责处理来自网络的、经过RTLT协议封装后的数据包. 首先,DHCP客户端与上层的驱动程序配合工作,在网络环境中注册该GPU节点,使其在网络中可用,并自动感知网络拓扑;其次,主机节点可以通过在RLTP头中发送具有正确字段的数据包来读写GPU节点内存的数据或者控制状态寄存器;iRDMA将预处理后的数据包分配给QP模块或CFG模块,分别负责完成对GPU内存/控制状态寄存器的读写操作. 再次,主机节点的CPU下发GPU节点之间的通信任务时,将通信操作所需的基本信息写入到CFG模块中. 最后,GPU节点根据该信息执行相应的RDMA操作,即从GPU内存中取出要发送的数据,iRDMA模块将封装并发送数据包到远程GPU节点. 由于目前没有充分利用GPU和网卡之间的紧耦合性,GPU之间的通信表现出的一系列行为并不完美. RISC-V的指令集具有高度的模块化和可定制性,未来考虑通过自定义指令直接向CFG模块提供RDMA操作所需的信息.
建立一个无损、可靠的网络链接的关键之一是实现高效的丢包重传. 在IB RDMA网络中,该重传机制基于Go-back-N实现,由于IB RDMA网络的丢包率接近于零,采用这种策略对网络性能的影响可以忽略不计. 标准的、基于UDP的RoCE RDMA网络继承了这种策略. 然而,与TCP协议相比,UDP协议缺乏滑动窗口和确认应答等机制来实现可靠传输[24]. 一旦发生丢包,需要依靠上层应用的检测来实现重传,大大降低RDMA的传输效率、增加通信时延. 类似地,如果我们不做改变,基于UDP的iRDMA也会受到该缺点的影响.
具体来说,Go-back-N策略意味着,如果一个数据包未能正确到达接收方,接收方将丢弃所有后续的数据包而不通知发送方. 而发送方只有检测到超时后,才会重新发送该丢失数据包之后的所有数据包.
在RLTP报头的“子包编号”和“标志位”等字段,我们基于IRN实现了选择性重传策略[18],该策略使用位图来跟踪数据包. 当发生数据包丢失时,接收端根据配置持续缓存一定范围内的后续分组,并通过发送带有附加NACK字段的响应主动通知发送方启动选择性重传. 图6显示了在有序传输情况下,Go-back-N与选择性重传的差异. 在Go-back-N策略中,发送方向接收方依次发送了pkt0~pkt4数据包,由于某种原因,pkt1数据包未能到达接收方,接收方会根据报头中的信息丢弃已收到但顺序不正确的数据包,同时反复发送pkt0的确认包ACK0;发送方无法接收到pkt1对应的应答包ACK1,超时之后,发送方会重传pkt1及其后续所有的数据包. 而在选择性重传策略下,当pkt1数据包未能到达接收方时,接收方会缓存后续顺序不正确的数据包,发送对应的ACK数据包,并通过发送NACK1主动通知发送方,以重传丢失的pkt1;当接收方收到发送方重传的pkt1数据包时,则从缓存中取出所有数据包pkt2~pkt4,进行后续处理. 实际场景中也存在乱序传输,但基本原理是相同的.
我们总结了3种RDMA实现的一些特性,如表2所示的丢包重传. 根据各自协议的标准,可以看出iRDMA在系统的可靠性、重传效率和实现难度之间取得了合理的平衡.
图7为Direct xPU系统的软件架构示意图. 在基于Direct xPU架构的分布式异构系统中,由于每个GPU节点可以作为网络上完全独立的通信节点,而不是僵化地由某个特定的主机节点所拥有,有利于大规模敏捷部署和资源池化. 为此,我们设计并开发了一个有效的软件基础设施,使GPU节点具有网络自动拓扑发现和管理等必要功能. 各层级的软件模块的设计遵循低耦合、高内聚的原则,逐层封装强硬件相关的实现细节,极大地提高了系统的可移植性和灵活性.
根据模块的功能和所服务的对象,软件基础架构主要由4个部分组成:后台服务软件库、GPU虚拟驱动、主机应用程序库和GPU应用程序库.
正如3.3节所述,当GPU节点启动并初始化时,它通过DHCP客户端完成对自身的注册,并从后台服务程序中的DHCP服务器获得自己的IP地址. 此外,后台服务程序也是具有唯一性的GPU节点管理器,通过发送或解析RLTP数据包,在已注册的GPU节点之间执行一些管理和同步任务,如为每个GPU分配一个唯一的ID,周期性或主动地将所有GPU节点的信息同步到网络中所有节点的节点表中,包括IP地址、mac地址和设备ID等. 为了增强GPU节点资源池的灵活性,还支持手动指定特定主机节点可以发现和使用的特定GPU节点.
GPU虚拟驱动是运行在内核态的底层模块,主要提供2个基本功能. 一是将GPU作为网络设备在Linux系统中进行枚举,以实现主机节点对GPU节点的远程访问;当某个主机节点通过GPU虚拟驱动从后台服务程序中获取某些GPU节点的必要信息后,将利用RLTP协议枚举并初始化对应的GPU节点. 二是向上层软件库——GPU MMD(memory mapped device)提供访问底层硬件的软件接口,以抽象主机节点与GPU节点之间的RDMA直接通信操作,生成或传递对应的RTLT数据包交付给网络协议栈.
Vortex GPU团队引入并修改了PoCL,使Vortex GPU具有良好的可移植性,这是本文选择Vortex作为xPU原型组件的重要原因. 对于主机节点的主机端程序来说,PoCL主机端库负责将OpenCL主机API转换为主机节点对GPU节点的各种操作,即GPU主机库. GPU主机库的API函数体使用了Intel OPAE接口,在本系统中使用自定义的MMD库替代基于PCIe的OPAE接口,以实现对GPU的远程调用,并向上保持接口的一致性. 此外,为了向终端用户提供控制GPU节点间通信的方法,在GPU主机库中添加了类OpenCL的APIs,该APIs通过直接调用GPU MMD库实现主机下发GPU节点间通信任务的操作.
图8进一步说明了GPU MMD库在软件栈中的关键作用. GPU MMD库向GPU主机库提供了一组更细粒度的操作——GPU的API. GPU MMD库工作在GPU虚拟驱动程序之上,公开了与GPU虚拟驱动程序直接交互的接口. GPU虚拟驱动程序实现了将远端的GPU节点枚举为本地的GPU设备的功能. 这种将远端GPU节点“虚拟”地暴露为同一节点中本地GPU设备的方法,为其他基于传统的PCIe链路的xPU的移植或开发提供了极大的便利性和可行性. 总而言之,选择这种实现方式一方面是移植Vortex GPU到原型系统中的需求,而另一方面则是松耦合的软件分层设计为Direct xPU带来了灵活和广阔的设计空间.
GPU节点采用OpenCL进行核函数的编程,其编程方式与其他支持OpenCL编程的计算设备没有区别,这种一致性消除了移植和开发应用程序代码的障碍. 不同的是,Vortex作为在RISC-V的ISA(instruction set architecture)上实现的GPU,需要对相应的编译器工具链和应用程序库提供针对性的支持. GPU内部的软硬件设计工作并非本文所关注的点,因此我们没有修改GPU的微架构,遵循了Vortex团队提供的库和工具链. 生成GPU二进制文件的整个过程如图9所示. 终端用户编写的OpenCL核函数通过PoCL编译器编译生成LLVM IR. 为了实现任务分配和线程调度的自动化,GPU核函数运行时库也将被编译,并与核函数生成的LLVM IR链接,生成最终的GPU二进制文件.
Direct xPU系统中的2种计算引擎GPU和CPU都具有专属的网络通信设备,因此传统的通过PCIe连接GPU和CPU的架构不再合理. 为了在空间上构建一个紧凑的计算资源池,并使其物理形态适应大规模分布式计算中心的需求,我们针对Direct xPU开发了一种xPU Box,如图10所示. xPU Box配备有冷却模块和电源模块,并提供了一定数量的仅供电的PCIe插槽,以实现与现有设备物理接口的最大兼容性. 箱内的GPU节点通过光纤连接到光纤交换机. 除了图10中展示的16个PCIe插槽的xPU Box,我们还设计了4个PCIe插槽的版本,以满足不同规模的部署需求.
在本次评估中,我们使用了1台PC作为主机节点,使用2个基于FPGA的Direct xPU原型机作为GPU节点. 表3给出测试环境的主要软硬件设施的详细信息. 主机节点是一台配备了光纤网卡的PC,通过10 Gbps光纤连接至交换机,2个GPU节点也通过10 Gbps光纤连接至交换机. 交换机来自锐捷,其支持最多48个设备以10 Gbps互联通信. 由于FPGA开发板所配备的MAC的性能限制,以及开发更高性能的iRDMA的硬件资源限制,目前所实现的系统网络性能距离主流的高性能RDMA网卡存在一定的差距. 然而,由于xPU和iRDMA在硬件电路设计上是解耦的,这使得我们可以很容易地在后续迭代升级至更高带宽的版本,这是我们的后续计划. Vortex支持配置1~32个计算核心,每个计算核心包含了4个物理线程. 由于FPGA芯片的资源限制,本原型中使用了32 Core的配置,在200 MHz的FPGA时钟下,通过执行简单的矩阵乘计算,其实际性能约为3.68 GFlops.
在进行稳定网络环境下的带宽和延迟测试之前,有必要对基于选择性重传的iRDMA的设备在不稳定环境中的网络带宽性能进行简单测试. 如3.3节中所述,iRDMA通过更复杂的状态管理,实现了网络性能开销相对更低的选择性重传机制. 商用GPU节点间通信工作[2-9]的通信性能取决于节点内所配备的与CPU共享的网络设备的性能,基于FPGA的领域专用加速器方的节点间通信工作[10-14]则未讨论不稳定网络下的重传机制所带来的负面影响. 因此,我们使用相同的2个x86平台主机,通过模拟不同的网络丢包率,在数据量大小为1 MB的条件下,测试了基于iRDMA网卡的主机间通信和基于RoCEv2的标准网卡——Mellanox ConnectX-4 Lx 10 GB的主机间通信,在丢包率为0%,1%,10%这3种情况下网络读写带宽性能的变化情况. 结果如图11所示,在丢包率达到1%时,基于Go-back-N的Mellanox网卡已处于几乎不可用的状态,而基于选择性重传的iRDMA网络具有更好的可靠性和稳定性.
图12展示了发送方视角下GPU节点之间完成1次通信测试过程所需要的时间和步骤. 在1次回环测试中,发送方如果执行的是写操作,则发送方发送数据并从接收方接收确认信号ACK;发送方如果执行的是读操作,则发送读命令并接收接收方返回的数据. 延迟主要由2阶段组成:一是主机节点向GPU节点发送命令,以启动GPU之间的直接通信,对应于图12中的t2–t0;二是发送方GPU节点向远端的接收方GPU节点发送数据/读命令进行写/读操作,并接收远端的GPU节点返回的ACK/数据,对应于$t_3-t'_2 $部分. $ t'_2-t_2 $段表示GPU节点根据命令信息触发网卡的读写操作的时间,与其他跨芯片传递信息的延迟相比,由于芯片内模块之间的紧耦合性,这一阶段通常只需要数十纳秒,可以忽略.
图13展示了测量$ t_3-t'_2 $延迟部分的过程. 由于所有节点之间都是使用光纤通过交换机互联,因此交换机的延迟开销对这一部分的延迟也有一定的影响. 测试结果如图14所示. 基于200 MHz的FPGA时钟频率,我们通过统计该操作所消耗的时钟数来测量数据包的读/写延迟. 当数据包大小为4 B时,写操作回环时延为2.65 µs,读操作回环时延为2.72 µs,两者可以认为是相等的.
据我们所知,目前广泛部署的商用GPU之间的RDMA通信是通过与CPU共享的网卡实现的,其延迟包括CPU引入的一系列软件栈开销,并且难以区分或精确测量GPU之间通信的延迟组成,因此,我们给出了一些基于FPGA实现的,且具备专属通信模块的分布式异构加速器的延迟比较,如表4所示. 值得注意的是,CoE和CIRCUS的延迟将会随着所执行的计算任务的复杂度的增加而增加,因为更复杂的任务将占用更多的FPGA资源,导致FPGA布局布线困难以及时钟频率降低. 然而,对于基于GPU的Direct xPU系统,执行不同的任务仅仅是执行不同的指令,不会增加硬件复杂度.
由于当前实现的iRDMA仍依赖于传统的网络协议栈,导致主机节点依赖于CPU进行协议解析和需要操作系统频繁地上下文切换,因此主机节点与GPU节点之间执行通信任务时的延迟$t_2-t_0 $相比于GPU节点之间的通信延迟大幅度增加. 我们使用GPU MMD库的API对主机节点和GPU节点之间的延迟进行了简单的测试,即测试主机节点写入GPU节点的CFG模块中不同数量的寄存器的延迟(每个寄存器为4 B). 图15显示了这部分开销会增加大约1 ms的延迟,并且随着所写入的寄存器数量的增加,延迟会有轻微的增加,最终趋于平稳.
图16给出了数据量大小为1 Mb和1 Gb时,GPU节点之间的带宽测试结果,基于iRDMA的Direct xPU实现了接近10 Gbps的理论带宽. 此外,这种基于网络的通信可以实现几乎相等的读写速度,避免了PCIe链路中读写速度不对等的问题.
我们基于Direct xPU,搭建了一个原型演示系统,如图17所示. 图中包含1个4PCIe插槽的FPGA Box,其配备了2块Direct xPU. Direct xPU通过光纤交换机与PC通信. 我们使用了OpenCL编程语言将SqueezeNet模型进行分割后,在该分布式计算平台上运行,即通过摄像头接收图像数据实时识别图像内容.
评估结果表明,基于Direct xPU的原型系统的网络性能基本接近硬件设备的理论上限. 特别是,我们使用新架构实现了任意GPU节点之间的超低延迟、完全独立的通信. 由于原型系统所使用FPGA板卡、网络模块等硬件条件限制,在延迟和带宽性能方面与目前的先进水平还有一定距离.
可以预见的是,通过对原型系统中限制其网络性能的硬件设备进行升级来解决其桶效应问题,各节点之间的通信性能将得到进一步提升. 此外,得益于RISC-V指令集的模块化设计和开源特性,增加自定义指令使GPU节点在指令层面上控制同一芯片内的网络设备成为可能,这是我们未来的工作计划.
作者贡献声明:李仁刚设计了论文整体逻辑架构;王彦伟提出了Direct xPU的软硬件架构方案;郝锐和肖麟阁设计了论文实验方案、完成实验并撰写论文;杨乐完善论文内容;杨广文和阚宏伟提出论文撰写指导意见.

参考文献 引证文献
排序方式:
1
Fröning H, Nüssle M, Litz H, et al. A case for FPGA based accelerated communication[C]//Proc of the 9th Int Conf on Networks. Piscataway, NJ: IEEE, 2010: 28−33
2
Shainer G, Ayoub A, Lui P, et al. The development of Mellanox/NVIDIA GPUDirect over InfiniBand—a new model for GPU to GPU communications[J]. Computer Science Research and Development, 2011, 26: 267−273
3
Ammendola R, Bernaschi M, Biagioni A, et al. GPU peer-to-peer techniques applied to a cluster interconnect[C]//Proc of 2013 IEEE Int Symp on Parallel & Distributed Processing, Workshops and PhD Forum. Piscataway, NJ: IEEE, 2013: 806−815
4
Agostini E, Rossetti D, Potluri S. Offloading communication control logic in GPU accelerated applications[C]//Proc of 17th IEEE/ACM Int Symp on Cluster, Cloud and Grid Computing (CCGRID). Piscataway, NJ: IEEE, 2017: 248−257
5
Agostini E, Rossetti D, Potluri S. GPUDirect Async: Exploring GPU synchronous communication techniques for InfiniBand clusters[J]. Journal of Parallel and Distributed Computing, 2018, 114: 28−45
6
Oden L, Fröning H. GGAS: Global GPU address spaces for efficient communication in heterogeneous clusters[C]//Proc of 2013 IEEE Int Conf on Cluster Computing (CLUSTER). Piscataway, NJ: IEEE, 2013: 1−8
7
Oden L, Fröning H. InfiniBand Verbs on GPU: A case study of controlling an InfiniBand network device from the GPU[J]. The International Journal of High Performance Computing Applications, 2017, 31(4): 274−284
8
Daoud F, Watad A, Silberstein M. GPUrdma: GPU-side library for high performance networking from GPU kernels[C]//Procof the 6th Int Workshop on Runtime and Operating Systems for Supercomputers. New York: ACM, 2016: 1−8
9
Silberstein M, Kim S, Huh S, et al. GPUnet: Networking abstractions for GPU programs[J]. ACM Transactions on Computer Systems, 2016, 34(3): 1−31
10
Balle S M, Tetreault M, Dicecco R. Inter-kernel links for direct inter-FPGA communication[DB/OL]. [2022-10-19]. https://cdrdv2-public.intel.com/650535/wp-01305-inter-kernel-links-for-direct-inter-fpga-communication.pdf.
11
Fujita N, Kobayashi R, Yamaguchi Y, et al. Parallel processing on FPGA combining computation and communication in OpenCL programming[C]/Proc of 2019 IEEE Int Parallel and Distributed Processing Symp Workshops (IPDPSW). Piscataway, NJ: IEEE, 2019: 479−488
12
Fujita N, Kobayashi R, Yamaguchi Y, et al. Performance evaluation of pipelined communication combined with computation in OpenCL programming on FPGA[C]//Proc of 2020 IEEE Int Parallel and Distributed Processing Symp Workshops (IPDPSW). Piscataway, NJ: IEEE, 2020: 450−459
13
Kobayashi R, Fujita N, Yamaguchi Y, et al. GPU-FPGA heterogeneous computing with OpenCL-enabled direct memory access[C]//Proc of 2019 IEEE Int Parallel and Distributed Processing Symp Workshops (IPDPSW). Piscataway, NJ: IEEE, 2019: 489−498
14
Kobayashi R, Fujita N, Yamaguchi Y, et al. OpenCL-enabled GPU-FPGA accelerated computing with inter-FPGA communication[C]//Proc of the Int Conf on High Performance Computing in Asia-Pacific Region Workshops. New York: ACM 2020: 17−20
15
Burstein I. Nvidia data center processing unit (DPU) architecture[C]//Proc of 2021 IEEE Hot Chips 33 Symp (HCS). Piscataway, NJ: IEEE, 2021: 1−20
16
Sundar N, Burres B, Li Y, et al. An in-depth look at the Intel IPU E2000[C]//Proc of 2023 IEEE Int Solid-State Circuits Conf (ISSCC). Piscataway, NJ: IEEE, 2023: 162−164
17
Lant J, Navaridas J, Luján M, et al. Toward FPGA-based HPC: Advancing interconnect technologies[J]. IEEE Micro, 2019, 40(1): 25−34
18
Mittal R, Shpiner A, Panda A, et al. Revisiting network support for RDMA[C]//Proc of the 2018 Conf of the ACM Special Interest Group on Data Communication. New York: ACM, 2018: 313−326
19
Wadekar M. Handbook of Fiber Optic Data Communication[M]//Cambridge, MA: Academic Press, 2013: 267−287
20
Jääskeläinen P, de La Lama C S, Schnetter E, et al. PoCL: A performance-portable OpenCL implementation[J]. International Journal of Parallel Programming, 2015, 43: 752−785
21
Tine B, Yalamarthy K P, Elsabbagh F, et al. Vortex: Extending the RISC-V ISA for GPGPU and 3D-graphics[C]//Proc of the 54th Annual IEEE/ACM Int Symp on Microarchitecture (MICRO-54). New York: ACM, 2021: 754−766
22
Elsabbagh F, Tine B, Roshan P, et al. Vortex: OpenCL Compatible RISC-V GPGPU[J]. arXiv preprint, arXiv: 2002.12151, 2020
23
Intel OPAE. Open programmable acceleration engine (OPAE) C API programming guide [DB/OL]. [2021-11-08] https://cdrdv2.intel.com/v1/dl/getContent/686262?explicitVersion=true&wapkw=opae.
24
曾高雄,胡水海,张骏雪,等. 数据中心网络传输协议综述[J]. 计算机研究与发展, 2020, 57(1): 74−84
Zeng Gaoxiong, Hu Shuihai, Zhang Junxue, et al. Transport protocols for data center networks: A survey[J]. Journal of Computer Research and Development, 2020, 57(1): 74−84 (in Chinese)
2024年第61卷第6期
57
26
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202440055
  • 接收时间:2024-01-29
  • 首发时间:2026-11-17
  • 出版时间:2024-06-01
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2024-01-29
  • 修回日期:2024-03-27
基金
作者信息
    1清华大学计算机科学与技术系 北京 100084
    2浪潮(北京)电子信息产业有限公司 北京 100085
    3广东浪潮智慧计算技术有限公司 广州 510623

通讯作者:

参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202440055
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏