Article(id=1325495704952635716, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1325495515563041259, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202550153, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=research-article, receivedDate=1740758400000, receivedDateStr=2025-03-01, revisedDate=1758643200000, revisedDateStr=2025-09-24, acceptedDate=1760112000000, acceptedDateStr=2025-10-11, onlineDate=1793952402580, onlineDateStr=2026-11-06, pubDate=1772294400000, pubDateStr=2026-03-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1793952402580, onlineIssueDateStr=2026-11-06, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1793952402580, creator=13701087609, updateTime=1793952402580, updator=13701087609, issue=Issue{id=1325495515563041259, tenantId=1146029695717560320, journalId=1324759914146529478, year='2026', volume='63', issue='3', pageStart='539', pageEnd='823', issueExtLink='null', onlineDate='null', pubDate='1772294400000', pubDateStr='2026-03-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1793952357425, creator='13701087609', updateTime=1793952549954, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext={EN=IssueExt(id=1325496323193057998, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1325495515563041259, language=EN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=), CN=IssueExt(id=1325496323193057999, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1325495515563041259, language=CN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=)}, issueFiles=null, downloadFileDto=null}, startPage=782, endPage=797, ext={EN=ArticleExt(id=1325495705204293957, articleId=1325495704952635716, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=SW-Inf: Efficient Transformer Inference Method for Sunway AI Accelerator Card, columnId=1325493593498059289, journalTitle=Journal of Computer Research and Development, columnName=High Performance Computing, runingTitle=null, highlight=null, articleAbstract=

Transformer has fundamentally revolutionized various fields of natural language processing through its self-attention mechanism that effectively models long-range contextual relationships. It overcomes traditional neural networks’ limitations in handling long sequences, and has emerged as the mainstream architecture for modern large language models. Currently, Transformer inference systems generally face a shortage of computing power and are commonly confronted with challenges like substantial scheduling latency, concurrency execution difficulties, and severe memory access limitations during deployment. The Sunway AI accelerator card, featuring a domestically-developed next-generation intelligent processor, has successfully empowered diverse edge inference deployments. However, conventional inference frameworks struggle to effectively explore the architectural potential in three critical dimensions: robust system control capabilities, multi-level parallel computing, and rich memory hierarchy. We propose SW-Inf: a high-efficiency Transformer inference method based on the Sunway AI accelerator card. First, an on-chip management processing element (MPE)-orchestrated computation graph scheduling mechanism is proposed to minimize kernel launch latency and inter-process data transfer overhead through graph aggregation and MPE coordination. Second, an LDM-prioritized heuristic tiling strategy is introduced to enable load-balanced parallel computation across processing elements. Finally, a hardware-aware tightly-coupled attention computation implementation is developed to eliminate redundant intermediate data movement between registers and main memory. We implement an end-to-end inference system on the Sunway AI accelerator card. Experimental results demonstrate that compared with the vLLM inference system on NVIDIA V100S which has higher memory bandwidth than the Sunway AI accelerator card, SW-Inf achieves 15% higher throughput and 20% better memory utilization rate.

, authors=Wenyuan Shen1, Lingtao Tang1, Haotian He1, Tiancheng Fei1, Yulong Zhao1, Lufei Zhang1, Xiaojun Qin1, Xin Liu2, *, Zuoning Chen2, authorsList=Wenyuan Shen, Lingtao Tang, Haotian He, Tiancheng Fei, Yulong Zhao, Lufei Zhang, Xiaojun Qin, Xin Liu, Zuoning Chen, authorCompany=null, correspAuthors=Xin Liu, authorNote=

born in 2000. Undergraduate. His main research interests include high performance computing and operating systems

born in 1994. PhD. His main research interests include machine learning, privacy protection, and intelligent operating systems

born in 1997. Master. His main research interests include high performance computing and operating systems

born in 1999. Master. His main research interests include high performance computing and operating systems

born in 1984. PhD candidate. His main research interests include artificial intelligence and operating system

born in 1986. PhD, senior engineer. His main research interests include high-performance computing, operating system, and machine learning

born in 1975. PhD, senior engineer. His main research interests include system security, software vulnerability analysis, and artificial intelligence

born in 1979. PhD, professor. Her main research interest includes parallel algorithms and applications

born in 1957. Academician of Chinese Academy of Engineering. Her main research interests include software theory, operating systems, and information security

, correspAuthorsNote=null, copyrightStatement=Copyright © 2026 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1325495710992433521, articleId=1325495704952635716, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=SW-Inf:基于申威智能加速卡的Transformer高效推理方法, columnId=1325493595259666971, journalTitle=计算机研究与发展, columnName=高性能计算, runingTitle=null, highlight=null, articleAbstract=

Transformer通过自注意力机制高效捕捉全局依赖,突破了传统神经网络处理长序列信息时的限制,推动了自然语言处理多个领域的革命性进展,成为现代大语言模型的核心架构。目前,国内Transformer推理应用整体上处于算力紧缺状态,且在部署时往往面临调度开销大、计算并发难、访存瓶颈明显等问题。申威智能加速卡搭载了国产自主研发的新一代智能处理器,已有效支撑各类端侧Transformer推理应用,然而常规推理框架难以发挥申威异构众核架构的管理功能完备、多级并行计算和存储层次丰富等特点。对此,提出一种基于申威智能加速卡的Transformer高效推理方法SW-Inf。首先,提出了片上主核驱动的整图下沉调度机制,通过计算图打包和片上主核调度,减少主机与设备间的核函数启动开销和设备内各流程数据传输开销;其次,设计了LDM优先的启发式分块并行策略,实现计算部件负载均衡,提高了硬件资源利用率;最后,实现了体系结构紧耦合的注意力计算方法,避免推理的中间结果频繁读写主存和寄存器,缓解端侧推理应用的访存瓶颈问题。为验证所提方法的效果,基于申威智能加速卡实现了一套端到端推理系统,测试结果表明,相较于访存带宽更高的英伟达V100S上的vLLM推理系统,SW-Inf的吞吐率和主存利用率分别提高了15个百分点和20个百分点。

, authors=沈文渊1, 汤凌韬1, 何昊天1, 费天成1, 赵玉龙1, 张鲁飞1, 秦晓军1, 刘鑫2, *, 陈左宁2, authorsList=沈文渊, 汤凌韬, 何昊天, 费天成, 赵玉龙, 张鲁飞, 秦晓军, 刘鑫, 陈左宁, authorCompany=null, correspAuthors=刘鑫, authorNote=

2000年生。本科生。主要研究方向为高性能计算、操作系统

1994年生。博士。主要研究方向为机器学习、隐私保护、智能操作系统

1997年生。硕士。主要研究方向为高性能计算、操作系统

1999年生。硕士。主要研究方向为高性能计算、操作系统

1984年生。博士研究生。主要研究方向为人工智能、操作系统

1986年生。博士,高级工程师。主要研究方向为高性能计算、操作系统、机器学习

1975年生。博士。高级工程师。主要研究方向为系统安全、软件脆弱性分析、人工智能

1979年生。博士,研究员。主要研究方向为并行算法和应用

1957年生。中国工程院院士。主要研究方向为软件理论、操作系统、信息安全

, correspAuthorsNote=
刘鑫()
, copyrightStatement=版权所有 © 2026 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=hi5m8fxvb3moWZmyJqPhSw==, magXml=jjylmpJnviqTSFL93A+Lzw==, pdfUrl=null, pdf=kASX+Okl6AhdaJAvXIiPDg==, pdfFileSize=1919353, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=ft/9NCVWF7igZbg/1TP6vQ==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=pAhO96A6uNq+vsBq+jOglg==, mapNumber=null, fund=null)}, authors=[Author(id=1325495713009893753, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=wenyuan17@yeah.net, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325495713106362747, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495713009893753, language=EN, stringName=Wenyuan Shen, firstName=Wenyuan, middleName=null, lastName=Shen, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 2000. Undergraduate. His main research interests include high performance computing and operating systems

"}, bioImg=null, bioContent=

born in 2000. Undergraduate. His main research interests include high performance computing and operating systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325495713219608956, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495713009893753, language=CN, stringName=沈文渊, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"9ITsID7Qfz8/HUmS/wj07Q==","content":"

2000年生。本科生。主要研究方向为高性能计算、操作系统

"}, bioImg=9ITsID7Qfz8/HUmS/wj07Q==, bioContent=

2000年生。本科生。主要研究方向为高性能计算、操作系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325495711239897458, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=1, ext=[AuthorCompanyExt(id=1325495711252480371, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1325495711260868980, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1325495713311883646, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325495713395769728, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495713311883646, language=EN, stringName=Lingtao Tang, firstName=Lingtao, middleName=null, lastName=Tang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1994. PhD. His main research interests include machine learning, privacy protection, and intelligent operating systems

"}, bioImg=null, bioContent=

born in 1994. PhD. His main research interests include machine learning, privacy protection, and intelligent operating systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325495713483850113, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495713311883646, language=CN, stringName=汤凌韬, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"uBXa9p3PP0OzJJ3pAJePfw==","content":"

1994年生。博士。主要研究方向为机器学习、隐私保护、智能操作系统

"}, bioImg=uBXa9p3PP0OzJJ3pAJePfw==, bioContent=

1994年生。博士。主要研究方向为机器学习、隐私保护、智能操作系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325495711239897458, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=1, ext=[AuthorCompanyExt(id=1325495711252480371, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1325495711260868980, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1325495713592902019, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325495713693565317, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495713592902019, language=EN, stringName=Haotian He, firstName=Haotian, middleName=null, lastName=He, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1997. Master. His main research interests include high performance computing and operating systems

"}, bioImg=null, bioContent=

born in 1997. Master. His main research interests include high performance computing and operating systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325495713798422918, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495713592902019, language=CN, stringName=何昊天, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"wgvL42o6ueHbH3QkZrFKcQ==","content":"

1997年生。硕士。主要研究方向为高性能计算、操作系统

"}, bioImg=wgvL42o6ueHbH3QkZrFKcQ==, bioContent=

1997年生。硕士。主要研究方向为高性能计算、操作系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325495711239897458, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=1, ext=[AuthorCompanyExt(id=1325495711252480371, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1325495711260868980, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1325495713882309000, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325495713974583690, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495713882309000, language=EN, stringName=Tiancheng Fei, firstName=Tiancheng, middleName=null, lastName=Fei, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1999. Master. His main research interests include high performance computing and operating systems

"}, bioImg=null, bioContent=

born in 1999. Master. His main research interests include high performance computing and operating systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325495714062664075, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495713882309000, language=CN, stringName=费天成, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"UomR7ExGXaaBuvcTD3QgWg==","content":"

1999年生。硕士。主要研究方向为高性能计算、操作系统

"}, bioImg=UomR7ExGXaaBuvcTD3QgWg==, bioContent=

1999年生。硕士。主要研究方向为高性能计算、操作系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325495711239897458, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=1, ext=[AuthorCompanyExt(id=1325495711252480371, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1325495711260868980, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1325495714167521677, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325495714263990671, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495714167521677, language=EN, stringName=Yulong Zhao, firstName=Yulong, middleName=null, lastName=Zhao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1984. PhD candidate. His main research interests include artificial intelligence and operating system

"}, bioImg=null, bioContent=

born in 1984. PhD candidate. His main research interests include artificial intelligence and operating system

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325495714343682448, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495714167521677, language=CN, stringName=赵玉龙, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"D3exj6Mvc4B6NVGIL38eYg==","content":"

1984年生。博士研究生。主要研究方向为人工智能、操作系统

"}, bioImg=D3exj6Mvc4B6NVGIL38eYg==, bioContent=

1984年生。博士研究生。主要研究方向为人工智能、操作系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325495711239897458, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=1, ext=[AuthorCompanyExt(id=1325495711252480371, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1325495711260868980, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1325495714482094482, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, orderNo=5, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325495714578563476, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495714482094482, language=EN, stringName=Lufei Zhang, firstName=Lufei, middleName=null, lastName=Zhang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1986. PhD, senior engineer. His main research interests include high-performance computing, operating system, and machine learning

"}, bioImg=null, bioContent=

born in 1986. PhD, senior engineer. His main research interests include high-performance computing, operating system, and machine learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325495714675032469, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495714482094482, language=CN, stringName=张鲁飞, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"+NAwwBD40sKFmfLNZR+UCA==","content":"

1986年生。博士,高级工程师。主要研究方向为高性能计算、操作系统、机器学习

"}, bioImg=+NAwwBD40sKFmfLNZR+UCA==, bioContent=

1986年生。博士,高级工程师。主要研究方向为高性能计算、操作系统、机器学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325495711239897458, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=1, ext=[AuthorCompanyExt(id=1325495711252480371, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1325495711260868980, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1325495714763112855, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, orderNo=6, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325495714851193241, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495714763112855, language=EN, stringName=Xiaojun Qin, firstName=Xiaojun, middleName=null, lastName=Qin, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1975. PhD, senior engineer. His main research interests include system security, software vulnerability analysis, and artificial intelligence

"}, bioImg=null, bioContent=

born in 1975. PhD, senior engineer. His main research interests include system security, software vulnerability analysis, and artificial intelligence

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325495714935079322, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495714763112855, language=CN, stringName=秦晓军, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"j229/SQRoT8tlzPI8/2RBA==","content":"

1975年生。博士。高级工程师。主要研究方向为系统安全、软件脆弱性分析、人工智能

"}, bioImg=j229/SQRoT8tlzPI8/2RBA==, bioContent=

1975年生。博士。高级工程师。主要研究方向为系统安全、软件脆弱性分析、人工智能

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325495711239897458, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=1, ext=[AuthorCompanyExt(id=1325495711252480371, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1325495711260868980, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1325495715027354012, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, orderNo=7, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=yyylx@263.net, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1325495715115434398, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495715027354012, language=EN, stringName=Xin Liu, firstName=Xin, middleName=null, lastName=Liu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, *, address=2National Parallel Computer Engineering Technology Research Center, Beijing 100083, bio={"content":"

born in 1979. PhD, professor. Her main research interest includes parallel algorithms and applications

"}, bioImg=null, bioContent=

born in 1979. PhD, professor. Her main research interest includes parallel algorithms and applications

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325495715195126175, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495715027354012, language=CN, stringName=刘鑫, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, *, address=2国家并行计算机工程技术研究中心 北京 100083, bio={"img":"1AGKD+nG+2XoQIR3FrCkgg==","content":"

1979年生。博士,研究员。主要研究方向为并行算法和应用

"}, bioImg=1AGKD+nG+2XoQIR3FrCkgg==, bioContent=

1979年生。博士,研究员。主要研究方向为并行算法和应用

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325495711332172149, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=2, ext=[AuthorCompanyExt(id=1325495711344755062, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711332172149, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2National Parallel Computer Engineering Technology Research Center, Beijing 100083), AuthorCompanyExt(id=1325495711353143671, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711332172149, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2国家并行计算机工程技术研究中心 北京 100083)])]), Author(id=1325495715279012257, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, orderNo=8, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325495715375481251, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495715279012257, language=EN, stringName=Zuoning Chen, firstName=Zuoning, middleName=null, lastName=Chen, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2National Parallel Computer Engineering Technology Research Center, Beijing 100083, bio={"content":"

born in 1957. Academician of Chinese Academy of Engineering. Her main research interests include software theory, operating systems, and information security

"}, bioImg=null, bioContent=

born in 1957. Academician of Chinese Academy of Engineering. Her main research interests include software theory, operating systems, and information security

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325495715467755940, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495715279012257, language=CN, stringName=陈左宁, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2国家并行计算机工程技术研究中心 北京 100083, bio={"img":"z857SmE9ENVNlfqZ5PIEVg==","content":"

1957年生。中国工程院院士。主要研究方向为软件理论、操作系统、信息安全

"}, bioImg=z857SmE9ENVNlfqZ5PIEVg==, bioContent=

1957年生。中国工程院院士。主要研究方向为软件理论、操作系统、信息安全

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325495711332172149, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=2, ext=[AuthorCompanyExt(id=1325495711344755062, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711332172149, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2National Parallel Computer Engineering Technology Research Center, Beijing 100083), AuthorCompanyExt(id=1325495711353143671, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711332172149, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2国家并行计算机工程技术研究中心 北京 100083)])])], keywords=[Keyword(id=1325495715576807845, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, orderNo=1, keyword=Sunway AI accelerator card), Keyword(id=1325495715669082534, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, orderNo=2, keyword=large model inference), Keyword(id=1325495715744580007, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, orderNo=3, keyword=scheduling optimization), Keyword(id=1325495717422301608, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, orderNo=4, keyword=parallel strategy), Keyword(id=1325495717506187689, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, orderNo=5, keyword=operator fusion), Keyword(id=1325495717594268074, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, orderNo=1, keyword=申威智能加速卡), Keyword(id=1325495717678154155, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, orderNo=2, keyword=大模型推理), Keyword(id=1325495717774623148, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, orderNo=3, keyword=调度优化), Keyword(id=1325495717900452269, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, orderNo=4, keyword=并行策略), Keyword(id=1325495717984338350, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, orderNo=5, keyword=算子融合)], refs=[Reference(id=1325495723382407635, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=Yu G, Jeong S, Kim G, et al. Orca: A distributed serving system for Transformer-based generative models[C]//Proc of the 16th Symp on Operating Systems Design and Implementation. Berkeley, CA: USENIX Association, 2022: 521−538, articleTitle=null, refAbstract=null), Reference(id=1325495723483070932, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Frantar E, Alistarh D. SparseGPT: Massive language models can be accurately pruned in one-shot[J]. arXiv preprint, arXiv: 2301.00774, 2023, articleTitle=null, refAbstract=null), Reference(id=1325495723579539925, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Zichang, Wang Jue, Dao T, et al. Deja Vu: Contextual sparsity for efficient LLMs at inference time[C]//Proc of the 40th Int Conf on Machine Learning. Cambridge, MA: PMLR, 2023: 919, articleTitle=null, refAbstract=null), Reference(id=1325495723663426006, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=王笑尘, 张坤, 张鹏. 多视角看大模型安全及实践[J]. 计算机研究与发展, 2024, 61(5): 1104−1112, articleTitle=null, refAbstract=null), Reference(id=1325495723755700695, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Xiaochen, Zhang Kun, Zhang Peng. Multi-perspective view of large model security and practice[J]. Journal of Computer Research and Development, 2024, 61(5): 1104−1112 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1325495723860558296, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Nallapati R, Zhou B, dos Santos C, et al. Abstractive text summarization using sequence-to-sequence RNNs and beyond[C]//Proc of the 20th SIGNLL Conf on Computational Natural Language Learning. Stroudsburg, PA: ACL, 2016: 280−290, articleTitle=null, refAbstract=null), Reference(id=1325495723944444377, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate[J]. arXiv preprint, arXiv: 1409.0473, 2014, articleTitle=null, refAbstract=null), Reference(id=1325495724049301978, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Adiwardana D, Luong M T, So D R, et al. Towards a human-like open-domain chatbot[J]. arXiv preprint, arXiv: 2001.09977, 2020, articleTitle=null, refAbstract=null), Reference(id=1325495724162548188, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 30: 5998−6008, articleTitle=null, refAbstract=null), Reference(id=1325495724271600093, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Kim S, Hooper C, Wattanawong T, et al. Full stack optimization of Transformer inference: A survey[J]. arXiv preprint, arXiv: 2302.14017, 2023, articleTitle=null, refAbstract=null), Reference(id=1325495724347097566, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Samuel W, Andrew W, David P. Roofline: An insightful visual performance model for multicore architectures[J]. Communications of the ACM, 2009, 52(4): 65−76, articleTitle=null, refAbstract=null), Reference(id=1325495724506481119, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Zheng Lianmin, Yin Liangsheng, Xie Zhiqiang, et al. SGLang: Efficient execution of structured language model programs[J]. Advances in Neural Information Processing Systems, 2024, 37: 62557−62583, articleTitle=null, refAbstract=null), Reference(id=1325495724590367200, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Dao T, Fu D, Ermon S, et al. FlashAttention: Fast and memory-efficient exact attention with IO-awareness[J]. Advances in Neural Information Processing Systems, 2022, 35: 16344−16359, articleTitle=null, refAbstract=null), Reference(id=1325495724674253281, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=DeepSeek-AI. DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model[J]. arXiv preprint, arxiv: 2405.04434, 2024, articleTitle=null, refAbstract=null), Reference(id=1325495726314226148, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Kwon W, Li Zhuohan, Zhuang Siyuan, et al. Efficient memory management for large language model serving with PagedAttention[C]//Pro of the 29th Symp on Operating Systems Principles. New York: ACM, 2023: 611–626, articleTitle=null, refAbstract=null), Reference(id=1325495726435860965, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Agrawal A, Kedia N, Panwar A, et al. Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve[C]//Proc of the 18th Symp on Operating Systems Design and Implementation. Berkeley, CA: USENIX Association, 2024: 117−134., articleTitle=null, refAbstract=null), Reference(id=1325495726528135654, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Sheng Ying, Zheng Lianmin, Yuan Binhang, et al. FlexGen: High-throughput generative inference of large language models with a single GPU[C]//Proc of the Int Conf on Machine Learning. Cambridge, MA: PMLR, 2023: 31094−31116, articleTitle=null, refAbstract=null), Reference(id=1325495726632993255, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Song Yixin, Mi Zeyu, Xie Haotong, et al. PowerInfer: Fast large language model serving with a consumer-grade GPU[C]//Proc of the 30th ACM SIGOPS Symp on Operating Systems Principles. New York: ACM, 2024: 590−606, articleTitle=null, refAbstract=null), Reference(id=1325495726725267944, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Michel P, Levy O, Neubig G. Are sixteen heads really better than one?[J]. Advances in Neural Information Processing Systems, 2019, 32: 14014−14024, articleTitle=null, refAbstract=null), Reference(id=1325495726809154025, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Bansal H, Gopalakrishnan K, Dingliwal S, et al. Rethinking the role of scale for in-context learning: An interpretability-based case study at 66 billion scale[J]. arXiv preprint, arXiv: 2212.09095, 2022, articleTitle=null, refAbstract=null), Reference(id=1325495726901428714, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Alizadeh K, Mirzadeh I, Belenko D, et al. LLM in a flash: Efficient large language model inference with limited memory[C]//Proc of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Stroudsburg, PA: ACL, 2024: 12562−12584, articleTitle=null, refAbstract=null), Reference(id=1325495726989509099, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=GGML-org. LLaMA. cpp[EB/OL]. [2024-10-01]. https://github.com/ggml-org/llama.cpp, articleTitle=null, refAbstract=null), Reference(id=1325495727094366700, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Jiang Xiaotang, Wang Huan, Chen Yiliu, et al. MNN: A universal and efficient inference engine[C]//Proc of Machine Learning and Systems. Miami, FL: MLSys Foundation, 2020: 1−3, articleTitle=null, refAbstract=null), Reference(id=1325495727178252781, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=Wu Ruohan, Zhu Xianyu, Chen Junshi, et al. SWattention: Designing fast and memory-efficient attention for a new Sunway supercomputer[J]. The Journal of Supercomputing, 2024, 80(10): 13657−13680, articleTitle=null, refAbstract=null), Reference(id=1325495727262138862, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=刘锟阳. 基于寒武纪MLU270的YOLOv5模型优化与应用研究[D]. 成都: 四川师范大学, 2023, articleTitle=null, refAbstract=null), Reference(id=1325495727358607855, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Kunyang. Optimization and application of YOLOv5 model based on Cambrian MLU270[D]. Chengdu: Sichuan Normal University, 2023 (in Chinese)., articleTitle=null, refAbstract=null), Reference(id=1325495727442493936, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=26, authorNames=null, journalName=null, refType=null, unstructuredReference=Ma Zixuan, He Jiaao, Qiu Jiezhong, et al. BaGuaLu: Targeting brain scale pretrained models with over 37 million cores[C]//Proc of the 27th ACM SIGPLAN Symp on Principles and Practice of Parallel Programming. New York: ACM, 2022: 192−204, articleTitle=null, refAbstract=null), Reference(id=1325495727517991409, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=27, authorNames=null, journalName=null, refType=null, unstructuredReference=Shah J, Bikshandi G, Zhang Ying, et al. FlashAttention-3: Fast and accurate attention with asynchrony and low-precision[J]. Advances in Neural Information Processing Systems, 2024, 37: 68658−68685, articleTitle=null, refAbstract=null), Reference(id=1325495727593488882, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=28, authorNames=null, journalName=null, refType=null, unstructuredReference=赵玉龙, 张鲁飞, 许国春, 等. SDAA: 面向申威智能加速卡的运行时系统[J]. 软件学报, 2024, 35(12): 5710−5724, articleTitle=null, refAbstract=null), Reference(id=1325495727652209139, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=29, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Yulong, Zhang Lufei, Xu Guochun, et al. SDAA: Runtime system for Sunway intelligent acceleration card[J]. Journal of Software, 2024, 35(12): 5710−5724 (in Chinese), articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1325495711239897458, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=1, ext=[AuthorCompanyExt(id=1325495711252480371, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1325495711260868980, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)]), AuthorCompany(id=1325495711332172149, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=2, ext=[AuthorCompanyExt(id=1325495711344755062, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711332172149, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2National Parallel Computer Engineering Technology Research Center, Beijing 100083), AuthorCompanyExt(id=1325495711353143671, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711332172149, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2国家并行计算机工程技术研究中心 北京 100083)])], figs=[ArticleFig(id=1325495718126944687, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, label=Figure 1, caption=Structure of Sunway AI processor, figureFileSmall=xWAmIOETiPYSDNv0bcIaLw==, figureFileBig=ft/9NCVWF7igZbg/1TP6vQ==, tableContent=null), ArticleFig(id=1325495718252773808, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, label=图1, caption=申威智能处理器结构, figureFileSmall=xWAmIOETiPYSDNv0bcIaLw==, figureFileBig=ft/9NCVWF7igZbg/1TP6vQ==, tableContent=null), ArticleFig(id=1325495718479266225, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, label=Figure 2, caption=Inference time consumption breakdown of Meta-LLaMA, figureFileSmall=jI4JbWKOpE0Z4ph7OhPT9A==, figureFileBig=5UTfLSHkMEEz3UNqmICTfA==, tableContent=null), ArticleFig(id=1325495718563152306, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, label=图2, caption=Meta-LLaMA推理耗时分解, figureFileSmall=jI4JbWKOpE0Z4ph7OhPT9A==, figureFileBig=5UTfLSHkMEEz3UNqmICTfA==, tableContent=null), ArticleFig(id=1325495718642844083, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, label=Figure 3, caption=Analysis of time consumption for MHA and FFN on Sunway AI processor, figureFileSmall=aFlIcrqqku14GiDk7f8X8A==, figureFileBig=GrH77XIJZfML1OrGvvhL1Q==, tableContent=null), ArticleFig(id=1325495718722535860, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, label=图3, caption=申威智能处理器上MHA和FFN耗时分析, figureFileSmall=aFlIcrqqku14GiDk7f8X8A==, figureFileBig=GrH77XIJZfML1OrGvvhL1Q==, tableContent=null), ArticleFig(id=1325495718810616245, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, label=Figure 4, caption=Overall architecture of SW-Inf, figureFileSmall=L4TZfFvAkJ8CbAkFVHYPUw==, figureFileBig=N3SlX1hmrC7HThBGckVOwA==, tableContent=null), ArticleFig(id=1325495718890308022, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, label=图4, caption=SW-Inf总体架构, figureFileSmall=L4TZfFvAkJ8CbAkFVHYPUw==, figureFileBig=N3SlX1hmrC7HThBGckVOwA==, tableContent=null), ArticleFig(id=1325495718969999799, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, label=Figure 5, caption=Illustration of the full-graph descent scheduling principle, figureFileSmall=XM21zkGmN56HfbcfK8UcHw==, figureFileBig=GIFdvO7tZn1jngpmNnyV9A==, tableContent=null), ArticleFig(id=1325495719053885880, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, label=图5, caption=整图下沉调度原理示意图, figureFileSmall=XM21zkGmN56HfbcfK8UcHw==, figureFileBig=GIFdvO7tZn1jngpmNnyV9A==, tableContent=null), ArticleFig(id=1325495719141966265, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, label=Figure 6, caption=Multi-level parallel computing strategy, figureFileSmall=TspHA4pNjJjjaeKSutVHKw==, figureFileBig=QK4EUJz0EocEhhFVWZGFzw==, tableContent=null), ArticleFig(id=1325495719213269434, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, label=图6, caption=多级并行计算策略, figureFileSmall=TspHA4pNjJjjaeKSutVHKw==, figureFileBig=QK4EUJz0EocEhhFVWZGFzw==, tableContent=null), ArticleFig(id=1325495719292961211, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, label=Figure 7, caption=Optimization logic for nonlinear functions, figureFileSmall=LM1eoBYiN2+Bb2+ftflsww==, figureFileBig=R/DAeie/n4PeTdb6xfzvOg==, tableContent=null), ArticleFig(id=1325495719368458684, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, label=图7, caption=非线性函数优化逻辑, figureFileSmall=LM1eoBYiN2+Bb2+ftflsww==, figureFileBig=R/DAeie/n4PeTdb6xfzvOg==, tableContent=null), ArticleFig(id=1325495719477510589, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, label=Figure 8, caption=Transforming rowmax operators to colmax in the transposed MHA algorithm, figureFileSmall=UNC7ITg/JMavFeKeGwc6Mg==, figureFileBig=//nXyN8viCEsGyy5W2tXig==, tableContent=null), ArticleFig(id=1325495719586562494, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, label=图8, caption=MHA转置算法中算子形态变化, figureFileSmall=UNC7ITg/JMavFeKeGwc6Mg==, figureFileBig=//nXyN8viCEsGyy5W2tXig==, tableContent=null), ArticleFig(id=1325495719657865663, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, label=Figure 9, caption=End-to-end inference throughput in different scenarios, figureFileSmall=BYDm4DtbSkbrGBjnBX1w8A==, figureFileBig=7/2guz+u3fKGDUcNWSDLjg==, tableContent=null), ArticleFig(id=1325495719737557440, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, label=图9, caption=不同场景下的端到端推理吞吐量, figureFileSmall=BYDm4DtbSkbrGBjnBX1w8A==, figureFileBig=7/2guz+u3fKGDUcNWSDLjg==, tableContent=null), ArticleFig(id=1325495719813054913, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, label=Figure 10, caption=Memory bandwidth usage with end-to-end inference in different scenarios, figureFileSmall=vbkaENEOBsRayrjbOnQfmA==, figureFileBig=Arxw5cyUYk8WaYK6wzaADg==, tableContent=null), ArticleFig(id=1325495719934689730, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, label=图10, caption=不同场景下的端到端推理主存带宽利用率, figureFileSmall=vbkaENEOBsRayrjbOnQfmA==, figureFileBig=Arxw5cyUYk8WaYK6wzaADg==, tableContent=null), ArticleFig(id=1325495720026964419, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, label=Figure 11, caption=Ablation study in end-to-end inference, figureFileSmall=ECtsZ17lxuSALOnrBqiK/A==, figureFileBig=h+nVqXQ79ksrLUNq8FWYww==, tableContent=null), ArticleFig(id=1325495720102461892, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, label=图11, caption=端到端推理消融实验, figureFileSmall=ECtsZ17lxuSALOnrBqiK/A==, figureFileBig=h+nVqXQ79ksrLUNq8FWYww==, tableContent=null), ArticleFig(id=1325495720198930885, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, label=Table 1, caption=Arithmetic Intensity of the Core Modules in LLaMA2 Model, figureFileSmall=null, figureFileBig=null, tableContent=
句长N模型选择MHA线性映射MHA核心计算FFN
128LLaMA2-7B12043123
51240951437
40961365621727
128LLaMA2-70B45543125
512204851474
40962048622493
), ArticleFig(id=1325495721922789830, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, label=表1, caption=

LLaMA2模型核心模块计算访存比

, figureFileSmall=null, figureFileBig=null, tableContent=
句长N模型选择MHA线性映射MHA核心计算FFN
128LLaMA2-7B12043123
51240951437
40961365621727
128LLaMA2-70B45543125
512204851474
40962048622493
), ArticleFig(id=1325495722061201863, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, label=Table 2, caption=Comparison of Kernel Launch Overhead Between Meta-LLaMA and SW-Inf, figureFileSmall=null, figureFileBig=null, tableContent=
词数量Meta-LLaMASW-Inf
交互次数交互耗时/s交互次数交互耗时/ms
50~5005672805.18190030.870
500~10006302085.346100054.345
500~150026020811.750200084.441
), ArticleFig(id=1325495722149282248, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, label=表2, caption=

Meta-LLaMA和SW-Inf的核函数启动开销对比

, figureFileSmall=null, figureFileBig=null, tableContent=
词数量Meta-LLaMASW-Inf
交互次数交互耗时/s交互次数交互耗时/ms
50~5005672805.18190030.870
500~10006302085.346100054.345
500~150026020811.750200084.441
), ArticleFig(id=1325495722291888585, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, label=Table 3, caption=Memory Traffic and Execution Time of Matrix Multiplication Under Different Parallelization Strategies, figureFileSmall=null, figureFileBig=null, tableContent=
矩阵规模从核拆分策略循环分块策略访存量/KB实际耗时/μs
m=2048
d=4096
n=4096
4,2,42,2,466562213.74
8,1,42,2,461441997.44
4,2,42,1,843521580.41
8,1,42,1,840961232.98
m=4096
d=4096
n=8192
2,8,24,1,23788813166.83
4,1,84,1,23686410732.33
2,8,22,1,8174085961.80
4,1,82,1,8163844862.31
), ArticleFig(id=1325495722396746186, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, label=表3, caption=

不同并行策略下的访存量和实际耗时

, figureFileSmall=null, figureFileBig=null, tableContent=
矩阵规模从核拆分策略循环分块策略访存量/KB实际耗时/μs
m=2048
d=4096
n=4096
4,2,42,2,466562213.74
8,1,42,2,461441997.44
4,2,42,1,843521580.41
8,1,42,1,840961232.98
m=4096
d=4096
n=8192
2,8,24,1,23788813166.83
4,1,84,1,23686410732.33
2,8,22,1,8174085961.80
4,1,82,1,8163844862.31
), ArticleFig(id=1325495722493215179, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, label=Table 4, caption=Performance Improvement of the Fused exp Operator, figureFileSmall=null, figureFileBig=null, tableContent=
向量维度单次运行平均耗时/μs
MathLibTBLExpFusedExp
1024885.491.140.21
20481783.622.270.42
40963567.244.520.83
61445350.856.771.23
81926364.429.021.64
), ArticleFig(id=1325495722581295564, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, label=表4, caption=

exp融合算子性能提升

, figureFileSmall=null, figureFileBig=null, tableContent=
向量维度单次运行平均耗时/μs
MathLibTBLExpFusedExp
1024885.491.140.21
20481783.622.270.42
40963567.244.520.83
61445350.856.771.23
81926364.429.021.64
), ArticleFig(id=1325495722686153165, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, label=Table 5, caption=Performance Improvement of the Transposed MHA Algorithm, figureFileSmall=null, figureFileBig=null, tableContent=
S→M单次运行平均耗时/μs
rowmaxcolmax
(32, 32)→(32, 1)24.860.20
(32, 128)→(32, 1)32.910.50
(192, 32)→(192, 1)146.510.85
), ArticleFig(id=1325495722811982286, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, label=表5, caption=

MHA转置算法性能提升

, figureFileSmall=null, figureFileBig=null, tableContent=
S→M单次运行平均耗时/μs
rowmaxcolmax
(32, 32)→(32, 1)24.860.20
(32, 128)→(32, 1)32.910.50
(192, 32)→(192, 1)146.510.85
), ArticleFig(id=1325495722908451279, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, label=Table 6, caption=Performance Comparison of Different Attention Calculation Methods, figureFileSmall=null, figureFileBig=null, tableContent=
Attention方法硬件环境耗时/ms
seq_len = 256seq_len = 4096
注:在Tesla-A100 40 GB下的多种Attention方法的数据与文献[12]一致,所以有一处为空。seg_len为句长。
PyTorch AttentionTesla-A100
40 GB
0.2216.47
Megatron0.26-
Local Attention0.5711.56
Longformer Attention0.9811.07
FlashAttention0.098.42
SWAI AttentionSW-AI0.04521.37
), ArticleFig(id=1325495723017503184, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, label=表6, caption=

不同注意力计算方法的性能对比

, figureFileSmall=null, figureFileBig=null, tableContent=
Attention方法硬件环境耗时/ms
seq_len = 256seq_len = 4096
注:在Tesla-A100 40 GB下的多种Attention方法的数据与文献[12]一致,所以有一处为空。seg_len为句长。
PyTorch AttentionTesla-A100
40 GB
0.2216.47
Megatron0.26-
Local Attention0.5711.56
Longformer Attention0.9811.07
FlashAttention0.098.42
SWAI AttentionSW-AI0.04521.37
), ArticleFig(id=1325495723122360785, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=EN, label=Table 7, caption=Time Consumption of MHA and FFN Under Different Models and Parameter Dimensions, figureFileSmall=null, figureFileBig=null, tableContent=
模型隐藏层维度注意力头数耗时/ms
MHAFFN
Qwen-1.8B2048160.343.06
LLaMA-7B4096321.0110.34
LLaMA-13B5120401.6615.85
Qwen-72B8192644.2940.81
), ArticleFig(id=1325495723218829778, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, language=CN, label=表7, caption=

不同模型和参数维度下MHA和FFN的耗时

, figureFileSmall=null, figureFileBig=null, tableContent=
模型隐藏层维度注意力头数耗时/ms
MHAFFN
Qwen-1.8B2048160.343.06
LLaMA-7B4096321.0110.34
LLaMA-13B5120401.6615.85
Qwen-72B8192644.2940.81
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202550153, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202550153, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202550153, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202550153, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1772294400000, fullTextJson=null, articleText=null, reference=null)
收藏切换
SW-Inf:基于申威智能加速卡的Transformer高效推理方法
沈文渊 1 , 汤凌韬 1 , 何昊天 1 , 费天成 1 , 赵玉龙 1 , 张鲁飞 1 , 秦晓军 1 , 刘鑫 2, * , 陈左宁 2
计算机研究与发展 | 高性能计算 2026,63(3): 782-797
收起
收藏切换
计算机研究与发展 |高性能计算 2026 , 63 (3) : 782 -797
SW-Inf:基于申威智能加速卡的Transformer高效推理方法
全屏
[Author(id=1325495713009893753, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=wenyuan17@yeah.net, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325495713106362747, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495713009893753, language=EN, stringName=Wenyuan Shen, firstName=Wenyuan, middleName=null, lastName=Shen, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 2000. Undergraduate. His main research interests include high performance computing and operating systems

"}, bioImg=null, bioContent=

born in 2000. Undergraduate. His main research interests include high performance computing and operating systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325495713219608956, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495713009893753, language=CN, stringName=沈文渊, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"9ITsID7Qfz8/HUmS/wj07Q==","content":"

2000年生。本科生。主要研究方向为高性能计算、操作系统

"}, bioImg=9ITsID7Qfz8/HUmS/wj07Q==, bioContent=

2000年生。本科生。主要研究方向为高性能计算、操作系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325495711239897458, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=1, ext=[AuthorCompanyExt(id=1325495711252480371, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1325495711260868980, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1325495713311883646, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325495713395769728, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495713311883646, language=EN, stringName=Lingtao Tang, firstName=Lingtao, middleName=null, lastName=Tang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1994. PhD. His main research interests include machine learning, privacy protection, and intelligent operating systems

"}, bioImg=null, bioContent=

born in 1994. PhD. His main research interests include machine learning, privacy protection, and intelligent operating systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325495713483850113, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495713311883646, language=CN, stringName=汤凌韬, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"uBXa9p3PP0OzJJ3pAJePfw==","content":"

1994年生。博士。主要研究方向为机器学习、隐私保护、智能操作系统

"}, bioImg=uBXa9p3PP0OzJJ3pAJePfw==, bioContent=

1994年生。博士。主要研究方向为机器学习、隐私保护、智能操作系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325495711239897458, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=1, ext=[AuthorCompanyExt(id=1325495711252480371, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1325495711260868980, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1325495713592902019, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325495713693565317, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495713592902019, language=EN, stringName=Haotian He, firstName=Haotian, middleName=null, lastName=He, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1997. Master. His main research interests include high performance computing and operating systems

"}, bioImg=null, bioContent=

born in 1997. Master. His main research interests include high performance computing and operating systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325495713798422918, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495713592902019, language=CN, stringName=何昊天, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"wgvL42o6ueHbH3QkZrFKcQ==","content":"

1997年生。硕士。主要研究方向为高性能计算、操作系统

"}, bioImg=wgvL42o6ueHbH3QkZrFKcQ==, bioContent=

1997年生。硕士。主要研究方向为高性能计算、操作系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325495711239897458, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=1, ext=[AuthorCompanyExt(id=1325495711252480371, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1325495711260868980, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1325495713882309000, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325495713974583690, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495713882309000, language=EN, stringName=Tiancheng Fei, firstName=Tiancheng, middleName=null, lastName=Fei, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1999. Master. His main research interests include high performance computing and operating systems

"}, bioImg=null, bioContent=

born in 1999. Master. His main research interests include high performance computing and operating systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325495714062664075, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495713882309000, language=CN, stringName=费天成, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"UomR7ExGXaaBuvcTD3QgWg==","content":"

1999年生。硕士。主要研究方向为高性能计算、操作系统

"}, bioImg=UomR7ExGXaaBuvcTD3QgWg==, bioContent=

1999年生。硕士。主要研究方向为高性能计算、操作系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325495711239897458, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=1, ext=[AuthorCompanyExt(id=1325495711252480371, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1325495711260868980, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1325495714167521677, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325495714263990671, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495714167521677, language=EN, stringName=Yulong Zhao, firstName=Yulong, middleName=null, lastName=Zhao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1984. PhD candidate. His main research interests include artificial intelligence and operating system

"}, bioImg=null, bioContent=

born in 1984. PhD candidate. His main research interests include artificial intelligence and operating system

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325495714343682448, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495714167521677, language=CN, stringName=赵玉龙, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"D3exj6Mvc4B6NVGIL38eYg==","content":"

1984年生。博士研究生。主要研究方向为人工智能、操作系统

"}, bioImg=D3exj6Mvc4B6NVGIL38eYg==, bioContent=

1984年生。博士研究生。主要研究方向为人工智能、操作系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325495711239897458, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=1, ext=[AuthorCompanyExt(id=1325495711252480371, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1325495711260868980, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1325495714482094482, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, orderNo=5, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325495714578563476, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495714482094482, language=EN, stringName=Lufei Zhang, firstName=Lufei, middleName=null, lastName=Zhang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1986. PhD, senior engineer. His main research interests include high-performance computing, operating system, and machine learning

"}, bioImg=null, bioContent=

born in 1986. PhD, senior engineer. His main research interests include high-performance computing, operating system, and machine learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325495714675032469, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495714482094482, language=CN, stringName=张鲁飞, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"+NAwwBD40sKFmfLNZR+UCA==","content":"

1986年生。博士,高级工程师。主要研究方向为高性能计算、操作系统、机器学习

"}, bioImg=+NAwwBD40sKFmfLNZR+UCA==, bioContent=

1986年生。博士,高级工程师。主要研究方向为高性能计算、操作系统、机器学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325495711239897458, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=1, ext=[AuthorCompanyExt(id=1325495711252480371, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1325495711260868980, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1325495714763112855, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, orderNo=6, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325495714851193241, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495714763112855, language=EN, stringName=Xiaojun Qin, firstName=Xiaojun, middleName=null, lastName=Qin, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125, bio={"content":"

born in 1975. PhD, senior engineer. His main research interests include system security, software vulnerability analysis, and artificial intelligence

"}, bioImg=null, bioContent=

born in 1975. PhD, senior engineer. His main research interests include system security, software vulnerability analysis, and artificial intelligence

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325495714935079322, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495714763112855, language=CN, stringName=秦晓军, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1数学工程与先进计算国家重点实验室 江苏无锡 214125, bio={"img":"j229/SQRoT8tlzPI8/2RBA==","content":"

1975年生。博士。高级工程师。主要研究方向为系统安全、软件脆弱性分析、人工智能

"}, bioImg=j229/SQRoT8tlzPI8/2RBA==, bioContent=

1975年生。博士。高级工程师。主要研究方向为系统安全、软件脆弱性分析、人工智能

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325495711239897458, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=1, ext=[AuthorCompanyExt(id=1325495711252480371, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125), AuthorCompanyExt(id=1325495711260868980, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711239897458, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1数学工程与先进计算国家重点实验室 江苏无锡 214125)])]), Author(id=1325495715027354012, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, orderNo=7, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=yyylx@263.net, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1325495715115434398, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495715027354012, language=EN, stringName=Xin Liu, firstName=Xin, middleName=null, lastName=Liu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, *, address=2National Parallel Computer Engineering Technology Research Center, Beijing 100083, bio={"content":"

born in 1979. PhD, professor. Her main research interest includes parallel algorithms and applications

"}, bioImg=null, bioContent=

born in 1979. PhD, professor. Her main research interest includes parallel algorithms and applications

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325495715195126175, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495715027354012, language=CN, stringName=刘鑫, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, *, address=2国家并行计算机工程技术研究中心 北京 100083, bio={"img":"1AGKD+nG+2XoQIR3FrCkgg==","content":"

1979年生。博士,研究员。主要研究方向为并行算法和应用

"}, bioImg=1AGKD+nG+2XoQIR3FrCkgg==, bioContent=

1979年生。博士,研究员。主要研究方向为并行算法和应用

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325495711332172149, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=2, ext=[AuthorCompanyExt(id=1325495711344755062, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711332172149, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2National Parallel Computer Engineering Technology Research Center, Beijing 100083), AuthorCompanyExt(id=1325495711353143671, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711332172149, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2国家并行计算机工程技术研究中心 北京 100083)])]), Author(id=1325495715279012257, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, orderNo=8, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325495715375481251, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495715279012257, language=EN, stringName=Zuoning Chen, firstName=Zuoning, middleName=null, lastName=Chen, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2National Parallel Computer Engineering Technology Research Center, Beijing 100083, bio={"content":"

born in 1957. Academician of Chinese Academy of Engineering. Her main research interests include software theory, operating systems, and information security

"}, bioImg=null, bioContent=

born in 1957. Academician of Chinese Academy of Engineering. Her main research interests include software theory, operating systems, and information security

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325495715467755940, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, authorId=1325495715279012257, language=CN, stringName=陈左宁, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2国家并行计算机工程技术研究中心 北京 100083, bio={"img":"z857SmE9ENVNlfqZ5PIEVg==","content":"

1957年生。中国工程院院士。主要研究方向为软件理论、操作系统、信息安全

"}, bioImg=z857SmE9ENVNlfqZ5PIEVg==, bioContent=

1957年生。中国工程院院士。主要研究方向为软件理论、操作系统、信息安全

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325495711332172149, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, xref=2, ext=[AuthorCompanyExt(id=1325495711344755062, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711332172149, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2National Parallel Computer Engineering Technology Research Center, Beijing 100083), AuthorCompanyExt(id=1325495711353143671, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325495704952635716, companyId=1325495711332172149, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2国家并行计算机工程技术研究中心 北京 100083)])])]
沈文渊1 , 汤凌韬1, 何昊天1, 费天成1, 赵玉龙1, 张鲁飞1, 秦晓军1, 刘鑫2, * , 陈左宁2
作者信息
  • 1数学工程与先进计算国家重点实验室 江苏无锡 214125
  • 2国家并行计算机工程技术研究中心 北京 100083
通讯作者:
刘鑫()
作者简介:

2000年生。本科生。主要研究方向为高性能计算、操作系统

1994年生。博士。主要研究方向为机器学习、隐私保护、智能操作系统

1997年生。硕士。主要研究方向为高性能计算、操作系统

1999年生。硕士。主要研究方向为高性能计算、操作系统

1984年生。博士研究生。主要研究方向为人工智能、操作系统

1986年生。博士,高级工程师。主要研究方向为高性能计算、操作系统、机器学习

1975年生。博士。高级工程师。主要研究方向为系统安全、软件脆弱性分析、人工智能

1979年生。博士,研究员。主要研究方向为并行算法和应用

1957年生。中国工程院院士。主要研究方向为软件理论、操作系统、信息安全

SW-Inf: Efficient Transformer Inference Method for Sunway AI Accelerator Card
Wenyuan Shen1 , Lingtao Tang1, Haotian He1, Tiancheng Fei1, Yulong Zhao1, Lufei Zhang1, Xiaojun Qin1, Xin Liu2, * , Zuoning Chen2
Affiliations
  • 1State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, Jiangsu 214125
  • 2National Parallel Computer Engineering Technology Research Center, Beijing 100083
  • born in 2000. Undergraduate. His main research interests include high performance computing and operating systems

    born in 1994. PhD. His main research interests include machine learning, privacy protection, and intelligent operating systems

    born in 1997. Master. His main research interests include high performance computing and operating systems

    born in 1999. Master. His main research interests include high performance computing and operating systems

    born in 1984. PhD candidate. His main research interests include artificial intelligence and operating system

    born in 1986. PhD, senior engineer. His main research interests include high-performance computing, operating system, and machine learning

    born in 1975. PhD, senior engineer. His main research interests include system security, software vulnerability analysis, and artificial intelligence

    born in 1979. PhD, professor. Her main research interest includes parallel algorithms and applications

    born in 1957. Academician of Chinese Academy of Engineering. Her main research interests include software theory, operating systems, and information security

出版时间: 2026-03-01 doi: 10.7544/issn1000-1239.202550153
文章导航
收藏切换

Transformer通过自注意力机制高效捕捉全局依赖,突破了传统神经网络处理长序列信息时的限制,推动了自然语言处理多个领域的革命性进展,成为现代大语言模型的核心架构。目前,国内Transformer推理应用整体上处于算力紧缺状态,且在部署时往往面临调度开销大、计算并发难、访存瓶颈明显等问题。申威智能加速卡搭载了国产自主研发的新一代智能处理器,已有效支撑各类端侧Transformer推理应用,然而常规推理框架难以发挥申威异构众核架构的管理功能完备、多级并行计算和存储层次丰富等特点。对此,提出一种基于申威智能加速卡的Transformer高效推理方法SW-Inf。首先,提出了片上主核驱动的整图下沉调度机制,通过计算图打包和片上主核调度,减少主机与设备间的核函数启动开销和设备内各流程数据传输开销;其次,设计了LDM优先的启发式分块并行策略,实现计算部件负载均衡,提高了硬件资源利用率;最后,实现了体系结构紧耦合的注意力计算方法,避免推理的中间结果频繁读写主存和寄存器,缓解端侧推理应用的访存瓶颈问题。为验证所提方法的效果,基于申威智能加速卡实现了一套端到端推理系统,测试结果表明,相较于访存带宽更高的英伟达V100S上的vLLM推理系统,SW-Inf的吞吐率和主存利用率分别提高了15个百分点和20个百分点。

申威智能加速卡  /  大模型推理  /  调度优化  /  并行策略  /  算子融合

Transformer has fundamentally revolutionized various fields of natural language processing through its self-attention mechanism that effectively models long-range contextual relationships. It overcomes traditional neural networks’ limitations in handling long sequences, and has emerged as the mainstream architecture for modern large language models. Currently, Transformer inference systems generally face a shortage of computing power and are commonly confronted with challenges like substantial scheduling latency, concurrency execution difficulties, and severe memory access limitations during deployment. The Sunway AI accelerator card, featuring a domestically-developed next-generation intelligent processor, has successfully empowered diverse edge inference deployments. However, conventional inference frameworks struggle to effectively explore the architectural potential in three critical dimensions: robust system control capabilities, multi-level parallel computing, and rich memory hierarchy. We propose SW-Inf: a high-efficiency Transformer inference method based on the Sunway AI accelerator card. First, an on-chip management processing element (MPE)-orchestrated computation graph scheduling mechanism is proposed to minimize kernel launch latency and inter-process data transfer overhead through graph aggregation and MPE coordination. Second, an LDM-prioritized heuristic tiling strategy is introduced to enable load-balanced parallel computation across processing elements. Finally, a hardware-aware tightly-coupled attention computation implementation is developed to eliminate redundant intermediate data movement between registers and main memory. We implement an end-to-end inference system on the Sunway AI accelerator card. Experimental results demonstrate that compared with the vLLM inference system on NVIDIA V100S which has higher memory bandwidth than the Sunway AI accelerator card, SW-Inf achieves 15% higher throughput and 20% better memory utilization rate.

Sunway AI accelerator card  /  large model inference  /  scheduling optimization  /  parallel strategy  /  operator fusion
沈文渊, 汤凌韬, 何昊天, 费天成, 赵玉龙, 张鲁飞, 秦晓军, 刘鑫, 陈左宁. SW-Inf:基于申威智能加速卡的Transformer高效推理方法. 计算机研究与发展, 2026 , 63 (3) : 782 -797 . DOI: 10.7544/issn1000-1239.202550153
Wenyuan Shen, Lingtao Tang, Haotian He, Tiancheng Fei, Yulong Zhao, Lufei Zhang, Xiaojun Qin, Xin Liu, Zuoning Chen. SW-Inf: Efficient Transformer Inference Method for Sunway AI Accelerator Card[J]. Journal of Computer Research and Development, 2026 , 63 (3) : 782 -797 . DOI: 10.7544/issn1000-1239.202550153
近年来,随着人工智能技术的飞速发展,以ChatGPT为代表的大语言模型在自然语言处理、计算机视觉和语音识别等通用人工智能领域取得了突破性进展[1-3]。凭借其强大的表示与学习能力,能够高效处理各类复杂任务[4],如高质量文本生成[5]、精准机器翻译[6]和智能问答[7]等。与此同时,智能监控、自动驾驶和工业自动化等端侧应用场景对人工智能模型的推理提出包括低延迟、低功耗和高资源利用率在内的更高要求。
Transformer[8]作为目前大语言模型的典型架构,凭借注意力机制高效处理长距离依赖问题,同时增强并行计算度显著提升了训练效率,同时Transformer架构在跨模态应用的广泛适应性,使其在多模态任务中展现出强大的整合能力。然而,随着模型规模的不断扩大,Transformer架构的注意力机制的计算复杂度与序列长度呈二次方关系增长,导致在处理长序列数据时计算资源消耗显著增加。此外,在推理阶段,Transformer架构对内存和计算资源的需求极高,尤其是当序列长度增加时,内存使用量会急剧上升。这种高计算复杂度和内存需求使其在边缘部署场景下面临严峻挑战[9-10]。
各界为此作了许多探索。硬件方面,英伟达(Nvidia)GPU凭借强大的并行计算能力、高带宽显存和专为深度学习优化的TensorCore,在大规模模型智能推理应用中表现出色,成为Transformer推理的首选设备。软件方面,SGLang[11]是一个用于高效执行复杂语言模型程序的开源系统,由前端语言简化编程,运行时利用RadixAttention和压缩有限状态机等优化技术提升效率,经多种任务评估表现优异。vLLM通过PagedAttention技术、连续批处理技术,并结合英伟达Triton的动态张量并行,在A100 GPU上实现1750 token/s的GPT-3吞吐量,较PyTorch提升6.2倍,成为推理系统的行业标杆。
申威智能处理器是申威系列面向智能计算加速的国产处理器,硬件上具备了包括环网总线、从核阵列和脉动阵列等,软件上包含了自研的驱动运行时、算子库、动态内存管理单元和自适应编译优化器等,已具备一定的Transformer推理能力。但是相较于Nvidia GPU,申威智能加速卡(SW-AI)采用异构众核架构,具备巨大的软硬协同优化空间,可弥补其在矩阵加速数据类型和片上网络带宽等方面等存在的限制。其面临的具体挑战与机遇如下:
1)Transformer推理流程中的大量算子均需发起主机和设备间跨PCIe的交互,使得调度开销成为推理耗时中不可忽视的一部分。而申威智能处理器具备异构众核架构,利用主核强大的自主控制能力,将整个Transformer 推理过程转移到主核上,把主机-设备间的PCIe交互转化为芯片内主从核间的交互。
2)申威智能加速卡的计算和存储层次丰富、特征明显,需要设计有效的任务划分和并行计算策略,同时各部件的使用在核内局存受限的情况下有各自的数据对齐和并行设计要求,加剧内存管理复杂度。这要求“LDM优先”设计必须兼顾计算单元适配、存储限制突破及通信优化3重目标,在硬件特性与软件易用性间寻求平衡。
3)申威智能加速卡拥有高计算能力,但在面临访存瓶颈的Decoder-Only模型推理场景下,访存带宽无法很好地匹配计算能力,因此如何减少算子间冗余访存,同时提高计算访存并发度,是提高模型推理效率的关键。
针对上述挑战与机遇,本文研究基于申威智能加速卡的Transformer高效推理方法SW-Inf,主要贡献有3点:
1)提出了基于片上主核整图下沉调度机制,利用申威处理器主核的调度管理能力,降低推理过程中主机与设备间的交互次数,减少调度开销。
2)设计了LDM优先的启发式并行策略,实现申威众核架构下计算和存储资源的负载均衡,通过计算、访存、通信相互掩盖的方式,减少不同部件间的数据等待,提高整体资源利用率,降低推理时延。
3)实现了多层内存高效的注意力机制,从算子、指令集和寄存器3个层面解决了FlashAttention算法在申威智能处理器上算子内冗余访存高、计算效率低和寄存器利用率不高等问题。
结合上述优化方法而实现的推理系统SW-Inf与申威处理器的计算单元深度对接,确保指令级兼容性,具有较强的推理效率和稳定性,可直接整合至申威云端服务器与超算平台,强化生态AI能力。
大语言模型因其在各类任务中展现出强大的理解和推理能力而受业界广泛关注。近年来,大模型参数规模的快速增长带来了更强的模型能力,同时也极大增加了模型推理复杂度,如何提高硬件利用效率和满足大模型在不同推理场景下的计算和存储需求,成为研究热点。
模型算法方面,FlashAttention[12]基于循环分块和计算融合的思想,结合online Softmax技术,提出IO感知的精确注意力算法,大幅减少了中间结果对主存的冗余访存,从而提升注意力计算性能。MLA(multi latent attention)是DeepSeek-V2[13]采用的创新注意力机制,通过低秩键值联合压缩,在性能上超越传统的多头注意力模块,同时大幅降低KV Cache存储量。
推理服务系统方面的主要挑战是在高并发请求情况下充分利用硬件资源减少推理气泡,同时优化资源管理,提高系统响应速度和吞吐率。Orca[1]提出了迭代级调度策略和连续批处理技术,实现了细粒度资源的利用和灵活请求响应,有效减少了请求的平均等待时间,提升系统的整体响应速度与资源利用率。PagedAttention[14]针对推理服务中KV Cache导致的内存碎片问题,借鉴操作系统虚拟内存和分页技术的思路进行缓存管理,实现灵活的内存共享,从而提升内存利用率。上述工作能有效提高系统吞吐率,但由于单批请求负载增大可能导致推理时延变长,对此Sarathi[15]将处于预填充阶段的请求拆分为几乎相等大小的块,并构建无停顿调度,在不暂停解码阶段请求的情况下添加新请求,从而平衡吞吐量和延迟之间的矛盾。
端侧推理应用方面的主要挑战是在硬件资源受限的情况下实现模型推理,尽可能提高硬件资源利用率。FlexGen[16]设计了一种运行在有限GPU资源上的LLM推理引擎,通过整合GPU、CPU、磁盘等计算和存储资源进行高效任务卸载,高效调度I/O操作。同时,采用细粒度分组量化将权重和缓存压缩至4 b以提升性能,实现了在单GPU上的千亿参数模型推理。类似的,PowerInfer[17]研究消费级GPU上的LLM推理优化方法,借鉴Deja Vu[3]中模型推理过程只有部分神经元被激活这一观察,对模型进行动态剪枝,并根据神经元的激活概率将其分为冷热2类神经元,通过GPU处理热神经元以及CPU处理冷神经元,实现了GPU-CPU混合推理方法,充分利用了整机资源。类似的工作还有SparseGPT[2],Michel[18],Bansal[19],LLMinFlash[20]等。另外,诸如LLaMA.cpp[21],MNN[22]等开发框架有效摆脱了大模型推理对AI框架和GPU设备的依赖,支持个人主机或各类边缘设备上的推理应用开发与部署。
随着国产智能处理器的发展,基于国产硬件平台优化大模型性能的研究逐渐涌现。例如,SWattention[23]是针对SW26010pro处理器的精确注意力高效计算方法,文献[24]基于国产寒武纪MLU270芯片提出了一种针对YOLOv5模型的推理优化方法。BaGuaLu[25]是世界上第1个支持人脑神经元突触规模的神经网络的训练平台,成功将预训练模型扩展到新一代神威超级计算机的整机规模,支持最大174万亿参数量模型的训练。
总体来说,目前大模型性能优化的大量研究工作仍聚焦于GPU平台,基于国产智能处理器的研究相对较少,缺乏具有针对性的算法与系统优化方案,难以直接移植现有技术。例如2024年Tri Dao联合Meta,Nvidia等机构提出了FlashAttention-3[26],针对H100新增的张量内存加速器(tensor memory accelerator,TMA)和FP8低精度运算支持,进一步提升了注意力计算效率,该方法为Hopper架构GPU量身定制。对于上述现状,本文聚焦大语言模型在国产申威平台上的端侧部署问题,首次系统性分析申威智能加速卡在推理流程中的计算与存储瓶颈,并围绕调度控制、并行计算与算子优化3个方面,提出一套端到端的高效推理方案SW-Inf。
申威异构众核智能处理器采用分层式架构设计,如图1所示,其核心组件包括环形网络总线、1个管理控制核(management processing element,MPE)和4个从核组(core group,CG)。其中,MPE作为主核,采用64位RISC指令集架构,承担全局资源调度、任务分配及系统管理等核心功能。每个CG包含1个高速存储控制器和32个运算核心(computing processing element,CPE)。CPE也称从核,以4×8的2维mesh拓扑结构互连,形成高效的阵列内数据交互网络。每个从核均配备核内局存(local data memory,LDM)、通信引擎(DMA)、向量计算部件以及专用于矩阵运算的脉动阵列。运算核心之间通过阵列内高速网络进行互连。
申威智能处理器的片上存储系统呈现3级层次结构:寄存器(register)→LDM→核组共享主存(HBM),访存带宽逐级递减。申威处理器的计算能力与英伟达A100相当,但在访存带宽指标上弱于V100S和A100。相较于V100S/A100,申威加速卡具备以下特征:1)采用大规模脉动阵列替代tensor core,但需严格满足数据形状与最小计算单元对齐;2)每个从核执行单线程任务,寄存器文件大小远小于GPU;3)取消2级缓存,依赖阵列广播实现数据复用;4)支持基于RMA的LDM间显式通信。这些架构特性为进一步优化访存能力提供了可能,但同时要求必须严格依赖LDM进行合理规划。一方面脉动阵列需依赖LDM空间排布实现严格的数据对齐,数据复用同样必须考虑LDM空间受限;另一方面DMA/RMA通信的高效性同样依赖LDM地址对齐。因此,申威异构众核智能处理器的架构特性要求在面对Transformer等大规模计算模型推理需求时必须严格依赖LDM的空间布局,以满足其容量限制和脉动阵列对齐要求。在此基础上实施的精细化分块策略是实现计算粒度适配、访存开销最小化以及核间通信优化,充分发挥申威处理器众核优势的关键。
以GPU为代表的AI加速卡通常是控制与计算分离式架构,会导致大模型推理过程存在控制流程冗余、主机设备交互频繁等问题,产生大量计算外的开销,包括核函数启动和数据拷贝等。模型推理流程的算子无论是cuDNN还是cuBLAS实现,都涉及众多launch或memcpy操作。例如,仅1个conv1d算子就涉及多达7次操作,包括1次Copy_kernel_Cuda、2次nchwToNhwcKernel、1次CUDAFunction_Add和1次computeOffsetsKernel等。此外,memcpy,getDevice,moduleUnload等处理CUDA资源分配和释放的辅助操作进一步增加了主机和设备之间的交互开销。
本文在NVIDIA V100S平台对Meta-LLaMA的LLaMA2-7B模型进行单批次推理耗时分析。将推理耗时分为模型加载、API调用、GPU计算及其他开销。如图2所示,在批处理数目(batch size)为1的典型端侧推理场景下,API调用时间占比高达48.1%,高于GPU实际计算耗时。基于此,在未深度优化的推理系统中,主机与加速卡间产生高频次控制流交互,每次kernel启动需经历指令下发、状态查询、结果回传等过程。当处理LLaMA2-7B这类包含120+异构算子的模型时,接近一半的推理时间用于上述交互。
申威智能加速卡片上主核MPE具备强大逻辑控制能力。在大模型推理时,将整个 Transformer推理过程转移到MPE上,可以将主机和设备间PCIe的数据交互转变为芯片内MPE和CPE间的数据交互,从而降低主机和设备之间交互的开销。
Transformer推理过程分为预填充(prefill)和解码(decoding)两个阶段。预填充阶段处理用户输入的多词元提示词,主要算子为矩阵乘,其需要的计算能力与访存带宽的比值高,且与提示词长度呈正相关;而解码阶段输入的是上一迭代产生的单个词元,主要算子为矩阵向量乘,计算访存比较低,但随着批处理数目的增加会有所提升。表1给出了LLaMA2模型中多头自注意力(multi-head attention,MHA)和前馈神经网络(feedforward neural network,FFN)两个关键模块在不同输入句长N下的计算访存比估计。
当前,AI加速卡针对大规模矩阵乘法中数据高复用的特性,提供了极高的计算能力。例如,英伟达V100S PCIe加速卡的半精度计算能力达到130 TFLOPS,访存带宽为1134 GBps,计算访存比约为117∶1;而A100 40 GB PCIe带宽高达1555 GBps,计算访存比分别增长到195∶1。随着GPU技术的不断发展,计算能力与访存能力的差距逐渐增大,这使得AI加速卡能够充分满足模型预训练和推理服务系统等计算密集型场景对高算力的需求。但在端侧推理场景中,通常只有少量并发请求,“单模型单用户”是一种典型的应用模式。在这种模式下,模型1次只处理1个推理请求,且自解码阶段的输入句长N=1,计算访存比较低,导致明显的访存瓶颈,计算单元常常处于等待数据的空闲状态,严重影响端侧推理效率。
申威智能加速卡作为一款适用于训练与推理场景的通用加速卡,其半精度计算能力与访存带宽的比值约320∶1。在批处理数目较小时,尤其是在单请求推理场景下,计算与访存能力失配问题愈发突出。如图3展示了基于申威加速卡MHA和FFN模块的执行耗时分解,其中注意力头数为8,序列长度为256,注意力头维度为64。可见对于2个核心模块,访存耗时都占总体耗时的90%以上,导致访存成为推理流程瓶颈。本文重点关注在小批处理数目和单请求推理场景下的性能优化策略,此类场景下访存瓶颈问题最为突出。
本文结合申威智能加速卡的体系结构特点,设计并实现了面向端侧推理的高效Transformer推理系统SW-Inf,总体架构如图4所示。SW-Inf推理系统的主机侧包含专用算子库、轻量级AI框架、申威智能加速卡的运行时系统SDAA[27](software defined accelerator architecture)等部分,以支持上层各类推理应用。当主机侧发起一个Transformer推理应用时,首先,SW-Inf将模型参数拷贝至加速卡主存,通过整图下沉技术将单次推理迭代涉及的算子打包成一个核函数进行发起。发起后输入至加速卡由片上主核进行调度,极大减少主机和设备间通过PCIe总线的交互次数,降低内核发起开销。其次,结合硬件的特性,设计多级并行策略将计算任务和相关数据分配至各从核,减少循环分块引入的冗余访存,并实现众核间的负载均衡。最后,对每个从核计算任务中的算子进行深度融合,在受限的LDM空间下尽可能合并相邻操作符,避免中间结果频繁读写HBM和寄存器,从而减少访存开销,提高执行效率。
Transformer推理过程涉及大量算子,如图5(a)所示,每次核函数启动均需要通过PCIe总线进行主机-设备交互,单次交互时间在微秒级别。针对此问题,GPU采用分层控制模式:CPU通过CUDA Graphs预编译静态计算图以降低Host干预,但其固化执行路径难以适配动态条件分支,必须将分支拆解为独立Kernel,且跨设备协调需依赖全局内存原子操作,底层由流式多处理器(streaming multiprocessor,SM)的Warp调度器执行线程级资源分配。然而,主从架构中主核的集中式控制与GPU的异构分层调度存在本质差异,使得前者在复杂逻辑密集型推理任务中更具扩展性与适应性。因此,本节主要结合申威架构主从核异构特点减少核函数启动开销。
基于上述分析,下面将详细介绍本文提出的基于整图下沉的多级调度技术,如图5(b)所示。该技术将输入文本转换为词元向量后,仅需进行1次 PCIe 启动,然后将任务转移到MPE执行。在MPE内,每个编码和解码步骤仅涉及1次启动操作,并且 MPE 利用计算图将计算任务分配给CPE。随后,CPE 根据计算图的内部结构执行这些任务,生成文本,并将结果返回给主机进行输出。
基于整图下沉多级调度机制分为模型预加载和模型执行2个阶段。在模型预加载阶段,多级调度机制利用了MPE强大的逻辑控制能力,引入分层弹性拆分机制将整个 Transformer 模型按层或算子粒度拆分为多个子图单元,并根据加速卡存储余量,动态选择子图粒度,最终将逻辑计算图转移到 MPE 上,把主机和设备间的PCIe交互转化为芯片内MPE和CPE间的交互。这种转化充分发挥了设备端的计算能力,减少了因主机端与设备端之间的协调而产生的额外开销。在模型执行阶段,为了减少过多的启动次数,完成模型预加载后,在MPE和CPE之间调度同样采用整图下沉方法。具体做法是利用MPE构建一个包含多个算子的计算图。对于参数量较大的模型,MPE会进一步将计算图按层拆分为多个子图,并通过异步流水线实现子图预取、参数换入换出。构建完成后,通过执行启动操作,以计算图的形式将多个计算任务分配给CPE,而不是针对每个单独的计算操作进行启动。根据计算图的规模,可将启动操作数量缩减至原来的千分之几甚至万分之几,显著提升了Transformer推理在核组和从核之间的片上交互效率,使系统资源得到更充分的利用。
整图下沉方法在图构建过程中会产生一定的开销。为了减少这种开销,本文采用了一种异步执行方法。具体来说,在CPE执行当前图的计算时,MPE 同时构建后续的图,这有效地掩盖了部分图构建开销。由于CPE和MPE相互独立,本文在CPE进行计算的同时,利用MPE进行图的构建。针对超大模型,MPE会进一步将后续图拆分为更细粒度的子图,并动态调整预加载顺序。在理想情况下,可实现图构建与计算的完全并行化。例如,可在当前Transformer层计算过程中,构建下一个Transformer层的计算图。从算子层面来看,算子内部数据交互可优化为整体数据搬运,一次性将该算子中各数据整体打包传输到CPE。在极端情况下,整个推理流程可合并为1张计算图,从而将单次完整推理的核函数启动次数降至2次。这不仅减少片内总体访存量,而且大数据量带来的传输带宽往往更高。
在2.1节中,通过与GPU架构的对比分析,指出深度挖掘申威架构计算潜力的关键在于利用片上局部存储器LDM。为此,本文提出了一种以最小化推理访存量为目标的LDM优先启发式分块并行策略,旨在显著提高带宽利用率并完善计算流水线。
以矩阵乘计算$ \boldsymbol {Y}=\boldsymbol {X}\times \boldsymbol {W} $,$ {\boldsymbol {X}\in \mathbb{R}}^{m\times d} $,${\boldsymbol {W}\in \mathbb{R}}^{d\times n} $,$\boldsymbol {Y}\in \mathbb{R}^{m\times n} $为例,本文深入探讨了该策略的应用。在典型的Transformer架构中,注意力层和全连接层通常涉及大量矩阵乘运算,通常有$ n=d $或$n=4d $。在端侧推理场景下,由于并发请求较少,推理过程主要集中在自解码阶段,此时有$ m\ll d $。特别地,单请求情况下,$ m=1 $。图6展示该策略整体流程,其3个子图分别阐述了策略在不同维度上的考量、具体步骤及其设计动机。
针对跨核组通信开销显著高于核组内通信开销的特性,分块策略的首要目标是将数据同步约束在核组内部,从而减少跨核组通信的频率和开销。通过将权值矩阵W的n维切分,实现4个核组间的数据独立性,如图6(a)所示。在处理单请求场景时,优先将W的n维划分至各核组HBM中,核组内矩阵乘运算维度为
$ \left(m,d\right)\times \left(d,\frac{n}{4}\right)=\left(m,\frac{n}{4}\right) 。$
完成核组间划分后,需在核组内进行细粒度任务分配,以充分发挥申威架构中32个从核的并行计算能力。假设m,d,n被平均划分为k1,k2,k3块,每个从核中处理的矩阵形状为
$ \left(\frac{m}{{k}_{1}},\frac{d}{{k}_{2}}\right)\times \left(\frac{d}{{k}_{2}},\frac{n}{{4k}_{3}}\right)=\left(\frac{m}{{k}_{1}},\frac{n}{{4k}_{3}}\right)\text{,} $
当块大小仍然超过LDM空间,需在从核内设计循环分块策略。考虑以下因素:首先,计算部件对输入数据形状和存储模式的要求存在最小计算单元的限制;其次,为实现访存与计算并行,需考虑流水操作对存取行为和内存分配的影响;最后,考虑内积计算数据换入换出频次高、外积计算LDM空间占用大,需采用混合策略以平衡访存量和空间限制,合理设计分块大小。所以设脉动阵列要求的最小计算单元维度如式(3)所示,通过循环分块策略(l1,l2,l3)确定单次存取任务规模如式(4)所示。在半精度条件下,分块大小需满足式(5)的双重约束:首先,内存占用不超过LDM可用空间M。采用双缓冲流水机制时,输入矩阵X和W需双份存储以重叠计算与访存,而结果矩阵通过实时累加仅需单份存储。其次,各维度不突破扩展上限。其中up(x, y)函数表示将x扩充到y的倍数,用于处理维度划分余项,在核组划分时需扩展到最小计算单元的整数倍;从核划分时则需对齐循环分块维度,下文推导的分块最优策略公式默认满足倍数关系。
$ (p,r )\times(r,q )=(p,q) \text{,} $
$ ({l}_{1}p,{l}_{2}r )\times ({l}_{2}r,{l}_{3}q )\to ({l}_{1}p,{l}_{3}q ) \text{,} $
$ \left\{\begin{aligned}&2{l}_{1}{l}_{2}pr+2{l}_{2}{l}_{3}qr+{l}_{2}{l}_{3}pq\le M/2\text{,}\\&{l}_{1}p\le up\left(\frac{m}{{k}_{1}},p\right)\text{,}\\&{l}_{2}r\le up\left(\frac{d}{{k}_{2}},r\right)\text{,}\\&{l}_{3}q\le up\left(\frac{n}{{4k}_{3}},q\right)\text{,}\\&{l}_{1}{,l}_{2}{,l}_{3}\in {\mathbb{N}}_+。\end{aligned}\right.$
在满足上述LDM容量限制和各部件对齐要求后,最小化各从核的访存复杂度以及从核之间的通信复杂度,进一步确定分块最优策略。若LDM空间能容纳式(2)中的输入输出分块,则优化问题可表示为
$ \mathrm{a}\mathrm{r}\mathrm{g}\;\mathrm{ }\mathrm{m}\mathrm{i}\mathrm{n}\frac{2d}{{k}_{2}}\left(\frac{m}{{k}_{1}}+\frac{n}{4{k}_{3}}\right)+\gamma \frac{mn}{{2k}_{1}{k}_{3}}\left({k}_{2}-1\right)\text{,} $
反之,当受LDM容量限制时,分块策略需在细粒度层面优化。当输入形状(m,n,d)和硬件参数(最小计算单元p,q,r及LDM空间M)固定时,从核拆分策略(k1,k2,k3)在32从核下仅有21种组合可能。循环分块策略(l1,l2,l3)受LDM空间约束取值有限,通过穷举所有排列组合,计算对应推理访存量并选取最小值,即可获得3级并行架构下以访存最优为目标的分块策略。如式(7)所示,该优化问题即为在多重硬件限制中的全局最优解。
$ \mathrm{a}\mathrm{r}\mathrm{g}\;\mathrm{ }\mathrm{m}\mathrm{i}\mathrm{n}\frac{mnd}{64}\left(\frac{1}{{l}_{1}}+\frac{1}{{l}_{3}}\right)+\gamma \frac{mn}{{2k}_{1}{k}_{3}}\left({k}_{2}-1\right)\text{,} $
式(6)和式(7)中$ \gamma $表示从核间数据同步速率与本地HBM访存带宽的比值,与具体的访存行为相关,包括数据量、数据跨步情况、同步核数、参与同步的从核的物理位置等。$ \gamma $一般指DMA广播行为与RMA行为速率的比值。
在上述LDM优先考量和最小化访存量之后,需进一步实现带宽优化与脉动阵列计算流的深度绑定。本文基于LDM私有化设计,通过DMA广播和流水操作实现硬件资源最大化利用。如图6(b)所示,采用DMA广播代替GPU的2级缓存的透明机制,实现核间数据复用;图6(c)通过双缓冲流水中的RMA广播同步数据,取代GPU共享内存操作。
为实现高效通信和复用相同子矩阵,采用行多播和列多播模式DMA广播操作。该操作需结合硬件层面上的数据传输逻辑,合理设计广播行为。例如,在4行8列的广播策略下,如图6(b)所示,17号从核的X2数据由16号从核行广播提供,W1由9号从核列广播提供,以实现数据复用并减少单从核模式导致的重复访存。硬件层面通过通道单向传输约束和动态起始点选择,例如交错选择列广播起始点,进一步优化数据传输效率。
为实现高效的计算访存流水,使用双缓冲机制协调独立计算与同步需求。如图6(c)所示,在非同步场景下,采用非阻塞DMA实现计算与预取的并行化:首个访存结束后立即启动计算,同时预取下一任务数据。在同步场景下,部分从核需通过RMA完成累加同步:非同步核保持独立流程,仅将存回操作替换为RMA传输;同步核需等待关联从核RMA完成后方可执行累加。每个等待队列元素的空间占用通常为最小计算单元的大小。通过时间重叠与操作解耦,该机制实现了计算的连续性并降低了同步开销。
在上述分析的基础上,算法1给出了完整的基于LDM优先的启发式分块并行过程。算法1的行①~③根据从核拆分因子将矩阵初步拆分,行④~⑰为内部循环分块过程,其中通过2个缓冲区交替预取和DMA异步传输来实现双缓冲流程;通过脉动阵列计算每个小块;根据不同情况通过RMA同步结果或是直接异步DMA返回结果。
算法1. LDM优先的启发式分块并行策略。
输入: 矩阵X维度[m, d],矩阵W维度[d, n], 从核拆分因子k1,k2,k3,循环分块因子l1,l2,l3;
输出:矩阵Y维度[m, n]。
术语定义:下标 part 指矩阵2级分块状态,block 指3级分块状态;ldm 标识 LDM 空间数据;next 表征双缓冲机制;Queue为任务等待队列。
① for 对每个核组并行执行 do
② for 对组内每个从核并行执行do
③  将输入矩阵拆分为局部块Xpart[m/k1][d/k2], Wpart[d/k2][n/4k3];
④   for网格划分范围(m /(k1×l1×p), n /(k3× l3×q))内遍历i×j do
⑤    初始化Yldm为全0;
⑥    DMA异步传输(Xblock,Wblock→Xldm, Wldm);
⑦    for 对维度方向分块(d/(k2×l2×r))遍历 k do
⑧     等待DMA传输完成;
⑨     异步预取下一块(Xnext_block, Wnext_block→Xnext_ldm,Wnext_ldm);
⑩     矩阵乘计算(Xldm,Wldm→Yldm);
⑪     if 该从核进行同步 do
⑫      RMA规约(Yldm, Queuesync → Yblock);
⑬     else
⑭      DMA异步传输(Yldm,Yblock);
⑮     end if
⑯      交换地址(Xldm,Wldm↔ Xnext_ldm, Wnext_ldm);
⑰    end for
⑱   end for
⑲ end for
⑳ end for
3.3节提出的启发式策略通过将大规模矩阵运算拆分到从核上进行分块并行运算,有效缓解了计算压力,并在Transformer流程中展现出一定的适用性,但访存受限问题仍是制约性能提升的关键因素。为此,本节提出一种与申威架构紧耦合的注意力机制SWAI-Attention,该机制严格遵循两大核心原则,对Transformer推理过程进行了深度适配与优化:一是减少访存通信,通过优化数据存储和访问模式降低访存开销;二是适配硬件架构,通过指令优化和并行化策略,充分发挥申威架构的计算能力。通过这种软硬件协同优化的方式,进一步解决访存瓶颈问题。
SWAI-Attention借鉴了Flash-Attention的设计思路,以多头注意力机制算子为基础。通过算子融合技术,在并行策略的驱动下将中间操作符合并至单个从核内。这一过程通过引入少量线性存储开销,避免了中间结果的反复读写,直接计算最终结果,从而显著减少了冗余的访存开销。
在此基础上,进一步结合申威硬件特性,针对Attention算子中的超越函数exp展开优化。exp函数涉及复杂的指数运算,不适合单指令多数据(single instruction multiple data,SIMD)处理的除法运算,导致计算和访存开销较大。本节通过并行分段线性查表法、并行指令硬件适配和寄存器汇编控制3个步骤优化非线性函数,有效解决了非线性函数在计算和访存上的高开销问题。具体优化措施如下:
1)并行分段线性查表法。采用并行分段线性查表法,通过2张表分别记录线段的斜率和截距以及通过乘加运算拟合非线性函数,如图7的步骤1优化HBM访存开销,显著减少了对HBM的读写操作,降低了访存开销。
2)并行指令硬件适配。利用申威架构的并行查表汇编指令,结合SIMD特性,实现单条指令同时处理多个数据的计算合并。这一优化贯穿于图7的步骤1和步骤2,显著提升了计算效率。
3)寄存器汇编控制。通过多寄存器控制数据流向,避免批处理查表操作中的冗余表加载和寄存器读取,如图7所示,步骤2优化LDM访存开销,确保数据不相互依赖并实现寄存器间的并行计算。
针对申威架构中online Softmax算子难以利用SIMD并行化的难题,提出了一种整体转置算法,有效解决了访存效率低下的问题,并显著提升了计算性能。
在传统实现中,计算矩阵每一行的最大值时,理想的并行化方案是利用SIMD一次性装载n行的某一列元素,然后逐列遍历n行所有列的元素,从而实现n行的rowmax并行计算。然而,如图8(a)所示,矩阵在HBM中默认按行主序存储,这与按列装载的要求存在冲突,导致rowmax算子难以高效利用SIMD并行化。此外,主存DMA不支持转置取数,进一步限制了矩阵原地快速转置的实现。
为解决这一问题,本节提出了一种整体转置算法。如图8(b)所示,该算法在不改变注意力模块核心计算逻辑的前提下,仅将中间结果以转置形式呈现,并通过DMA put的转置存储功能将最终输出存回HBM。通过这种方式,将rowmax算子转换为colmax算子,使得行向量在LDM中连续存储,从而满足硬件的装载要求,并能够高效利用SIMD进行并行计算。
本节对SW-Inf的关键技术和整体优化效果进行实验验证与分析。测试硬件环境为1台型号 NF5468M5 的浪潮英信服务器,其 PCIe 插槽上分别安装一块 Tesla-V100S 32 GB加速卡,NVIDIA -A100 40 GB PCle加速卡和1块申威智能加速卡,以确保主机侧环境一致性。Tesla-V100S和NVIDIA -A100的CUDA版本号为12.2,申威智能加速卡的SDAA版本号为1.1。
本节对 SW-Inf 推理系统的性能进行整体评估,以 LLaMA2-7B Instruct Inference 为评估对象,并在V100S,A100 和申威加速卡上对4个推理系统进行对比测试,包括一个标准推理系统(面向GPU的Meta-LLaMA(git commit:b8348da))、一个申威加速卡上直接移植cuda生态后的推理系统(Vanilla),以及2个高度优化的先进推理系统:面向GPU的VLLM(版本为0.6.1)和本文提出的SW-Inf。这种对比设置能够更直观地反映在一定硬件性能差距的情况下不同推理系统之间的性能和效率差异。
实验的参数设置如下:batch size为1,上下文长度分别设置为50→500,500→1000,1000→2000,其中m→n表示提示词长度为m,提示词与完整回答总长度为n,m<n。主要选取吞吐率TPS(tokens per second)和主存带宽利用率(memory bandwidth usage,MBU)作为测试指标。MBU定义如下:
$ {M}{B}{U}=\left(\frac{(\mathrm{模}\mathrm{型}\mathrm{参}\mathrm{数}\mathrm{大}\mathrm{小}+\mathrm{K}\mathrm{V}\;\mathrm{ }\mathrm{C}\mathrm{a}\mathrm{c}\mathrm{h}\mathrm{e})/{T}{P}{O}{T}}{\mathrm{峰}\mathrm{值}\mathrm{访}\mathrm{问}\mathrm{带}\mathrm{宽}}\right)\text{,} $
其中TPOT(time per output token)是单个token的生成时间。TPS表示系统除首词外在单位时间内输出的词元数目,用于衡量推理系统基于特定硬件平台的实际性能,而MBU用于衡量推理系统能否充分利用硬件主存带宽的能力。
由图9和图10所示,SW-Inf在性能上展现出显著优势。相比于直接移植的Vanilla系统,SW-Inf的TPS提升1.83倍,MBU都提升35%以上;与在V100S,A100上运行的典型推理系统Meta-LLaMA相比,SW-Inf的TPS提升2倍,MBU都提升35%以上;相比于V100S上最先进的推理系统VLLM,在3种不同的上下文长度下,SW-Inf的TPS提升了1.12倍,MBU都提升15%以上。此外,尽管SW-Inf的TPS受限于主存访问带宽不如运行在A100上的VLLM。但在3种不同的上下文长度下,MBU都提升9%以上。
实验结果表明,尽管申威加速卡在硬件性能上与V100S和A100存在差距,尤其是在主存带宽方面,SW-Inf通过综合应用3种针对性优化策略,成功弥补了硬件性能的不足,基于申威智能加速卡实现了较高的推理吞吐率,在某些场景下的推理性能甚至超过了基于V100S的先进推理系统。这一结果不仅证明了SW-Inf在资源受限条件下的高效性,还为国产硬件在大模型推理领域的应用提供了有力支持,展示了其在实际部署中的巨大潜力。
为验证整图下沉调度机制对申威智能加速卡架构推理性能的优化效果,本节以主机端与设备端的交互次数和交互总耗时(具体涉及PCIe传输)为核心指标,设计了对照实验。整图下沉前的对照组采用传统的实时交互模式,控制流程由主机端主导;下沉后的实验组则采用整图下沉调度,通过计算图打包与片上主核调度优化流程。
由表2可见,整图下沉优化方案带来了显著增益:1)交互次数显著降低,单次词生成的交互次数从平均千次级降至固定的2次,包括1次预处理和1次词元解码;2)总耗时大幅减少,下沉后总耗时较传统模式减少1~2个数量级,当词数为500~1500时,耗时从11.75 s降至84.44 ms,效率提升了139倍。
实验表明,整图下沉调度通过计算图预编译与片上主核自主调度,将动态控制逻辑从主机端迁移至设备端,有效减少跨端交互频率。同时,利用申威架构的硬件并行性掩盖数据搬运延迟,为大规模语言模型推理提供了低延迟保障。
本节以矩阵乘运算为例评估多级并行计算策略的有效性,主要验证最小化LDM访存量的启发式策略能否搜索出高效并行方式。测试矩阵规模(m,d,n)为(2048,4096,4096)或(4096,4096,8192)两种典型维度的矩阵乘法,最小计算单元(p,r,q)设置为(128,32,32),可用的本地高速缓冲空间M=200 KB,并统一使用DMA/RMA广播机制优化数据传输与同步来对比不同并行分块方式的访存量和实际耗时。
如表3所示,在同一矩阵形状下,拆分和分块后的访存量越小,实际的耗时往往也越小。在相近的总访存量时,涉及需要RMA同步的流程往往对应的平均带宽要比核组间独立计算的流程要低,实际在处理RMA同步问题时,往往同步会在一定程度上影响流水操作。最后表明在实际场景,尽管不同传输模式有速率差异比,但总体还是以访存量为推理流程性能的主要因素,在多级并行策略下,往往选取最小化访存量的情况为实际最优结果。
本节实验讨论注意力机制在与申威体系结构紧耦合前后的性能变化,重点对非线性函数优化和整体转置优化进行了消融实验,并最终进行了整体性能测试。
1)非线性函数优化实验
本节验证了非线性函数优化对算子性能的提升效果,主要测试了softmax算子中的指数函数exp。表4给出了在单核组相同输入规模下3种exp实现方法的性能对比:MathLib基于C语言标准数学库的标量计算实现;TBLExp采用并行分段线性查表法,并结合并行查表指令,以适配硬件加速计算;FusedExp在TBLExp基础上,进一步结合寄存器汇编控制,减少寄存器交互,合并指令层的计算,从而提升整体效率。
由表4可见,在多种输入规模下,MathLib的计算耗时显著高于其他2种方法。这是由于C语言标准数学库中的函数实现主要是为了保证通用性,未针对硬件特性进行优化。相比之下,TBLExp通过分段线性拟合和硬件并行查表指令,大幅减少了寄存器和LDM之间的数据传输。而FusedExp进一步优化了寄存器分配和指令流水,根据单从核计算任务指定所需寄存器,从而更有效地实现指令流水化和平均性能提升,且在某些场景下表现更优。
2)整体转置实验
本节验证了整体转置算法在注意力机制中对max相关计算的性能提升效果。由于分块策略的约束,矩阵S形状由矩阵Q和K的分块决定,例如,Q分块为(192, 128),K的转置为(128, 32),此时S为(192, 32),结果向量M为(192, 1),如表5所示,实验覆盖了3种典型分块对应的形状,并测试了max步骤的完整流程,包括读取、计算和存回。
SIMD向量化指令实现了多元素的并行计算,显著提升了计算效率。如表5所示,在不同矩阵形状下,colmax相比于rowmax的加速比可达124~172。特别的,在矩阵形状为(32, 32)时,计算耗时从24.86 μs显著降低至0.20 μs。此外,随着矩阵列数的增加,加速比进一步提升,充分体现了SIMD向量化规模的优势。实验结果表明,整体转置算法通过将rowmax转换为colmax,显著适配了申威架构的SIMD硬件特性,为注意力机制的分块策略提供了坚实的硬件环境。
3)MHA算子整体实验
本节实验将SWAI Attention 与文献[12]中的FlashAttention在Tesla-A100 40 GB下的多种主流注意力机制,包括PyTorch Attention,Megatron,Local Attention,Longformer Attention,FlashAttention进行性能对比,并且这些主流注意力机制数据与原论文一致。旨在衡量在计算能力相当的情况下,访存带宽存在一定差距时,SWAI Attention 在缓解访存瓶颈方面的效果。评估对象为注意力模块的单次正向计算。实验参数设置如下:批样本数为16,注意力头数目为8,注意力头维度为64,且不进行mask和dropout处理。
如表6所示,尽管申威智能加速卡在访存带宽上相对A100存在较大差距,导致其在长句长任务中的性能表现劣于对比方案。但在短句长场景下,SWAI Attention的计算耗时显著低于其他方法。特别是在句长为256时,SWAI Attention在SW-AI架构上的耗时相较于Tesla-A100上的先进方法FlashAttention减少了50%,充分证明了体系结构紧耦合的注意力计算方法的有效性。
本节主要开展消融实验,探究本文所提的3个优化方法对总体推理性能给出了不同情况下的端到端推理TPS和相对于基准方法的性能提升比。其中“原始”场景与4.1节的Vanilla意义相同,表示申威平台上未经优化的推理系统,“+1”表示在Vanilla基础上进行整图下沉优化,“+2”表示在Vanilla基础上使用LDM优先的多级并行优化, “+3”表示在Vanilla基础上进行SWAI-Attention优化,按此记号记录不同组合下的TPS和性能提升比。
由图11可见,3种优化方法在单独引入时可带来60%~80%的性能提升,其中,多级并行策略在减少总体访存量的同时实现负载均衡,充分利用所有从核的计算能力,构成性能提升的核心基础;整图下沉与注意力优化通过降低数据搬运开销和计算冗余度进一步提高性能,最终从存储、计算、通信3个层面进行协同优化,总体可带来超过120%的端到端推理TPS提升。
本节验证所提方法在不同模型和参数规模的情况下的适应性,针对Transformer模型的核心模块MHA和FFN,测试LLaMA和Qwen这2类模型,共4种参数规模下MHA和FFN解码阶段的推理耗时。
如表7所示,模型参数规模(隐藏层维度从2048扩展至8192)与多头注意力机制的结构复杂度(注意力头数从16增至64)的同步提升,会导致FFN和Attention的计算时延显著增长,其运行耗时分别增长12.6倍与13.3倍,可见SW-Inf推理耗时与Transformer规模成近线性关系,随着模型规模增大,各核心模块的算子规模增大,如GEMM等关键算子的计算访存比提升及数据复用度增大,如2.3节分析,由于申威智能加速卡在端侧推理场景下存在计算能力溢出的现象,因此模型规模的增长不会导致实际推理耗时等比例增长。
申威智能加速卡的众核架构、主从核特性以及丰富的片上计算资源和存储层次,为端侧大模型智能应用的性能优化提供了坚实基础,同时也对推理系统设计提出了新的挑战。本文提出了一种基于申威智能加速卡的Transformer高效推理方法SW-Inf,通过整图下沉调度机制减少主机与设备间的频繁交互开销,设计多级并行计算策略充分利用片上所有计算资源以及利用深度算子融合技术缓解推理过程中的访存瓶颈问题。实验结果表明,与英伟达V100S上的先进推理系统vLLM相比,SW-Inf在推理吞吐率和主存利用率方面均取得了显著提升。这一成果不仅验证了本文提出的3种针对性优化策略的有效性,更凸显出国产硬件在边缘侧大模型推理部署中的广阔前景和巨大潜力。本文工作主要面向单卡端侧推理场景,在未来的工作中,将进一步研究面向推理服务系统和模型训练场景的软硬协同优化方法与技术,以充分发挥申威加速卡的潜力,推动其在更广泛场景中的应用。
作者贡献声明:沈文渊参与理论分析、优化全过程和整体代码编写以及撰写论文;汤凌韬负责研究构思设计与理论分析,完成跨平台性能测试和论文撰写;何昊天、费天成撰写论文、制作图表和修订关键段落;赵玉龙、张鲁飞指导研究框架和协调实验设备;秦晓军、刘鑫和陈左宁负责质量控制、数据复核和方法论改进。

参考文献 引证文献
排序方式:
1
Yu G, Jeong S, Kim G, et al. Orca: A distributed serving system for Transformer-based generative models[C]//Proc of the 16th Symp on Operating Systems Design and Implementation. Berkeley, CA: USENIX Association, 2022: 521−538
2
Frantar E, Alistarh D. SparseGPT: Massive language models can be accurately pruned in one-shot[J]. arXiv preprint, arXiv: 2301.00774, 2023
3
Liu Zichang, Wang Jue, Dao T, et al. Deja Vu: Contextual sparsity for efficient LLMs at inference time[C]//Proc of the 40th Int Conf on Machine Learning. Cambridge, MA: PMLR, 2023: 919
4
王笑尘, 张坤, 张鹏. 多视角看大模型安全及实践[J]. 计算机研究与发展, 2024, 61(5): 1104−1112
Wang Xiaochen, Zhang Kun, Zhang Peng. Multi-perspective view of large model security and practice[J]. Journal of Computer Research and Development, 2024, 61(5): 1104−1112 (in Chinese)
5
Nallapati R, Zhou B, dos Santos C, et al. Abstractive text summarization using sequence-to-sequence RNNs and beyond[C]//Proc of the 20th SIGNLL Conf on Computational Natural Language Learning. Stroudsburg, PA: ACL, 2016: 280−290
6
Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate[J]. arXiv preprint, arXiv: 1409.0473, 2014
7
Adiwardana D, Luong M T, So D R, et al. Towards a human-like open-domain chatbot[J]. arXiv preprint, arXiv: 2001.09977, 2020
8
Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 30: 5998−6008
9
Kim S, Hooper C, Wattanawong T, et al. Full stack optimization of Transformer inference: A survey[J]. arXiv preprint, arXiv: 2302.14017, 2023
10
Samuel W, Andrew W, David P. Roofline: An insightful visual performance model for multicore architectures[J]. Communications of the ACM, 2009, 52(4): 65−76
11
Zheng Lianmin, Yin Liangsheng, Xie Zhiqiang, et al. SGLang: Efficient execution of structured language model programs[J]. Advances in Neural Information Processing Systems, 2024, 37: 62557−62583
12
Dao T, Fu D, Ermon S, et al. FlashAttention: Fast and memory-efficient exact attention with IO-awareness[J]. Advances in Neural Information Processing Systems, 2022, 35: 16344−16359
13
DeepSeek-AI. DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model[J]. arXiv preprint, arxiv: 2405.04434, 2024
14
Kwon W, Li Zhuohan, Zhuang Siyuan, et al. Efficient memory management for large language model serving with PagedAttention[C]//Pro of the 29th Symp on Operating Systems Principles. New York: ACM, 2023: 611–626
15
Agrawal A, Kedia N, Panwar A, et al. Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve[C]//Proc of the 18th Symp on Operating Systems Design and Implementation. Berkeley, CA: USENIX Association, 2024: 117−134.
16
Sheng Ying, Zheng Lianmin, Yuan Binhang, et al. FlexGen: High-throughput generative inference of large language models with a single GPU[C]//Proc of the Int Conf on Machine Learning. Cambridge, MA: PMLR, 2023: 31094−31116
17
Song Yixin, Mi Zeyu, Xie Haotong, et al. PowerInfer: Fast large language model serving with a consumer-grade GPU[C]//Proc of the 30th ACM SIGOPS Symp on Operating Systems Principles. New York: ACM, 2024: 590−606
18
Michel P, Levy O, Neubig G. Are sixteen heads really better than one?[J]. Advances in Neural Information Processing Systems, 2019, 32: 14014−14024
19
Bansal H, Gopalakrishnan K, Dingliwal S, et al. Rethinking the role of scale for in-context learning: An interpretability-based case study at 66 billion scale[J]. arXiv preprint, arXiv: 2212.09095, 2022
20
Alizadeh K, Mirzadeh I, Belenko D, et al. LLM in a flash: Efficient large language model inference with limited memory[C]//Proc of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Stroudsburg, PA: ACL, 2024: 12562−12584
21
GGML-org. LLaMA. cpp[EB/OL]. [2024-10-01]. https://github.com/ggml-org/llama.cpp
22
Jiang Xiaotang, Wang Huan, Chen Yiliu, et al. MNN: A universal and efficient inference engine[C]//Proc of Machine Learning and Systems. Miami, FL: MLSys Foundation, 2020: 1−3
23
Wu Ruohan, Zhu Xianyu, Chen Junshi, et al. SWattention: Designing fast and memory-efficient attention for a new Sunway supercomputer[J]. The Journal of Supercomputing, 2024, 80(10): 13657−13680
24
刘锟阳. 基于寒武纪MLU270的YOLOv5模型优化与应用研究[D]. 成都: 四川师范大学, 2023
Liu Kunyang. Optimization and application of YOLOv5 model based on Cambrian MLU270[D]. Chengdu: Sichuan Normal University, 2023 (in Chinese).
25
Ma Zixuan, He Jiaao, Qiu Jiezhong, et al. BaGuaLu: Targeting brain scale pretrained models with over 37 million cores[C]//Proc of the 27th ACM SIGPLAN Symp on Principles and Practice of Parallel Programming. New York: ACM, 2022: 192−204
26
Shah J, Bikshandi G, Zhang Ying, et al. FlashAttention-3: Fast and accurate attention with asynchrony and low-precision[J]. Advances in Neural Information Processing Systems, 2024, 37: 68658−68685
27
赵玉龙, 张鲁飞, 许国春, 等. SDAA: 面向申威智能加速卡的运行时系统[J]. 软件学报, 2024, 35(12): 5710−5724
Zhao Yulong, Zhang Lufei, Xu Guochun, et al. SDAA: Runtime system for Sunway intelligent acceleration card[J]. Journal of Software, 2024, 35(12): 5710−5724 (in Chinese)
2026年第63卷第3期
218
102
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202550153
  • 接收时间:2025-03-01
  • 首发时间:2026-11-06
  • 出版时间:2026-03-01
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2025-03-01
  • 修回日期:2025-09-24
  • 录用日期:2025-10-11
基金
作者信息
    1数学工程与先进计算国家重点实验室 江苏无锡 214125
    2国家并行计算机工程技术研究中心 北京 100083

通讯作者:

刘鑫()
参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202550153
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏