Article(id=1329420141516927786, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1329420000110161984, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202550301, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=null, receivedDate=1745769600000, receivedDateStr=2025-04-28, revisedDate=1770307200000, revisedDateStr=2026-02-06, acceptedDate=1770652800000, acceptedDateStr=2026-02-10, onlineDate=1794888061169, onlineDateStr=2026-11-17, pubDate=1782835200000, pubDateStr=2026-07-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1794888061169, onlineIssueDateStr=2026-11-17, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1794888061169, creator=13701087609, updateTime=1794888061169, updator=13701087609, issue=Issue{id=1329420000110161984, tenantId=1146029695717560320, journalId=1324759914146529478, year='2026', volume='63', issue='7', pageStart='1639', pageEnd='1903', issueExtLink='null', onlineDate='null', pubDate='1782835200000', pubDateStr='2026-07-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1794888027455, creator='13701087609', updateTime=1794892162905, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext={EN=IssueExt(id=1329437345918595829, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1329420000110161984, language=EN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=), CN=IssueExt(id=1329437345918595830, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1329420000110161984, language=CN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=)}, issueFiles=null, downloadFileDto=null}, startPage=1816, endPage=1830, ext={EN=ArticleExt(id=1329420141718254380, articleId=1329420141516927786, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=Locality-Aware Decoding Accelerator for Large Language Models Based on Hybrid Bonding Architecture, columnId=1325489901852041729, journalTitle=Journal of Computer Research and Development, columnName=Computer Architecture, runingTitle=null, highlight=null, articleAbstract=

Large language models (LLMs) have demonstrated remarkable capabilities across various domains, including natural language processing, computer vision, chat bots and content generation. Deploying LLMs on edge devices has attracted significant attention due to its advantages in service stability and data privacy. However, the decoding inference of LLMs is composed of memory-bound operations, and with model sizes growing and complex tasks demanding increasingly longer context lengths, the resulting memory access overhead becomes critical bottleneck limiting the deployment of LLMs on edge platforms. To lower the memory access demand of long context decoding, locality aware attention mechanism is proposed. By leveraging the numerical locality of attention scores during decoding, this mechanism compresses the key-value cache (KV cache), significantly reducing redundant memory access without compromising model accuracy. We present a locality-aware inference accelerator for LLMs with hybrid-bonded 3D DRAM. The accelerator not only reduces the overall memory traffic but also significantly improves the main memory bandwidth. It integrates load-balancing and prediction-based prefetching mechanisms tailored to the structural characteristics of hybrid-bonded 3D DRAM, thereby enhancing the bandwidth utilization. On the evaluated models, the proposed accelerator achieves on average 1971 token/s, 3752 token/s and 6849 token/s throughputs for batch sizes of 1, 2 and 4, respectively, with corresponding energy efficiencies of 0.23 J/token, 0.12 J/token and 0.07 J/token. The achieved throughput corresponds to 50.5 times, 49.4 times, 50.0 times that of the A100 GPU, and 23.7 times, 22.9 times, 21.5 times that of the LAD-HBM accelerator. Meanwhile, the energy consumption amounts to only 5.1%, 5.1%, 4.9% of GPU, and 46.2%, 46.9%, 48.2% of LAD-HBM.

, authors=Haoran Wang1, 2, 3, Haobo Xu1, 2, *, Ying Wang1, 2, Yinhe Han1, 2, authorsList=Haoran Wang, Haobo Xu, Ying Wang, Yinhe Han, authorCompany=null, correspAuthors=Haobo Xu, authorNote=

born in 1997. PhD. His main research interests include computer architecture, large language model system and accelerator design, and memory architecture based on advanced packaging

born in 1990. PhD, associate professor. Member of CCF. His main research interests include domain-specific processor chips and 2.5D/3D chiplet-based integrated chips

born in 1985. PhD, professor. Senior member of CCF. His main research interests include dedicated processor architecture and integrated chip systems

born in 1980. PhD, professor. Senior member of CCF. His main research interests include computer architecture and chips, and data computing chips and computing systems

, correspAuthorsNote=null, copyrightStatement=Copyright © 2026 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1329420145052726089, articleId=1329420141516927786, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=基于混合键合架构的大模型局部性感知解码加速器, columnId=1325489904427344387, journalTitle=计算机研究与发展, columnName=体系结构, runingTitle=null, highlight=null, articleAbstract=

大模型在自然语言处理、计算机视觉、聊天机器人以及内容生成等多个领域中展现出了优秀的能力。大模型在边缘设备上的部署因其在稳定性和数据安全方面的优势受到广泛关注。然而,大模型的解码推理过程主要由访存受限的算子构成,随着模型规模的扩展以及复杂任务对上下文长度的需求持续增长,访存开销成为了限制大模型在边缘设备上部署的关键瓶颈。为了降低大模型在长上下文解码时的访存开销,局部性感知的注意力机制被提出。该机制基于注意力分数在解码过程中的数值局部性,对键值缓存(KV cache)进行压缩,实现了在不影响模型精度的情况下,有效削减了冗余的访存。设计了一种基于混合键合3维DRAM的局部性感知大模型推理加速器,在有效降低总体访存量的同时显著提升了主存带宽。该加速器集成了负载均衡与基于预测的预取机制,来适配混合键合3维DRAM的结构特性,从而提升带宽利用率。在所评估的模型上,该加速器在批处理大小为1,2,4时,平均实现了分别为1971 token/s,3752 token/s,6849 token/s的吞吐,对应的能耗分别为0.23 J/token,0.12 J/token,0.07 J/token。该吞吐相当于A100 GPU的50.5倍、49.4倍、50.0倍,LAD-HBM加速器的23.7倍、22.9倍、21.5倍,而能耗仅相当于GPU的5.1%,5.1%,4.9%,LAD-HBM的46.2%,46.9%,48.2%。

, authors=王浩然1, 2, 3, 许浩博1, 2, *, 王颖1, 2, 韩银和1, 2, authorsList=王浩然, 许浩博, 王颖, 韩银和, authorCompany=null, correspAuthors=许浩博, authorNote=

作者贡献声明

王浩然设计了大模型局部性感知解码的混合键合加速器,并进行了性能和功耗实验;许浩博指导了实验的细节,搜集了混合键合DRAM的架构参数,制定了论文写作大纲;王颖、韩银和指导了论文总体的技术方向和实验方法。

1997年生。博士。主要研究方向为计算机体系结构、大模型系统与加速器设计、基于先进封装的存储架构

1990年生。 博士,副研究员。CCF会员。 主要研究方向为领域专用处理器芯片、2.5维/3维芯粒集成芯片

1985年生。 博士,研究员。CCF高级会员。 主要研究方向为专用处理器体系结构、集成芯片系统

1980年生。博士,研究员。CCF高级会员。主要研究方向为计算机体系结构和芯片、数据计算芯片和计算系统

, correspAuthorsNote=
许浩博()
, copyrightStatement=版权所有 © 2026 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=+lj7kq0+CwErxpt4JUs2dQ==, magXml=dYjpM5zQOYYzdBnFYDbV4w==, pdfUrl=null, pdf=ujuB3kTb5hg1Lw1yPPBxLQ==, pdfFileSize=1638638, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=0FMbM8ox6RYuMLWnOfzMxw==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=CqMeQ2kQt7M9r1/pYor6Vw==, mapNumber=null, fund=null)}, authors=[Author(id=1329420145597985620, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=wanghaoran20g@ict.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329420145686066008, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, authorId=1329420145597985620, language=EN, stringName=Haoran Wang, firstName=Haoran, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, 3, address=1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
3University of Chinese Academy of Sciences, Beijing 100190, bio={"content":"

born in 1997. PhD. His main research interests include computer architecture, large language model system and accelerator design, and memory architecture based on advanced packaging

"}, bioImg=null, bioContent=

born in 1997. PhD. His main research interests include computer architecture, large language model system and accelerator design, and memory architecture based on advanced packaging

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329420145753174873, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, authorId=1329420145597985620, language=CN, stringName=王浩然, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, 3, address=1中国科学院计算技术研究所智能计算机研究中心 北京 100190
2处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190
3中国科学院大学 北京 100190, bio={"img":"DbpS4j7Vw7jgQxoLVSWDPQ==","content":"

1997年生。博士。主要研究方向为计算机体系结构、大模型系统与加速器设计、基于先进封装的存储架构

"}, bioImg=DbpS4j7Vw7jgQxoLVSWDPQ==, bioContent=

1997年生。博士。主要研究方向为计算机体系结构、大模型系统与加速器设计、基于先进封装的存储架构

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329420145308578634, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=1, ext=[AuthorCompanyExt(id=1329420145316967243, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145308578634, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1329420145329550156, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145308578634, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所智能计算机研究中心 北京 100190)]), AuthorCompany(id=1329420145409241933, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=2, ext=[AuthorCompanyExt(id=1329420145413436238, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145409241933, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1329420145421824847, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145409241933, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)]), AuthorCompany(id=1329420145493128016, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=3, ext=[AuthorCompanyExt(id=1329420145505710929, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145493128016, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3University of Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1329420145514099538, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145493128016, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3中国科学院大学 北京 100190)])]), Author(id=1329420145841255259, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=xuhaobo@ict.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329420145912558430, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, authorId=1329420145841255259, language=EN, stringName=Haobo Xu, firstName=Haobo, middleName=null, lastName=Xu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, *, address=1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190, bio={"content":"

born in 1990. PhD, associate professor. Member of CCF. His main research interests include domain-specific processor chips and 2.5D/3D chiplet-based integrated chips

"}, bioImg=null, bioContent=

born in 1990. PhD, associate professor. Member of CCF. His main research interests include domain-specific processor chips and 2.5D/3D chiplet-based integrated chips

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329420145971278687, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, authorId=1329420145841255259, language=CN, stringName=许浩博, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, *, address=1中国科学院计算技术研究所智能计算机研究中心 北京 100190
2处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190, bio={"img":"KymBk6OMJcAe0ujAkmaO6w==","content":"

1990年生。 博士,副研究员。CCF会员。 主要研究方向为领域专用处理器芯片、2.5维/3维芯粒集成芯片

"}, bioImg=KymBk6OMJcAe0ujAkmaO6w==, bioContent=

1990年生。 博士,副研究员。CCF会员。 主要研究方向为领域专用处理器芯片、2.5维/3维芯粒集成芯片

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329420145308578634, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=1, ext=[AuthorCompanyExt(id=1329420145316967243, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145308578634, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1329420145329550156, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145308578634, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所智能计算机研究中心 北京 100190)]), AuthorCompany(id=1329420145409241933, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=2, ext=[AuthorCompanyExt(id=1329420145413436238, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145409241933, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1329420145421824847, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145409241933, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)])]), Author(id=1329420146038387553, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329420146134856548, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, authorId=1329420146038387553, language=EN, stringName=Ying Wang, firstName=Ying, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190, bio={"content":"

born in 1985. PhD, professor. Senior member of CCF. His main research interests include dedicated processor architecture and integrated chip systems

"}, bioImg=null, bioContent=

born in 1985. PhD, professor. Senior member of CCF. His main research interests include dedicated processor architecture and integrated chip systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329420146252297061, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, authorId=1329420146038387553, language=CN, stringName=王颖, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1中国科学院计算技术研究所智能计算机研究中心 北京 100190
2处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190, bio={"img":"i+msuHgF0Fox8T3IUgeEMA==","content":"

1985年生。 博士,研究员。CCF高级会员。 主要研究方向为专用处理器体系结构、集成芯片系统

"}, bioImg=i+msuHgF0Fox8T3IUgeEMA==, bioContent=

1985年生。 博士,研究员。CCF高级会员。 主要研究方向为专用处理器体系结构、集成芯片系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329420145308578634, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=1, ext=[AuthorCompanyExt(id=1329420145316967243, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145308578634, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1329420145329550156, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145308578634, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所智能计算机研究中心 北京 100190)]), AuthorCompany(id=1329420145409241933, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=2, ext=[AuthorCompanyExt(id=1329420145413436238, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145409241933, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1329420145421824847, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145409241933, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)])]), Author(id=1329420146365543271, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=yinhes@ict.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329420146449429354, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, authorId=1329420146365543271, language=EN, stringName=Yinhe Han, firstName=Yinhe, middleName=null, lastName=Han, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190, bio={"content":"

born in 1980. PhD, professor. Senior member of CCF. His main research interests include computer architecture and chips, and data computing chips and computing systems

"}, bioImg=null, bioContent=

born in 1980. PhD, professor. Senior member of CCF. His main research interests include computer architecture and chips, and data computing chips and computing systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329420146533315435, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, authorId=1329420146365543271, language=CN, stringName=韩银和, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1中国科学院计算技术研究所智能计算机研究中心 北京 100190
2处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190, bio={"img":"mu9hqS/N2jwq9LFklnR2ng==","content":"

1980年生。博士,研究员。CCF高级会员。主要研究方向为计算机体系结构和芯片、数据计算芯片和计算系统

"}, bioImg=mu9hqS/N2jwq9LFklnR2ng==, bioContent=

1980年生。博士,研究员。CCF高级会员。主要研究方向为计算机体系结构和芯片、数据计算芯片和计算系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329420145308578634, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=1, ext=[AuthorCompanyExt(id=1329420145316967243, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145308578634, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1329420145329550156, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145308578634, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所智能计算机研究中心 北京 100190)]), AuthorCompany(id=1329420145409241933, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=2, ext=[AuthorCompanyExt(id=1329420145413436238, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145409241933, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1329420145421824847, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145409241933, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)])])], keywords=[Keyword(id=1329420146633978732, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=EN, orderNo=1, keyword=large language model), Keyword(id=1329420146701087597, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=EN, orderNo=2, keyword=auto-regressive decoding), Keyword(id=1329420146772390766, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=EN, orderNo=3, keyword=hybrid bonding), Keyword(id=1329420146847888239, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=EN, orderNo=4, keyword=accelerator), Keyword(id=1329420146931774320, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=EN, orderNo=5, keyword=edge device), Keyword(id=1329420147007271793, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=CN, orderNo=1, keyword=大模型), Keyword(id=1329420147082769266, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=CN, orderNo=2, keyword=自回归解码), Keyword(id=1329420147162461043, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=CN, orderNo=3, keyword=混合键合), Keyword(id=1329420147242152820, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=CN, orderNo=4, keyword=加速器), Keyword(id=1329420147309261685, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=CN, orderNo=5, keyword=边缘设备)], refs=[Reference(id=1329420149863592848, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 30: 6000−6010, articleTitle=null, refAbstract=null), Reference(id=1329420149922313105, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Brown T, Mann B, Ryder N, et al. Language models are few-shot learners[J]. Advances in Neural Information Processing Systems, 2020, 33: 1877−1901, articleTitle=null, refAbstract=null), Reference(id=1329420149989421970, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Achiam J, Adler S, Agarwal S, et al. Gpt−4 technical report[J]. arXiv preprint, arXiv: 2303.08774, 2023, articleTitle=null, refAbstract=null), Reference(id=1329420150069113747, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Touvron H, Martin L, Stone K, et al. LLaMA 2: Open foundation and fine-tuned chat models[J]. arXiv preprint, arXiv: 2307.09288, 2023, articleTitle=null, refAbstract=null), Reference(id=1329420150132028308, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Aixin, Feng Bei, Wang Bin, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model[J]. arXiv preprint, arXiv: 2405.04434, 2024, articleTitle=null, refAbstract=null), Reference(id=1329420150236885910, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Susan, Roller S, Goyal N, et al. OPT: Open pre-trained transformer language models[J]. arXiv preprint, arXiv: 2205.01068, 2022, articleTitle=null, refAbstract=null), Reference(id=1329420150312383383, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=He Kaiming, Chen Xinlei, Xie Saining, et al. Masked autoencoders are scalable vision learners[C]//Proc of the 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. 2022: 16000−16009, articleTitle=null, refAbstract=null), Reference(id=1329420150375297944, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Roziere B, Gehring J, Gloeckle F, et al. Code LLaMA: Open foundation models for code[J]. arXiv preprint, arXiv: 2308.12950, 2023, articleTitle=null, refAbstract=null), Reference(id=1329420150446601113, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Aixin, Feng Bei, Xue Bing, et al. Deepseek-v3 technical report[J]. arXiv preprint, arXiv: 2412.19437, 2024, articleTitle=null, refAbstract=null), Reference(id=1329420150505321370, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Guo Daya, Zhu Qihao, Yang Dejian, et al. DeepSeek-Coder: When the large language model meets programming–The rise of code intelligence[J]. arXiv preprint, arXiv: 2401.14196, 2024, articleTitle=null, refAbstract=null), Reference(id=1329420150568235931, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Kaplan J, McCandlish S, Henighan T, et al. Scaling laws for neural language models[J]. arXiv preprint, arXiv: 2001.08361, 2020, articleTitle=null, refAbstract=null), Reference(id=1329420150626956188, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Wei J, Wang Xuezhi, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models[J]. Advances in Neural Information Processing Systems, 2022, 35: 24824−24837, articleTitle=null, refAbstract=null), Reference(id=1329420150698259357, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Haoran, Li Yuming, Xu Haobo, et al. LAD: Efficient accelerator for generative inference of LLM with locality aware decoding[C]//Proc of the 31st IEEE Int Symp on High Performance Computer Architecture (HPCA). Piscataway, NJ: IEEE, 2025: 1482−1495, articleTitle=null, refAbstract=null), Reference(id=1329420150777951134, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=葛旭冉, 欧洋, 王博, 等. 大语言模型推理中的存储优化技术综述[J]. 计算机研究与发展, 2025, 62(3): 545−562, articleTitle=null, refAbstract=null), Reference(id=1329420150866031519, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Ge Xuran, Ou Yang, Wang Bo, et al. Survey of storage optimization techniques in large language model inference[J]. Journal of Computer Research and Development, 2025, 62(3): 545−562 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1329420150945723296, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Yu Zhongkai, Liang Shengwen, Ma Tianyun, et al. Cambricon-LLM: A chiplet-based hybrid architecture for on-device inference of 70b LLM[C]//Proc of the 57th IEEE/ACM Int Symp on Microarchitecture (MICRO). Piscataway, NJ: IEEE, 2024: 1474−1488, articleTitle=null, refAbstract=null), Reference(id=1329420151017026465, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Bai Fujun, Jiang Xiping, Wang Song, et al. A stacked embedded DRAM array for LPDDR4/4X using hybrid bonding 3D integration with 34GB/s/1Gb 0.88 pJ/b logic-to-memory interface[C]//Proc of the 68th IEEE Int Electron Devices Meeting (IEDM). Piscataway, NJ: IEEE, 2020: 6.6. 1−6.6. 4, articleTitle=null, refAbstract=null), Reference(id=1329420151079941026, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Choquette J, Gandhi W, Giroux O, et al. Nvidia a100 tensor core GPU: Performance and innovation[J]. IEEE Micro, 2021, 41(2): 29−35, articleTitle=null, refAbstract=null), Reference(id=1329420151159632803, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Yue Zhiheng, Wang Huizheng, Fang Jiahao, et al. Exploiting similarity opportunities of emerging vision AI models on hybrid bonding architecture[C]//Proc of the 51st ACM/IEEE Annual Int Symp on Computer Architecture (ISCA). Piscataway, NJ: IEEE, 2024: 396−409, articleTitle=null, refAbstract=null), Reference(id=1329420151226741668, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Niu Dimin, Li Shuangchen, Wang Yuhao, et al. 184QPS/W 64Mb/mm2 3D logic-to-DRAM hybrid bonding with process-near-memory engine for recommendation system[C]//Proc of the 2022 IEEE Int Solid-State Circuits Conf (ISSCC). Piscataway, NJ: IEEE, 2022, 65: 1−3, articleTitle=null, refAbstract=null), Reference(id=1329420151302239141, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Park J, Choi J, Kyung K, et al. AttAcc! Unleashing the power of PIM for batched transformer-based generative model inference[C]//Proc of the 29th ACM Int Conf on Architectural Support for Programming Languages and Operating Systems, Volume 2. New York: ACM, 2024: 103−119, articleTitle=null, refAbstract=null), Reference(id=1329420151386125222, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Amin H, Curtis K M, Hayes-Gill B R. Piecewise linear approximation applied to nonlinear function of a neural network[J]. IEE Proceedings-Circuits, Devices and Systems, 1997, 144(6): 313−317, articleTitle=null, refAbstract=null), Reference(id=1329420151461622695, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Khan H, Khan A, Khan Z, et al. NPE: An FPGA-based overlay processor for natural language processing[J]. arXiv preprint, arXiv: 2104.06535, 2021, articleTitle=null, refAbstract=null), Reference(id=1329420151520342952, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=Jiang Xiping, Zuo Fengguo, Wang Song, et al. A 1596-GB/s 48-Gb stacked embedded DRAM 384-core SoC with hybrid bonding integration[J]. IEEE Solid-State Circuits Letters, 2022, 5: 110−113, articleTitle=null, refAbstract=null), Reference(id=1329420151591646121, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=Jiménez D A, Lin C. Dynamic branch prediction with perceptrons[C]//Proc of the 7th HPCA Int Symp on High-Performance Computer Architecture. Piscataway, NJ: IEEE, 2001: 197−206, articleTitle=null, refAbstract=null), Reference(id=1329420151667143594, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=Kwon W, Li Zhuohan, Zhuang Siyuan, et al. Efficient memory management for large language model serving with pagedattention[C]//Proc of the 29th Symp on Operating Systems Principles. New York: ACM, 2023: 611−626, articleTitle=null, refAbstract=null), Reference(id=1329420151738446763, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=26, authorNames=null, journalName=null, refType=null, unstructuredReference=Muralimanohar N, Balasubramonian R, Jouppi N P. CACTI 6.0: A tool to model large caches[J]. HP Laboratories, 2009, 27: 28, articleTitle=null, refAbstract=null), Reference(id=1329420151805555628, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=27, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Shang, Yang Zhiyuan, Reddy D, et al. DRAMSim3: A cycle-accurate, thermal-capable DRAM simulator[J]. IEEE Computer Architecture Letters, 2020, 19(2): 106−109, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1329420145308578634, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=1, ext=[AuthorCompanyExt(id=1329420145316967243, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145308578634, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1329420145329550156, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145308578634, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所智能计算机研究中心 北京 100190)]), AuthorCompany(id=1329420145409241933, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=2, ext=[AuthorCompanyExt(id=1329420145413436238, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145409241933, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1329420145421824847, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145409241933, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)]), AuthorCompany(id=1329420145493128016, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=3, ext=[AuthorCompanyExt(id=1329420145505710929, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145493128016, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3University of Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1329420145514099538, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145493128016, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3中国科学院大学 北京 100190)])], figs=[ArticleFig(id=1329420147435090806, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=EN, label=Figure 1, caption=The structure of hybrid bonding[18], figureFileSmall=aI3Rrv6DV2IYRtZd9MntoQ==, figureFileBig=0FMbM8ox6RYuMLWnOfzMxw==, tableContent=null), ArticleFig(id=1329420147535754103, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=CN, label=图1, caption=混合键合结构[18], figureFileSmall=aI3Rrv6DV2IYRtZd9MntoQ==, figureFileBig=0FMbM8ox6RYuMLWnOfzMxw==, tableContent=null), ArticleFig(id=1329420147720303480, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=EN, label=Figure 2, caption=Exponential function computed by piecewise linear approach, figureFileSmall=Ng5j8NX66K8o666hiO/vNQ==, figureFileBig=5o2Ak5r2pKEKZgl9x1AtCA==, tableContent=null), ArticleFig(id=1329420147787412345, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=CN, label=图2, caption=分段线性方法计算指数函数, figureFileSmall=Ng5j8NX66K8o666hiO/vNQ==, figureFileBig=5o2Ak5r2pKEKZgl9x1AtCA==, tableContent=null), ArticleFig(id=1329420147854521210, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=EN, label=Figure 3, caption=The ratio of maximum to minimum active position load for each model under different context lengths, figureFileSmall=71B2hksDFonzvd/isz7SIQ==, figureFileBig=6ErjHBf6FhvGY5cOd1H8Ug==, tableContent=null), ArticleFig(id=1329420148005516155, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=CN, label=图3, caption=不同上下文长度下各模型最大与最小活跃位置负载比值, figureFileSmall=71B2hksDFonzvd/isz7SIQ==, figureFileBig=6ErjHBf6FhvGY5cOd1H8Ug==, tableContent=null), ArticleFig(id=1329420148081013628, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=EN, label=Figure 4, caption=The overall architecture of the accelerator, figureFileSmall=IXcAK6x/5ow+gCKP60L7JQ==, figureFileBig=40wPh6bvgnUcEjOzxXtKIg==, tableContent=null), ArticleFig(id=1329420148143928189, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=CN, label=图4, caption=加速器的整体架构, figureFileSmall=IXcAK6x/5ow+gCKP60L7JQ==, figureFileBig=40wPh6bvgnUcEjOzxXtKIg==, tableContent=null), ArticleFig(id=1329420148206842750, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=EN, label=Figure 5, caption=The parallelization strategy across LAD-3D computational units, figureFileSmall=L5j0VIeIt4ETJSMKepCneQ==, figureFileBig=rqSjG2suV3rWvBjIWNCikA==, tableContent=null), ArticleFig(id=1329420148265563007, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=CN, label=图5, caption=LAD-3D 计算单元之间的并行策略, figureFileSmall=L5j0VIeIt4ETJSMKepCneQ==, figureFileBig=rqSjG2suV3rWvBjIWNCikA==, tableContent=null), ArticleFig(id=1329420148328477568, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=EN, label=Figure 6, caption=The architectural details of LAD-3D accelerated modules, figureFileSmall=6ISeEDux5PVsYBbFWdOqgw==, figureFileBig=vqHgmeQbM8PU4jDWHLN6LQ==, tableContent=null), ArticleFig(id=1329420148571747201, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=CN, label=图6, caption=LAD-3D 加速模块的架构细节, figureFileSmall=6ISeEDux5PVsYBbFWdOqgw==, figureFileBig=vqHgmeQbM8PU4jDWHLN6LQ==, tableContent=null), ArticleFig(id=1329420148664021890, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=EN, label=Figure 7, caption=The heatmap of the distribution of active positions across LAD-3D computational elements, figureFileSmall=jluMne7AVPunqV6AD14dDw==, figureFileBig=4CzQ3RnfoW3aVWYFbmZsLQ==, tableContent=null), ArticleFig(id=1329420148735325059, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=CN, label=图7, caption=活跃位置在各 LAD-3D 计算单元间分布的热力图, figureFileSmall=jluMne7AVPunqV6AD14dDw==, figureFileBig=4CzQ3RnfoW3aVWYFbmZsLQ==, tableContent=null), ArticleFig(id=1329420148903097220, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=EN, label=Figure 8, caption=The architectural details of the active position predictor, figureFileSmall=JWTM+/e8uc3SwKudLoTv4Q==, figureFileBig=2gwFk3oemC1P44doC0AFEA==, tableContent=null), ArticleFig(id=1329420148957623173, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=CN, label=图8, caption=活跃位置预测器的架构细节, figureFileSmall=JWTM+/e8uc3SwKudLoTv4Q==, figureFileBig=2gwFk3oemC1P44doC0AFEA==, tableContent=null), ArticleFig(id=1329420149041509254, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=EN, label=Figure 9, caption=Decoding performance of LLMs under different context lengths, figureFileSmall=1GQ/pHF8JSk5D9unUJyxSQ==, figureFileBig=nVkv1avbz5oIHZjjDHIDOg==, tableContent=null), ArticleFig(id=1329420149112812423, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=CN, label=图9, caption=大模型在不同上下文长度下的解码性能, figureFileSmall=1GQ/pHF8JSk5D9unUJyxSQ==, figureFileBig=nVkv1avbz5oIHZjjDHIDOg==, tableContent=null), ArticleFig(id=1329420149221864328, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=EN, label=Figure 10, caption=Decoding energy of LLMs under different context lengths, figureFileSmall=kABLeP2Ehd3csrb76MhuqA==, figureFileBig=bVUJt3lNsTEwMDKRDbsa6A==, tableContent=null), ArticleFig(id=1329420149305750409, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=CN, label=图10, caption=大模型在不同上下文长度下的解码能耗, figureFileSmall=kABLeP2Ehd3csrb76MhuqA==, figureFileBig=bVUJt3lNsTEwMDKRDbsa6A==, tableContent=null), ArticleFig(id=1329420149377053578, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=EN, label=Table 1, caption=

Balance Ratios of Active Position

, figureFileSmall=null, figureFileBig=null, tableContent=
上下文
长度
模型
OPT-2.7BOPT-6.7BLLaMA-13BLLaMA-7B
5120.97900.96490.96760.9685
10240.97320.97320.98110.9850
15360.97390.98480.98130.9866
20480.98230.98230.98520.9889
2560--0.98760.9872
3072--0.98970.9863
3548--0.99070.9881
4096--0.99190.9894
), ArticleFig(id=1329420149448356747, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=CN, label=表1, caption=

活跃位置的均衡比值

, figureFileSmall=null, figureFileBig=null, tableContent=
上下文
长度
模型
OPT-2.7BOPT-6.7BLLaMA-13BLLaMA-7B
5120.97900.96490.96760.9685
10240.97320.97320.98110.9850
15360.97390.98480.98130.9866
20480.98230.98230.98520.9889
2560--0.98760.9872
3072--0.98970.9863
3548--0.99070.9881
4096--0.99190.9894
), ArticleFig(id=1329420149515465612, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=EN, label=Table 2, caption=

Predicting Missing Probability of Active Positions

, figureFileSmall=null, figureFileBig=null, tableContent=
上下文
长度
模型
OPT-2.7BOPT-6.7BLLaMA-13BLLaMA-7B
5120.02890.02620.01040.0322
10240.02180.01610.01080.0282
15360.02460.01810.01720.0262
20480.01680.02010.01910.0209
2560--0.01490.0173
3072--0.01320.0212
3548--0.01520.0223
4096--0.01380.0151
), ArticleFig(id=1329420149590963085, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=CN, label=表2, caption=

活跃位置的预测未命中概率

, figureFileSmall=null, figureFileBig=null, tableContent=
上下文
长度
模型
OPT-2.7BOPT-6.7BLLaMA-13BLLaMA-7B
5120.02890.02620.01040.0322
10240.02180.01610.01080.0282
15360.02460.01810.01720.0262
20480.01680.02010.01910.0209
2560--0.01490.0173
3072--0.01320.0212
3548--0.01520.0223
4096--0.01380.0151
), ArticleFig(id=1329420149691626382, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=EN, label=Table 3, caption=

Area and Power of Logic Modules of LAD-3D Computational Elements

, figureFileSmall=null, figureFileBig=null, tableContent=
模块名面积/mm2动态功耗/mW静态功耗/mW
VPU0.1706125.048633.2571
SFU0.068643.293416.8810
EAS0.00483.23101.1673
APID0.00592.71391.2116
MD0.00251.75290.4855
AC0.085660.205419.8291
SRAM6.15561004.8288466.3696
预测器0.051918.97259.3295
), ArticleFig(id=1329420149767123855, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, language=CN, label=表3, caption=

LAD-3D计算单元逻辑模块的面积与功耗

, figureFileSmall=null, figureFileBig=null, tableContent=
模块名面积/mm2动态功耗/mW静态功耗/mW
VPU0.1706125.048633.2571
SFU0.068643.293416.8810
EAS0.00483.23101.1673
APID0.00592.71391.2116
MD0.00251.75290.4855
AC0.085660.205419.8291
SRAM6.15561004.8288466.3696
预测器0.051918.97259.3295
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202550301, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202550301, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202550301, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202550301, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1782835200000, fullTextJson=null, articleText=null, reference=null)
收藏切换
基于混合键合架构的大模型局部性感知解码加速器
王浩然 1, 2, 3 , 许浩博 1, 2, * , 王颖 1, 2 , 韩银和 1, 2
计算机研究与发展 | 体系结构 2026,63(7): 1816-1830
收起
收藏切换
计算机研究与发展 |体系结构 2026 , 63 (7) : 1816 -1830
基于混合键合架构的大模型局部性感知解码加速器
全屏
[Author(id=1329420145597985620, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=wanghaoran20g@ict.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329420145686066008, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, authorId=1329420145597985620, language=EN, stringName=Haoran Wang, firstName=Haoran, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, 3, address=1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
3University of Chinese Academy of Sciences, Beijing 100190, bio={"content":"

born in 1997. PhD. His main research interests include computer architecture, large language model system and accelerator design, and memory architecture based on advanced packaging

"}, bioImg=null, bioContent=

born in 1997. PhD. His main research interests include computer architecture, large language model system and accelerator design, and memory architecture based on advanced packaging

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329420145753174873, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, authorId=1329420145597985620, language=CN, stringName=王浩然, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, 3, address=1中国科学院计算技术研究所智能计算机研究中心 北京 100190
2处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190
3中国科学院大学 北京 100190, bio={"img":"DbpS4j7Vw7jgQxoLVSWDPQ==","content":"

1997年生。博士。主要研究方向为计算机体系结构、大模型系统与加速器设计、基于先进封装的存储架构

"}, bioImg=DbpS4j7Vw7jgQxoLVSWDPQ==, bioContent=

1997年生。博士。主要研究方向为计算机体系结构、大模型系统与加速器设计、基于先进封装的存储架构

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329420145308578634, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=1, ext=[AuthorCompanyExt(id=1329420145316967243, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145308578634, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1329420145329550156, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145308578634, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所智能计算机研究中心 北京 100190)]), AuthorCompany(id=1329420145409241933, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=2, ext=[AuthorCompanyExt(id=1329420145413436238, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145409241933, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1329420145421824847, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145409241933, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)]), AuthorCompany(id=1329420145493128016, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=3, ext=[AuthorCompanyExt(id=1329420145505710929, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145493128016, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3University of Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1329420145514099538, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145493128016, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3中国科学院大学 北京 100190)])]), Author(id=1329420145841255259, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=xuhaobo@ict.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329420145912558430, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, authorId=1329420145841255259, language=EN, stringName=Haobo Xu, firstName=Haobo, middleName=null, lastName=Xu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, *, address=1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190, bio={"content":"

born in 1990. PhD, associate professor. Member of CCF. His main research interests include domain-specific processor chips and 2.5D/3D chiplet-based integrated chips

"}, bioImg=null, bioContent=

born in 1990. PhD, associate professor. Member of CCF. His main research interests include domain-specific processor chips and 2.5D/3D chiplet-based integrated chips

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329420145971278687, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, authorId=1329420145841255259, language=CN, stringName=许浩博, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, *, address=1中国科学院计算技术研究所智能计算机研究中心 北京 100190
2处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190, bio={"img":"KymBk6OMJcAe0ujAkmaO6w==","content":"

1990年生。 博士,副研究员。CCF会员。 主要研究方向为领域专用处理器芯片、2.5维/3维芯粒集成芯片

"}, bioImg=KymBk6OMJcAe0ujAkmaO6w==, bioContent=

1990年生。 博士,副研究员。CCF会员。 主要研究方向为领域专用处理器芯片、2.5维/3维芯粒集成芯片

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329420145308578634, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=1, ext=[AuthorCompanyExt(id=1329420145316967243, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145308578634, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1329420145329550156, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145308578634, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所智能计算机研究中心 北京 100190)]), AuthorCompany(id=1329420145409241933, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=2, ext=[AuthorCompanyExt(id=1329420145413436238, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145409241933, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1329420145421824847, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145409241933, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)])]), Author(id=1329420146038387553, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329420146134856548, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, authorId=1329420146038387553, language=EN, stringName=Ying Wang, firstName=Ying, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190, bio={"content":"

born in 1985. PhD, professor. Senior member of CCF. His main research interests include dedicated processor architecture and integrated chip systems

"}, bioImg=null, bioContent=

born in 1985. PhD, professor. Senior member of CCF. His main research interests include dedicated processor architecture and integrated chip systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329420146252297061, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, authorId=1329420146038387553, language=CN, stringName=王颖, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1中国科学院计算技术研究所智能计算机研究中心 北京 100190
2处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190, bio={"img":"i+msuHgF0Fox8T3IUgeEMA==","content":"

1985年生。 博士,研究员。CCF高级会员。 主要研究方向为专用处理器体系结构、集成芯片系统

"}, bioImg=i+msuHgF0Fox8T3IUgeEMA==, bioContent=

1985年生。 博士,研究员。CCF高级会员。 主要研究方向为专用处理器体系结构、集成芯片系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329420145308578634, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=1, ext=[AuthorCompanyExt(id=1329420145316967243, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145308578634, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1329420145329550156, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145308578634, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所智能计算机研究中心 北京 100190)]), AuthorCompany(id=1329420145409241933, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=2, ext=[AuthorCompanyExt(id=1329420145413436238, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145409241933, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1329420145421824847, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145409241933, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)])]), Author(id=1329420146365543271, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=yinhes@ict.ac.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329420146449429354, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, authorId=1329420146365543271, language=EN, stringName=Yinhe Han, firstName=Yinhe, middleName=null, lastName=Han, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190, bio={"content":"

born in 1980. PhD, professor. Senior member of CCF. His main research interests include computer architecture and chips, and data computing chips and computing systems

"}, bioImg=null, bioContent=

born in 1980. PhD, professor. Senior member of CCF. His main research interests include computer architecture and chips, and data computing chips and computing systems

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329420146533315435, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, authorId=1329420146365543271, language=CN, stringName=韩银和, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1中国科学院计算技术研究所智能计算机研究中心 北京 100190
2处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190, bio={"img":"mu9hqS/N2jwq9LFklnR2ng==","content":"

1980年生。博士,研究员。CCF高级会员。主要研究方向为计算机体系结构和芯片、数据计算芯片和计算系统

"}, bioImg=mu9hqS/N2jwq9LFklnR2ng==, bioContent=

1980年生。博士,研究员。CCF高级会员。主要研究方向为计算机体系结构和芯片、数据计算芯片和计算系统

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329420145308578634, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=1, ext=[AuthorCompanyExt(id=1329420145316967243, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145308578634, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190), AuthorCompanyExt(id=1329420145329550156, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145308578634, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1中国科学院计算技术研究所智能计算机研究中心 北京 100190)]), AuthorCompany(id=1329420145409241933, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, xref=2, ext=[AuthorCompanyExt(id=1329420145413436238, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145409241933, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190), AuthorCompanyExt(id=1329420145421824847, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420141516927786, companyId=1329420145409241933, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190)])])]
王浩然1, 2, 3 , 许浩博1, 2, * , 王颖1, 2, 韩银和1, 2
作者信息
  • 1中国科学院计算技术研究所智能计算机研究中心 北京 100190
  • 2处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190
  • 3中国科学院大学 北京 100190
通讯作者:
许浩博()
作者简介:

作者贡献声明

王浩然设计了大模型局部性感知解码的混合键合加速器,并进行了性能和功耗实验;许浩博指导了实验的细节,搜集了混合键合DRAM的架构参数,制定了论文写作大纲;王颖、韩银和指导了论文总体的技术方向和实验方法。

1997年生。博士。主要研究方向为计算机体系结构、大模型系统与加速器设计、基于先进封装的存储架构

1990年生。 博士,副研究员。CCF会员。 主要研究方向为领域专用处理器芯片、2.5维/3维芯粒集成芯片

1985年生。 博士,研究员。CCF高级会员。 主要研究方向为专用处理器体系结构、集成芯片系统

1980年生。博士,研究员。CCF高级会员。主要研究方向为计算机体系结构和芯片、数据计算芯片和计算系统

Locality-Aware Decoding Accelerator for Large Language Models Based on Hybrid Bonding Architecture
Haoran Wang1, 2, 3 , Haobo Xu1, 2, * , Ying Wang1, 2, Yinhe Han1, 2
Affiliations
  • 1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
  • 2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
  • 3University of Chinese Academy of Sciences, Beijing 100190
  • born in 1997. PhD. His main research interests include computer architecture, large language model system and accelerator design, and memory architecture based on advanced packaging

    born in 1990. PhD, associate professor. Member of CCF. His main research interests include domain-specific processor chips and 2.5D/3D chiplet-based integrated chips

    born in 1985. PhD, professor. Senior member of CCF. His main research interests include dedicated processor architecture and integrated chip systems

    born in 1980. PhD, professor. Senior member of CCF. His main research interests include computer architecture and chips, and data computing chips and computing systems

出版时间: 2026-07-01 doi: 10.7544/issn1000-1239.202550301
文章导航
收藏切换

大模型在自然语言处理、计算机视觉、聊天机器人以及内容生成等多个领域中展现出了优秀的能力。大模型在边缘设备上的部署因其在稳定性和数据安全方面的优势受到广泛关注。然而,大模型的解码推理过程主要由访存受限的算子构成,随着模型规模的扩展以及复杂任务对上下文长度的需求持续增长,访存开销成为了限制大模型在边缘设备上部署的关键瓶颈。为了降低大模型在长上下文解码时的访存开销,局部性感知的注意力机制被提出。该机制基于注意力分数在解码过程中的数值局部性,对键值缓存(KV cache)进行压缩,实现了在不影响模型精度的情况下,有效削减了冗余的访存。设计了一种基于混合键合3维DRAM的局部性感知大模型推理加速器,在有效降低总体访存量的同时显著提升了主存带宽。该加速器集成了负载均衡与基于预测的预取机制,来适配混合键合3维DRAM的结构特性,从而提升带宽利用率。在所评估的模型上,该加速器在批处理大小为1,2,4时,平均实现了分别为1971 token/s,3752 token/s,6849 token/s的吞吐,对应的能耗分别为0.23 J/token,0.12 J/token,0.07 J/token。该吞吐相当于A100 GPU的50.5倍、49.4倍、50.0倍,LAD-HBM加速器的23.7倍、22.9倍、21.5倍,而能耗仅相当于GPU的5.1%,5.1%,4.9%,LAD-HBM的46.2%,46.9%,48.2%。

大模型  /  自回归解码  /  混合键合  /  加速器  /  边缘设备

Large language models (LLMs) have demonstrated remarkable capabilities across various domains, including natural language processing, computer vision, chat bots and content generation. Deploying LLMs on edge devices has attracted significant attention due to its advantages in service stability and data privacy. However, the decoding inference of LLMs is composed of memory-bound operations, and with model sizes growing and complex tasks demanding increasingly longer context lengths, the resulting memory access overhead becomes critical bottleneck limiting the deployment of LLMs on edge platforms. To lower the memory access demand of long context decoding, locality aware attention mechanism is proposed. By leveraging the numerical locality of attention scores during decoding, this mechanism compresses the key-value cache (KV cache), significantly reducing redundant memory access without compromising model accuracy. We present a locality-aware inference accelerator for LLMs with hybrid-bonded 3D DRAM. The accelerator not only reduces the overall memory traffic but also significantly improves the main memory bandwidth. It integrates load-balancing and prediction-based prefetching mechanisms tailored to the structural characteristics of hybrid-bonded 3D DRAM, thereby enhancing the bandwidth utilization. On the evaluated models, the proposed accelerator achieves on average 1971 token/s, 3752 token/s and 6849 token/s throughputs for batch sizes of 1, 2 and 4, respectively, with corresponding energy efficiencies of 0.23 J/token, 0.12 J/token and 0.07 J/token. The achieved throughput corresponds to 50.5 times, 49.4 times, 50.0 times that of the A100 GPU, and 23.7 times, 22.9 times, 21.5 times that of the LAD-HBM accelerator. Meanwhile, the energy consumption amounts to only 5.1%, 5.1%, 4.9% of GPU, and 46.2%, 46.9%, 48.2% of LAD-HBM.

large language model  /  auto-regressive decoding  /  hybrid bonding  /  accelerator  /  edge device
王浩然, 许浩博, 王颖, 韩银和. 基于混合键合架构的大模型局部性感知解码加速器. 计算机研究与发展, 2026 , 63 (7) : 1816 -1830 . DOI: 10.7544/issn1000-1239.202550301
Haoran Wang, Haobo Xu, Ying Wang, Yinhe Han. Locality-Aware Decoding Accelerator for Large Language Models Based on Hybrid Bonding Architecture[J]. Journal of Computer Research and Development, 2026 , 63 (7) : 1816 -1830 . DOI: 10.7544/issn1000-1239.202550301
以Transformer[1]解码器为核心结构的大模型[2-6]在多个领域的任务中表现出卓越的性能,不仅推动了自然语言处理、计算机视觉、聊天机器人和内容生成[7-10]等技术的突破,还在不断重塑各行业的工作流程。将大模型推理部署在边缘设备上正成为一个重要的需求。部署在边缘设备上的大模型不仅能够更好地满足用户的个性化需求,还能有效保障隐私安全。此外,这种部署方式还能提升模型服务的稳定性,使其不受网络状况和云端并发请求数量等因素的影响,从而为用户提供更可靠的使用体验。
大模型的强大能力源于其参数规模、训练数据规模[11]以及推理上下文的扩展[12]。参数规模和上下文长度的增长显著增加了推理开销,使得在边缘设备上高效部署大模型推理面临挑战。大模型的推理分为预填充(prefill)和解码(decode)两个阶段,其中后者需要多次自回归的解码计算,每次只生成1个新的词元(token)。在解码过程中,模型的注意力机制层会将每次生成的词元所对应的键值(key value,KV)向量存储起来,形成键值缓存(KV cache)。键值缓存记录了解码过程中的上下文历史,其大小随着上下文长度的增加而线性增长。与模型权重不同,键值缓存是特定于单个请求的,其访存无法在不同请求之间复用。因此,在长上下文解码任务中,对键值缓存的访存开销成为了性能瓶颈[13-14]。此外,在边缘设备的应用场景中,推理负载通常采用较小的批处理规模,甚至以单请求的方式运行。这种特性显著降低了线性层权重的访存复用机会,使得线性层也成为访存密集型计算。单请求的大模型解码负载的计算访存比[15]是BERT和VGG网络的近1/100。现有边缘设备访存带宽难以支撑大模型解码推理的有效加速,即便具备足够的算力,仍会因访存带宽低而使计算资源利用率低。
为了缓解大模型长序列解码时的访存压力,Wang等人[13]提出了局部性感知的注意力机制加速技术,显著减少了键值缓存的访存量并实现端到端加速。该方法通过统计已生成的上下文中各位置注意力分数最常落入的数值区间,将键值缓存压缩至固定大小的中间缓存。在每次解码时,仅需访问少数注意力分数未落入其最常出现区间的位置(活跃位置)所对应的键值,从而使得注意力机制的访存量仅随上下文长度呈现亚线性增长。在序列长度达到4096时,该技术将访存量降低至原始注意力机制的10%。然而,该技术并未对负载均衡进行优化,导致在进行细粒度任务划分的并行计算时,局部性感知的注意力机制的性能受到负载不均的限制。此外,该工作提出的加速器采用HBM2作为主存,难以满足大模型日益增长的带宽需求。
为解决传统DRAM在带宽与能效方面的瓶颈,基于混合键合(hybrid bonding)技术的3维DRAM架构被提出[16]。该架构将控制、电源管理等外围电路集成在逻辑芯片上,并与存储阵列芯片采用铜-铜键合技术连接。这种技术可以实现高密度低电阻的I/O,显著提升了DRAM的带宽和能效。但是,混合键合的DRAM通常以较细的粒度(通常为bank)进行独立控制以实现高访存并行度,且不采用传统的内存交错技术,因此确保访存负载在各banks之间的均衡至关重要。如何在基于混合键合的架构上实现局部性感知的大模型稀疏推理,同时兼顾访存的高并行度与均衡性,是一个具有挑战性的研究问题。
本文设计了一种基于混合键合3维DRAM的局部性感知大模型加速器,在显著降低长文本解码的访存量的同时,提供比现有的GPU/加速器高得多的访存带宽,有效解决了上面提到的问题。本文的主要贡献为3个方面:
1)提出了一种面向局部性感知注意力机制的活跃位置均衡算法,该算法根据大模型解码过程中活跃位置的动态稀疏性进行负载划分。实验表明,该算法可将划分后的平均负载与最大负载之比提升至0.96以上。该算法开销低且硬件友好,适合在硬件加速器中轻量化地实现。
2)提出了一种活跃位置预测算法,用于在当前解码计算时预测下一次解码将访问的活跃位置。实验表明,该算法的平均活跃位置未命中率仅为1.96%。基于该算法的硬件预测器能够高效地支持活跃位置预取和访存调度优化,从而提升访存带宽的利用率。
3)设计了一种面向边缘部署场景的大模型推理加速器。在所评估的大模型上,该加速器在批处理大小为1,2,4时,分别实现了平均1971 token/s,3752 token/s,6849 token/s的吞吐,相当于A100 GPU[17]的50.5倍、49.4倍、50.0倍,LAD-HBM加速器[13]的23.7倍、22.9倍、21.5倍。同时,其平均能耗分别为0.23 J/token,0.12 J/token,0.07 J/token,相当于GPU的5.1%,5.1%,4.9%,LAD-HBM的46.2%,46.9%,48.2%。
DRAM因其较低的访存延迟和较高的存储密度,被广泛采用为各类硬件系统的主存介质。然而,面对近年来大模型、推荐系统及图分析等访存密集型任务的发展,传统DRAM在容量与带宽方面均难以满足快速增长的需求。混合键合3维DRAM为应对上述挑战提供了有效的解决方案,其主要结构如图1所示。
Bai等人[16]设计制造了一款基于LPDDR4/4X的混合键合3维堆叠DRAM。将逻辑芯片与DRAM芯片分别制造后以超高密度、低阻抗的铜-铜键合方式进行面对面堆叠,实现了每1 Gb高达34 GBps的访存带宽与0.88 pJ/b的低能耗。他们基于该技术开发出可量产的4Gb LPDDR4/4X芯片,该芯片由4颗1Gb DRAM芯片与1颗逻辑芯片组成,内部集成了超64000个混合键和连接。Yue等人[18]研究了如何在基于混合键合DRAM的加速器上,利用视觉数据在空间上的聚类相似性来加速视觉AI任务。他们将专用加速逻辑单元与DRAM bank紧密集成,实现了高效的稀疏检测,并提出稀疏差值表示和稀疏均衡技术来提升bank间负载的均衡性。Niu等人[19]设计了面向推荐系统的混合键合加速器。该设计将25 nm DRAM芯片与55 nm逻辑芯片通过3 μm间距的铜-铜键合进行连接,实现了64 Mb/mm2的存储密度,提供高达2.4 GBps/mm2的带宽密度。逻辑芯片内集成了匹配引擎与神经网络引擎,其中匹配引擎包含距离计算模块和基于最大堆的top-k模块来支持推荐系统的匹配阶段,而神经网络引擎则集成了INT8脉动阵列以支持多层感知机(MLP)的计算。
然而混合键合DRAM技术在实际部署中仍存在着一些困难。1)混合键合DRAM在提升了存储系统并行度和总带宽的同时,引入了更强的分布式特性,这使得确保混合键合DRAM系统的利用率成为了新的挑战。2)混合键合DRAM的制造成本高,存在着对准精度、晶圆翘曲、良率以及I/O面积挤占有效存储面积等多方面的挑战。通常,键合设备的对准精度需为焊盘直径(pad diameter)的0.1~0.25倍。例如对于1 μm直径的焊盘,对准精度就需要在100~250 nm之间。3)逻辑层位于DRAM层和封装基板之间,通过铜柱向上方冷却系统导热,需控制逻辑层的厚度在10 μm左右[18]以解决积热问题。
大模型解码推理具有较低的计算强度,即计算量与对主存储器的访存数据量之比较低。随着模型参数规模和上下文长度的不断增长,大模型解码推理的访存开销日益突出。Park等人[20]提出了存内计算的AttAcc加速系统来加速大模型解码阶段的注意力计算。该系统在高带宽存储器(high bandwidth memory,HBM)内部集成矩阵向量乘计算单元,并构建GPU和AttAcc之间的流水线协同处理机制来加速端到端的大模型推理。Yu等人[15]提出一种基于芯粒技术的混合异构架构,结合具备高计算能力的NPU与具备近存处理能力的NAND闪存芯片,通过高带宽芯粒间互联实现NPU与近存处理单元的高效协同。该架构引入硬件感知的分块策略以及轻量级NAND闪存片上纠错机制,有效提升了带宽利用率,并减少了闪存数据错误对模型精度的影响。Wang等人[13]利用大模型解码过程中注意力分数的数值局部性,提出了一种将键值缓存压缩为固定大小的中间缓存的算法。他们分模块设计了所提出的算法中的主要步骤,并实现了专用加速器,能够高效地维护中间缓存,从而显著降低访存开销,实现端到端大模型解码推理的吞吐和能效的提升。
大模型的结构由多层Transformer层堆叠而成,每个Transformer层主要由注意力机制和线性层组成。大模型的推理分为2个阶段,即预填充阶段和解码阶段。在预填充阶段,模型接收提示(prompt),并行地计算提示中所有词元之间的注意力关系,生成第1个输出词元。由于该阶段主要涉及计算强度较高的矩阵乘法运算,通常不会产生访存瓶颈。
在解码阶段,计算以自回归迭代的方式进行,每次解码仅生成1个新的词元,且该词元的生成依赖于先前已生成的所有词元(包括提示中的词元)。对所有已生成词元的记忆能力是大模型性能的关键,而这一能力依赖于注意力机制。下面详细解释注意力机制在解码过程中的计算。记已生成的词元的键值缓存为$ \boldsymbol{K},\boldsymbol{V} $,它们都是大小为$ n\times d $的张量($ n $和$ d $分别是已生成的词元数量和单个注意力头的嵌入维度)。在新的一次解码时,注意力机制以新词元的查询、键和值为输入,分别记为$ {\boldsymbol{q}}_{n+1},{\boldsymbol{k}}_{n+1},{\boldsymbol{v}}_{n+1} $,它们均为$ 1\times d $的张量。首先,注意力机制需访存键值缓存,并通过将当前新的键值拼接至末尾来更新键值缓存:
$ \boldsymbol{K}\leftarrow {\left({\boldsymbol{K}}^{\mathrm{T}},\boldsymbol{k}_{n+1}^{\mathrm{T}}\right)}^{\mathrm{T}},\;\;\boldsymbol{V}\leftarrow {\left({\boldsymbol{V}}^{\mathrm{T}},\boldsymbol{v}_{n+1}^{\mathrm{T}}\right)}^{\mathrm{T}}\; 。$
之后,计算新查询和更新后的$ \boldsymbol{K},\boldsymbol{V} $之间的注意力机制:
$ \begin{aligned}&\boldsymbol{s}=(s_1,s_2,…,s_{n+1})=\frac{{\boldsymbol{q}}_{n+1}{\boldsymbol{K}}^{\rm T}}{\sqrt{d}}\;,\\ &\boldsymbol{o}=\dfrac{\left({e}^{{{s}_{1}}},{e}^{{{s}_{2}}},… ,{e}^{{{s}_{n+1}}}\right)\cdot \boldsymbol{V}}{\displaystyle\sum\limits_{i}{e}^{{{s}_{i}}}}\;。\end{aligned} $
从式(1)(2)可见,注意力机制在解码阶段的主要计算为矩阵-向量乘法,其计算强度低会产生访存瓶颈。同时,键值缓存的规模随着已生成的词元数量线性增长,在长序列生成任务中,对庞大键值缓存的频繁访存会造成显著的延迟。
分段线性方法[21-22]是加速器和边缘设备中实现超越函数的常用技术。在注意力机制中,主要的超越函数是softmax中的指数运算。局部性感知的注意力解码机制[13]LAD(locality aware decoding)提出将注意力机制中的指数函数定义域分为若干个子区间,在每个子区间内用线性函数做近似计算,如图2所示,并验证了这种近似方法几乎不影响模型的精度。
在采用了分段线性方法后,单次解码中的注意力机制计算可以表达为:
$ \boldsymbol{o}=\dfrac{\displaystyle\sum\limits_{i}\left({a}_{i}\left(\boldsymbol{q}\boldsymbol{k}_{i}^{\mathrm{T}}-m\right)+{b}_{i}\right){\boldsymbol{v}}_{i}}{\displaystyle\sum\limits_{i}\left({a}_{i}\left(\boldsymbol{q}\boldsymbol{k}_{i}^{\mathrm{T}}-m\right)+{b}_{i}\right)}\text{,} $
其中$ i $为遍历上下文中所有位置的索引,$ \boldsymbol{q} $为查询向量,$ {\boldsymbol{k}}_{i},{\boldsymbol{v}}_{i} $为位置$ i $的键值向量,$ m $为注意力分数$ \boldsymbol{q}\boldsymbol{k}_{i}^{\mathrm{T}} $中的最大值,而$ {a}_{i} $,${b}_{i} $为分段线性系数。在不同的解码步骤中,$ {a}_{i} $,${b}_{i} $随着注意力分数的变化而变化。局部性是指同一个位置在多次解码步骤中,其注意力分数以较高概率落入某一特定区间。利用这种局部性,LAD统计每个位置最常落入的区间,即众数区间(mode interval),记位置$ i $的众数区间的系数为$ a_{i}^{*} $,$b_{i}^{*} $。在1次解码当中,注意力分数落在众数区间之外的位置被定义为活跃位置,记活跃位置构成的集合为$ J $,位置$ i $的真实区间和活跃区间的系数差为$ {\alpha }_{i}={a}_{i}-a_{i}^{*} $,${\beta }_{i}={b}_{i}-b_{i}^{*} $(非活跃位置的$ {\alpha }_{i}=0 $,${\beta }_{i}=0 $)。式(3)可等价地改写为
$ \begin{aligned}&{\Delta }_{\boldsymbol{n}\boldsymbol{u}\boldsymbol{m}}=\displaystyle\sum\limits_{i\in J}{\alpha }_{i}\boldsymbol{q}\boldsymbol{k}_{i}^{\mathrm{T}}{\boldsymbol{v}}_{i}-m\displaystyle\sum\limits_{i\in J}{\alpha }_{i}{\boldsymbol{v}}_{i}+\displaystyle\sum\limits_{i\in J}{\beta }_{i}{\boldsymbol{v}}_{i}\;,\\ &{\Delta }_{den}=\displaystyle\sum\limits_{i\in J}{\alpha }_{i}\boldsymbol{q}\boldsymbol{k}_{i}^{\mathrm{T}}-m\displaystyle\sum\limits_{i\in J}{\alpha }_{i}+\displaystyle\sum\limits_{i\in J}{\beta }_{i}\;。 \end{aligned} $
$ \boldsymbol{o}=\dfrac{\boldsymbol{q}\boldsymbol{A}-m\boldsymbol{B}+\boldsymbol{C}+{\Delta }_{\boldsymbol{n}\boldsymbol{u}\boldsymbol{m}}}{\boldsymbol{q}\boldsymbol{D}-mE+F+{\Delta }_{den}}\; 。$
式(5)中$ \boldsymbol{A} $~$ F $代表基于各位置的众数计算出的中间缓存:$ \boldsymbol{A}= \displaystyle\sum\limits_{i}a_{i}^{*}\boldsymbol{k}_{i}^{\mathrm{T}}{\boldsymbol{v}}_{i} $,$\boldsymbol{B}= \displaystyle\sum\limits_{i}a_{i}^{*}{\boldsymbol{v}}_{i} $,$\boldsymbol{C}= \displaystyle\sum\limits_{i}b_{i}^{*}{\boldsymbol{v}}_{i} $,$\boldsymbol{D}= \displaystyle\sum\limits_{i}a_{i}^{*}\boldsymbol{k}_{i}^{\mathrm{T}} $,$E=\displaystyle\sum\limits_{i}a_{i}^{*} $,$F=\displaystyle\sum\limits_{i}b_{i}^{*} $。当某些位置的众数更新时,中间缓存也应相应地更新。设需要更新众数的位置的集合为$ U $,则更新公式为
$ \begin{aligned}&\boldsymbol{A}=\boldsymbol{A}+\sum \limits_{i\in U}{\alpha }_{i}\boldsymbol{k}_{i}^{\mathrm{T}}{\boldsymbol{v}}_{i},\;\;\boldsymbol{B}=\boldsymbol{B}+\sum \limits_{i\in U}{\alpha }_{i}{\boldsymbol{v}}_{i}\;,\\ &\boldsymbol{C}=\boldsymbol{C}+\sum \limits_{i\in U}{\beta }_{i}{\boldsymbol{v}}_{i},\;\;\boldsymbol{D}+=\sum \limits_{i\in U}{\alpha }_{i}\boldsymbol{k}_{i}^{\mathrm{T}}\;,\\&E=E+\sum \limits_{i\in U}{\alpha }_{i},\;\;F=F+\sum \limits_{i\in U}{\beta }_{i}\;。 \end{aligned} $
总结来说,LAD将注意力机制解码计算分解为先利用固定大小的中间缓存计算初始分子$ \boldsymbol{q}\boldsymbol{A}- m\boldsymbol{B}+ \boldsymbol{C} $和分母$ \boldsymbol{q}\boldsymbol{D}-mE+F $,再根据活跃位置的系数差计算修正项$ {\Delta }_{\boldsymbol{n}\boldsymbol{u}\boldsymbol{m}} $和$ {\Delta }_{den} $以对分子和分母进行修正。这种做法的优势在于,每次解码仅需访存固定大小的中间缓存和少量活跃位置的键值向量,相较于原始注意力机制,在长序列解码时可减少大量冗余访存。
为了应对DRAM性能滞后于计算部件访存需求的挑战,混合键合技术被引入,以同时提升访存带宽和能效[16,18,20,23]。混合键合技术将DRAM的控制电路、DFT模块等从DRAM芯片分离出来,集成在了逻辑芯片上,并利用铜-铜键合技术在垂直方向上将DRAM芯片和逻辑芯片堆叠连接。混合键合技术具有3种优势[16]:
1)高I/O密度。相邻I/O间可实现3 μm间距,实现高达110000/mm2的I/O密度,该密度较HBM中使用的微凸点技术高出数百倍。
2)低寄生电阻。DRAM数据通过铜互连直接传输至逻辑单元,有效降低寄生电阻至0.5 Ω以下,同时将数据存取能效提升至0.88 pJ/b。
3)存储与逻辑芯片分离。DRAM芯片与逻辑芯片可采用相互独立的制程工艺,从而能够针对各自的性能需求分别进行设计与工艺优化,可以在逻辑芯片上实现除传统DRAM外围电路之外的更复杂的逻辑功能。
混合键合DRAM的上述优势对于构建高性能边缘侧推理平台具有重要意义。在2.2节中介绍的LAD机制显著降低了注意力机制解码阶段的访存需求,但其主要计算负载仍是低计算强度的特殊算子和矩阵向量乘法。因此,LAD机制仍然是访存受限的。实现混合键合3维DRAM的架构来加速基于LAD的大模型推理,不仅可以大幅减少长序列解码时的访存量,还能显著提升访存性能与能效,这为边缘侧大模型推理提供了强有力的支撑。然而,在混合键合DRAM的架构上部署基于LAD的大模型面临着一些问题,本文将这些问题归纳为2个方面,即访存的空间均匀性和时间均匀性问题。
1)空间均匀性。混合键合技术显著提升了I/O密度,从而实现了更高的访存并行度,并支持更细粒度的独立访存操作。与传统DRAM结构相比,这种变化在数据映射优化和高效利用带宽方面引入了新的挑战。实际上,现有混合键合DRAM可实现独立访存每个bank[18],因此当负载在bank间分布不均时,会导致部分bank成为瓶颈,显著降低整体带宽利用率,此即访存的空间均匀性问题。在LAD机制中,访存和计算围绕着活跃位置进行,但活跃位置在上下文序列中的分布是不均匀的,且在解码过程中动态变化,这使得保持访存的空间均匀性具有挑战性。图3展示了采用朴素的均分策略将上下文分配至所有LAD-3D计算单元(参考3.1节)的情况下,各模型最大与最小活跃位置负载的比值。可以观察到,在所有测试模型中,活跃位置的分布均存在不均衡现象,且其严重程度随解码上下文长度的增加而加剧。
2)时间均匀性。LAD机制中的活跃位置是根据每次解码时注意力分数所在的区间动态确定的,其随着查询向量的不同不断变化。因此,活跃位置的访存模式是动态变化且在每次解码计算发生之前不能确定的。此外,虽然线性层权重的访存模式是静态的,但在不同的批处理大小下,单位时间内的访存需求也存在差异。如何保障访存的时间均匀性,使DRAM持续处于高带宽利用率的状态,是部署基于LAD的大模型的另一个挑战。
本文提出的加速器架构由8个LAD-3D芯片组成,每个LAD-3D芯片内含32个LAD-3D计算单元,这些计算单元通过环形拓扑NoC互连。同时,LAD-3D芯片之间通过D2D互连形成链式拓扑。图4展示了LAD-3D加速器的总体架构。
与此种互连拓扑相适应地,本文提出芯片间流水线并行与芯片内张量并行相结合的并行策略。流水线并行策略将大模型的所有Transformer层均分为8组,并分别映射到8个LAD-3D芯片上执行。后一个LAD-3D芯片接收前一个LAD-3D芯片的输出作为输入,由于在解码过程中,Transformer层之间传递的张量仅为单个词元(在批处理大小大于1时,为批大小个词元),因此D2D互连上的通信压力小,可以采用开销较低的D2D互连设计。
本文根据大模型中每个算子的特性,采用相应的算子内张量并行策略,以高效利用LAD-3D计算单元之间的环形NoC。线性算子采用列并行的策略,将权重矩阵沿着列维度均分为32份,并分别映射至不同的LAD-3D计算单元。每个LAD-3D计算单元仅计算线性层输出中对应于自身权重的通道,因此需通过All-Gather聚合得到完整的输出,而All-Gather在环形拓扑上效率很高。
对于式(4)(5)(6)所示的LAD注意力算子(下文记为算子1,2,3),本文采用不同的并行策略,以优化它们的通信开销,如图5所示。算子1不适合采用头并行,因为不同头的活跃位置数量不同,若采用头并行则会使计算单元之间的活跃位置数量难以平衡,从而引发负载不均,影响性能。为解决这一问题,本文提出一种针对活跃负载均衡优化的序列并行(sequence parallelism)策略,将上下文序列划分为多个子集,并分配至不同的计算单元,使得每个计算单元所分配到的位置中,活跃位置的数量大致相当(详见3.3节)。设划分后,式(4)中的活跃位置集合$ J $被划分为子集$ {J}_{1},{J}_{2}… ,{J}_{m} $,第$ i $个计算单元负责计算$ {J}_{i} $对应的分子与分母修正项:
$ \begin{aligned}&\Delta _{\boldsymbol{n}\boldsymbol{u}\boldsymbol{m}}^{{J}_{i}}=\sum \limits_{i\in {J}_{i}}{\alpha }_{i}\boldsymbol{q}\boldsymbol{k}_{i}^{\mathrm{T}}{\boldsymbol{v}}_{i}-m\sum \limits_{i\in {J}_{i}}{\alpha }_{i}{\boldsymbol{v}}_{i}+\sum \limits_{i\in {J}_{i}}{\beta }_{i}{\boldsymbol{v}}_{i}\;\text{,}\\&\Delta _{den}^{{J}_{i}}=\sum \limits_{i\in {J}_{i}}{\alpha }_{i}\boldsymbol{q}\boldsymbol{k}_{i}^{\mathrm{T}}-m\sum \limits_{i\in {J}_{i}}{\alpha }_{i}+\sum \limits_{i\in {J}_{i}}{\beta }_{i}\;。 \end{aligned} $
需要注意的是,这里的公式仅展示了对单个注意力头的计算,而每个计算单元需对所有注意力头执行该计算。这样,每个计算单元算得所有注意力头的分子分母修正项的一个部分和。由于算子2采用头并行,因此需在计算单元之间进行Reduce-Scatter,使得每个计算单元获得其负责的注意力头对应的修正项的全部和。在环形拓扑上,Reduce-Scatter能够实现最高的理论通信效率。
算子2和算子3的计算主要围绕中间缓存展开,而每个注意力头都拥有独立的中间缓存,并且不同注意力头上的计算相互独立。因此,采用头并行策略能够在这2个算子上实现最高的计算与通信效率。算子1末尾的Reduce-Scatter确保每个计算单元获得与其中间缓存匹配的修正项,可直接按照式(3)加至分子分母中。对于算子3,由于采用序列并行,需要更新众数的位置集合$ U $也被划分为子集$ {U}_{1},{U}_{2},… ,{U}_{m} $。若每个计算单元基于子集$ {U}_{i} $计算:
$ \begin{aligned}&\Delta {\boldsymbol{A}}^{{{U}_{i}}}=\sum \limits_{i\in {U}_{i}}{\alpha }_{i}\boldsymbol{k}_{i}^{\mathrm{T}}{\boldsymbol{v}}_{i},\;\;\Delta {\boldsymbol{B}}^{{{U}_{i}}}=\sum \limits_{i\in {U}_{i}}{\alpha }_{i}{\boldsymbol{v}}_{i}\;,\\&\Delta {\boldsymbol{C}}^{{{U}_{i}}}=\sum \limits_{i\in {U}_{i}}{\beta }_{i}{\boldsymbol{v}}_{i},\;\;\Delta {\boldsymbol{D}}^{{{U}_{i}}}=\sum \limits_{i\in {U}_{i}}{\alpha }_{i}\boldsymbol{k}_{i}^{\mathrm{T}}\;,\\&\Delta {E}^{{{U}_{i}}}=\sum \limits_{i\in {U}_{i}}{\alpha }_{i},\;\;\Delta {F}^{{{U}_{i}}}=\sum \limits_{i\in {U}_{i}}{\beta }_{i}\;,\end{aligned} $
则会得到中间缓存更新量$ \Delta \boldsymbol{A} $~$ \Delta F $的部分和,需要对它们进行Reduce-Scatter。然而,$ \Delta \boldsymbol{A} $~$ \Delta F $的大小较大,会造成高昂的Reduce-Scatter通信开销。例如,$ \Delta \boldsymbol{A} $的大小就是算子1中分子修正项的$ d $倍($ d $为单个注意力头的嵌入维度,通常d=128)。为解决这个问题,在算子3执行之前,每个计算单元需要执行All-Gather来收集其负责的注意力头对应的集合$ U $中所有位置的键值。由于众数的稳定性,集合$ U $的规模是很小的(在本文的实验中仅占序列长度的$ {10}^{-3} $量级),因此这一通信开销极低。在执行完All-Gather后,每个计算单元即可计算其所负责注意力头的更新量,并完成对应的中间缓存的更新。
如图5所示,LAD-3D计算单元包含高效注意力分数(efficient attention score,EAS)计算模块、活跃位置识别(active position identification,APID)模块、众数差异(mode discrepancy,MD)模块、注意力计算(attention computation,AC)模块、特殊函数单元(special function unit,SFU)以及负责核心计算任务的向量处理单元(vector processing unit,VPU)。图6展示了这些模块的详细结构。在一个LAD-3D计算单元中,EAS,APID,MD,AC模块共用同一套VPU,它们分时调用VPU依次执行注意力机制层的各部分计算。SFU负责处理特殊函数,包括层归一化(LayerNorm)、旋转位置编码(rotary positional embedding)以及激活函数等。线性层的计算完全由VPU负责执行。
EAS模块的核心功能是计算查询和键中心(key center)之间的注意力分数,并根据新产生的键实时更新键中心。EAS计算的注意力分数仅用于判定是否落在众数区间以识别活跃位置,故而无需精确计算。EAS模块动态地从键向量序列中选取方向中心作为键中心,使得即便仅访存这部分占比很小的中心,依然能够以足够的精度识别出活跃位置。为此,EAS模块首先计算查询和键中心之间的点积,并将计算结果进行缩放,即乘以每个位置的键模长与其对应的键中心模长的比值,从而得到近似的注意力分数。对于新生成的键,EAS模块计算其与现有键中心之间的余弦相似度,并找出相似度绝对值的最大值。随后,该最大值与预设阈值0.98进行比较,若其大于0.98,则新键归属于该最大值对应的中心,反之新键自身成为一个新的中心。
APID模块的主要功能包括识别活跃位置、更新活跃概率并维护活跃位置在计算单元间分布的均衡。该模块包含2个查找表,分别存储了注意力分数区间的上下界。APID模块以EAS模块算得的注意力分数,以及各位置当前的众数区间、众数区间计数为输入。APID模块从查找表中检索众数区间的上下界并与注意力分数进行比较以判定该位置是否为活跃状态。若判定为活跃,则将该位置添加到活跃位置FIFO中;若判定为不活跃,则将该位置的众数区间计数加1,以记录其众数区间的又一次出现。活跃位置信号输入至活跃概率计算模块,以计算各位置的活跃概率,并累加在概率和累加器中,用于维护活跃位置的均衡分布(详见3.3节)。
基于APID识别出的活跃位置,MD模块计算这些活跃位置的精确注意力分数,并将所得分数与各区间的下界进行比较,从而确定活跃位置所属的实际区间。对于每个被处理的活跃位置,将其所有区间的计数加载到一组寄存器中。MD模块将活跃位置的实际区间的计数加1,而后与其当前的众数区间的计数比较,若实际区间的计数更大,则表明该位置的众数区间发生了改变,将其添加到众数更新FIFO中。MD模块还包含存储了各区间的线性系数的查找表,用以查找活跃位置的精确区间和众数区间对应的系数,进而计算系数差作为后续AC模块的输入,以支持对活跃位置进行修正计算。
AC模块负责执行LAD注意力计算的主要步骤,即式(4)(5)(6)。该模块包含1个向量累加寄存器和1个向量归约模块,支持在单个周期内计算累加寄存器中的向量与另外3个向量的和,并将结果存至累加寄存器。AC模块首先基于中间缓存计算初始的注意力输出分子$ \boldsymbol{q}\boldsymbol{A}-m\boldsymbol{B}+\boldsymbol{C} $和分母$ \boldsymbol{q}\boldsymbol{D}-mE+F $,并将计算结果存入向量累加寄存器中。随后,AC模块开始对分子和分母进行修正计算。在实际计算中,修正项主要由2部分组成:一部分是来源于活跃位置的系数差,应根据MD模块算得的活跃位置系数差,分别计算分子和分母的修正项$ \displaystyle\sum\limits_{i\in J}\big({\alpha }_{i}\big(\boldsymbol{q}\boldsymbol{k}_{i}^{\mathrm{T}}- m\big)+ {\beta }_{i}\big){\boldsymbol{v}}_{i} $和$ \displaystyle\sum\limits_{i\in J}\left({\alpha }_{i}\left(\boldsymbol{q}\boldsymbol{k}_{i}^{\mathrm{T}}-m\right)+{\beta }_{i}\right) $。另一部分修正是来源于尚未纳入中间缓存的位置,该部分的计算形式同第1部分,只是系数差需要修改为这些位置当前所在区间的系数。修正计算分多周期完成,每周期计算修正项的1个部分和并通过归约模块更新到向量累加寄存器中。待所有部分和计算完成后,寄存器中便累加了最终的分子和分母,可直接用于计算注意力输出。为了更新中间缓存,同一芯片上的每个LAD-3D计算单元通过NoC收集所有MD模块,众数更新FIFO中记录的位置的键值向量中自己负责处理的注意力头对应的部分(见3.1节并行策略)。随后,每个LAD-3D计算单元的AC模块即可执行式(6)以更新其负责的注意力头对应的中间缓存。在AC模块中,更新计算采用逐向量的方式进行。具体而言,中间缓存被拆解为一系列向量,每次将1个待更新向量加载到向量累加寄存器中,完成累加更新后,再处理下一个向量,直至更新完成。
EAS,MD,AC模块的硬件并行度与单个LAD-3D计算单元内的VPU数量保持一致,以确保这些模块能够恰好调用所有的VPU资源。APID模块不进行向量运算,故而不需要调用VPU,其硬件并行度不受VPU数量的限制。为了在较少的周期内完成活跃位置识别,APID的硬件并行度需要足够高。在实验中,本文设定EAS,MD,AC模块的并行度皆为3,而APID的并行度为16。
如3.1节所述,本文对LAD注意力算子采用了序列并行的策略。设解码上下文的长度为$ n $,将词元位置序号的集合$ S=\{1{,}2,… ,n\} $分为$ m $个不相交的子集$ {S}_{1},{S}_{2},… ,{S}_{m} $,且$ {S}_{1}\cup {S}_{2}\cup … \cup {S}_{m}=S $。通过这种划分将词元序列分配到$ m $个计算单元上,即计算单元$ i $处理$ {S}_{i} $中位置的对应词元。现有的序列并行策略通常按照位置顺序将序列均分为$ m $块,然而,这种简单的划分方式在LAD机制中会造成性能损失。具体而言,不同位置的活跃概率存在差异,均分的划分方法可能导致各子集内位置的活跃概率之和差异较大。在解码过程中,活跃概率和较高的子集往往会产生更多的活跃位置,从而使对应的计算单元有更高的计算和访存负载。如图7所示,在LLaMA2-13B模型的解码过程中,采用均分划分方式会导致不同的LAD-3D计算单元的活跃负载差距较大,且该不均衡程度呈现随着上下文长度的增加而加剧的趋势。由于LAD计算过程中存在同步需求(3.1节的Reduce-Scatter),系统的整体性能受限于负载最高的计算单元,活跃位置分布的不均衡会成为制约性能的关键瓶颈。
为了解决上述问题,需要维护各子集的活跃概率之和的动态平衡。受负载均衡(MakeSpan)问题的启发,本文提出一种基于活跃概率的贪心算法,如算法1所示。
算法1. 活跃位置均衡算法。
输入:第$ j $次解码的活跃状态$ y_{1}^{j},… ,y_{{n}_{j}}^{j} $(其中$ j=1{,}2,… ,k $($ k $为解码次数),$y_i^j=1 $表示活跃);
输出:位置序号集合$ S $的划分$ {S}_{1},{S}_{2},… ,{S}_{m} $。
① 初始化:$ {T}_{i}=0,{p}_{i}=0,{S}_{i}=\varnothing ,c_{i}^{j}=0 $;
② for $ j $ = $ 1 $ to $ k $ do
③  for $ i $ = $ 1 $ to $ {n}_{j} $ do
④    $ {p}_{i}\leftarrow \left({p}_{i}c_{i}^{j}+\left(y_{i}^{j}==1\right)\right)/\left(c_{i}^{j}+1\right) $;
⑤    $ c_{i}^{j+1}\leftarrow c_{i}^{j}+1 $;
⑥  end for
⑦  for $ i $ = $ 1 $ to $ m $ do
⑧    $ {T}_{i}\leftarrow 0 $;
⑨   for all $ k $ in $ {S}_{i} $ do
⑩     $ {T}_{i}\leftarrow {T}_{i}+{p}_{k} $;
⑪   end for
⑫  end for
⑬   $ {i}^{*}\leftarrow \arg $ $ \min {T}_{i} $;
⑭  if $ j==1 $ then
⑮   将$ \left\{1{,}2,… ,{n}_{1}\right\} $平均分配到$ {S}_{1},{S}_{2},… ,{S}_{m} $中;
⑯  else
⑰    $ {S}_{{{i}^{*}}}\leftarrow {S}_{{{i}^{*}}}\cup \left\{{n}_{j-1},{n}_{j-1}+1… ,{n}_{j}\right\} $;
⑱  end if
⑲ end for
在第$ j $次解码时,利用最新的活跃状态$ y_{i}^{j} $更新每个位置$ i $的活跃概率$ {p}_{i} $,根据已有的划分计算每个子集$ {S}_{i} $对应的概率和$ {T}_{i} $,并找出所有概率和中最小的$ {T}_{{{i}^{*}}} $。若是预填充阶段(算法1的行⑦)则将输入提示中的所有位置平均分配到各集合$ {S}_{i} $中。若是解码阶段(算法1的行⑧),则将新生成的位置并入$ {S}_{{{i}^{*}}} $。
控制均衡机制的硬件模块集成在APID模块中,其中包含活跃概率计算模块、概率求和模块以及用于维护子集划分的相关逻辑(图6(b))。概率计算以及概率求和模块的硬件并行度与APID的活跃位置识别并行度保持一致(见3.2节),当待处理的位置总数超出硬件并行度时,采用分批次处理方式多周期完成计算。每个时钟周期识别出的活跃位置信号被送入概率计算模块,该模块实现算法1行③的逻辑,更新当前批次位置的活跃概率。由于一个APID模块仅负责处理其所属的LAD-3D计算单元所分配到的位置,即当前LAD-3D计算单元对应的子集中的所有位置(见3.1节并行策略),因此概率求和模块直接将概率计算模块的输出进行累加。概率求和模块的主要结构为一个加法树和加法树输出的累加器。概率求和模块的输出在同一芯片上的各个LAD-3D计算单元之间通过环形NoC进行reduce-min运算。概率和最小的计算单元将当前解码步骤新生成的位置序号添加到其子集中。
在自回归解码的过程中,每次解码的活跃位置是不固定的,这种不确定性对访存效率构成挑战。具体而言,某个位置在当前解码步骤中是否活跃,取决于其注意力分数所处的区间,而该分数由当前查询向量和键向量之间的点积决定。若总是在计算出注意力分数后才对活跃位置进行访存操作,访存延迟会显著影响LAD注意力机制的性能。故而,本文提出一种高精度的活跃位置预测器,如图8所示。被预测为活跃位置的键值向量可提前从DRAM预取至片上缓存,这为联合调度线性层和注意力层的访存请求提供了优化空间,从而实现解码过程中访存负载的时间均匀性,以提升系统的整体性能。
算法2展示了预测器的预测逻辑,其分为预测和更新2个部分。该预测器是基于活跃历史记录来预测下一次解码的活跃状态的。对于上下文中的每个位置$ i $,记录最近$ hlen $次解码的活跃状态,其中“1”表示活跃,“−1”表示不活跃。这样,每个位置$ i $仅需维护一个$ hlen $比特大小的向量$ {\boldsymbol{h}}_{i} $,其带来的存储和访存开销影响很小。 例如,按照本文的实验设定$ hlen= 32 $,一个32头的注意力层在上下文长度为4096时,其活跃历史仅占0.5 MB。预测器的核心在于利用感知机学习活跃历史中的模式[24],而这些模式很大程度上受活跃概率的影响。因此,预测器按照活跃概率对活跃模式进行分类学习:概率值被分为$ \left[0{,}0.1\right], \left[0.1{,}0.2\right],… ,[0.9\sim 1.0] $这10个区间,所有位置按照其活跃概率所在的区间被分为10类,分别用$ {w}_{1},{w}_{2}… , {w}_{10} $这10个独立的感知机权重进行学习和预测。
在预测阶段,对于每个位置$ i $,预测器首先确定其活跃概率$ {p}_{i} $所属的第$ j\left[i\right] $个区间,并据此选取对应的权重$ {w}_{j\left[i\right]} $,随后计算该权重与活跃历史向量之间的点积$ {x}_{i} $。若点积$ {x}_{i}> 0 $或位置$ i $在当前解码步骤中是活跃的,则预测其在下一次解码是活跃的(算法2的行③)。
在更新阶段,预测器根据预测状态和真实状态之间的差异来更新感知机权重。当位置$ i $的输出$ {x}_{i} $的绝对值小于阈值$ \delta $时,表明感知机预测的稳定性较低;而若预测状态$ {y}_{i} $和真实状态$ {\hat{y}}_{i} $不一致,则说明感知机预测错误。只有在稳定性低且预测错误的情况下,预测器才会更新感知机权重(算法3的行①)。位置$ i $将自身模式$ {y}_{i}{\boldsymbol{h}}_{i} $加到权重中来更新感知机(算法3的行②)。若为$ {y}_{i}=1 $(活跃),则加$ {\boldsymbol{h}}_{i} $使得未来该权重与$ {\boldsymbol{h}}_{i} $的点积增大,提高将$ {\boldsymbol{h}}_{i} $预测为活跃的可能性;而若$ {y}_{i}=-1 $(不活跃),则加$ -{\boldsymbol{h}}_{i} $使得点积减小,提高预测为不活跃的可能性。在运行时,同一权重会被多个位置共享。如果直接累加所有位置的更新,会由于更新向量的模长过大而导致权重不稳定。因此,预测器采用投票机制来更新权重。如算法3的行⑦所示,对累加的更新向量取符号函数,确保更新向量的每个元素值为$ \pm 1 $,并且更新方向符合大多数位置的模式。
算法2. 预测器算法。
输入:位置$ i $的活跃历史$ {\boldsymbol{h}}_{i}\in {\left\{\pm 1\right\}}^{hlen} $,位置$ i $的活跃概率$ {p}_{i} $,预测器权重$ {\boldsymbol{w}}_{j}\in {\mathbb{Z}}^{hlen} $(其中$ j=1{,}2,… ,10 $);
输出:位置$ i $的预测值$ {x}_{i} $以及预测活跃状态$ {\hat{y}}_{i} $。
① 确定$ {p}_{i} $所在的区间$ {I}_{j\left[i\right]} $,其中${I}_{j[i]}=[0.1(j[i]-1) $,$ 0.1j[i]] $;
② $ {x}_{i}\leftarrow \left\langle {\boldsymbol{h}}_{i},{\boldsymbol{w}}_{j\left[i\right]}\right\rangle $;
③ if $ \left({x}_{i}> 0\right)|~({\boldsymbol{h}}_{i}\left[-1\right]==1) $ then
④   $ {\hat{y}}_{i}\leftarrow 1 $;
⑤ else
⑥   $ {\hat{y}}_{i}\leftarrow -1 $;
⑦ end if
算法3. 更新算法。
输入:预测算法的输出$ {x}_{i} $,$ {\hat{y}}_{i} $,真实活跃状态$ {y}_{i} $,阈值$ \delta $;
输出:更新后的权重$ {\boldsymbol{w}}_{j} $。
① $ u{m}_{i}\leftarrow \left(\left| {x}_{i}\right| \leq \delta \right)\& \left({y}_{i}\neq {\hat{y}}_{i}\right) $;
② $ {\Delta }_{i}\leftarrow {y}_{i}{\boldsymbol{h}}_{i}u{m}_{i} $;
③ for $ j $ = $ 1 $ to $ 10 $ do
④   $ \boldsymbol{u}\boldsymbol{p}\boldsymbol{d}\boldsymbol{a}\boldsymbol{t}\boldsymbol{e}\leftarrow \mathbf{0} $;
⑤  for all $ i $ in $ \left\{i|{p}_{i}\in {I}_{j}\right\} $ do
⑥    $ \boldsymbol{u}\boldsymbol{p}\boldsymbol{d}\boldsymbol{a}\boldsymbol{t}\boldsymbol{e}\leftarrow \boldsymbol{u}\boldsymbol{p}\boldsymbol{d}\boldsymbol{a}\boldsymbol{t}\boldsymbol{e}+{\Delta }_{i} $;
⑦  end for
⑧   $ {\boldsymbol{w}}_{j}\leftarrow {\boldsymbol{w}}_{j}+\text{sgn}\left({\boldsymbol{update}}\right) $;
⑨ end for
在硬件实现方面,由于历史$ {\boldsymbol{h}}_{i} $的元素仅取值为$ \pm 1 $,预测阶段计算点积 $ \left\langle {\boldsymbol{h}}_{i},{\boldsymbol{w}}_{j\left[i\right]}\right\rangle $ 时并不需要乘法器,可以通过加法树来实现该点积计算,省去了乘法器的硬件开销,如图8所示。预测器中负责更新的硬件包括用于根据预测结果来生成更新掩码$ u{m}_{i} $的逻辑单元、生成更新量$ \Delta $的逻辑单元以及权重更新模块。预测器根据$ u{m}_{i},{y}_{i} $,$ {\boldsymbol{h}}_{i} $,通过2级选择器生成$ {\Delta }_{i} $的取值,可为$ \mathbf{0},+{\boldsymbol{h}}_{i} $或$ -{\boldsymbol{h}}_{i} $。权重更新模块由权重更新累加器和符号函数逻辑组成,在计算每个感知机的更新量累加值后取其符号函数作为最终更新值。
基于活跃位置预测器,加速器可引入预取机制:在执行当前层计算的同时,从DRAM预取下一层预测活跃位置对应的键值向量、线性层权重等必要数据到片上SRAM。在实验中,预测未命中的活跃位置平均仅占1.96%,故绝大部分访存都在预取阶段完成,能够充分进行调度优化。
1)测试基准与对比方案
本文测试了各个硬件加速器在OPT 2.7B,OPT 6.7B,LLaMA2 7B,LLaMA2 13B[4,6]这4个具有代表性的大语言模型上的表现。本文选取英伟达A100-40 GB PCIe[17] GPU以及LAD-HBM加速器[13]作为对比平台。其中,A100-40GB部署了流行的大模型推理框架vLLM[25],该框架集成了包括PagedAttention在内的多种优化技术,能够充分发挥GPU的大模型推理性能。LAD-HBM加速器采用其论文中推理吞吐最高的版本(3.5 MB片上缓存),其采用了和A100-40GB配置相同的HBM作为主存,访存带宽为1555 GBps。
2)硬件实现
本文基于Verilog实现了LAD-3D加速器,且所有功能模块均通过功能验证。随后,使用Synopsys Design Compiler基于TSMC 22 nm工艺库对RTL代码进行了综合,综合结果表明LAD-3D计算单元满足了1ns的时序约束,从而确保LAD-3D逻辑芯片能够运行于1GHz的工作频率。LAD-3D的计算部件采用fp16的数值表示,与作为对比测试的GPU以及LAD-HBM采用相同的数值表示。本文采用CACTI[26]对SRAM进行模拟,通过配置阵列大小、端口数等参数来满足LAD-3D计算部件的访存需求,并模拟出功耗与面积数据。本文实现了精确到周期的LAD-3D模拟器,并拓展了开源的模拟器DRAMSim-3[27]以模拟混合键合DRAM,其DRAM参数采用了文献[18]中提供的数据配置。每个DRAM bank大小为32 MB,拥有1024个工作在500 MHz的I/O,提供59.6 GBps的峰值带宽。每个LAD-3D计算单元的算力为0.768 TFLOPS,垂直方向上混合键合了4个DRAM banks,提供238.4 GBps的峰值带宽以及128 MB的DRAM存储。为了准确模拟混合键合DRAM访存的延迟,本文在受测的PyTorch代码中插入访存记录(memory trace)抓取函数以生成每个测试例运行时的访存命令(包含访存命令类型和地址信息)序列。拓展的DRAMSim-3模拟器以所记录的访存命令序列为输入,模拟出混合键合DRAM的访存延迟。由于本文实现的加速器系统由8个LAD-3D芯片组成的流水线构成,每个LAD-3D芯片集成了32个计算单元,所以系统的峰值算力为196.608 TFLOPS,峰值访存带宽为59.6 TBps,总DRAM容量为32 GB。
本节展示了LAD-3D,LAD-HBM,A100 GPU的端到端大模型解码推理性能对比。其中,LAD-3D还做了无基于活跃概率的活跃位置均衡机制以及无活跃预取机制的消融实验,以评估这2项技术的有效性。为评估不同上下文长度下的模型性能,本文测试了在上下文长度为512,1024,…,4096时的解码吞吐性能(其中OPT模型仅支持最长2048的上下文长度)。
如图9所示,LAD-3D在所有测试例下都取得了显著好于GPU-vLLM和LAD-HBM的解码性能。在所有测试例子中,LAD-3D相较于GPU-vLLM在OPT-2.7B,OPT-6.7B,LLaMA2-7B,LLaMA2-13B模型上分别实现了55.2倍、35.7倍、44.5倍、50.0倍的几何平均加速比。相较于LAD-HBM,也实现了平均17.6倍、18.5倍、17.9倍、22.6倍的加速比。在执行LLaMA2-13B的解码推理时,LAD-3D在批处理大小分别为1,2,4的情况下,平均实现了1971 token/s,3752 token/s,6849 token/s的吞吐,而GPU-vLLM在相同负载下的平均吞吐仅为39 token/s,76 token/s,137 token/s,LAD-HBM的平均吞吐也仅为83 token/s,164 token/s,319 token/s。无活跃位置均衡机制的LAD-3D解码性能平均下降了24%,而无活跃位置预测预取机制的LAD-3D解码性能平均下降了22%。LAD-3D的性能优势来源于3个方面:一是活跃位置的稀疏性;二是混合键合的DRAM拥有远高于HBM的带宽;三是通过活跃位置的均衡调度以及预取机制,有效提升了混合键合DRAM的带宽利用率。
表1展示了在不同上下文长度下的各模型测试中,所有LAD计算单元的活跃位置的平均值与最大值的比值。该比值越接近于1,表示活跃位置负载在计算单元之间分布得越均衡。在所有测试场景中,该比值均高于0.96,且平均值达到0.98,这证明了LAD-3D引入的活跃位置均衡机制的有效性。表2展示了活跃位置预测器的活跃位置未命中的概率。在所有测试场景下,未命中概率均小于4%,平均值为1.96%。少数未命中的活跃位置所引起的访存开销对加速器性能的影响极小。
表3列出了单个LAD-3D计算单元的逻辑芯片中主要模块的面积和功耗。除表中所列模块外,单个LAD-3D计算单元还包含NoC以及DRAM I/O与控制器,其面积分别为0.1925 mm2和2.1875 mm2。单个LAD-3D芯片包含面积为57.3056 mm2的D2D模块。值得注意的是,混合键合DRAM的I/O与控制器集成在了逻辑芯片而非DRAM芯片上,且为实现更高的I/O数量面积较大。
DRAM芯片中,每个bank面积为2.74 mm2,容量为32 MB,有1024个I/O,运行频率为500 MHz,工作主电压VDD为1.2V±0.02 V。各类时序参数包括:行到列延迟16.4 ns,CAS延迟1.7 ns,行激活保持时间33.6 ns,预充电时间11.7 ns,行周期45.3 ns。
图10中展示了A100 GPU、LAD-HBM、LAD-3D(无活跃位置均衡机制)、LAD-3D(无活跃位置预取机制)和LAD-3D在大模型解码任务中的能耗表现。在批处理大小为1,2,4时,LAD-3D的DRAM平均能耗分别为0.09 J/token,0.05 J/token,0.03 J/token,而计算部件的平均能效分别为0.14 J/token,0.07 J/token,0.04 J/token。在批处理大小为1,2,4时LAD-3D的解码单个词元的能耗分别是GPU的5.1%,5.1%,4.9%,LAD-HBM的46.2%,46.9%,48.2%。无活跃位置均衡机制的LAD-3D推理能耗平均上升了14%,而无活跃位置预测预取机制的LAD-3D推理能耗平均上升了13%。在本文的实验中,混合键合DRAM展现出4~5倍于HBM的能耗,这主要得益于铜-铜键合的低电阻以及显著低于TSV的寄生电容。
本文详细分析了大模型的推理机制及其高访存需求的特性。混合键合DRAM因其高I/O密度、低寄生电阻的特性,提供了较传统DRAM显著更高的总带宽和能效。本文提出了一种基于混合键合DRAM的端侧大模型推理加速器LAD-3D。LAD-3D能够利用大模型推理过程中注意力分数的数值局部性减少不必要的键值缓存的访存,并以混合键合DRAM为主存实现了高能效高性能的端侧大模型推理。LAD-3D实现了基于活跃概率的活跃位置均衡机制和活跃位置预测预取机制,有效解决了访存的空间和时间均匀性问题,确保LAD-3D能够充分利用混合键合DRAM的带宽。

参考文献 引证文献
排序方式:
1
Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 30: 6000−6010
2
Brown T, Mann B, Ryder N, et al. Language models are few-shot learners[J]. Advances in Neural Information Processing Systems, 2020, 33: 1877−1901
3
Achiam J, Adler S, Agarwal S, et al. Gpt−4 technical report[J]. arXiv preprint, arXiv: 2303.08774, 2023
4
Touvron H, Martin L, Stone K, et al. LLaMA 2: Open foundation and fine-tuned chat models[J]. arXiv preprint, arXiv: 2307.09288, 2023
5
Liu Aixin, Feng Bei, Wang Bin, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model[J]. arXiv preprint, arXiv: 2405.04434, 2024
6
Zhang Susan, Roller S, Goyal N, et al. OPT: Open pre-trained transformer language models[J]. arXiv preprint, arXiv: 2205.01068, 2022
7
He Kaiming, Chen Xinlei, Xie Saining, et al. Masked autoencoders are scalable vision learners[C]//Proc of the 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. 2022: 16000−16009
8
Roziere B, Gehring J, Gloeckle F, et al. Code LLaMA: Open foundation models for code[J]. arXiv preprint, arXiv: 2308.12950, 2023
9
Liu Aixin, Feng Bei, Xue Bing, et al. Deepseek-v3 technical report[J]. arXiv preprint, arXiv: 2412.19437, 2024
10
Guo Daya, Zhu Qihao, Yang Dejian, et al. DeepSeek-Coder: When the large language model meets programming–The rise of code intelligence[J]. arXiv preprint, arXiv: 2401.14196, 2024
11
Kaplan J, McCandlish S, Henighan T, et al. Scaling laws for neural language models[J]. arXiv preprint, arXiv: 2001.08361, 2020
12
Wei J, Wang Xuezhi, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models[J]. Advances in Neural Information Processing Systems, 2022, 35: 24824−24837
13
Wang Haoran, Li Yuming, Xu Haobo, et al. LAD: Efficient accelerator for generative inference of LLM with locality aware decoding[C]//Proc of the 31st IEEE Int Symp on High Performance Computer Architecture (HPCA). Piscataway, NJ: IEEE, 2025: 1482−1495
14
葛旭冉, 欧洋, 王博, 等. 大语言模型推理中的存储优化技术综述[J]. 计算机研究与发展, 2025, 62(3): 545−562
Ge Xuran, Ou Yang, Wang Bo, et al. Survey of storage optimization techniques in large language model inference[J]. Journal of Computer Research and Development, 2025, 62(3): 545−562 (in Chinese)
15
Yu Zhongkai, Liang Shengwen, Ma Tianyun, et al. Cambricon-LLM: A chiplet-based hybrid architecture for on-device inference of 70b LLM[C]//Proc of the 57th IEEE/ACM Int Symp on Microarchitecture (MICRO). Piscataway, NJ: IEEE, 2024: 1474−1488
16
Bai Fujun, Jiang Xiping, Wang Song, et al. A stacked embedded DRAM array for LPDDR4/4X using hybrid bonding 3D integration with 34GB/s/1Gb 0.88 pJ/b logic-to-memory interface[C]//Proc of the 68th IEEE Int Electron Devices Meeting (IEDM). Piscataway, NJ: IEEE, 2020: 6.6. 1−6.6. 4
17
Choquette J, Gandhi W, Giroux O, et al. Nvidia a100 tensor core GPU: Performance and innovation[J]. IEEE Micro, 2021, 41(2): 29−35
18
Yue Zhiheng, Wang Huizheng, Fang Jiahao, et al. Exploiting similarity opportunities of emerging vision AI models on hybrid bonding architecture[C]//Proc of the 51st ACM/IEEE Annual Int Symp on Computer Architecture (ISCA). Piscataway, NJ: IEEE, 2024: 396−409
19
Niu Dimin, Li Shuangchen, Wang Yuhao, et al. 184QPS/W 64Mb/mm2 3D logic-to-DRAM hybrid bonding with process-near-memory engine for recommendation system[C]//Proc of the 2022 IEEE Int Solid-State Circuits Conf (ISSCC). Piscataway, NJ: IEEE, 2022, 65: 1−3
20
Park J, Choi J, Kyung K, et al. AttAcc! Unleashing the power of PIM for batched transformer-based generative model inference[C]//Proc of the 29th ACM Int Conf on Architectural Support for Programming Languages and Operating Systems, Volume 2. New York: ACM, 2024: 103−119
21
Amin H, Curtis K M, Hayes-Gill B R. Piecewise linear approximation applied to nonlinear function of a neural network[J]. IEE Proceedings-Circuits, Devices and Systems, 1997, 144(6): 313−317
22
Khan H, Khan A, Khan Z, et al. NPE: An FPGA-based overlay processor for natural language processing[J]. arXiv preprint, arXiv: 2104.06535, 2021
23
Jiang Xiping, Zuo Fengguo, Wang Song, et al. A 1596-GB/s 48-Gb stacked embedded DRAM 384-core SoC with hybrid bonding integration[J]. IEEE Solid-State Circuits Letters, 2022, 5: 110−113
24
Jiménez D A, Lin C. Dynamic branch prediction with perceptrons[C]//Proc of the 7th HPCA Int Symp on High-Performance Computer Architecture. Piscataway, NJ: IEEE, 2001: 197−206
25
Kwon W, Li Zhuohan, Zhuang Siyuan, et al. Efficient memory management for large language model serving with pagedattention[C]//Proc of the 29th Symp on Operating Systems Principles. New York: ACM, 2023: 611−626
26
Muralimanohar N, Balasubramonian R, Jouppi N P. CACTI 6.0: A tool to model large caches[J]. HP Laboratories, 2009, 27: 28
27
Li Shang, Yang Zhiyuan, Reddy D, et al. DRAMSim3: A cycle-accurate, thermal-capable DRAM simulator[J]. IEEE Computer Architecture Letters, 2020, 19(2): 106−109
2026年第63卷第7期
55
24
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202550301
  • 接收时间:2025-04-28
  • 首发时间:2026-11-17
  • 出版时间:2026-07-01
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2025-04-28
  • 修回日期:2026-02-06
  • 录用日期:2026-02-10
基金
作者信息
    1中国科学院计算技术研究所智能计算机研究中心 北京 100190
    2处理器芯片全国重点实验室(中国科学院计算技术研究所) 北京 100190
    3中国科学院大学 北京 100190

通讯作者:

许浩博()
参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202550301
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏