收藏切换
Locality-Aware Decoding Accelerator for Large Language Models Based on Hybrid Bonding Architecture
收藏切换
Haoran Wang1, 2, 3, Haobo Xu1, 2, *, Ying Wang1, 2, Yinhe Han1, 2
Journal of Computer Research and Development | 2026, 63(7) : 1816 - 1830
Less
收藏切换
Journal of Computer Research and Development | 2026, 63(7): 1816-1830
• Computer Architecture •
Locality-Aware Decoding Accelerator for Large Language Models Based on Hybrid Bonding Architecture
Full
Haoran Wang1, 2, 3, Haobo Xu1, 2, *, Ying Wang1, 2, Yinhe Han1, 2
Affiliations
  • 1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
  • 2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
  • 3University of Chinese Academy of Sciences, Beijing 100190
  • born in 1997. PhD. His main research interests include computer architecture, large language model system and accelerator design, and memory architecture based on advanced packaging

    born in 1990. PhD, associate professor. Member of CCF. His main research interests include domain-specific processor chips and 2.5D/3D chiplet-based integrated chips

    born in 1985. PhD, professor. Senior member of CCF. His main research interests include dedicated processor architecture and integrated chip systems

    born in 1980. PhD, professor. Senior member of CCF. His main research interests include computer architecture and chips, and data computing chips and computing systems

About Author:

born in 1997. PhD. His main research interests include computer architecture, large language model system and accelerator design, and memory architecture based on advanced packaging

born in 1990. PhD, associate professor. Member of CCF. His main research interests include domain-specific processor chips and 2.5D/3D chiplet-based integrated chips

born in 1985. PhD, professor. Senior member of CCF. His main research interests include dedicated processor architecture and integrated chip systems

born in 1980. PhD, professor. Senior member of CCF. His main research interests include computer architecture and chips, and data computing chips and computing systems

Published: 2026-07-01 doi: 10.7544/issn1000-1239.202550301
Outline
收藏切换

Large language models (LLMs) have demonstrated remarkable capabilities across various domains, including natural language processing, computer vision, chat bots and content generation. Deploying LLMs on edge devices has attracted significant attention due to its advantages in service stability and data privacy. However, the decoding inference of LLMs is composed of memory-bound operations, and with model sizes growing and complex tasks demanding increasingly longer context lengths, the resulting memory access overhead becomes critical bottleneck limiting the deployment of LLMs on edge platforms. To lower the memory access demand of long context decoding, locality aware attention mechanism is proposed. By leveraging the numerical locality of attention scores during decoding, this mechanism compresses the key-value cache (KV cache), significantly reducing redundant memory access without compromising model accuracy. We present a locality-aware inference accelerator for LLMs with hybrid-bonded 3D DRAM. The accelerator not only reduces the overall memory traffic but also significantly improves the main memory bandwidth. It integrates load-balancing and prediction-based prefetching mechanisms tailored to the structural characteristics of hybrid-bonded 3D DRAM, thereby enhancing the bandwidth utilization. On the evaluated models, the proposed accelerator achieves on average 1971 token/s, 3752 token/s and 6849 token/s throughputs for batch sizes of 1, 2 and 4, respectively, with corresponding energy efficiencies of 0.23 J/token, 0.12 J/token and 0.07 J/token. The achieved throughput corresponds to 50.5 times, 49.4 times, 50.0 times that of the A100 GPU, and 23.7 times, 22.9 times, 21.5 times that of the LAD-HBM accelerator. Meanwhile, the energy consumption amounts to only 5.1%, 5.1%, 4.9% of GPU, and 46.2%, 46.9%, 48.2% of LAD-HBM.

large language model  /  auto-regressive decoding  /  hybrid bonding  /  accelerator  /  edge device
王浩然, 许浩博, 王颖, 韩银和. 基于混合键合架构的大模型局部性感知解码加速器. 计算机研究与发展, 2026 , 63 (7) : 1816 -1830 . DOI: 10.7544/issn1000-1239.202550301
Haoran Wang, Haobo Xu, Ying Wang, Yinhe Han. Locality-Aware Decoding Accelerator for Large Language Models Based on Hybrid Bonding Architecture[J]. Journal of Computer Research and Development, 2026 , 63 (7) : 1816 -1830 . DOI: 10.7544/issn1000-1239.202550301
Year 2026 volume 63 Issue 7
51
22
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202550301
  • Receive Date:2025-04-28
  • Online Date:2026-11-17
  • Published:2026-07-01
Article Data
Affiliations
History
  • Received:2025-04-28
  • Revised:2026-02-06
  • Accepted:2026-02-10
Affiliations
    1Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190
    2State Key Lab of Processors (Institute of Computing Technology, Chinese Academy of Sciences), Beijing 100190
    3University of Chinese Academy of Sciences, Beijing 100190
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202550301
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT