born in 1997. PhD. His main research interests include computer architecture, large language model system and accelerator design, and memory architecture based on advanced packaging
born in 1990. PhD, associate professor. Member of CCF. His main research interests include domain-specific processor chips and 2.5D/3D chiplet-based integrated chips
born in 1985. PhD, professor. Senior member of CCF. His main research interests include dedicated processor architecture and integrated chip systems
born in 1980. PhD, professor. Senior member of CCF. His main research interests include computer architecture and chips, and data computing chips and computing systems
Large language models (LLMs) have demonstrated remarkable capabilities across various domains, including natural language processing, computer vision, chat bots and content generation. Deploying LLMs on edge devices has attracted significant attention due to its advantages in service stability and data privacy. However, the decoding inference of LLMs is composed of memory-bound operations, and with model sizes growing and complex tasks demanding increasingly longer context lengths, the resulting memory access overhead becomes critical bottleneck limiting the deployment of LLMs on edge platforms. To lower the memory access demand of long context decoding, locality aware attention mechanism is proposed. By leveraging the numerical locality of attention scores during decoding, this mechanism compresses the key-value cache (KV cache), significantly reducing redundant memory access without compromising model accuracy. We present a locality-aware inference accelerator for LLMs with hybrid-bonded 3D DRAM. The accelerator not only reduces the overall memory traffic but also significantly improves the main memory bandwidth. It integrates load-balancing and prediction-based prefetching mechanisms tailored to the structural characteristics of hybrid-bonded 3D DRAM, thereby enhancing the bandwidth utilization. On the evaluated models, the proposed accelerator achieves on average
| 科 Family | 属数 Number of genus | 种数 Number of species | 占总种数比例 Percentage of total species (%) | 属 Genus | 种数 Number of species | 占总种数比例 Percentage of total species (%) |
|---|---|---|---|---|---|---|
| 鹅膏菌科Amanitaceae | 2 | 11 | 5.26 | 鹅膏菌属 Amanita | 10 | 4.78 |
| 小菇科 Mycenaceae | 2 | 12 | 5.74 | 丝盖伞属 Inocybe | 5 | 2.39 |
| 多孔菌科 Polyporaceae | 8 | 14 | 6.70 | 蜡蘑属 Laccaria | 5 | 2.39 |
| 红菇科 Russulaceae | 3 | 23 | 11.00 | 小皮伞属 Marasmius | 6 | 2.87 |
| 小菇属 Mycena | 11 | 5.26 | ||||
| 光柄菇属 Pluteus | 5 | 2.39 | ||||
| 红菇属 Russula | 17 | 8.13 | ||||
| 栓菌属 Trametes | 5 | 2.39 |