收藏切换
Survey on KV Cache Compression Methods for Large Language Model Inference
收藏切换
Shipeng Hu, Guangyan Zhang*, Weimin Zheng
Journal of Computer Research and Development | 2026, 63(4) : 1021 - 1038
Less
收藏切换
Journal of Computer Research and Development | 2026, 63(4): 1021-1038
• Computer Architecture •
Survey on KV Cache Compression Methods for Large Language Model Inference
Full
Shipeng Hu, Guangyan Zhang*, Weimin Zheng
Affiliations
  • Department of Computer Science and Technology, Tsinghua University, Beijing 100084
  • born in 1999. PhD candidate. His main research interests include storage systems and large language models

    born in 1976. PhD, associate professor, PhD supervisor. Distinguished member of CCF. His main research interests include big data computing, storage systems, and distributed processing

    born in 1946. Master, professor, PhD supervisor, academician of Chinese Academy of Engineering. Fellow of CCF. His main research interests include high performance computing, network storage, and parallel compiler

About Author:

born in 1999. PhD candidate. His main research interests include storage systems and large language models

born in 1976. PhD, associate professor, PhD supervisor. Distinguished member of CCF. His main research interests include big data computing, storage systems, and distributed processing

born in 1946. Master, professor, PhD supervisor, academician of Chinese Academy of Engineering. Fellow of CCF. His main research interests include high performance computing, network storage, and parallel compiler

Published: 2026-04-01 doi: 10.7544/issn1000-1239.202550461
Outline
收藏切换

Large language model (LLM) inference requires buffering and reusing intermediate data, namely KV cache. However, the rapid growth in the size of KV cache imposes significant storage pressure on systems. On one hand, the large-scale KV cache of ongoing inference requests leads to GPU memory bottlenecks, thereby limiting inference parallelism. On the other hand, the long-term memory capability enabled by LLM service providers results in large-scale KV cache being stored on persistent storage devices, incurring high storage costs and loading overhead. To alleviate the storage pressure caused by KV cache, existing research proposes various KV cache compression strategies to reduce its size. Nonetheless, it remains challenging to achieve high compression ratios while minimizing the impact on model response quality and keeping the additional computational overhead at an acceptable level. We present a systematic survey of recent advances in KV cache compression strategies. First, we analyze the scale of KV cache and its impact on inference efficiency and storage cost. Based on this, we identify three major challenges in designing KV cache compression strategies: preserving model response quality, minimizing computational overhead, and ensuring generalizability. We then categorize existing KV cache compression approaches into five types: merge-based, quantization-based, token eviction-based, sharing-based, and attention head pruning-based strategies. Each category is elaborated upon with representative examples, discussing their design principles and implementation techniques. Finally, we outline potential future research directions in KV cache compression.

large language model  /  inference service  /  KV cache  /  storage system  /  data compression
胡世鹏, 张广艳, 郑纬民. 大语言模型推理中的键值缓存压缩方法综述. 计算机研究与发展, 2026 , 63 (4) : 1021 -1038 . DOI: 10.7544/issn1000-1239.202550461
Shipeng Hu, Guangyan Zhang, Weimin Zheng. Survey on KV Cache Compression Methods for Large Language Model Inference[J]. Journal of Computer Research and Development, 2026 , 63 (4) : 1021 -1038 . DOI: 10.7544/issn1000-1239.202550461
Year 2026 volume 63 Issue 4
90
40
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202550461
  • Receive Date:2025-06-09
  • Online Date:2026-11-17
  • Published:2026-04-01
Article Data
Affiliations
History
  • Received:2025-06-09
  • Revised:2025-11-13
  • Accepted:2025-11-28
Affiliations
    Department of Computer Science and Technology, Tsinghua University, Beijing 100084
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202550461
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT