收藏切换
Knowledge Distillation in Visual Algorithms: A Survey
收藏切换
Haiwei Pan, Fengming Yu, Kejia Zhang*, Haiyan Lan, Qingyu Meng, Zhe Li
Journal of Computer Research and Development | 2026, 63(1) : 90 - 122
Less
收藏切换
Journal of Computer Research and Development | 2026, 63(1): 90-122
• Artificial Intelligence •
Knowledge Distillation in Visual Algorithms: A Survey
Full
Haiwei Pan, Fengming Yu, Kejia Zhang*, Haiyan Lan, Qingyu Meng, Zhe Li
Affiliations
  • College of Computer Science and Technology, Harbin Engineering University, Harbin 150001
  • born in 1975. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include remote sensing image processing, medical image processing, data mining, machine learning, and deep learning

    born in 1998. PhD candidate. Student member of CCF. His main research interests include knowledge distillation, model compression, and machine learning

    born in 1981. PhD, associate professor, PhD supervisor. Member of CCF. His main research interests include machine learning, smart healthcare, and big data

    born in 1983. PhD, lecturer, master supervisor. Senior member of CCF. Her main research interests include signal processing and information security

    born in 1998. Master. His main research interests include model compression, natural language processing, and machine learning

    born in 1999. PhD candidate. Student member of CCF. His main research interests include computer vision, image generation, and multimodal image processing

About Author:

born in 1975. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include remote sensing image processing, medical image processing, data mining, machine learning, and deep learning

born in 1998. PhD candidate. Student member of CCF. His main research interests include knowledge distillation, model compression, and machine learning

born in 1981. PhD, associate professor, PhD supervisor. Member of CCF. His main research interests include machine learning, smart healthcare, and big data

born in 1983. PhD, lecturer, master supervisor. Senior member of CCF. Her main research interests include signal processing and information security

born in 1998. Master. His main research interests include model compression, natural language processing, and machine learning

born in 1999. PhD candidate. Student member of CCF. His main research interests include computer vision, image generation, and multimodal image processing

Published: 2026-01-01 doi: 10.7544/issn1000-1239.202440694
Outline
收藏切换

Knowledge distillation, as a key technique in deep learning, achieves model compression and acceleration by transferring knowledge from a large teacher model to a smaller student model. Under the premise of maintaining performance, this technology significantly reduces the requirements of computational resources and storage, and facilitates the deployment of high-performance models on resource-constrained edge devices. Firstly, we provide a systematic review of the recent research in knowledge distillation and categorize it from two perspectives: the type of knowledge and teacher-student model architectures. We comprehensively summarize the distillation methods based on three typical types of knowledge: output feature knowledge, intermediate feature knowledge, and relational feature knowledge, as well as distillation methods based on CNN to CNN architecture, CNN to ViT (vision Transformer) architecture, ViT to CNN architecture, and ViT to ViT architecture. Next, we explore various learning paradigms such as offline distillation, online distillation, self-distillation, data-free distillation, multi-teacher distillation, and assistant distillation. Then, we summarize distillation optimization methods based on the distillation process, knowledge structure, temperature coefficient, and loss functions and analyze improvements in distillation brought by adversarial techniques, automated machine learning, reinforcement learning, and diffusion models, and conclude with the implementation of distillation technology in common applications. Despite significant advancements in knowledge distillation, numerous challenges remain in both practical applications and theoretical research. Finally, we provide an in-depth analysis of these issues and offer insights into future development directions.

knowledge distillation  /  model compression  /  deep learning  /  convolutional neural network  /  vision Transformer
潘海为, 于丰铭, 张可佳, 兰海燕, 孟庆宇, 李哲. 面向视觉算法的知识蒸馏研究综述. 计算机研究与发展, 2026 , 63 (1) : 90 -122 . DOI: 10.7544/issn1000-1239.202440694
Haiwei Pan, Fengming Yu, Kejia Zhang, Haiyan Lan, Qingyu Meng, Zhe Li. Knowledge Distillation in Visual Algorithms: A Survey[J]. Journal of Computer Research and Development, 2026 , 63 (1) : 90 -122 . DOI: 10.7544/issn1000-1239.202440694
Year 2026 volume 63 Issue 1
254
112
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202440694
  • Receive Date:2024-08-29
  • Online Date:2026-11-06
  • Published:2026-01-01
Article Data
Affiliations
History
  • Received:2024-08-29
  • Revised:2025-04-29
  • Accepted:2025-06-11
Affiliations
    College of Computer Science and Technology, Harbin Engineering University, Harbin 150001
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202440694
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT