born in 1975. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include remote sensing image processing, medical image processing, data mining, machine learning, and deep learning
born in 1998. PhD candidate. Student member of CCF. His main research interests include knowledge distillation, model compression, and machine learning
born in 1981. PhD, associate professor, PhD supervisor. Member of CCF. His main research interests include machine learning, smart healthcare, and big data
born in 1983. PhD, lecturer, master supervisor. Senior member of CCF. Her main research interests include signal processing and information security
born in 1998. Master. His main research interests include model compression, natural language processing, and machine learning
born in 1999. PhD candidate. Student member of CCF. His main research interests include computer vision, image generation, and multimodal image processing
Knowledge distillation, as a key technique in deep learning, achieves model compression and acceleration by transferring knowledge from a large teacher model to a smaller student model. Under the premise of maintaining performance, this technology significantly reduces the requirements of computational resources and storage, and facilitates the deployment of high-performance models on resource-constrained edge devices. Firstly, we provide a systematic review of the recent research in knowledge distillation and categorize it from two perspectives: the type of knowledge and teacher-student model architectures. We comprehensively summarize the distillation methods based on three typical types of knowledge: output feature knowledge, intermediate feature knowledge, and relational feature knowledge, as well as distillation methods based on CNN to CNN architecture, CNN to ViT (vision Transformer) architecture, ViT to CNN architecture, and ViT to ViT architecture. Next, we explore various learning paradigms such as offline distillation, online distillation, self-distillation, data-free distillation, multi-teacher distillation, and assistant distillation. Then, we summarize distillation optimization methods based on the distillation process, knowledge structure, temperature coefficient, and loss functions and analyze improvements in distillation brought by adversarial techniques, automated machine learning, reinforcement learning, and diffusion models, and conclude with the implementation of distillation technology in common applications. Despite significant advancements in knowledge distillation, numerous challenges remain in both practical applications and theoretical research. Finally, we provide an in-depth analysis of these issues and offer insights into future development directions.
| 科 Family | 属数 Number of genus | 种数 Number of species | 占总种数比例 Percentage of total species (%) | 属 Genus | 种数 Number of species | 占总种数比例 Percentage of total species (%) |
|---|---|---|---|---|---|---|
| 鹅膏菌科Amanitaceae | 2 | 11 | 5.26 | 鹅膏菌属 Amanita | 10 | 4.78 |
| 小菇科 Mycenaceae | 2 | 12 | 5.74 | 丝盖伞属 Inocybe | 5 | 2.39 |
| 多孔菌科 Polyporaceae | 8 | 14 | 6.70 | 蜡蘑属 Laccaria | 5 | 2.39 |
| 红菇科 Russulaceae | 3 | 23 | 11.00 | 小皮伞属 Marasmius | 6 | 2.87 |
| 小菇属 Mycena | 11 | 5.26 | ||||
| 光柄菇属 Pluteus | 5 | 2.39 | ||||
| 红菇属 Russula | 17 | 8.13 | ||||
| 栓菌属 Trametes | 5 | 2.39 |