Article(id=1325490414429545387, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1325489890678420381, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202440694, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=research-article, receivedDate=1724860800000, receivedDateStr=2024-08-29, revisedDate=1745856000000, revisedDateStr=2025-04-29, acceptedDate=1749571200000, acceptedDateStr=2025-06-11, onlineDate=1793951141220, onlineDateStr=2026-11-06, pubDate=1767196800000, pubDateStr=2026-01-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1793951141220, onlineIssueDateStr=2026-11-06, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1793951141220, creator=13701087609, updateTime=1793951141220, updator=13701087609, issue=Issue{id=1325489890678420381, tenantId=1146029695717560320, journalId=1324759914146529478, year='2026', volume='63', issue='1', pageStart='1', pageEnd='272', issueExtLink='null', onlineDate='null', pubDate='1767196800000', pubDateStr='2026-01-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1793951016348, creator='13701087609', updateTime=1793952522017, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext={EN=IssueExt(id=1325496205974839692, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1325489890678420381, language=EN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=), CN=IssueExt(id=1325496205974839693, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1325489890678420381, language=CN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=)}, issueFiles=null, downloadFileDto=null}, startPage=90, endPage=122, ext={EN=ArticleExt(id=1325490425359901614, articleId=1325490414429545387, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=Knowledge Distillation in Visual Algorithms: A Survey, columnId=1325489891789910943, journalTitle=Journal of Computer Research and Development, columnName=Artificial Intelligence, runingTitle=null, highlight=null, articleAbstract=

Knowledge distillation, as a key technique in deep learning, achieves model compression and acceleration by transferring knowledge from a large teacher model to a smaller student model. Under the premise of maintaining performance, this technology significantly reduces the requirements of computational resources and storage, and facilitates the deployment of high-performance models on resource-constrained edge devices. Firstly, we provide a systematic review of the recent research in knowledge distillation and categorize it from two perspectives: the type of knowledge and teacher-student model architectures. We comprehensively summarize the distillation methods based on three typical types of knowledge: output feature knowledge, intermediate feature knowledge, and relational feature knowledge, as well as distillation methods based on CNN to CNN architecture, CNN to ViT (vision Transformer) architecture, ViT to CNN architecture, and ViT to ViT architecture. Next, we explore various learning paradigms such as offline distillation, online distillation, self-distillation, data-free distillation, multi-teacher distillation, and assistant distillation. Then, we summarize distillation optimization methods based on the distillation process, knowledge structure, temperature coefficient, and loss functions and analyze improvements in distillation brought by adversarial techniques, automated machine learning, reinforcement learning, and diffusion models, and conclude with the implementation of distillation technology in common applications. Despite significant advancements in knowledge distillation, numerous challenges remain in both practical applications and theoretical research. Finally, we provide an in-depth analysis of these issues and offer insights into future development directions.

, authors=Haiwei Pan, Fengming Yu, Kejia Zhang*, Haiyan Lan, Qingyu Meng, Zhe Li, authorsList=Haiwei Pan, Fengming Yu, Kejia Zhang, Haiyan Lan, Qingyu Meng, Zhe Li, authorCompany=null, correspAuthors=Kejia Zhang, authorNote=

born in 1975. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include remote sensing image processing, medical image processing, data mining, machine learning, and deep learning

born in 1998. PhD candidate. Student member of CCF. His main research interests include knowledge distillation, model compression, and machine learning

born in 1981. PhD, associate professor, PhD supervisor. Member of CCF. His main research interests include machine learning, smart healthcare, and big data

born in 1983. PhD, lecturer, master supervisor. Senior member of CCF. Her main research interests include signal processing and information security

born in 1998. Master. His main research interests include model compression, natural language processing, and machine learning

born in 1999. PhD candidate. Student member of CCF. His main research interests include computer vision, image generation, and multimodal image processing

, correspAuthorsNote=null, copyrightStatement=Copyright © 2026 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1325490573792124945, articleId=1325490414429545387, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=面向视觉算法的知识蒸馏研究综述, columnId=1325489892104483745, journalTitle=计算机研究与发展, columnName=人工智能, runingTitle=null, highlight=null, articleAbstract=

知识蒸馏作为深度学习中的关键技术,通过将大型教师模型的知识传递给较小的学生模型,实现了模型的压缩与加速。在保证性能的前提下,显著减少了计算资源和存储需求,促进了高性能模型在资源受限的边缘设备上的部署。围绕知识蒸馏的最新研究进展进行了系统性的综述,从知识类型和师生模型架构2个角度对知识蒸馏进行分类,详细汇总了输出特征知识、中间特征知识、关系特征知识3种典型知识类型的蒸馏方法,以及卷积架构到卷积架构、卷积架构到ViT(vision Transformer)架构、ViT架构到卷积架构和ViT架构到ViT架构的蒸馏方法;探讨了离线蒸馏、在线蒸馏、自蒸馏、无数据蒸馏、多教师蒸馏和助理蒸馏的学习方式;归纳了基于蒸馏过程、知识结构、温度系数及损失函数的蒸馏优化方法,分析了对抗性技术、自动机器学习、强化学习和扩散模型对蒸馏的改进,并总结了蒸馏技术在常见应用中的实现。尽管知识蒸馏取得了显著进展,但在实际应用和理论研究中仍面临诸多挑战。最后,对这些问题进行了深入分析,并对未来发展方向提出了见解。

, authors=潘海为, 于丰铭, 张可佳*, 兰海燕, 孟庆宇, 李哲, authorsList=潘海为, 于丰铭, 张可佳, 兰海燕, 孟庆宇, 李哲, authorCompany=null, correspAuthors=张可佳, authorNote=

1975年生。博士,教授,博士生导师。CCF高级会员。主要研究方向为遥感图像处理、医学图像处理、数据挖掘、机器学习、深度学习

1998年生。博士研究生。CCF学生会员。主要研究方向为知识蒸馏、模型压缩、机器学习

1981年生。博士,副教授,博士生导师。CCF会员。主要研究方向为机器学习、智慧医疗、大数据

1983年生。博士,讲师,硕士生导师。CCF高级会员。主要研究方向为信号处理、信息安全

1998年生。硕士。主要研究方向为模型压缩、自然语言处理、机器学习

1999年生。博士研究生。CCF学生会员。主要研究方向为计算机视觉、图像生成、多模态图像处理

, correspAuthorsNote=
张可佳()
, copyrightStatement=版权所有 © 2026 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=Cg+Yj0fWJF4mVglgl1ZhNw==, magXml=+dbfugX4DU/JND2BgNBVhw==, pdfUrl=null, pdf=XYICTV+GezntksZjoVsUNw==, pdfFileSize=2175625, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=c5TffK5Y2OVaiAIy+407Gg==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=ik4womjjWLX17AEJT9QBPg==, mapNumber=null, fund=null)}, authors=[Author(id=1325490584328216600, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=panhaiwei@hrbeu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325490586010132506, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490584328216600, language=EN, stringName=Haiwei Pan, firstName=Haiwei, middleName=null, lastName=Pan, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001, bio={"content":"

born in 1975. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include remote sensing image processing, medical image processing, data mining, machine learning, and deep learning

"}, bioImg=null, bioContent=

born in 1975. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include remote sensing image processing, medical image processing, data mining, machine learning, and deep learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490588136644635, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490584328216600, language=CN, stringName=潘海为, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001, bio={"img":"RvLRdxKhIw4RYAitV5qsfw==","content":"

1975年生。博士,教授,博士生导师。CCF高级会员。主要研究方向为遥感图像处理、医学图像处理、数据挖掘、机器学习、深度学习

"}, bioImg=RvLRdxKhIw4RYAitV5qsfw==, bioContent=

1975年生。博士,教授,博士生导师。CCF高级会员。主要研究方向为遥感图像处理、医学图像处理、数据挖掘、机器学习、深度学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325490582310756372, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, xref=null, ext=[AuthorCompanyExt(id=1325490582361088021, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001), AuthorCompanyExt(id=1325490582587580438, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001)])]), Author(id=1325490589852114973, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325490593136255008, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490589852114973, language=EN, stringName=Fengming Yu, firstName=Fengming, middleName=null, lastName=Yu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001, bio={"content":"

born in 1998. PhD candidate. Student member of CCF. His main research interests include knowledge distillation, model compression, and machine learning

"}, bioImg=null, bioContent=

born in 1998. PhD candidate. Student member of CCF. His main research interests include knowledge distillation, model compression, and machine learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490594843336737, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490589852114973, language=CN, stringName=于丰铭, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001, bio={"img":"4cLSSbHqrsnl91CyY3GZyA==","content":"

1998年生。博士研究生。CCF学生会员。主要研究方向为知识蒸馏、模型压缩、机器学习

"}, bioImg=4cLSSbHqrsnl91CyY3GZyA==, bioContent=

1998年生。博士研究生。CCF学生会员。主要研究方向为知识蒸馏、模型压缩、机器学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325490582310756372, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, xref=null, ext=[AuthorCompanyExt(id=1325490582361088021, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001), AuthorCompanyExt(id=1325490582587580438, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001)])]), Author(id=1325490597276033062, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=kejiazhang@hrbeu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1325490598555295784, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490597276033062, language=EN, stringName=Kejia Zhang, firstName=Kejia, middleName=null, lastName=Zhang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001, bio={"content":"

born in 1981. PhD, associate professor, PhD supervisor. Member of CCF. His main research interests include machine learning, smart healthcare, and big data

"}, bioImg=null, bioContent=

born in 1981. PhD, associate professor, PhD supervisor. Member of CCF. His main research interests include machine learning, smart healthcare, and big data

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490599914250281, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490597276033062, language=CN, stringName=张可佳, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001, bio={"img":"2DN+pW3Sz4H4IJYowRSC0A==","content":"

1981年生。博士,副教授,博士生导师。CCF会员。主要研究方向为机器学习、智慧医疗、大数据

"}, bioImg=2DN+pW3Sz4H4IJYowRSC0A==, bioContent=

1981年生。博士,副教授,博士生导师。CCF会员。主要研究方向为机器学习、智慧医疗、大数据

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325490582310756372, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, xref=null, ext=[AuthorCompanyExt(id=1325490582361088021, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001), AuthorCompanyExt(id=1325490582587580438, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001)])]), Author(id=1325490602325975083, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325490605404594222, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490602325975083, language=EN, stringName=Haiyan Lan, firstName=Haiyan, middleName=null, lastName=Lan, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001, bio={"content":"

born in 1983. PhD, lecturer, master supervisor. Senior member of CCF. Her main research interests include signal processing and information security

"}, bioImg=null, bioContent=

born in 1983. PhD, lecturer, master supervisor. Senior member of CCF. Her main research interests include signal processing and information security

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490607765987377, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490602325975083, language=CN, stringName=兰海燕, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001, bio={"img":"LcbrCCD2Mr8BKhKqmKb0kw==","content":"

1983年生。博士,讲师,硕士生导师。CCF高级会员。主要研究方向为信号处理、信息安全

"}, bioImg=LcbrCCD2Mr8BKhKqmKb0kw==, bioContent=

1983年生。博士,讲师,硕士生导师。CCF高级会员。主要研究方向为信号处理、信息安全

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325490582310756372, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, xref=null, ext=[AuthorCompanyExt(id=1325490582361088021, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001), AuthorCompanyExt(id=1325490582587580438, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001)])]), Author(id=1325490611121430579, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325490612409081910, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490611121430579, language=EN, stringName=Qingyu Meng, firstName=Qingyu, middleName=null, lastName=Meng, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001, bio={"content":"

born in 1998. Master. His main research interests include model compression, natural language processing, and machine learning

"}, bioImg=null, bioContent=

born in 1998. Master. His main research interests include model compression, natural language processing, and machine learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490614904692792, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490611121430579, language=CN, stringName=孟庆宇, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001, bio={"img":"lbXEooh6sVk23lCVpi7YtQ==","content":"

1998年生。硕士。主要研究方向为模型压缩、自然语言处理、机器学习

"}, bioImg=lbXEooh6sVk23lCVpi7YtQ==, bioContent=

1998年生。硕士。主要研究方向为模型压缩、自然语言处理、机器学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325490582310756372, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, xref=null, ext=[AuthorCompanyExt(id=1325490582361088021, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001), AuthorCompanyExt(id=1325490582587580438, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001)])]), Author(id=1325490616150401082, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, orderNo=5, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325490617878454333, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490616150401082, language=EN, stringName=Zhe Li, firstName=Zhe, middleName=null, lastName=Li, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001, bio={"content":"

born in 1999. PhD candidate. Student member of CCF. His main research interests include computer vision, image generation, and multimodal image processing

"}, bioImg=null, bioContent=

born in 1999. PhD candidate. Student member of CCF. His main research interests include computer vision, image generation, and multimodal image processing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490619988189246, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490616150401082, language=CN, stringName=李哲, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001, bio={"img":"2EItHUXjrCrylIjyNkGmfQ==","content":"

1999年生。博士研究生。CCF学生会员。主要研究方向为计算机视觉、图像生成、多模态图像处理

"}, bioImg=2EItHUXjrCrylIjyNkGmfQ==, bioContent=

1999年生。博士研究生。CCF学生会员。主要研究方向为计算机视觉、图像生成、多模态图像处理

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325490582310756372, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, xref=null, ext=[AuthorCompanyExt(id=1325490582361088021, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001), AuthorCompanyExt(id=1325490582587580438, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001)])])], keywords=[Keyword(id=1325490624123772991, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, orderNo=1, keyword=knowledge distillation), Keyword(id=1325490625482727488, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, orderNo=2, keyword=model compression), Keyword(id=1325490628737507393, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, orderNo=3, keyword=deep learning), Keyword(id=1325490629685420098, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, orderNo=4, keyword=convolutional neural network), Keyword(id=1325490633766477891, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, orderNo=5, keyword=vision Transformer), Keyword(id=1325490635532279877, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, orderNo=1, keyword=知识蒸馏), Keyword(id=1325490637251944518, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, orderNo=2, keyword=模型压缩), Keyword(id=1325490639244238919, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, orderNo=3, keyword=深度学习), Keyword(id=1325490642570321994, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, orderNo=4, keyword=卷积神经网络), Keyword(id=1325490643845390412, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, orderNo=5, keyword=视觉Transformer)], refs=[Reference(id=1325490767606718615, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=Lecun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278−2324, articleTitle=null, refAbstract=null), Reference(id=1325490769661927575, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks[C]//Proc of the 26th Annual Conference on Neural Information Processing Systems. Cambridge, MA: MIT, 2012: 1106−1114, articleTitle=null, refAbstract=null), Reference(id=1325490772237230233, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint, arXiv: 1409.1556, 2014, articleTitle=null, refAbstract=null), Reference(id=1325490774401491098, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Szegedy C, Liu Wei, Jia Yangqing, et al. Going deeper with convolutions[C]//Proc of 2015 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2015: 1−9, articleTitle=null, refAbstract=null), Reference(id=1325490778558046363, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision[C]//Proc of 2016 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2016: 2818−2826, articleTitle=null, refAbstract=null), Reference(id=1325490780659392669, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Szegedy C, Ioffe S, Vanhoucke V, et al. Inception-v4, inception-ResNet and the impact of residual connections on learning[C]//Proc of the 31st AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2017: 4278−4284, articleTitle=null, refAbstract=null), Reference(id=1325490783310192798, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=He Kaiming, Zhang Xiangyu, Ren Shaoqing, et al. Deep residual learning for image recognition[C]//Proc of 2016 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2016: 770−778, articleTitle=null, refAbstract=null), Reference(id=1325490785193435297, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Gao, Liu Zhuang, van der M L, et al. Densely connected convolutional networks[C]//Proc of 2017 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2017: 4700−4708, articleTitle=null, refAbstract=null), Reference(id=1325490787286392994, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Howard A G, Zhu Menglong, Chen Bo, et al. MobileNets: Efficient convolutional neural networks for mobile vision applications[J]. arXiv preprint, arXiv: 1704.04861, 2017, articleTitle=null, refAbstract=null), Reference(id=1325490789912027301, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Sandler M, Howard A, Zhu Menglong, et al. MobileNetV2: Inverted residuals and linear bottlenecks[C]//Proc of 2018 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 4510−4520, articleTitle=null, refAbstract=null), Reference(id=1325490792768348326, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Howard A, Sandler M, Chu G, et al. Searching for MobileNetV3[C]//Proc of 2019 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 1314−1324, articleTitle=null, refAbstract=null), Reference(id=1325490795268153512, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Tan Mingxing, Le Q. EfficientNet: Rethinking model scaling for convolutional neural networks[C]//Proc of the 36th Int Conf on Machine Learning. New York: ACM, 2019: 6105−6114, articleTitle=null, refAbstract=null), Reference(id=1325490797344333993, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Tan Mingxing, Le Q. EfficientNetV2: Smaller models and faster training[C]//Proc of the 36th Int Conf on Machine Learning. New York: ACM, 2021: 10096−10106, articleTitle=null, refAbstract=null), Reference(id=1325490799449874604, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proc of the 31st Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2017: 5998−6008, articleTitle=null, refAbstract=null), Reference(id=1325490801983234222, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16×16 words: Transformers for image recognition at scale[J]. arXiv preprint, arXiv: 2010.11929, 2021, articleTitle=null, refAbstract=null), Reference(id=1325490804571119791, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Ze, Lin Yutong, Cao Yue, et al. Swin Transformer: Hierarchical vision Transformer using shifted windows[C]//Proc of 2021 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2021: 10012−10022, articleTitle=null, refAbstract=null), Reference(id=1325490808719286451, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Touvron H, Cord M, Douze M, et al. Training data-efficient image transformers & distillation through attention[C]//Proc of the 38th Int Conf on Machine Learning. New York: ACM, 2021: 10347−10357, articleTitle=null, refAbstract=null), Reference(id=1325490811672076468, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Wenhai, Xie Enze, Li Xiang, et al. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions[C]//Proc of 2021 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2021: 568−578, articleTitle=null, refAbstract=null), Reference(id=1325490814201241782, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//Proc of the 38th Int Conf on Machine Learning. New York: ACM, 2021: 8748−8763, articleTitle=null, refAbstract=null), Reference(id=1325490817103700151, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Ramesh A, Pavlov M, Goh G, et al. Zero-shot text-to-image generation[C]//Proc of the 38th Int Conf on Machine Learning. New York: ACM, 2021: 8821−8831, articleTitle=null, refAbstract=null), Reference(id=1325490819246989498, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network[J]. arXiv preprint, arXiv: 1503.02531, 2015, articleTitle=null, refAbstract=null), Reference(id=1325490822183002299, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Gou Jianping, Yu Baosheng, Maybank S J, et al. Knowledge distillation: A survey[J]. International Journal of Computer Vision, 2021, 129(6): 1789−1819, articleTitle=null, refAbstract=null), Reference(id=1325490824712167612, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Lin, Yoon K J. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(6): 3048−3068, articleTitle=null, refAbstract=null), Reference(id=1325490827589460158, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=黄震华, 杨顺志, 林威, 等. 知识蒸馏研究综述[J]. 计算机学报, 2022, 45(3): 624−653, articleTitle=null, refAbstract=null), Reference(id=1325490831314002111, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Zhenhua, Yang Shunzhi, Lin Wei, et al. Knowledge distillation: A survey[J]. Chinese Journal of Computers, 2022, 45(3): 624−653 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1325490833503428801, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=邵仁荣, 刘宇昂, 张伟, 等. 深度学习中知识蒸馏研究综述[J]. 计算机学报, 2022, 45(8): 1638−1673, articleTitle=null, refAbstract=null), Reference(id=1325490835130818755, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=26, authorNames=null, journalName=null, refType=null, unstructuredReference=Shao Renrong, Liu Yuang, Zhang Wei, et al. A survey of knowledge distillation in deep learning[J]. Chinese Journal of Computers, 2022, 45(8): 1638−1673 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1325490836863066310, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=27, authorNames=null, journalName=null, refType=null, unstructuredReference=司兆峰, 齐洪钢. 知识蒸馏方法研究与应用综述[J]. 中国图象图形学报, 2023, 28(9): 2817−2832, articleTitle=null, refAbstract=null), Reference(id=1325490838914080968, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=28, authorNames=null, journalName=null, refType=null, unstructuredReference=Si Zhaofeng, Qi Honggang. Survey on knowledge distillation and its application[J]. Journal of Image and Graphics, 2023, 28(9): 2817−2832 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1325490840575025353, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=29, authorNames=null, journalName=null, refType=null, unstructuredReference=Oord A, Li Yazhe, Vinyals O. Representation learning with contrastive predictive coding[J]. arXiv preprint, arXiv: 1807.03748, 2019, articleTitle=null, refAbstract=null), Reference(id=1325490843989188811, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=28, rfOrder=30, authorNames=null, journalName=null, refType=null, unstructuredReference=Bucila C, Caruana R, Niculescu-Mizil A. Model compression[C]//Proc of the 12th ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining. New York: ACM, 2006: 535−541, articleTitle=null, refAbstract=null), Reference(id=1325490846447050956, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=29, rfOrder=31, authorNames=null, journalName=null, refType=null, unstructuredReference=Sau B B, Balasubramanian V N. Deep model compression: Distilling knowledge from noisy teachers[J]. arXiv preprint, arXiv: 1610.09650, 2016, articleTitle=null, refAbstract=null), Reference(id=1325490848598728910, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=30, rfOrder=32, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Ying, Xiang Tao, Hospedales T M, et al. Deep mutual learning[C]//Proc of 2018 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 4320−4328, articleTitle=null, refAbstract=null), Reference(id=1325490851027230927, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=31, rfOrder=33, authorNames=null, journalName=null, refType=null, unstructuredReference=Furlanello T, Lipton Z, Tschannen M, et al. Born again neural networks[C]//Proc of the 35th Int Conf on Machine Learning. New York: ACM, 2018: 1607−1616, articleTitle=null, refAbstract=null), Reference(id=1325490852361019601, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=32, rfOrder=34, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Chenglin, Xie Lingxi, Qiao Siyuan, et al. Training deep neural networks in generations: A more tolerant teacher educates better students[C]//Proc of the 33rd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 5628−5635, articleTitle=null, refAbstract=null), Reference(id=1325490853963243730, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=33, rfOrder=35, authorNames=null, journalName=null, refType=null, unstructuredReference=Lee H, Hwang S J, Shin J. Self-supervised label augmentation via input transformations[C]//Proc of the 37th Int Conf on Machine Learning. New York: ACM, 2020: 5714−5724, articleTitle=null, refAbstract=null), Reference(id=1325490856945393876, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=34, rfOrder=36, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Jing, Martinez B, Bulat A, et al. Knowledge distillation via softmax regression representation learning[C/OL]//Proc of 9th Int Conf on Learning Representations, New York: Ithaca, 2020[2024-08-20]. https://openreview.net/forum?id=ZzwDy_wiWv, articleTitle=null, refAbstract=null), Reference(id=1325490859482947797, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=35, rfOrder=37, authorNames=null, journalName=null, refType=null, unstructuredReference=Ba J, Caruana R. Do deep nets really need to be deep?[C]//Proc of the 28th Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2014: 2654−2662, articleTitle=null, refAbstract=null), Reference(id=1325490861131309272, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=36, rfOrder=38, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhou Guorui, Fan Ying, Cui Runpeng, et al. Rocket launching: A universal and efficient framework for training well-performing light net[C]//Proc of the 32nd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2018: 4580−4587, articleTitle=null, refAbstract=null), Reference(id=1325490864449003737, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=37, rfOrder=39, authorNames=null, journalName=null, refType=null, unstructuredReference=Son W, Na J, Choi J, et al. Densely guided knowledge distillation using multiple teacher assistants[C]//Proc of 2021 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2021: 9375−9384, articleTitle=null, refAbstract=null), Reference(id=1325490866198028508, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=38, rfOrder=40, authorNames=null, journalName=null, refType=null, unstructuredReference=Mirzadeh S I, Farajtabar M, Li Ang, et al. Improved knowledge distillation via teacher assistant[C]//Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 5191−5198, articleTitle=null, refAbstract=null), Reference(id=1325490868228071645, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=39, rfOrder=41, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Xuewei, Li Songyuan, Omar B, et al. ResKD: Residual-guided knowledge distillation[J]. IEEE Transactions on Image Processing, 2021, 30: 4735−4746, articleTitle=null, refAbstract=null), Reference(id=1325490870413304031, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=40, rfOrder=42, authorNames=null, journalName=null, refType=null, unstructuredReference=Romero A, Ballas N, Kahou S E, et al. FitNets: Hints for thin deep nets[J]. arXiv preprint, arXiv: 1412.6550, 2015, articleTitle=null, refAbstract=null), Reference(id=1325490871705149664, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=41, rfOrder=43, authorNames=null, journalName=null, refType=null, unstructuredReference=Zagoruyko S, Komodakis N. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer[J]. arXiv preprint, arXiv: 1612.03928, 2017, articleTitle=null, refAbstract=null), Reference(id=1325490873340928227, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=42, rfOrder=44, authorNames=null, journalName=null, refType=null, unstructuredReference=Lin Sihao, Xie Hongwei, Wang Bing, et al. Knowledge distillation via the target-aware transformer[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 10915−10924, articleTitle=null, refAbstract=null), Reference(id=1325490875412914404, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=43, rfOrder=45, authorNames=null, journalName=null, refType=null, unstructuredReference=Lao Shanshan, Song Guanglu, Liu Boxiao, et al. Masked autoencoders are stronger knowledge distillers[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 6361−6370, articleTitle=null, refAbstract=null), Reference(id=1325490877505872104, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=44, rfOrder=46, authorNames=null, journalName=null, refType=null, unstructuredReference=Guo Ziyao, Yan Haonan, Li Hui, et al. Class attention transfer based knowledge distillation[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 11868−11877, articleTitle=null, refAbstract=null), Reference(id=1325490879150039273, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=45, rfOrder=47, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Yuan, Huang Tao, Liu Jiaming, et al. FreeKD: Knowledge distillation via semantic frequency prompt[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 15931−15940, articleTitle=null, refAbstract=null), Reference(id=1325490881675010284, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=46, rfOrder=48, authorNames=null, journalName=null, refType=null, unstructuredReference=Jia Menglin, Tang Luming, Chen B, et al. Visual prompt tuning[C]//Proc of the 17th European Conf on Computer Vision. Berlin: Springer, 2022: 709−727, articleTitle=null, refAbstract=null), Reference(id=1325490884233535725, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=47, rfOrder=49, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Zehao, Wang Naiyan. Like what you like: Knowledge distill via neuron selectivity transfer[J]. arXiv preprint, arXiv: 1707.01219, 2017, articleTitle=null, refAbstract=null), Reference(id=1325490886724952303, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=48, rfOrder=50, authorNames=null, journalName=null, refType=null, unstructuredReference=Heo B, Lee M, Yun S, et al. Knowledge transfer via distillation of activation boundaries formed by hidden neurons[C]//Proc of the 33rd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 3779−3787, articleTitle=null, refAbstract=null), Reference(id=1325490888364925171, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=49, rfOrder=51, authorNames=null, journalName=null, refType=null, unstructuredReference=Shen Chengchao, Wang Xinchao, Song Jie, et al. Amalgamating knowledge towards comprehensive classification[C]//Proc of the 33rd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 3068−3075, articleTitle=null, refAbstract=null), Reference(id=1325490890126532852, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=50, rfOrder=52, authorNames=null, journalName=null, refType=null, unstructuredReference=Wei Yi, Pan Xinyu, Qin Hongwei, et al. Quantization mimic: Towards very tiny CNN for object detection[C]//Proc of the 15th European Conf on Computer Vision. Berlin: Springer, 2018: 274−290, articleTitle=null, refAbstract=null), Reference(id=1325490892626338037, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=51, rfOrder=53, authorNames=null, journalName=null, refType=null, unstructuredReference=Ahn S, Hu S X, Damianou A, et al. Variational information distillation for knowledge transfer[C]//Proc of 2019 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 9155−9163, articleTitle=null, refAbstract=null), Reference(id=1325490894727684342, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=52, rfOrder=54, authorNames=null, journalName=null, refType=null, unstructuredReference=Heo B, Kim J, Yun S, et al. A comprehensive overhaul of feature distillation[C]//Proc of 2019 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 1921−1930, articleTitle=null, refAbstract=null), Reference(id=1325490898427060472, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=53, rfOrder=55, authorNames=null, journalName=null, refType=null, unstructuredReference=Yim J, Joo D, Bae J, et al. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning[C]//Proc of 2017 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2017: 7130−7138, articleTitle=null, refAbstract=null), Reference(id=1325490902696861945, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=54, rfOrder=56, authorNames=null, journalName=null, refType=null, unstructuredReference=Srinivas S, Fleuret F. Knowledge transfer with Jacobian matching[C]//Proc of the 35th Int Conf on Machine Learning. New York: ACM, 2018: 4730−4738, articleTitle=null, refAbstract=null), Reference(id=1325490905993584890, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=55, rfOrder=57, authorNames=null, journalName=null, refType=null, unstructuredReference=Kim J, Park S, Kwak N. Paraphrasing complex network: Network compression via factor transfer[C]//Proc of the 32nd Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2018: 2765−2774, articleTitle=null, refAbstract=null), Reference(id=1325490907352539391, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=56, rfOrder=58, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Chuanguang, An Zhulin, Cai Linhang, et al. Hierarchical self-supervised augmented knowledge distillation[C]//Proc of the 30th Int Joint Conf on Artificial Intelligence. San Francisco, CA: Morgan Kaufmann, 2021: 1217−1223, articleTitle=null, refAbstract=null), Reference(id=1325490910590542081, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=57, rfOrder=59, authorNames=null, journalName=null, refType=null, unstructuredReference=Tian Yonglong, Krishnan D, Isola P. Contrastive representation distillation[J]. arXiv preprint, arXiv: 1910.10699, 2022, articleTitle=null, refAbstract=null), Reference(id=1325490913199399171, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=58, rfOrder=60, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Defang, Mei Jian Ping, Zhang Hailin, et al. Knowledge distillation with the reused teacher classifier[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 11923−11932, articleTitle=null, refAbstract=null), Reference(id=1325490915720175878, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=59, rfOrder=61, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Xiaobo, Fu Tianyu, Liao Shengcai, et al. Exclusivity-consistency regularized knowledge distillation for face recognition[C]//Proc of the 16th European Conf on Computer Vision. Berlin: Springer, 2020: 325−342, articleTitle=null, refAbstract=null), Reference(id=1325490918555525384, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=60, rfOrder=62, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Pengguang, Liu Shu, Zhao Hengshuang, et al. Distilling knowledge via knowledge review[C]//Proc of 2021 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2021: 5006−5015, articleTitle=null, refAbstract=null), Reference(id=1325490920753340682, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=61, rfOrder=63, authorNames=null, journalName=null, refType=null, unstructuredReference=You Shan, Xu Chang, Xu Chao, et al. Learning from multiple teacher networks[C]//Proc of the 23rd ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining. New York: ACM, 2017: 1285−1294, articleTitle=null, refAbstract=null), Reference(id=1325490924519825676, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=62, rfOrder=64, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Chenrui, Peng Yuxin. Better and faster: Knowledge transfer from multiple self-supervised learning tasks via graph distillation for video classification[C]//Proc of the 27th Int Joint Conf on Artificial Intelligence. San Francisco, CA: Morgan Kaufmann, 2018: 1135−1141, articleTitle=null, refAbstract=null), Reference(id=1325490926235296014, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=63, rfOrder=65, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Defang, Mei Jianping, Zhang Yuan, et al. Cross-layer distillation with semantic calibration[C]//Proc of the 35th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2021: 7028−7036, articleTitle=null, refAbstract=null), Reference(id=1325490928366002449, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=64, rfOrder=66, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Li, Huang Qingle, Lin Sihao, et al. Exploring inter-channel correlation for diversity-preserved knowledge distillation[C]//Proc of 2021 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2021: 8251−8260, articleTitle=null, refAbstract=null), Reference(id=1325490929989198098, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=65, rfOrder=67, authorNames=null, journalName=null, refType=null, unstructuredReference=Jin Ying, Wang Jiaqi, Lin Dahua. Multi-level logit distillation[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 24276−24285, articleTitle=null, refAbstract=null), Reference(id=1325490933235589396, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=66, rfOrder=68, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Yuntao, Wang Naiyan, Zhang Zhaoxiang. DarkRank: Accelerating deep metric learning via cross sample similarities transfer[C]//Proc of the 32nd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2018: 2852−2859, articleTitle=null, refAbstract=null), Reference(id=1325490935383073046, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=67, rfOrder=69, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Yufan, Cao Jiajiong, Li Bing, et al. Knowledge distillation via instance relationship graph[C]//Proc of 2019 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 7089−7097, articleTitle=null, refAbstract=null), Reference(id=1325490938277142808, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=68, rfOrder=70, authorNames=null, journalName=null, refType=null, unstructuredReference=Peng Baoyun, Jin Xiao, Li Dongsheng, et al. Correlation congruence for knowledge distillation[C]//Proc of 2019 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 5006−5015, articleTitle=null, refAbstract=null), Reference(id=1325490940869222681, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=69, rfOrder=71, authorNames=null, journalName=null, refType=null, unstructuredReference=Tung F, Mori G. Similarity-preserving knowledge distillation[C]//Proc of 2019 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 1365−1374, articleTitle=null, refAbstract=null), Reference(id=1325490944228860187, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=70, rfOrder=72, authorNames=null, journalName=null, refType=null, unstructuredReference=Xu Haohang, Fang Jiemin, Zhang Xiaopeng, et al. Bag of instances aggregation boosts self-supervised distillation[J]. arXiv preprint, arXiv: 2107.01691, 2022, articleTitle=null, refAbstract=null), Reference(id=1325490947076792604, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=71, rfOrder=73, authorNames=null, journalName=null, refType=null, unstructuredReference=刘静, 郑铜亚, 郝沁汾. 图知识蒸馏综述: 算法分类与应用分析[J]. 软件学报, 2023, 35(2): 675−710, articleTitle=null, refAbstract=null), Reference(id=1325490949652095262, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=71, rfOrder=74, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Jing, Zheng Tongya, Hao Qinfen. Survey on knowledge distillation with graph: Algorithms classification and application analysis[J]. Journal of Software, 2023, 35(2): 675−710 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1325490951724081439, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=72, rfOrder=75, authorNames=null, journalName=null, refType=null, unstructuredReference=Park W, Kim D, Lu Yan, et al. Relational knowledge distillation[C]//Proc of 2019 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 3967−3976, articleTitle=null, refAbstract=null), Reference(id=1325490953020121376, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=73, rfOrder=76, authorNames=null, journalName=null, refType=null, unstructuredReference=Ren Sucheng, Gao Zhengqi, Hua Tianyu, et al. Co-advise: Cross inductive bias distillation[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 16752−16761, articleTitle=null, refAbstract=null), Reference(id=1325490955939356962, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=74, rfOrder=77, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Duo, Hu Jie, Wang Changhu, et al. Involution: Inverting the inherence of convolution for visual recognition[C]//Proc of 2021 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2021: 12316−12325, articleTitle=null, refAbstract=null), Reference(id=1325490957642244388, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=75, rfOrder=78, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Borui, Song Renjie, Liang Jiajun. Cumulative spatial knowledge distillation for vision transformers[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 6123−6132, articleTitle=null, refAbstract=null), Reference(id=1325490959668093222, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=76, rfOrder=79, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Yufan, Cao Jiajiong, Li Bing, et al. Cross-architecture knowledge distillation[C]//Proc of the 16th Asian Conf on Computer Vision. Berlin: Springer, 2022: 179−195, articleTitle=null, refAbstract=null), Reference(id=1325490961375174951, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=77, rfOrder=80, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Weisong, Zhu Xiangyu, He Zhixiang, et al. Cross-architecture distillation for face recognition[C]//Proc of the 31st ACM Int Conf on Multimedia. New York: ACM, 2023: 8076−8085, articleTitle=null, refAbstract=null), Reference(id=1325490965133271337, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=78, rfOrder=81, authorNames=null, journalName=null, refType=null, unstructuredReference=Hao Zhiwei, Guo Jianyuan, Han Kai, et al. One-for-all: Bridge the gap between heterogeneous architectures in knowledge distillation[C]//Proc of the 37th Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2023: 79570−79582, articleTitle=null, refAbstract=null), Reference(id=1325490967729545515, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=79, rfOrder=82, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Guopeng, Wang Qiang, Yan Ke, et al. Fuse before transfer: Knowledge fusion for heterogeneous distillation[C]// Proc of 2025 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2025: 3445-3454, articleTitle=null, refAbstract=null), Reference(id=1325490970262905132, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=80, rfOrder=83, authorNames=null, journalName=null, refType=null, unstructuredReference=Ni Jianyuan, Tang Hao, Shang Yuzhang, et al. Adaptive cross-architecture mutual knowledge distillation[C]//Proc of the 18th Int Conf on Automatic Face and Gesture Recognition. Piscataway, NJ: IEEE, 2024: 1−5, articleTitle=null, refAbstract=null), Reference(id=1325490974851473710, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=81, rfOrder=84, authorNames=null, journalName=null, refType=null, unstructuredReference=Zheng Zhimeng, Huang Tao, Li Gongsheng, et al. Promoting CNNs with cross-architecture knowledge distillation for efficient monocular depth estimation[J]. arXiv preprint, arXiv: 2404.16386, 2024, articleTitle=null, refAbstract=null), Reference(id=1325490978974474543, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=82, rfOrder=85, authorNames=null, journalName=null, refType=null, unstructuredReference=Wu Kan, Zhang Jinnian, Peng Houwen, et al. TinyViT: Fast pretraining distillation for small vision transformers[C]//Proc of the 17th European Conf on Computer Vision. Berlin: Springer, 2022: 68−85, articleTitle=null, refAbstract=null), Reference(id=1325490981117763889, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=83, rfOrder=86, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Jinnian, Peng Houwen, Wu Kan, et al. MiniViT: Compressing vision transformers with weight multiplexing[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 12135−12144, articleTitle=null, refAbstract=null), Reference(id=1325490983223304498, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=84, rfOrder=87, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Zhendong, Li Zhe, Zeng Ailing, et al. ViTKD: Practical guidelines for ViT feature knowledge distillation[J]. arXiv preprint, arXiv: 2209.02432, 2022, articleTitle=null, refAbstract=null), Reference(id=1325490987023343926, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=85, rfOrder=88, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Tianyang, Singh K Y, Appalaraju S, et al. No head left behind–multi-head alignment distillation for transformers[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 7514−7524, articleTitle=null, refAbstract=null), Reference(id=1325490989896442167, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=86, rfOrder=89, authorNames=null, journalName=null, refType=null, unstructuredReference=Ren Sucheng, Wei Fangyun, Zhang Zheng, et al. TinyMIM: An empirical study of distilling MIM pre-trained models[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 3687−3697, articleTitle=null, refAbstract=null), Reference(id=1325490992811483449, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=87, rfOrder=90, authorNames=null, journalName=null, refType=null, unstructuredReference=Lin Han, Han Guangxing, Ma Jiawei, et al. Supervised masked knowledge distillation for few-shot transformers[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 19649−19659, articleTitle=null, refAbstract=null), Reference(id=1325490997060313402, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=88, rfOrder=91, authorNames=null, journalName=null, refType=null, unstructuredReference=Bai Yutong, Wang Zeyu, Xiao Junfei, et al. Masked autoencoders enable efficient knowledge distillers[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 24256−24265, articleTitle=null, refAbstract=null), Reference(id=1325491001632104765, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=89, rfOrder=92, authorNames=null, journalName=null, refType=null, unstructuredReference=He Kaiming, Chen Xinlei, Xie Saining, et al. Masked autoencoders are scalable vision learners[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 16000−16009, articleTitle=null, refAbstract=null), Reference(id=1325491004538757441, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=90, rfOrder=93, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Zhiyu, Huang Bingkun, Xing Sen, et al. Asymmetric masked distillation for pre-training small foundation models[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 18516−18526, articleTitle=null, refAbstract=null), Reference(id=1325491005499253059, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=91, rfOrder=94, authorNames=null, journalName=null, refType=null, unstructuredReference=Ye Hanjia, Lu Su, Zhan Dechuan. Distilling cross-task knowledge via relationship matching[C]//Proc of 2020 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 12396−12405, articleTitle=null, refAbstract=null), Reference(id=1325491006677852484, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=92, rfOrder=95, authorNames=null, journalName=null, refType=null, unstructuredReference=Lê H Â, Pham M T. Self-training and multi-task learning for limited data: Evaluation study on object detection[C]//Proc of 2019 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 1003−1009, articleTitle=null, refAbstract=null), Reference(id=1325491009219600709, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=93, rfOrder=96, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Deng, Wu Aming, Han Yahong, et al. Prototype-guided cross-task knowledge distillation for large-scale models[J]. arXiv preprint, arXiv: 2212.13180, 2022, articleTitle=null, refAbstract=null), Reference(id=1325491011283198279, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=94, rfOrder=97, authorNames=null, journalName=null, refType=null, unstructuredReference=Auty D, Miles R, Kolbeinsson B, et al. Learning to project for cross-task knowledge distillation[J]. arXiv preprint, arXiv: 2403.14494, 2024, articleTitle=null, refAbstract=null), Reference(id=1325491014282125640, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=95, rfOrder=98, authorNames=null, journalName=null, refType=null, unstructuredReference=Yuan Mingkuan, Peng Yuxin. CKD: Cross-task knowledge distillation for text-to-image synthesis[J]. IEEE Transactions on Multimedia, 2020, 22(8): 1955−1968, articleTitle=null, refAbstract=null), Reference(id=1325491016714821962, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=96, rfOrder=99, authorNames=null, journalName=null, refType=null, unstructuredReference=Bayramli B, Ding Yue, Lu Hongtao. Integrating semantic segmentation model for self-supervised scene flow estimation via cross task distillation[C]//Proc of 2024 Int Joint Conf on Neural Networks. Piscataway, NJ: IEEE, 2024: 1−8, articleTitle=null, refAbstract=null), Reference(id=1325491019432730955, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=97, rfOrder=100, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Lin, Chae Y, Yoon S H, et al. EvDistill: Asynchronous events to end-task learning via bidirectional reconstruction-guided cross-modal knowledge distillation[C]//Proc of 2021 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2021: 608−619, articleTitle=null, refAbstract=null), Reference(id=1325491020586164557, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=98, rfOrder=101, authorNames=null, journalName=null, refType=null, unstructuredReference=Qian Biao, Wang Yang, Yin Hongzhi, et al. Switchable online knowledge distillation[C]//Proc of the 17th European Conf on Computer Vision. Berlin: Springer, 2022: 449−466, articleTitle=null, refAbstract=null), Reference(id=1325491023023055183, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=99, rfOrder=102, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Chuanguang, An Zhulin, Zhou Helong, et al. Online knowledge distillation via mutual contrastive learning for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(8): 10212−10227, articleTitle=null, refAbstract=null), Reference(id=1325491025510277456, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=100, rfOrder=103, authorNames=null, journalName=null, refType=null, unstructuredReference=Jacob G M, Agarwal V, Stenger B. Online knowledge distillation for multi-task learning[C]//Proc of 2023 IEEE/CVF Winter Conf on Applications of Computer Vision. Piscataway, NJ: IEEE, 2023: 2358−2367, articleTitle=null, refAbstract=null), Reference(id=1325491028119134546, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=101, rfOrder=104, authorNames=null, journalName=null, refType=null, unstructuredReference=Rao Jun, Meng Xv, Ding Liang, et al. Parameter-efficient and student-friendly knowledge distillation[J]. IEEE Transactions on Multimedia, 2024, 26: 4230−4241, articleTitle=null, refAbstract=null), Reference(id=1325491030144983379, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=102, rfOrder=105, authorNames=null, journalName=null, refType=null, unstructuredReference=Anil R, Pereyra G, Passos A, et al. Large scale distributed neural network training through online distillation[J]. arXiv preprint, arXiv: 1804.03235, 2020, articleTitle=null, refAbstract=null), Reference(id=1325491031910785364, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=103, rfOrder=106, authorNames=null, journalName=null, refType=null, unstructuredReference=Song Guocong, Chai Wei. Collaborative learning for deep neural networks[C]//Proc of the 32nd Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2018: 1837−1846, articleTitle=null, refAbstract=null), Reference(id=1325491033500426582, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=104, rfOrder=107, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Defang, Mei Jianping, Wang Can, et al. Online knowledge distillation with diverse peers[C]//Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 3430−3437, articleTitle=null, refAbstract=null), Reference(id=1325491036184781143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=105, rfOrder=108, authorNames=null, journalName=null, refType=null, unstructuredReference=Gong Linrui, Lin Shaohui, Zhang Baochang, et al. Adaptive hierarchy-branch fusion for online knowledge distillation[C]//Proc of the 37th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023: 7731−7739, articleTitle=null, refAbstract=null), Reference(id=1325491038550368602, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=106, rfOrder=109, authorNames=null, journalName=null, refType=null, unstructuredReference=Wu Guile, Gong Shaogang. Peer collaborative learning for online knowledge distillation[C]//Proc of the 35th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2021: 10302−10310, articleTitle=null, refAbstract=null), Reference(id=1325491040706240860, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=107, rfOrder=110, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Shaojie, Lin Mingbao, Wang Yan, et al. Distilling a powerful student model via online knowledge distillation[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 34(11): 8743−8752, articleTitle=null, refAbstract=null), Reference(id=1325491043549978973, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=108, rfOrder=111, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Linfeng, Song Jiebo, Gao Anni, et al. Be your own teacher: Improve the performance of convolutional neural networks via self distillation[C]//Proc of 2019 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 3712−3721, articleTitle=null, refAbstract=null), Reference(id=1325491048167907679, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=109, rfOrder=112, authorNames=null, journalName=null, refType=null, unstructuredReference=Lan Xu, Zhu Xiatian, Gong Shaogang. Self-referenced deep learning[C]//Proc of the 14th Asian Conf on Computer Vision. Berlin: Springer, 2018: 284−300, articleTitle=null, refAbstract=null), Reference(id=1325491050676101474, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=110, rfOrder=113, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Chenglin, Xie Lingxi, Su Chi, et al. Snapshot distillation: Teacher-student optimization in one generation[C]//Proc of 2019 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 2854−2863, articleTitle=null, refAbstract=null), Reference(id=1325491054530666853, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=111, rfOrder=114, authorNames=null, journalName=null, refType=null, unstructuredReference=Mobahi H, Farajtabar M, Bartlett P L. Self-distillation amplifies regularization in Hilbert space[C]//Proc of the 34th Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2020: 3351−3361, articleTitle=null, refAbstract=null), Reference(id=1325491056652984678, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=112, rfOrder=115, authorNames=null, journalName=null, refType=null, unstructuredReference=Shen Yiqing, Xu Liwu, Yang Yuzhe, et al. Self-distillation from the last mini-batch for consistency regularization[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 11933−11942, articleTitle=null, refAbstract=null), Reference(id=1325491060771791211, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=113, rfOrder=116, authorNames=null, journalName=null, refType=null, unstructuredReference=Gou Jianping, Xiong Xiangshuo, Yu Baosheng, et al. Multi-target knowledge distillation via student self-reflection[J]. International Journal of Computer Vision, 2023, 131(7): 1857−1874, articleTitle=null, refAbstract=null), Reference(id=1325491063695221100, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=114, rfOrder=117, authorNames=null, journalName=null, refType=null, unstructuredReference=余鹰, 危伟, 汤洪, 等. 多层次知识自蒸馏联合多步骤训练的细粒度图像识别[J]. 计算机研究与发展, 2023, 60(8): 1834−1845, articleTitle=null, refAbstract=null), Reference(id=1325491068774523250, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=114, rfOrder=118, authorNames=null, journalName=null, refType=null, unstructuredReference=Yu Ying, Wei Wei, Tang Hong, et al. Multi-stage training with multi-level knowledge self-distillation for fine-grained image recognition[J]. Journal of Computer Research and Development, 2023, 60(8): 1834−1845 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1325491072935272819, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=115, rfOrder=119, authorNames=null, journalName=null, refType=null, unstructuredReference=Lopes R G, Fenu S, Starner T. Data-free knowledge distillation for deep neural networks[J]. arXiv preprint, arXiv: 1710.07535, 2017, articleTitle=null, refAbstract=null), Reference(id=1325491076697563510, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=116, rfOrder=120, authorNames=null, journalName=null, refType=null, unstructuredReference=Nayak G K, Mopuri K R, Shaj V, et al. Zero-shot knowledge distillation in deep networks[C]//Proc of the 36th Int Conf on Machine Learning. New York: ACM, 2019: 4743−4751, articleTitle=null, refAbstract=null), Reference(id=1325491078874407287, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=117, rfOrder=121, authorNames=null, journalName=null, refType=null, unstructuredReference=Fang Gongfan, Mo Kanya, Wang Xinchao, et al. Up to 100× faster data-free knowledge distillation[C]//Proc of the 36th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2022: 6597−6604, articleTitle=null, refAbstract=null), Reference(id=1325491082166935928, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=118, rfOrder=122, authorNames=null, journalName=null, refType=null, unstructuredReference=Yu Shikang, Chen Jiachen, Han Hu, et al. Data-free knowledge distillation via feature exchange and activation region constraint[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 24266−24275, articleTitle=null, refAbstract=null), Reference(id=1325491085505601916, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=119, rfOrder=123, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Jingru, Zhou Sheng, Li Liangcheng, et al. Dynamic data-free knowledge distillation by easy-to-hard learning strategy[J]. Information Sciences, 2023, 642: 119202, articleTitle=null, refAbstract=null), Reference(id=1325491087187517822, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=120, rfOrder=124, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Yang, Qian Biao, Liu Haipeng, et al. Unpacking the gap box against data-free knowledge distillation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 46(9): 6280−6291, articleTitle=null, refAbstract=null), Reference(id=1325491089817346432, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=121, rfOrder=125, authorNames=null, journalName=null, refType=null, unstructuredReference=张晶, 鞠佳良, 任永功. 基于双生成器网络的Data-Free知识蒸馏[J]. 计算机研究与发展, 2023, 60(7): 1615−1627, articleTitle=null, refAbstract=null), Reference(id=1325491091469902210, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=121, rfOrder=126, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Jing, Ju Jialiang, Ren Yonggong. Double-generators network for data-free knowledge distillation[J]. Journal of Computer Research and Development, 2023, 60(7): 1615−1627 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1325491094829539716, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=122, rfOrder=127, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Hanting, Wang Yunhe, Xu Chang, et al. Data-free learning of student networks[C]//Proc of 2019 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 3513−3521, articleTitle=null, refAbstract=null), Reference(id=1325491096842805637, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=123, rfOrder=128, authorNames=null, journalName=null, refType=null, unstructuredReference=Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets[C]//Proc of the 28th Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2014: 2672−2680, articleTitle=null, refAbstract=null), Reference(id=1325491099749458311, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=124, rfOrder=129, authorNames=null, journalName=null, refType=null, unstructuredReference=Fang Gongfan, Song Jie, Shen Chengchao, et al. Data-free adversarial distillation[J]. arXiv preprint, arXiv: 1912.11006, 2020, articleTitle=null, refAbstract=null), Reference(id=1325491102303789449, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=125, rfOrder=130, authorNames=null, journalName=null, refType=null, unstructuredReference=Luo Liangchen, Sandler M, Lin Zi, et al. Large-scale generative data-free distillation[J]. arXiv preprint, arXiv: 2012.05578, 2020, articleTitle=null, refAbstract=null), Reference(id=1325491104455467404, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=126, rfOrder=131, authorNames=null, journalName=null, refType=null, unstructuredReference=Qu Xiaoyang, Wang Jianzong, Xiao Jing. Enhancing data-free adversarial distillation with activation regularization and virtual interpolation[C]//Proc of 2021 IEEE Int Conf on Acoustics, Speech and Signal Processing. Piscataway, NJ: IEEE, 2021: 3340−3344, articleTitle=null, refAbstract=null), Reference(id=1325491108230341005, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=127, rfOrder=132, authorNames=null, journalName=null, refType=null, unstructuredReference=Micaelli P, Storkey A J. Zero-shot knowledge transfer via adversarial belief matching[C]//Proc of the 33rd Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2019: 9547−9557, articleTitle=null, refAbstract=null), Reference(id=1325491109492826512, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=128, rfOrder=133, authorNames=null, journalName=null, refType=null, unstructuredReference=Do K, Le T H, Nguyen D, et al. Momentum adversarial distillation: Handling large distribution shifts in data-free knowledge distillation[C]//Proc of the 36th Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2022: 10055−10067, articleTitle=null, refAbstract=null), Reference(id=1325491111996826001, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=129, rfOrder=134, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Haoran, Sun Xin, Dong Junyu, et al. Dual discriminator adversarial distillation for data-free model compression[J]. International Journal of Machine Learning and Cybernetics, 2021, 13(5): 1213−1230, articleTitle=null, refAbstract=null), Reference(id=1325491113657770386, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=130, rfOrder=135, authorNames=null, journalName=null, refType=null, unstructuredReference=Shao Renrong, Zhang Wei, Wang Jun. Conditional pseudo-supervised contrast for data-free knowledge distillation[J]. Pattern Recognition, 2023, 143: 109781, articleTitle=null, refAbstract=null), Reference(id=1325491117365535123, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=131, rfOrder=136, authorNames=null, journalName=null, refType=null, unstructuredReference=Patel G, Mopuri K R, Qiu Qiang. Learning to retain while acquiring: Combating distribution-shift in adversarial data-free knowledge distillation[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 7786−7794, articleTitle=null, refAbstract=null), Reference(id=1325491120393822612, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=132, rfOrder=137, authorNames=null, journalName=null, refType=null, unstructuredReference=Fukuda T, Suzuki M, Kurata G, et al. Efficient knowledge distillation from an ensemble of teachers[C]//Proc of the 18th Annual Conf of the Int Speech Communication Association. Grenoble, FR: ISCA, 2017: 3697−3701, articleTitle=null, refAbstract=null), Reference(id=1325491121673085333, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=133, rfOrder=138, authorNames=null, journalName=null, refType=null, unstructuredReference=Tarvainen A, Valpola H. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[C]//Proc of the 31st Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2017: 1195−1204, articleTitle=null, refAbstract=null), Reference(id=1325491122931376534, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=134, rfOrder=139, authorNames=null, journalName=null, refType=null, unstructuredReference=Xiang Liuyu, Ding Guiguang, Han Jungong. Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification[C]//Proc of the 16th European Conf on Computer Vision. Berlin: Springer, 2020: 247−263, articleTitle=null, refAbstract=null), Reference(id=1325491125414404503, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=135, rfOrder=140, authorNames=null, journalName=null, refType=null, unstructuredReference=Nguyen L T, Lee K, Shim B. Stochasticity and skip connection improve knowledge transfer[C]//Proc of the 28th European Signal Processing Conf. Piscataway, NJ: IEEE, 2021: 1537−1541, articleTitle=null, refAbstract=null), Reference(id=1325491127859683736, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=136, rfOrder=141, authorNames=null, journalName=null, refType=null, unstructuredReference=Shi Luyao, Jiang Ning, Tang Jialiang, et al. Correlation guided multi-teacher knowledge distillation[C]//Proc of the 30th Int Conf on Neural Information Processing. Berlin: Springer, 2023: 562−574, articleTitle=null, refAbstract=null), Reference(id=1325491131236098457, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=137, rfOrder=142, authorNames=null, journalName=null, refType=null, unstructuredReference=Pham C, Hoang T, Do T T. Collaborative multi-teacher knowledge distillation for learning low bit-width deep neural networks[C]//Proc of 2023 IEEE/CVF Winter Conf on Applications of Computer Vision. Piscataway, NJ: IEEE, 2023: 6424−6432, articleTitle=null, refAbstract=null), Reference(id=1325491133396165019, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=138, rfOrder=143, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Hailin, Chen Defang, Wang Can. Confidence-aware multi-teacher knowledge distillation[C]//Proc of 2022 IEEE Int Conf on Acoustics, Speech and Signal Processing. Piscataway, NJ: IEEE, 2022: 4498−4502, articleTitle=null, refAbstract=null), Reference(id=1325491135426208157, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=139, rfOrder=144, authorNames=null, journalName=null, refType=null, unstructuredReference=Cheng Xin, Zhang Zhiqiang, Weng Wei, et al. DE-MKD: Decoupled multi-teacher knowledge distillation based on entropy[J]. Mathematics, 2024, 12(11): 1672, articleTitle=null, refAbstract=null), Reference(id=1325491137175232927, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=140, rfOrder=145, authorNames=null, journalName=null, refType=null, unstructuredReference=Klingner M, Borse S, Kumar V R, et al. X3KD: Knowledge distillation across modalities, tasks and stages for multi-camera 3D object detection[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 13343−13353, articleTitle=null, refAbstract=null), Reference(id=1325491138416746913, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=141, rfOrder=146, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Lingjun, Song Jingyu, Skinner Katherine A. CRKD: Enhanced camera-radar object detection with cross-modality knowledge distillation[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 15470−15480, articleTitle=null, refAbstract=null), Reference(id=1325491139607929250, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=142, rfOrder=147, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Yunhe, Xu Chang, Xu Chao, et al. Adversarial learning of portable student networks[C]//Proc of the 32nd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2018: 4260−4267, articleTitle=null, refAbstract=null), Reference(id=1325491140509704611, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=143, rfOrder=148, authorNames=null, journalName=null, refType=null, unstructuredReference=Passalis N, Tzelepi M, Tefas A. Heterogeneous knowledge distillation using information flow modeling[C]//Proc of 2020 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 2339−2348, articleTitle=null, refAbstract=null), Reference(id=1325491143076618662, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=144, rfOrder=149, authorNames=null, journalName=null, refType=null, unstructuredReference=Ganta D P, Gupta H D, Sheng V S. Knowledge distillation via weighted ensemble of teaching assistants[C]//Proc of 2021 IEEE Int Conf on Big Knowledge. Piscataway, NJ: IEEE, 2021: 30−37, articleTitle=null, refAbstract=null), Reference(id=1325491144754340263, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=145, rfOrder=150, authorNames=null, journalName=null, refType=null, unstructuredReference=Gao Mengya, Wang Yujun, Wan Liang. Residual error based knowledge distillation[J]. Neurocomputing, 2021, 433: 154−161, articleTitle=null, refAbstract=null), Reference(id=1325491148013314473, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=146, rfOrder=151, authorNames=null, journalName=null, refType=null, unstructuredReference=Morikawa T, Kameyama K. Multi-stage model compression using teacher assistant and distillation with hint-based training[C]//Proc of 2022 IEEE Int Conf on Pervasive Computing and Communications Workshops and other Affiliated Events. Piscataway, NJ: IEEE, 2022: 484−490, articleTitle=null, refAbstract=null), Reference(id=1325491149749756331, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=147, rfOrder=152, authorNames=null, journalName=null, refType=null, unstructuredReference=He Limuxuan, Hua Bei. Semi-supervised knowledge distillation via teaching assistant[J]. Highlights in Science, Engineering and Technology, 2023, 72: 429−436, articleTitle=null, refAbstract=null), Reference(id=1325491152186646956, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=148, rfOrder=153, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Lei, Ng W W Y, Zhang Jianjun. Distillation of ensemble adaptive teacher assistant knowledge[J]. International Journal of Machine Learning and Cybernetics, 2025, 16(10): 8063-8078, articleTitle=null, refAbstract=null), Reference(id=1325491154283798957, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=149, rfOrder=154, authorNames=null, journalName=null, refType=null, unstructuredReference=Huo Fushuo, Xu Wenchao, Guo Jingcai, et al. C2KD: Bridging the modality gap for cross-modal knowledge distillation[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 16006−16015, articleTitle=null, refAbstract=null), Reference(id=1325491156074766766, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=150, rfOrder=155, authorNames=null, journalName=null, refType=null, unstructuredReference=Cho J H, Hariharan B. On the efficacy of knowledge distillation[C]//Proc of 2019 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 4793−4801, articleTitle=null, refAbstract=null), Reference(id=1325491156901044655, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=151, rfOrder=156, authorNames=null, journalName=null, refType=null, unstructuredReference=Yuan Li, Tay F E, Li Guilin, et al. Revisiting knowledge distillation via label smoothing regularization[C]//Proc of 2020 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 3902−3910, articleTitle=null, refAbstract=null), Reference(id=1325491158062866864, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=152, rfOrder=157, authorNames=null, journalName=null, refType=null, unstructuredReference=Stanton S, Izmailov P, Kirichenko P, et al. Does knowledge distillation really work?[C]//Proc of the 35th Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2021: 6906−6919, articleTitle=null, refAbstract=null), Reference(id=1325491160671723954, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=153, rfOrder=158, authorNames=null, journalName=null, refType=null, unstructuredReference=He Ruifei, Sun Shuyang, Yang Jihan, et al. Knowledge distillation as efficient pre-training: Faster convergence, higher data-efficiency, and better transferability[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 9151−9161, articleTitle=null, refAbstract=null), Reference(id=1325491161967763892, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=154, rfOrder=159, authorNames=null, journalName=null, refType=null, unstructuredReference=Shen Zhiqiang, Xing E. A fast knowledge distillation framework for visual recognition[C]//Proc of the 17th European Conf on Computer Vision. Berlin: Springer, 2022: 673−690, articleTitle=null, refAbstract=null), Reference(id=1325491163989418421, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=155, rfOrder=160, authorNames=null, journalName=null, refType=null, unstructuredReference=Song Jie, Chen Ying, Ye Jingwen, et al. Spot-adaptive knowledge distillation[J]. IEEE Transactions on Image Processing, 2022, 31: 3359−3370, articleTitle=null, refAbstract=null), Reference(id=1325491165688111542, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=156, rfOrder=161, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Tianli, Xue Mengqi, Zhang Jiangtao, et al. Generalization matters: Loss minima flattening via parameter hybridization for efficient online knowledge distillation[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 20176−20185, articleTitle=null, refAbstract=null), Reference(id=1325491166967374264, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=157, rfOrder=162, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Jiabao, Chen Yuming, Zheng Zhaohui, et al. CrossKD: Cross-head knowledge distillation for object detection[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 16520−16530, articleTitle=null, refAbstract=null), Reference(id=1325491168137585081, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=158, rfOrder=163, authorNames=null, journalName=null, refType=null, unstructuredReference=Xu Guodong, Liu Ziwei, Li Xiaoxiao, et al. Knowledge distillation meets self-supervision[C]//Proc of the 16th European Conf on Computer Vision. Berlin: Springer, 2020: 588−604, articleTitle=null, refAbstract=null), Reference(id=1325491169437819322, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=159, rfOrder=164, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Chenxin, Lin Mingbao, Ding Zhiyuan, et al. Knowledge condensation distillation[C]//Proc of the 17th European Conf on Computer Vision. Berlin: Springer, 2022: 19−35, articleTitle=null, refAbstract=null), Reference(id=1325491171077792187, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=160, rfOrder=165, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Tao, You Shan, Wang Fei, et al. Knowledge distillation from a stronger teacher[C]//Proc of the 36th Annual Conference on Neural Information Processing Systems. Cambridge, MA: MIT, 2022: 33716−33727, articleTitle=null, refAbstract=null), Reference(id=1325491172453523900, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=161, rfOrder=166, authorNames=null, journalName=null, refType=null, unstructuredReference=Choi J, Cho H, Cheung S, et al. ORC: Network group-based knowledge distillation using online role change[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 17335−17344, articleTitle=null, refAbstract=null), Reference(id=1325491174009610686, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=162, rfOrder=167, authorNames=null, journalName=null, refType=null, unstructuredReference=Shen Zhiqiang. FerKD: Surgical label adaptation for efficient distillation[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 1666−1675, articleTitle=null, refAbstract=null), Reference(id=1325491175683137983, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=163, rfOrder=168, authorNames=null, journalName=null, refType=null, unstructuredReference=Miles R, Elezi I, Deng Jiankang. VkD: Improving knowledge distillation using orthogonal projections[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 15720−15730, articleTitle=null, refAbstract=null), Reference(id=1325491178669482432, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=164, rfOrder=169, authorNames=null, journalName=null, refType=null, unstructuredReference=Miles R, Mikolajczyk K. Understanding the role of the projector in knowledge distillation[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 4233−4241, articleTitle=null, refAbstract=null), Reference(id=1325491180758245826, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=165, rfOrder=170, authorNames=null, journalName=null, refType=null, unstructuredReference=Sun Wujie, Chen Defang, Lyu Siwei, et al. Knowledge distillation with refined logits[J]. arXiv preprint, arXiv: 2408.07703, 2024, articleTitle=null, refAbstract=null), Reference(id=1325491183316771267, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=166, rfOrder=171, authorNames=null, journalName=null, refType=null, unstructuredReference=Xu Kunran, Rui Lai, Li Yishi, et al. Feature normalized knowledge distillation for image classification[C]//Proc of the 16th European Conf on Computer Vision. Berlin: Springer, 2020: 664−680, articleTitle=null, refAbstract=null), Reference(id=1325491184898023877, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=167, rfOrder=172, authorNames=null, journalName=null, refType=null, unstructuredReference=Jafari A, Rezagholizadeh M, Sharma P, et al. Annealing knowledge distillation[C]//Proc of the 16th Conf of the European Chapter of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2021: 2493−2504, articleTitle=null, refAbstract=null), Reference(id=1325491187838230982, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=168, rfOrder=173, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Zheng, Li Xiang, Yang Lingfeng, et al. Curriculum temperature for knowledge distillation[C]//Proc of the 37th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023: 1504−1512, articleTitle=null, refAbstract=null), Reference(id=1325491189142659528, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=169, rfOrder=174, authorNames=null, journalName=null, refType=null, unstructuredReference=Chi Zhihao, Zheng Tu, Li Hengjia, et al. NormKD: Normalized logits for knowledge distillation[J]. arXiv preprint, arXiv: 2308.00520, 2023, articleTitle=null, refAbstract=null), Reference(id=1325491190396756425, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=170, rfOrder=175, authorNames=null, journalName=null, refType=null, unstructuredReference=Wei Yukang, Bai Yu. Dynamic temperature knowledge distillation[J]. arXiv preprint, arXiv: 2404.12711, 2024, articleTitle=null, refAbstract=null), Reference(id=1325491192477131211, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=171, rfOrder=176, authorNames=null, journalName=null, refType=null, unstructuredReference=Sun Shangquan, Ren Wenqi, Li Jingzhi, et al. Logit standardization in knowledge distillation[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 15731−15740, articleTitle=null, refAbstract=null), Reference(id=1325491194163241420, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=172, rfOrder=177, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Borui, Cui Quan, Song Renjie, et al. Decoupled knowledge distillation[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 11943−11952, articleTitle=null, refAbstract=null), Reference(id=1325491196738544078, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=173, rfOrder=178, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Zhendong, Zeng Ailing, Li Zhe, et al. From knowledge distillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 17139−17148, articleTitle=null, refAbstract=null), Reference(id=1325491197971669455, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=174, rfOrder=179, authorNames=null, journalName=null, refType=null, unstructuredReference=Park H, Lee J S. Adaptive explicit knowledge transfer for knowledge distillation[J]. arXiv preprint, arXiv: 2409.01679, 2024, articleTitle=null, refAbstract=null), Reference(id=1325491199221572048, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=175, rfOrder=180, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Borui, Cui Quan, Song Renjie, et al. DOT: A distillation-oriented trainer[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 6166−6175, articleTitle=null, refAbstract=null), Reference(id=1325491201255809490, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=176, rfOrder=181, authorNames=null, journalName=null, refType=null, unstructuredReference=Shen Zhiqiang, He Zhankui, Xue Xiangyang. MEAL: Multi-model ensemble via adversarial learning[C]//Proc of the 33rd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 4886−4893, articleTitle=null, refAbstract=null), Reference(id=1325491202992251347, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=177, rfOrder=182, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Peng, Shu Chang, Xie Yuan, et al. Hierarchical knowledge squeezed adversarial network compression[C]//Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 11370−11377, articleTitle=null, refAbstract=null), Reference(id=1325491206284779989, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=178, rfOrder=183, authorNames=null, journalName=null, refType=null, unstructuredReference=Ma Haoyu, Chen Tianlong, Hu T K, et al. Undistillable: Making a nasty teacher that cannot teach students[J]. arXiv preprint, arXiv: 2105.07381, 2021, articleTitle=null, refAbstract=null), Reference(id=1325491208407097816, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=179, rfOrder=184, authorNames=null, journalName=null, refType=null, unstructuredReference=Heo B, Lee M, Yun S, et al. Knowledge distillation with adversarial samples supporting decision boundary[C]//Proc of the 33rd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 3771−3778, articleTitle=null, refAbstract=null), Reference(id=1325491210575552985, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=180, rfOrder=185, authorNames=null, journalName=null, refType=null, unstructuredReference=Nguyen-Duc T, Le T, Zhao He, et al. Adversarial local distribution regularization for knowledge distillation[C]//Proc of 2023 IEEE/CVF Winter Conf on Applications of Computer Vision. Piscataway, NJ: IEEE, 2023: 4670−4679, articleTitle=null, refAbstract=null), Reference(id=1325491213071163867, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=181, rfOrder=186, authorNames=null, journalName=null, refType=null, unstructuredReference=Bai Tao, Zhao Jun, Wen Bihan. Guided adversarial contrastive distillation for robust students[J]. IEEE Transactions on Information Forensics and Security, 2024, 19: 9643−9655, articleTitle=null, refAbstract=null), Reference(id=1325491215982010845, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=182, rfOrder=187, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Bo, Chen Mingyang, Wang Yi, et al. Boosting accuracy and robustness of student models via adaptive adversarial distillation[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 24668−24677, articleTitle=null, refAbstract=null), Reference(id=1325491218053997022, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=183, rfOrder=188, authorNames=null, journalName=null, refType=null, unstructuredReference=Yue Xinli, Mou Ningping, Wang Qian, et al. Revisiting adversarial robustness distillation from the perspective of robust fairness[C]//Proc of the 37th Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2023: 30390−30401, articleTitle=null, refAbstract=null), Reference(id=1325491220981621216, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=184, rfOrder=189, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Yuzheng, Chen Zhaoyu, Yang Dingkang, et al. Adversarial contrastive distillation with adaptive denoising[C]//Proc of 2023 IEEE Int Conf on Acoustics, Speech and Signal Processing, Piscataway, NJ: IEEE, 2023: 1−5, articleTitle=null, refAbstract=null), Reference(id=1325491223590478306, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=185, rfOrder=190, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Changlin, Peng Jiefeng, Yuan Liuchun, et al. Block-wisely supervised neural architecture search with knowledge distillation[C]//Proc of 2020 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 1986−1995, articleTitle=null, refAbstract=null), Reference(id=1325491225322725859, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=186, rfOrder=191, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Yu, Jia Xuhui, Tan Mingxing, et al. Search to distill: Pearls are everywhere but not the eyes[C]//Proc of 2020 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 7536−7545, articleTitle=null, refAbstract=null), Reference(id=1325491227717673446, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=187, rfOrder=192, authorNames=null, journalName=null, refType=null, unstructuredReference=Kang M, Mun J, Han B. Towards oracle knowledge distillation with neural architecture search[C]//Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 4404−4411, articleTitle=null, refAbstract=null), Reference(id=1325491229407977959, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=188, rfOrder=193, authorNames=null, journalName=null, refType=null, unstructuredReference=Wei Zimian, Pan Hengyue, Li Lujun, et al. TVT: Training-free vision transformer search on tiny datasets[J]. arXiv preprint, arXiv: 2311.14337, 2023, articleTitle=null, refAbstract=null), Reference(id=1325491232343990761, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=189, rfOrder=194, authorNames=null, journalName=null, refType=null, unstructuredReference=Dong Peijie, Li Lujun, Wei Zimian. DisWOT: Student architecture search for distillation without training[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 11898−11908, articleTitle=null, refAbstract=null), Reference(id=1325491234038489578, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=190, rfOrder=195, authorNames=null, journalName=null, refType=null, unstructuredReference=Lu Liming, Chen Zhenghan, Lu Xiaoyu, et al. UniADS: Universal architecture-distiller search for distillation gap[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 14167−14174, articleTitle=null, refAbstract=null), Reference(id=1325491236597015021, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=191, rfOrder=196, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Lujun, Dong Peijie, Wei Zimian, et al. Automated knowledge distillation via monte carlo tree search[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 17367−17378, articleTitle=null, refAbstract=null), Reference(id=1325491237867889135, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=192, rfOrder=197, authorNames=null, journalName=null, refType=null, unstructuredReference=Sohl-Dickstein J, Weiss E A, Maheswaranathan N, et al. Deep unsupervised learning using nonequilibrium thermodynamics[C]//Proc of the 32nd Int Conf on Machine Learning. New York: ACM, 2015: 2256−2265, articleTitle=null, refAbstract=null), Reference(id=1325491239046488560, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=193, rfOrder=198, authorNames=null, journalName=null, refType=null, unstructuredReference=Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models[C]//Proc of the 34th Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2020: 6840−6851, articleTitle=null, refAbstract=null), Reference(id=1325491241533710835, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=194, rfOrder=199, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Tao, Zhang Yuan, Zheng Mingkai, et al. Knowledge diffusion for distillation[C]//Proc of the 37th Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2023, 36: 65299−65316, articleTitle=null, refAbstract=null), Reference(id=1325491242473234933, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=195, rfOrder=200, authorNames=null, journalName=null, refType=null, unstructuredReference=Guo Qiushan, Mu Yao, Chen Jianyu, et al. Scale-equivalent distillation for semi-supervised object detection[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 14502−14511, articleTitle=null, refAbstract=null), Reference(id=1325491244465529334, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=196, rfOrder=201, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhou Huajun, Qiao Bo, Yang Lingxiao, et al. Texture-guided saliency distilling for unsupervised salient object detection[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 7257−7267, articleTitle=null, refAbstract=null), Reference(id=1325491246629790200, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=197, rfOrder=202, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Luting, Liu Yi, Du Penghui, et al. Object-aware distillation pyramid for open-vocabulary object detection[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 11186−11196, articleTitle=null, refAbstract=null), Reference(id=1325491247900664313, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=198, rfOrder=203, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Liangqi, Miao Jiaxu, Shi Dahu, et al. Distilling DETR with visual-linguistic knowledge for open-vocabulary object detection[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 6478−6487, articleTitle=null, refAbstract=null), Reference(id=1325491250748596731, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=199, rfOrder=204, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhou Shengchao, Liu Weizhou, Hu Chen, et al. UniDistill: A universal cross-modality knowledge distillation framework for 3D object detection in bird’s-eye view[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 5116−5125, articleTitle=null, refAbstract=null), Reference(id=1325491252870914556, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=200, rfOrder=205, authorNames=null, journalName=null, refType=null, unstructuredReference=Wu Aming, Deng Cheng. Single-domain generalized object detection in urban scene via cyclic-disentangled self-distillation[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 837−846, articleTitle=null, refAbstract=null), Reference(id=1325491255907590654, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=201, rfOrder=206, authorNames=null, journalName=null, refType=null, unstructuredReference=Lao Shanshan, Song Guanglu, Liu Boxiao, et al. UniKD: Universal knowledge distillation for mimicking homogeneous or heterogeneous object detectors[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 6362−6372, articleTitle=null, refAbstract=null), Reference(id=1325491258298343935, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=202, rfOrder=207, authorNames=null, journalName=null, refType=null, unstructuredReference=Zeng Jia, Chen Li, Deng Hanming, et al. Distilling focal knowledge from imperfect expert for 3D object detection[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 992−1001, articleTitle=null, refAbstract=null), Reference(id=1325491260529713664, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=203, rfOrder=208, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Zhe, Ye Xiaoqing, Tan Xiao, et al. StereoDistill: Pick the cream from LiDAR for distilling stereo-based 3D object detection[C]//Proc of the 37th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023: 1790−1798, articleTitle=null, refAbstract=null), Reference(id=1325491264237478401, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=204, rfOrder=209, authorNames=null, journalName=null, refType=null, unstructuredReference=Wu Zizhang, Wu Yunzhe, Pu Jian, et al. Attention-based depth distillation with 3D-aware positional encoding for monocular 3D object detection[C]//Proc of the 37th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023: 2892−2900, articleTitle=null, refAbstract=null), Reference(id=1325491265957143042, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=205, rfOrder=210, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Zeyu, Li Dingwen, Luo Chenxu, et al. DistillBEV: Boosting multi-camera 3D object detection with cross-modal knowledge distillation[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 8603−8612, articleTitle=null, refAbstract=null), Reference(id=1325491268805075459, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=206, rfOrder=211, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Yifan, Dong Zhen, Yang Huanrui, et al. QD-BEV: Quantization-aware view-guided distillation for multi-view 3D object detection[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 3802−3812, articleTitle=null, refAbstract=null), Reference(id=1325491270562488836, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=207, rfOrder=212, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Haimei, Zhang Qiming, Zhao Shanshan, et al. SimDistill: Simulated multi-modal distillation for BEV 3D object detection[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 7460−7468, articleTitle=null, refAbstract=null), Reference(id=1325491273829851653, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=208, rfOrder=213, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Yanjing, Xu Sheng, Lin Mingbao, et al. Representation disparity-aware distillation for 3D object detection[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 6692−6701, articleTitle=null, refAbstract=null), Reference(id=1325491276421931526, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=209, rfOrder=214, authorNames=null, journalName=null, refType=null, unstructuredReference=Gambashidze A, Dadukin A, Golyadkin M, et al. Weak-to-strong 3D object detection with X-ray distillation[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 15055−15064, articleTitle=null, refAbstract=null), Reference(id=1325491278103847432, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=210, rfOrder=215, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Xun, Wu Hai, Li Xin, et al. Sunshine to rainstorm: Cross-weather knowledge distillation for robust 3D object detection[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 2409−2416, articleTitle=null, refAbstract=null), Reference(id=1325491279710265865, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=211, rfOrder=216, authorNames=null, journalName=null, refType=null, unstructuredReference=Ji Deyi, Wang Haoran, Tao Mingyuan, et al. Structural and statistical texture knowledge distillation for semantic segmentation[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 16855−16864, articleTitle=null, refAbstract=null), Reference(id=1325491281471873546, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=212, rfOrder=217, authorNames=null, journalName=null, refType=null, unstructuredReference=Gao Huan, Guo Jichang, Wang Guoli, et al. Cross-domain correlation distillation for unsupervised domain adaptation in nighttime semantic segmentation[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 9903−9913, articleTitle=null, refAbstract=null), Reference(id=1325491283040543243, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=213, rfOrder=218, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Chuanguang, Zhou Helong, An Zhulin, et al. Cross-image relational knowledge distillation for semantic segmentation[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 12309−12318, articleTitle=null, refAbstract=null), Reference(id=1325491285330633228, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=214, rfOrder=219, authorNames=null, journalName=null, refType=null, unstructuredReference=Peng Bohao, Tian Zhuotao, Wu Xiaoyang, et al. Hierarchical dense correlation distillation for few-shot segmentation[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 23641−23651, articleTitle=null, refAbstract=null), Reference(id=1325491286840582670, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=215, rfOrder=220, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Yuang, Zhang Wei, Wang Jun. Self-decoupling and ensemble distillation for efficient segmentation[C]//Proc of the 37th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023: 1772−1780, articleTitle=null, refAbstract=null), Reference(id=1325491288195342863, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=216, rfOrder=221, authorNames=null, journalName=null, refType=null, unstructuredReference=Hou Yuenan, Zhu Xinge, Ma Yuexin, et al. Point-to-voxel knowledge distillation for LiDAR semantic segmentation[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 8469−8478, articleTitle=null, refAbstract=null), Reference(id=1325491290233774608, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=217, rfOrder=222, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Zisheng, Xu Hongbin, Chen Weitao, et al. PointDC: Unsupervised semantic segmentation of 3D point clouds via cross-modal distillation and super-voxel clustering[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 14244−14253, articleTitle=null, refAbstract=null), Reference(id=1325491291966022161, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=218, rfOrder=223, authorNames=null, journalName=null, refType=null, unstructuredReference=Xu Jianqing, Li Shen, Deng Ailin, et al. Probabilistic knowledge distillation of face ensembles[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 3489−3498, articleTitle=null, refAbstract=null), Reference(id=1325491293593412114, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=219, rfOrder=224, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Jingzhi, Guo Zidong, Li Hui, et al. Rethinking feature-based knowledge distillation for face recognition[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 20156−20165, articleTitle=null, refAbstract=null), Reference(id=1325491295644426771, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=220, rfOrder=225, authorNames=null, journalName=null, refType=null, unstructuredReference=Yu Zhipeng, Liu Jiaheng, Qin Haoyu, et al. ICD-face: Intra-class compactness distillation for face recognition[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 20985−20995, articleTitle=null, refAbstract=null), Reference(id=1325491296994992660, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=221, rfOrder=226, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Weisong, Zhu Xiangyu, Guo Kaiwen, et al. Grouped knowledge distillation for deep face recognition[C]//Proc of the 37th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023: 3615−3623, articleTitle=null, refAbstract=null), Reference(id=1325491299515769365, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=222, rfOrder=227, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Zhenduo, Chen Yongru, Yang Wenming, et al. Pose-invariant face recognition via adaptive angular distillation[C]//Proc of the 36th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2022: 3390−3398, articleTitle=null, refAbstract=null), Reference(id=1325491301583561238, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=223, rfOrder=228, authorNames=null, journalName=null, refType=null, unstructuredReference=Hu Siyuan, Wang Zheng, Hu Peng, et al. PrefAce: Face-centric pretraining with self-structure aware distillation[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 12538−12546, articleTitle=null, refAbstract=null), Reference(id=1325491304439882263, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=224, rfOrder=229, authorNames=null, journalName=null, refType=null, unstructuredReference=Deng Hanqiu, Li Xingyu. Anomaly detection via reverse distillation from one-class embedding[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 9727−9736, articleTitle=null, refAbstract=null), Reference(id=1325491306553811480, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=225, rfOrder=230, authorNames=null, journalName=null, refType=null, unstructuredReference=Tien T D, Nguyen A T, Tran N H, et al. Revisiting reverse distillation for anomaly detection[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 24511−24520, articleTitle=null, refAbstract=null), Reference(id=1325491308265087513, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=226, rfOrder=231, authorNames=null, journalName=null, refType=null, unstructuredReference=Gu Zhihao, Liu Liang, Chen Xu, et al. Remembering normality: Memory-guided knowledge distillation for unsupervised anomaly detection[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 16355−16363, articleTitle=null, refAbstract=null), Reference(id=1325491309217194523, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=227, rfOrder=232, authorNames=null, journalName=null, refType=null, unstructuredReference=Binh L M, Woo S. ADD: Frequency attention and multi-view based knowledge distillation to detect low-quality compressed deepfake images[C]//Proc of the 36th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2022: 122−130, articleTitle=null, refAbstract=null), Reference(id=1325491309963780636, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=228, rfOrder=233, authorNames=null, journalName=null, refType=null, unstructuredReference=Ma Ye, Jiang Xu, Guan Nan, et al. Anomaly detection based on multi-teacher knowledge distillation[J]. Journal of Systems Architecture, 2023, 138: 102861, articleTitle=null, refAbstract=null), Reference(id=1325491312035766814, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=229, rfOrder=234, authorNames=null, journalName=null, refType=null, unstructuredReference=Ristea N C, Croitoru F A, Ionescu R T, et al. Self-distilled masked auto-encoders are efficient video anomaly detectors[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 15984−15995, articleTitle=null, refAbstract=null), Reference(id=1325491313386332703, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=230, rfOrder=235, authorNames=null, journalName=null, refType=null, unstructuredReference=邢鹏, 蒋鑫, 潘永华, 等. 基于特征约束蒸馏学习的视觉异常检测[J]. 软件学报, 2022, 34(9): 4378−4391, articleTitle=null, refAbstract=null), Reference(id=1325491315370238496, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=230, rfOrder=236, authorNames=null, journalName=null, refType=null, unstructuredReference=Xing Peng, Jiang Xin, Pan Yonghua, et al. Feature constrained restricted distillation learning for visual anomaly detection[J]. Journal of Software, 2022, 34(9): 4378−4391 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1325491317857460769, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=231, rfOrder=237, authorNames=null, journalName=null, refType=null, unstructuredReference=Ju Chen, Zheng Kunhao, Liu Jinxiang, et al. Distilling vision-language pre-training to collaborate with weakly-supervised temporal action localization[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 14751−14762, articleTitle=null, refAbstract=null), Reference(id=1325491318755041826, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=232, rfOrder=238, authorNames=null, journalName=null, refType=null, unstructuredReference=Wu Mingrui, Gu Jiaxin, Shen Yunhang, et al. End-to-end zero-shot HOI detection via vision and language knowledge distillation[C]//Proc of the 37th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023: 2839−2846, articleTitle=null, refAbstract=null), Reference(id=1325491320009138723, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=233, rfOrder=239, authorNames=null, journalName=null, refType=null, unstructuredReference=Monti A, Porrello A, Calderara S, et al. How many observations are enough? Knowledge distillation for trajectory forecasting[C]//Proc of 2018 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 6543−6552, articleTitle=null, refAbstract=null), Reference(id=1325491321720414756, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=234, rfOrder=240, authorNames=null, journalName=null, refType=null, unstructuredReference=Ma Hengbo, Li Jiachen, Hosseini R, et al. Multi-objective diverse human motion prediction with knowledge distillation[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 8151−8161, articleTitle=null, refAbstract=null), Reference(id=1325491323804983845, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=235, rfOrder=241, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang C, Kong K, Min S, et al. SEFD: Learning to distill complex pose and occlusion[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 14895−14906, articleTitle=null, refAbstract=null), Reference(id=1325491326736802342, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=236, rfOrder=242, authorNames=null, journalName=null, refType=null, unstructuredReference=Ye Suhang, Zhang Yingyi, Hu Jie, et al. DistilPose: Tokenized pose regression with heatmap distillation[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 2163−2172, articleTitle=null, refAbstract=null), Reference(id=1325491328817177128, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=237, rfOrder=243, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Sichen, Zhang Yingyi, Huang Siming, et al. SDPose: Tokenized pose estimation via circulation-guide self-distillation[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 1082−1090, articleTitle=null, refAbstract=null), Reference(id=1325491331795132970, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=238, rfOrder=244, authorNames=null, journalName=null, refType=null, unstructuredReference=Lu Yichen, Wang Mei, Deng Weihong. Augmented geometric distillation for data-free incremental person ReID[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 7319−7328, articleTitle=null, refAbstract=null), Reference(id=1325491332977926701, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=239, rfOrder=245, authorNames=null, journalName=null, refType=null, unstructuredReference=Shang Chao, Li Hongliang, Meng Fanman, et al. Incrementer: Transformer for class-incremental semantic segmentation with knowledge distillation focusing on old class[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 7214−7224, articleTitle=null, refAbstract=null), Reference(id=1325491335075078703, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=240, rfOrder=246, authorNames=null, journalName=null, refType=null, unstructuredReference=Phan M H, Ta T A, Phung S L, et al. Class similarity weighted knowledge distillation for continual semantic segmentation[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 16845−16854, articleTitle=null, refAbstract=null), Reference(id=1325491335985242672, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=241, rfOrder=247, authorNames=null, journalName=null, refType=null, unstructuredReference=Gu Peiyan, Zhang Chuyu, Xu Ruijie, et al. Class-relation knowledge distillation for novel class discovery[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 16428−16437, articleTitle=null, refAbstract=null), Reference(id=1325491337616826929, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=242, rfOrder=248, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Jiyong, Azizov D, Li Yang, et al. Contrastive continual learning with importance sampling and prototype-instance relation distillation[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 13554−13562, articleTitle=null, refAbstract=null), Reference(id=1325491340246655538, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=243, rfOrder=249, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Linglan, Lu Jing, Xu Yunlu, et al. Few-shot class-incremental learning via class-aware bilateral distillation[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 11838−11847, articleTitle=null, refAbstract=null), Reference(id=1325491341844685364, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=244, rfOrder=250, authorNames=null, journalName=null, refType=null, unstructuredReference=Yan Hongwei, Wang Liyuan, Ma Kaisheng, et al. Orchestrate latent expertise: Advancing online continual learning with multi-level supervision and reverse self-distillation[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 23670−23680, articleTitle=null, refAbstract=null), Reference(id=1325491343971197493, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=245, rfOrder=251, authorNames=null, journalName=null, refType=null, unstructuredReference=Ye Fei, Bors A G. Continual variational autoencoder via continual generative knowledge distillation[C]//Proc of the 37th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023: 10918−10926, articleTitle=null, refAbstract=null), Reference(id=1325491346374533687, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=246, rfOrder=252, authorNames=null, journalName=null, refType=null, unstructuredReference=Wen Haitao, Pan Lili, Dai Yu, et al. Class incremental learning with multi-teacher distillation[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 28443−28452, articleTitle=null, refAbstract=null), Reference(id=1325491348505240120, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=247, rfOrder=253, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Ze, Li Ruibo, Ling Evan, et al. Label-guided knowledge distillation for continual semantic segmentation on 2D images and 3D point clouds[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 18555−18566, articleTitle=null, refAbstract=null), Reference(id=1325491349805474361, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=248, rfOrder=254, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Lin, Shen Li, Ding Liang, et al. Fine-tuning global model via data-free knowledge distillation for non-IID federated learning[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 10164−10173, articleTitle=null, refAbstract=null), Reference(id=1325491354033332796, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=249, rfOrder=255, authorNames=null, journalName=null, refType=null, unstructuredReference=Shen Yiqing, Zhou Yuyin, Yu Lequan. CD2-pFed: Cyclic distillation-guided channel decoupling for model personalization in federated learning[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 10031−10040, articleTitle=null, refAbstract=null), Reference(id=1325491356034015805, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=250, rfOrder=256, authorNames=null, journalName=null, refType=null, unstructuredReference=Kim H, Kwak Y, Jung M, et al. ProtoFL: Unsupervised federated learning via prototypical distillation[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 6447−6456, articleTitle=null, refAbstract=null), Reference(id=1325491357791429182, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=251, rfOrder=257, authorNames=null, journalName=null, refType=null, unstructuredReference=Lu Yang, Chen Lin, Zhang Yonggang, et al. Federated learning with extremely noisy clients via negative distillation[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 14184−14192, articleTitle=null, refAbstract=null), Reference(id=1325491358970028607, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=252, rfOrder=258, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Haozhao, Li Yichen, Xu Wenchao, et al. DaFKD: Domain-aware federated knowledge distillation[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 20412−20421, articleTitle=null, refAbstract=null), Reference(id=1325491360249291329, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=253, rfOrder=259, authorNames=null, journalName=null, refType=null, unstructuredReference=乐俊青, 谭州勇, 张迪, 等. 面向车联网数据持续共享的安全高效联邦学习[J]. 计算机研究与发展, 2024, 61(9): 2199−2212, articleTitle=null, refAbstract=null), Reference(id=1325491362371609154, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=253, rfOrder=260, authorNames=null, journalName=null, refType=null, unstructuredReference=Le Junqing, Tan Zhouyong, Zhang Di, et al. Secure and efficient federated learning for continuous IoV data sharing[J]. Journal of Computer Research and Development, 2024, 61(9): 2199−2212 (in Chinese), articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1325490582310756372, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, xref=null, ext=[AuthorCompanyExt(id=1325490582361088021, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001), AuthorCompanyExt(id=1325490582587580438, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001)])], figs=[ArticleFig(id=1325490651911037005, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Figure 1, caption=Overview of the general structure in our paper, figureFileSmall=vKzwRXKs5ApH7FR+XmapLg==, figureFileBig=c5TffK5Y2OVaiAIy+407Gg==, tableContent=null), ArticleFig(id=1325490653899137102, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=图1, caption=本文总体结构概览, figureFileSmall=vKzwRXKs5ApH7FR+XmapLg==, figureFileBig=c5TffK5Y2OVaiAIy+407Gg==, tableContent=null), ArticleFig(id=1325490662329688145, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Figure 2, caption=Convolutional network model architecture, figureFileSmall=Yk9wvvvidmUB3FfHgjYh8Q==, figureFileBig=8O/FUP1mV1W3XRNNqaGzyg==, tableContent=null), ArticleFig(id=1325490665261506643, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=图2, caption=卷积网络模型架构, figureFileSmall=Yk9wvvvidmUB3FfHgjYh8Q==, figureFileBig=8O/FUP1mV1W3XRNNqaGzyg==, tableContent=null), ArticleFig(id=1325490667761311828, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Figure 3, caption=ViT model architecture, figureFileSmall=Ganfc+xyx+7S3OBDTdQnqw==, figureFileBig=F3qCwuMK9AeFotTO/nRZ7A==, tableContent=null), ArticleFig(id=1325490670747656277, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=图3, caption=ViT 模型架构, figureFileSmall=Ganfc+xyx+7S3OBDTdQnqw==, figureFileBig=F3qCwuMK9AeFotTO/nRZ7A==, tableContent=null), ArticleFig(id=1325490675306864729, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Figure 4, caption=Basic architecture of knowledge distillation, figureFileSmall=CgGn37Ckj6BoVj2Ca+CyVQ==, figureFileBig=hUfpVIuRW0+hVzcM99njXw==, tableContent=null), ArticleFig(id=1325490678230294619, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=图4, caption=知识蒸馏的基本架构, figureFileSmall=CgGn37Ckj6BoVj2Ca+CyVQ==, figureFileBig=hUfpVIuRW0+hVzcM99njXw==, tableContent=null), ArticleFig(id=1325490680742682716, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Figure 5, caption=Illustration of distillation method based on probability distribution output, figureFileSmall=4l6kdGw2qC6m2+a1/xtslw==, figureFileBig=vzDbLtXU4bFAnh0lzo1WNw==, tableContent=null), ArticleFig(id=1325490682911137886, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=图5, caption=基于概率分布输出的蒸馏方法示意图, figureFileSmall=4l6kdGw2qC6m2+a1/xtslw==, figureFileBig=vzDbLtXU4bFAnh0lzo1WNw==, tableContent=null), ArticleFig(id=1325490685423525983, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Figure 6, caption=Illustration of distillation method based on Logits output, figureFileSmall=+aiJvHYdwQ0u4xLxD8kYZg==, figureFileBig=2FVY/JHgbd+8NYY8/jAdpg==, tableContent=null), ArticleFig(id=1325490688774774880, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=图6, caption=基于Logits 输出的蒸馏方法示意图, figureFileSmall=+aiJvHYdwQ0u4xLxD8kYZg==, figureFileBig=2FVY/JHgbd+8NYY8/jAdpg==, tableContent=null), ArticleFig(id=1325490692176355427, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Figure 7, caption=Illustration of distillation method based on intermediate features, figureFileSmall=jdXbTvM+r75PoaGPZP8Eng==, figureFileBig=NfsXNY/f8RVwEYyS1RnSdA==, tableContent=null), ArticleFig(id=1325490694193815652, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=图7, caption=基于中间特征的蒸馏方法示意图, figureFileSmall=jdXbTvM+r75PoaGPZP8Eng==, figureFileBig=NfsXNY/f8RVwEYyS1RnSdA==, tableContent=null), ArticleFig(id=1325490696265801829, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Figure 8, caption=Illustration of distillation method based on relational features, figureFileSmall=GhWAzPMHYuEvjmrePAaYOw==, figureFileBig=gv+za33hpcbv0k+VQgDEAQ==, tableContent=null), ArticleFig(id=1325490698434256999, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=图8, caption=基于关系特征的蒸馏方法示意图, figureFileSmall=GhWAzPMHYuEvjmrePAaYOw==, figureFileBig=gv+za33hpcbv0k+VQgDEAQ==, tableContent=null), ArticleFig(id=1325490701760340072, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Figure 9, caption=Illustration of CNN2ViT method, figureFileSmall=D+ExZABaGV0jMdjIWM/QMQ==, figureFileBig=+auuYPcBGDx3G4CkKwZloA==, tableContent=null), ArticleFig(id=1325490705585545321, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=图9, caption=CNN2ViT 方法示意图, figureFileSmall=D+ExZABaGV0jMdjIWM/QMQ==, figureFileBig=+auuYPcBGDx3G4CkKwZloA==, tableContent=null), ArticleFig(id=1325490708508975210, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Figure 10, caption=Illustration of ViT2CNN method, figureFileSmall=P5ibvUjHhLhNTU1G6eFrGA==, figureFileBig=8jhK7oZ3s3xTbTalk1xl8A==, tableContent=null), ArticleFig(id=1325490712212545643, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=图10, caption=ViT2CNN 方法示意图, figureFileSmall=P5ibvUjHhLhNTU1G6eFrGA==, figureFileBig=8jhK7oZ3s3xTbTalk1xl8A==, tableContent=null), ArticleFig(id=1325490714007707758, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Figure 11, caption=Illustration of ViT2ViT method, figureFileSmall=othK7q2yk35X2lXiBoddew==, figureFileBig=uEQVNRlCo2te9zFzISKcyA==, tableContent=null), ArticleFig(id=1325490716012585072, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=图11, caption=ViT2ViT 方法示意图, figureFileSmall=othK7q2yk35X2lXiBoddew==, figureFileBig=uEQVNRlCo2te9zFzISKcyA==, tableContent=null), ArticleFig(id=1325490718550138994, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Figure 12, caption=Style of learning of knowledge distillation, figureFileSmall=MrjZRXQBim+O6xuxePHusw==, figureFileBig=SG9UoU5unQoBPtPd6lkXmA==, tableContent=null), ArticleFig(id=1325490720659873907, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=图12, caption=知识蒸馏的学习方式, figureFileSmall=MrjZRXQBim+O6xuxePHusw==, figureFileBig=SG9UoU5unQoBPtPd6lkXmA==, tableContent=null), ArticleFig(id=1325490723579109493, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Figure 13, caption=Illustration of optimized location of knowledge distillation, figureFileSmall=ChTSfxEWdBwJILhAFGLbBQ==, figureFileBig=Xsj0Af1j6zjs/gh5QH4J+A==, tableContent=null), ArticleFig(id=1325490725630124151, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=图13, caption=知识蒸馏的优化位置示意图, figureFileSmall=ChTSfxEWdBwJILhAFGLbBQ==, figureFileBig=Xsj0Af1j6zjs/gh5QH4J+A==, tableContent=null), ArticleFig(id=1325490727853105273, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Table 1, caption=Reference Table of Distance Symbols, figureFileSmall=null, figureFileBig=null, tableContent=
符号缩写英文全称中文名称
${{L}_1}$${{L}_{1}}{ \text{-Norm}}$曼哈顿距离
${{L}_2}$$ {{L}_{2}}{ \text{-Norm}} $欧氏距离
${{L}_\infty }$$ {{L}_\infty }{ \text{-Norm}} $切比雪夫距离
CEcross entropy交叉熵
KLKullback-Leibler divergenceKL散度
MSEmean-square error均方误差
MMDmaximum mean discrepancy最大均值差异
EMDearth mover’s distance推土机距离
InfoNCE[27]information noise-contrastive estimation
), ArticleFig(id=1325490729874759803, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=表1, caption=

距离符号对照表

, figureFileSmall=null, figureFileBig=null, tableContent=
符号缩写英文全称中文名称
${{L}_1}$${{L}_{1}}{ \text{-Norm}}$曼哈顿距离
${{L}_2}$$ {{L}_{2}}{ \text{-Norm}} $欧氏距离
${{L}_\infty }$$ {{L}_\infty }{ \text{-Norm}} $切比雪夫距离
CEcross entropy交叉熵
KLKullback-Leibler divergenceKL散度
MSEmean-square error均方误差
MMDmaximum mean discrepancy最大均值差异
EMDearth mover’s distance推土机距离
InfoNCE[27]information noise-contrastive estimation
), ArticleFig(id=1325490732852715644, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Table 2, caption=Summary of Knowledge Distillation Methods Based on Output Features, figureFileSmall=null, figureFileBig=null, tableContent=
方法来源知识的来源距离度量关键技术
注:L2,KL,CE分别表示欧氏距离、KL散度和交叉熵。
文献[35]Logits输出${{L}_2}$提出模仿复杂模型的Logits输出,获得与复杂模型精度相当的轻量模型
文献[21]概率分布输出KL通过温度系数软化标签强调暗知识的作用
文献[29]概率分布输出${{L}_2}$提出对教师模型概率分布输出添加噪声,模拟同专业的不同教师模型,缓解知识多样性问题
文献[30]概率分布输出KL提出让多个模型互相学习来获得更有效更紧凑的模型
文献[36]Logits输出${{L}_2}$通过共享师生模型架构的低级别参数来降低推理所需要的时间
文献[31]概率分布输出CE提出了一种新的学习范式,通过序列化建模获得高性能的学生模型
文献[32]概率分布输出最高分差考虑了次要类别概率分布输出的知识,缓解严格教师模型带来的性能损失
文献[33]概率分布输出CE提出标签与增强标签联合优化,处理样本经过数据增强后不一定线性可分的问题
文献[34]概率分布输出
倒数第2层输出
CE
L2
提出将表示学习和分类解耦,利用教师分类器辅助学生模型训练,改善学生模型的特征表示
文献[37]Logits输出概率分布输出CE
KL
始终将上级教师模型的知识作为多级下级学生模型的指导信息,缓解教师模型错误累计问题
文献[39]师生Logits输出差KL利用师生模型Logits输出的差异作为知识来缩减师生模型之间的能力差距
), ArticleFig(id=1325490735323160703, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=表2, caption=

基于输出特征的知识蒸馏方法汇总

, figureFileSmall=null, figureFileBig=null, tableContent=
方法来源知识的来源距离度量关键技术
注:L2,KL,CE分别表示欧氏距离、KL散度和交叉熵。
文献[35]Logits输出${{L}_2}$提出模仿复杂模型的Logits输出,获得与复杂模型精度相当的轻量模型
文献[21]概率分布输出KL通过温度系数软化标签强调暗知识的作用
文献[29]概率分布输出${{L}_2}$提出对教师模型概率分布输出添加噪声,模拟同专业的不同教师模型,缓解知识多样性问题
文献[30]概率分布输出KL提出让多个模型互相学习来获得更有效更紧凑的模型
文献[36]Logits输出${{L}_2}$通过共享师生模型架构的低级别参数来降低推理所需要的时间
文献[31]概率分布输出CE提出了一种新的学习范式,通过序列化建模获得高性能的学生模型
文献[32]概率分布输出最高分差考虑了次要类别概率分布输出的知识,缓解严格教师模型带来的性能损失
文献[33]概率分布输出CE提出标签与增强标签联合优化,处理样本经过数据增强后不一定线性可分的问题
文献[34]概率分布输出
倒数第2层输出
CE
L2
提出将表示学习和分类解耦,利用教师分类器辅助学生模型训练,改善学生模型的特征表示
文献[37]Logits输出概率分布输出CE
KL
始终将上级教师模型的知识作为多级下级学生模型的指导信息,缓解教师模型错误累计问题
文献[39]师生Logits输出差KL利用师生模型Logits输出的差异作为知识来缩减师生模型之间的能力差距
), ArticleFig(id=1325490738250784898, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Table 3, caption=Summary of Distillation Methods Based on Intermediate Features, figureFileSmall=null, figureFileBig=null, tableContent=
方法来源知识的来源知识的转换距离度量关键技术
注:${{L}_1}$和${{L}_2}$表示曼哈顿距离和欧氏距离,KL和CE表示KL散度和交叉熵,MMD表示最大均值差异,dp[52]表示修正的欧氏距离。
文献[40]中间层特征图线性层${{L}_2}$提出从教师模型中间层抽取知识来改善训练过程和学生模型性能
文献[41]注意力图归一化层${{L}_2}$利用网络层块注意力图转移特征知识模仿人类视觉系统来提高网络性能
文献[47]神经元激活模式核方法MMD提出转移教师模型神经元激活模式来传递知识以提高蒸馏知识的有效性
文献[50]中间层特征图线性层${{L}_2}$通过结合量化和蒸馏2种方法来专注于非常微小网络的训练
文献[51]中间层特征图互信息提出知识转移信息论框架,将知识转移表述为师生模型的互信息最大化
文献[52]中间层特征图margin-ReLUdp提出新的激活函数来保存教师模型信息,引入新距离度量用于特征蒸馏
文献[59]概率分布输出中间层特征图线性层CE,${{L}_2}$鼓励同一卷积层的不同卷积核生成多样性特征图,
缓解学生模型性能低下的问题
文献[56]联合分布特征中间层特征图概率分布输出线性层CE,KL将联合标签增强自蒸馏方法迁移到传统蒸馏,缓解自监督对比
学习对原始类表征的损害
文献[34]倒数第2层特征
概率分布输出
线性层${{L}_2}$,CE提出将表示学习和分类解耦,利用教师分类器辅助学生模型训练,
改善学生模型的特征表示
文献[60]金字塔网络特征图注意力聚合${{L}_2}$提出跨阶段连接路径,学生模型高阶层学习教师模型低层
特征以提高模型的性能表现
文献[58]倒数第2层特征教师模型分类器线性层${{L}_2}$提出对齐师生模型倒数第2层特征并直接复用教师模型分类器
来降低蒸馏算法的复杂性
文献[42]中间层特征图相似加权线性层${{L}_2}$利用教师模型对学生模型特征重建,解决模型架构差异带来的语义偏差
文献[43]金字塔网络特征图随机掩码、线性层${{L}_2}$利用掩码图像机制进行知识传递,提高教师模型知识的完整性和有效性
文献[44]类别注意力图归一化层${{L}_2}$提出类别注意力图来进行知识的迁移,缓解蒸馏可解释性差的问题
文献[45]中间层频率特征线性层${{L}_1}$通过离散小波变换获得频率域特征,缓解空间域连续下采样的信息丢失
), ArticleFig(id=1325490742264733828, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=表3, caption=

基于中间特征的蒸馏方法汇总

, figureFileSmall=null, figureFileBig=null, tableContent=
方法来源知识的来源知识的转换距离度量关键技术
注:${{L}_1}$和${{L}_2}$表示曼哈顿距离和欧氏距离,KL和CE表示KL散度和交叉熵,MMD表示最大均值差异,dp[52]表示修正的欧氏距离。
文献[40]中间层特征图线性层${{L}_2}$提出从教师模型中间层抽取知识来改善训练过程和学生模型性能
文献[41]注意力图归一化层${{L}_2}$利用网络层块注意力图转移特征知识模仿人类视觉系统来提高网络性能
文献[47]神经元激活模式核方法MMD提出转移教师模型神经元激活模式来传递知识以提高蒸馏知识的有效性
文献[50]中间层特征图线性层${{L}_2}$通过结合量化和蒸馏2种方法来专注于非常微小网络的训练
文献[51]中间层特征图互信息提出知识转移信息论框架,将知识转移表述为师生模型的互信息最大化
文献[52]中间层特征图margin-ReLUdp提出新的激活函数来保存教师模型信息,引入新距离度量用于特征蒸馏
文献[59]概率分布输出中间层特征图线性层CE,${{L}_2}$鼓励同一卷积层的不同卷积核生成多样性特征图,
缓解学生模型性能低下的问题
文献[56]联合分布特征中间层特征图概率分布输出线性层CE,KL将联合标签增强自蒸馏方法迁移到传统蒸馏,缓解自监督对比
学习对原始类表征的损害
文献[34]倒数第2层特征
概率分布输出
线性层${{L}_2}$,CE提出将表示学习和分类解耦,利用教师分类器辅助学生模型训练,
改善学生模型的特征表示
文献[60]金字塔网络特征图注意力聚合${{L}_2}$提出跨阶段连接路径,学生模型高阶层学习教师模型低层
特征以提高模型的性能表现
文献[58]倒数第2层特征教师模型分类器线性层${{L}_2}$提出对齐师生模型倒数第2层特征并直接复用教师模型分类器
来降低蒸馏算法的复杂性
文献[42]中间层特征图相似加权线性层${{L}_2}$利用教师模型对学生模型特征重建,解决模型架构差异带来的语义偏差
文献[43]金字塔网络特征图随机掩码、线性层${{L}_2}$利用掩码图像机制进行知识传递,提高教师模型知识的完整性和有效性
文献[44]类别注意力图归一化层${{L}_2}$提出类别注意力图来进行知识的迁移,缓解蒸馏可解释性差的问题
文献[45]中间层频率特征线性层${{L}_1}$通过离散小波变换获得频率域特征,缓解空间域连续下采样的信息丢失
), ArticleFig(id=1325490745783754885, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Table 4, caption=Summary of Distillation Methods Based on Relational Features, figureFileSmall=null, figureFileBig=null, tableContent=
方法来源知识的来源相似性知识距离度量关键技术
注:${{L}_2}$,${{L}_\infty }$分别表示欧氏距离和切比雪夫距离,EMD,MMD,MSE,KL分别表示推土机距离、最大均值差异、均方误差、KL散度, InfoNCE[27]表示一项对比损失。
文献[61]中间层特征图${{L}_2}$${{L}_\infty }$利用不同样本的中间层特征之间的相对差异作为知识,提高知识的多样性
文献[53]中间层特征图哈达玛乘积${{L}_2}$提出将教师模型求解问题流程抽象为关系矩阵来作为知识提高学生模型的性能
文献[62]Logits输出
中间层特征图
Logits图表示图EMD
MMD
利用教师模型Logits输出特征和中间特征构造图结构,解决特征之间互补性的缺失问题
文献[66]样本特征向量${{L}_2}$KL提出一种新的知识类型,给出样本相似性打分函数并根据相似性来进行蒸馏
文献[67]不同层样本
特征向量
实例关系图${{L}_2}$利用实例特征构造图关系知识,解决由于师生模型架构不同
导致的性能下降和类间差异性低的问题
文献[68]样本特征向量核方法${{L}_2}$考虑样本内容一致性和样本关系一致性,解除师生模型的实例强约束限制
文献[69]中间层特征图转置相乘${{L}_2}$利用样本相似性向学生模型传递教师模型的激活模式,推动知识获取的发展
文献[57]样本特征向量互信息最大互信息引入对比损失表达样本关系,处理先前方法忽略教师模型结构信息的问题
文献[63]中间层特征图转置相乘MSE利用相似度矩阵和注意力机制捕获中间层语义关联,解决手动关联不匹配问题
文献[64]中间层特征图转置相乘${{L}_2}$提出通道间相似性知识蒸馏,解决教师模型特征多样性不足的问题
文献[70]样本特征向量余弦相似度InfoNCE提出词袋聚合蒸馏传递关系知识,解决基于对比学习方法优化困难的问题
文献[65]概率分布输出
中间层特征图
转置相乘KL
L2
从样本实例、批次和类别角度构造关系知识,解决基于概率
分布输出蒸馏方法性能欠佳的问题
), ArticleFig(id=1325490747897684103, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=表4, caption=

基于关系特征的蒸馏方法汇总

, figureFileSmall=null, figureFileBig=null, tableContent=
方法来源知识的来源相似性知识距离度量关键技术
注:${{L}_2}$,${{L}_\infty }$分别表示欧氏距离和切比雪夫距离,EMD,MMD,MSE,KL分别表示推土机距离、最大均值差异、均方误差、KL散度, InfoNCE[27]表示一项对比损失。
文献[61]中间层特征图${{L}_2}$${{L}_\infty }$利用不同样本的中间层特征之间的相对差异作为知识,提高知识的多样性
文献[53]中间层特征图哈达玛乘积${{L}_2}$提出将教师模型求解问题流程抽象为关系矩阵来作为知识提高学生模型的性能
文献[62]Logits输出
中间层特征图
Logits图表示图EMD
MMD
利用教师模型Logits输出特征和中间特征构造图结构,解决特征之间互补性的缺失问题
文献[66]样本特征向量${{L}_2}$KL提出一种新的知识类型,给出样本相似性打分函数并根据相似性来进行蒸馏
文献[67]不同层样本
特征向量
实例关系图${{L}_2}$利用实例特征构造图关系知识,解决由于师生模型架构不同
导致的性能下降和类间差异性低的问题
文献[68]样本特征向量核方法${{L}_2}$考虑样本内容一致性和样本关系一致性,解除师生模型的实例强约束限制
文献[69]中间层特征图转置相乘${{L}_2}$利用样本相似性向学生模型传递教师模型的激活模式,推动知识获取的发展
文献[57]样本特征向量互信息最大互信息引入对比损失表达样本关系,处理先前方法忽略教师模型结构信息的问题
文献[63]中间层特征图转置相乘MSE利用相似度矩阵和注意力机制捕获中间层语义关联,解决手动关联不匹配问题
文献[64]中间层特征图转置相乘${{L}_2}$提出通道间相似性知识蒸馏,解决教师模型特征多样性不足的问题
文献[70]样本特征向量余弦相似度InfoNCE提出词袋聚合蒸馏传递关系知识,解决基于对比学习方法优化困难的问题
文献[65]概率分布输出
中间层特征图
转置相乘KL
L2
从样本实例、批次和类别角度构造关系知识,解决基于概率
分布输出蒸馏方法性能欠佳的问题
), ArticleFig(id=1325490749529268361, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Table 5, caption=Summary of ViT Architecture Relative Distillation Methods, figureFileSmall=null, figureFileBig=null, tableContent=
方法来源知识的来源知识的转换架构距离度量关键技术
注:KL,CE,MSE,${{L}_{1}}$,${{L}_{2}}$分别表示KL散度、交叉熵、均方误差、曼哈顿距离和欧氏距离。
文献[17]CNN概率分布输出蒸馏tokenCNN→ViTKL引入一个蒸馏token接收卷积教师模型的知识,
解决ViT在数据不足时的泛化性问题
文献[76]QKV矩阵线性层ViT→CNN${{L}_2}$提出交叉注意力分组线性映射对齐师生模型特征,
缓解跨架构蒸馏的语义对齐问题
文献[73]CNN概率分布输出、
INN概率分布输出
蒸馏tokenCNN→ViTKL引入多个蒸馏token接收卷积网络和内卷网络的知识,
探究了归纳偏差的作用
文献[82]分类token稀疏编码ViT→ViTCE将教师模型推理与蒸馏分离,缓解资源有限设备上的局限性
文献[83]分类token、QKV
矩阵、MLP输出
自注意力矩阵ViT→ViTCE提出权重复用和权重蒸馏,缓解在资源有限设备上的适用性
文献[84]MLP输出线性层、掩码ViT→ViT${{L}_2}$提出了一种基于特征的ViT蒸馏方法,缓解CNN架构的
蒸馏方法无法应用到ViT架构上的问题
文献[75]CNN Logits输出全局池化层CNN→ViTKL提出累计空间知识蒸馏,缓解跨架构语义对齐和CNN
知识抑制模型后期训练的问题
文献[86]QKV矩阵自注意力矩阵ViT→ViTKL将MIM的知识转移到较小的模型中,解决小模型
只能从预训练方法中获益的问题
文献[87]分类token、
图像掩码块
无ViT→ViTCE通过分类token和MIM损失进行自监督训练,
缓解小样本下ViT的性能损失问题
文献[88]MLP输出线性层ViT→ViT${{L}_1}$利用MAE预训练模型提取知识,缓解蒸馏计算效率的问题
文献[90]MLP输出线性层 MHAViT→ViT${{L}_1}$提出非对称掩码蒸馏,缓解教师模型高掩码率信息丢失问题
文献[77]MHA特征图自注意力矩阵ViT→CNNMSE将像素级特征映射到具有统一感受野的局部特征中,
解决师生模型特征的对齐问题
文献[85]MHA特征图余弦相似度ViT→ViT${{L}_1}$提出多头注意力对齐策略,解决师生模型输出头对齐问题
文献[78]MLP输出
中间层特征图
特征投影层ViT→CNN
CNN→ViT
KL提出通过将特征映射到Logits空间,解决异构模型的特征对齐问题
文献[79]MLP输出中间
层特征图
助理模型ViT→CNN
CNN→ViT
InfoNCE
KL
引入一个辅助模型来促进异构教师模型和学生模型之间的特征传递
文献[80]MHA特征、
分类token
线性层ViT→CNNKL
CE
通过多个不同归纳偏差的学生模型相互学习,来缓解跨架构蒸馏的性能问题
文献[81]MLP输出特征适应模块ViT→CNN${{L}_2}$提出特征适应块处理教师模型特征,缓解异构特征的不匹配
), ArticleFig(id=1325490751211184267, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=表5, caption=

ViT 架构相关蒸馏方法汇总

, figureFileSmall=null, figureFileBig=null, tableContent=
方法来源知识的来源知识的转换架构距离度量关键技术
注:KL,CE,MSE,${{L}_{1}}$,${{L}_{2}}$分别表示KL散度、交叉熵、均方误差、曼哈顿距离和欧氏距离。
文献[17]CNN概率分布输出蒸馏tokenCNN→ViTKL引入一个蒸馏token接收卷积教师模型的知识,
解决ViT在数据不足时的泛化性问题
文献[76]QKV矩阵线性层ViT→CNN${{L}_2}$提出交叉注意力分组线性映射对齐师生模型特征,
缓解跨架构蒸馏的语义对齐问题
文献[73]CNN概率分布输出、
INN概率分布输出
蒸馏tokenCNN→ViTKL引入多个蒸馏token接收卷积网络和内卷网络的知识,
探究了归纳偏差的作用
文献[82]分类token稀疏编码ViT→ViTCE将教师模型推理与蒸馏分离,缓解资源有限设备上的局限性
文献[83]分类token、QKV
矩阵、MLP输出
自注意力矩阵ViT→ViTCE提出权重复用和权重蒸馏,缓解在资源有限设备上的适用性
文献[84]MLP输出线性层、掩码ViT→ViT${{L}_2}$提出了一种基于特征的ViT蒸馏方法,缓解CNN架构的
蒸馏方法无法应用到ViT架构上的问题
文献[75]CNN Logits输出全局池化层CNN→ViTKL提出累计空间知识蒸馏,缓解跨架构语义对齐和CNN
知识抑制模型后期训练的问题
文献[86]QKV矩阵自注意力矩阵ViT→ViTKL将MIM的知识转移到较小的模型中,解决小模型
只能从预训练方法中获益的问题
文献[87]分类token、
图像掩码块
无ViT→ViTCE通过分类token和MIM损失进行自监督训练,
缓解小样本下ViT的性能损失问题
文献[88]MLP输出线性层ViT→ViT${{L}_1}$利用MAE预训练模型提取知识,缓解蒸馏计算效率的问题
文献[90]MLP输出线性层 MHAViT→ViT${{L}_1}$提出非对称掩码蒸馏,缓解教师模型高掩码率信息丢失问题
文献[77]MHA特征图自注意力矩阵ViT→CNNMSE将像素级特征映射到具有统一感受野的局部特征中,
解决师生模型特征的对齐问题
文献[85]MHA特征图余弦相似度ViT→ViT${{L}_1}$提出多头注意力对齐策略,解决师生模型输出头对齐问题
文献[78]MLP输出
中间层特征图
特征投影层ViT→CNN
CNN→ViT
KL提出通过将特征映射到Logits空间,解决异构模型的特征对齐问题
文献[79]MLP输出中间
层特征图
助理模型ViT→CNN
CNN→ViT
InfoNCE
KL
引入一个辅助模型来促进异构教师模型和学生模型之间的特征传递
文献[80]MHA特征、
分类token
线性层ViT→CNNKL
CE
通过多个不同归纳偏差的学生模型相互学习,来缓解跨架构蒸馏的性能问题
文献[81]MLP输出特征适应模块ViT→CNN${{L}_2}$提出特征适应块处理教师模型特征,缓解异构特征的不匹配
), ArticleFig(id=1325490755422265484, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Table 6, caption=Summary of Learning Methods of Knowledge Distillation, figureFileSmall=null, figureFileBig=null, tableContent=
方法来源学习方法关键技术
文献[115]无数据学习提出利用网络的元数据蒸馏,无需访问原始数据,解决数据隐私问题
文献[30]互学习提出让多个模型互相学习其输出,在没有强大教师模型也能提高模型性能
文献[102]共享学习使每个模型在不相交的数据子集上训练并迁移其他模型知识来降低复杂性
文献[103]协同学习提出协同学习,在相同数据上训练多个分类器,提高模型的泛化能力
文献[36]共享学习通过共享师生模型架构的低级别参数来降低推理所需要的时间
文献[122]无数据学习提出一种利用GAN合成数据的方法,缓解蒸馏中的隐私问题
文献[109]自学习提出将模型中发现的知识提炼回自身来避免训练计算开销大的教师模型
文献[108]自学习提出自蒸馏的概念来缓解网络规模增加带来的计算存储成本
文献[116]无数据学习将softmax函数建模为狄利克雷分布抽取教师模型数据分布,缓解数据隐私问题
文献[110]自学习将模型训练早期的知识转移到模型训练后期来降低训练的计算资源消耗
文献[104]协同学习提出多个辅助网络与主网络进行2级蒸馏,提高多样性来缓解学生模型同质化问题
文献[111]自学习对自蒸馏进行理论分析,自蒸馏迭代逐渐限制表示解决方案的基函数数量
文献[38]助理学习提出通过助理模型缓解师生模型的能力差距问题
文献[143]助理学习通过助理模型建立师生模型层之间1对1的匹配关系,缓解师生模型的能力差距问题
文献[106]协作学习构建分支网络,分支增强后与另一分支分类器集成后互相学习来构建高水平教师模型
文献[144]助理学习集成了多个助理模型来缩减师生模型性能差异,实现了更好的蒸馏性能
文献[145]助理学习提出通过助理模型学习师生模型之间的残差来弥合两者学习能力的差异
文献[37]助理学习利用高层助理模型和教师模型指导低层助理模型和学生模型缓解错误累计问题
文献[128]无数据学习提出动量对抗蒸馏来缓解合成数据的分布偏移问题
文献[98]互学习通过自适应切换阈值互换师生模型角色,缓解师生模型能力差距带来的性能影响
文献[129]无数据学习提出通过双判别器对抗蒸馏来解决数据不可用的问题
文献[112]自学习通过利用上一次迭代结果更新自蒸馏模型来保持模型的一致性
文献[130]无数据学习利用条件生成对抗网络改进合成样本的歧义性和多样性不足的问题
文献[117]无数据学习提出利用重用共享的共同特征来解决数据合成效率低下的问题
文献[146]助理学习同时考虑了输出和中间特征来训练助理模型,更好地缓解了师生模型之间的性能差异
文献[101]互学习在教师模型中添加可学习适配器模块来缓解师生模型预测分布尖锐度差异
文献[99]互学习提出对比互学习框架来从其他网络学习对比知识以增强特征表示能力
文献[131]无数据学习提出一个元学习启发框架来解决生成的伪样本分布不稳定的问题
文献[100]互学习提出自适应特征蒸馏和在线任务加权来缓解多任务学习中准确率受损问题
文献[105]协作学习提出自适应分支融合框架提高模型多样性来缓解学生模型集成的同质化
文献[113]自学习将模型前几层知识蒸馏到后几层,通过自我反思进行更有效的学习
文献[118]无数据学习提出信道特征交换和多尺度区域一致性,处理数据合成在多样性和效率方面的局限性
文献[119]无数据学习通过逐渐生成由易到难的伪样本,模拟人类学习方式解决学习不充分不稳定的问题
文献[147]助理学习令助理模型和学生模型相互学习,通过离线在线联合蒸馏的方法缓解师生模型差异
文献[148]助理学习构造自适应助理模型,提供多样化知识并缩小师生模型之间的能力差距
文献[120]无数据学习利用师生模型的差异生成样本来解决无数据蒸馏中样本质量不理想的问题
文献[79]助理学习通过助理模型来进行跨架构的蒸馏,弥合异构模型之间的巨大特征差异
), ArticleFig(id=1325490758492496016, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=表6, caption=

知识蒸馏的学习方式汇总

, figureFileSmall=null, figureFileBig=null, tableContent=
方法来源学习方法关键技术
文献[115]无数据学习提出利用网络的元数据蒸馏,无需访问原始数据,解决数据隐私问题
文献[30]互学习提出让多个模型互相学习其输出,在没有强大教师模型也能提高模型性能
文献[102]共享学习使每个模型在不相交的数据子集上训练并迁移其他模型知识来降低复杂性
文献[103]协同学习提出协同学习,在相同数据上训练多个分类器,提高模型的泛化能力
文献[36]共享学习通过共享师生模型架构的低级别参数来降低推理所需要的时间
文献[122]无数据学习提出一种利用GAN合成数据的方法,缓解蒸馏中的隐私问题
文献[109]自学习提出将模型中发现的知识提炼回自身来避免训练计算开销大的教师模型
文献[108]自学习提出自蒸馏的概念来缓解网络规模增加带来的计算存储成本
文献[116]无数据学习将softmax函数建模为狄利克雷分布抽取教师模型数据分布,缓解数据隐私问题
文献[110]自学习将模型训练早期的知识转移到模型训练后期来降低训练的计算资源消耗
文献[104]协同学习提出多个辅助网络与主网络进行2级蒸馏,提高多样性来缓解学生模型同质化问题
文献[111]自学习对自蒸馏进行理论分析,自蒸馏迭代逐渐限制表示解决方案的基函数数量
文献[38]助理学习提出通过助理模型缓解师生模型的能力差距问题
文献[143]助理学习通过助理模型建立师生模型层之间1对1的匹配关系,缓解师生模型的能力差距问题
文献[106]协作学习构建分支网络,分支增强后与另一分支分类器集成后互相学习来构建高水平教师模型
文献[144]助理学习集成了多个助理模型来缩减师生模型性能差异,实现了更好的蒸馏性能
文献[145]助理学习提出通过助理模型学习师生模型之间的残差来弥合两者学习能力的差异
文献[37]助理学习利用高层助理模型和教师模型指导低层助理模型和学生模型缓解错误累计问题
文献[128]无数据学习提出动量对抗蒸馏来缓解合成数据的分布偏移问题
文献[98]互学习通过自适应切换阈值互换师生模型角色,缓解师生模型能力差距带来的性能影响
文献[129]无数据学习提出通过双判别器对抗蒸馏来解决数据不可用的问题
文献[112]自学习通过利用上一次迭代结果更新自蒸馏模型来保持模型的一致性
文献[130]无数据学习利用条件生成对抗网络改进合成样本的歧义性和多样性不足的问题
文献[117]无数据学习提出利用重用共享的共同特征来解决数据合成效率低下的问题
文献[146]助理学习同时考虑了输出和中间特征来训练助理模型,更好地缓解了师生模型之间的性能差异
文献[101]互学习在教师模型中添加可学习适配器模块来缓解师生模型预测分布尖锐度差异
文献[99]互学习提出对比互学习框架来从其他网络学习对比知识以增强特征表示能力
文献[131]无数据学习提出一个元学习启发框架来解决生成的伪样本分布不稳定的问题
文献[100]互学习提出自适应特征蒸馏和在线任务加权来缓解多任务学习中准确率受损问题
文献[105]协作学习提出自适应分支融合框架提高模型多样性来缓解学生模型集成的同质化
文献[113]自学习将模型前几层知识蒸馏到后几层,通过自我反思进行更有效的学习
文献[118]无数据学习提出信道特征交换和多尺度区域一致性,处理数据合成在多样性和效率方面的局限性
文献[119]无数据学习通过逐渐生成由易到难的伪样本,模拟人类学习方式解决学习不充分不稳定的问题
文献[147]助理学习令助理模型和学生模型相互学习,通过离线在线联合蒸馏的方法缓解师生模型差异
文献[148]助理学习构造自适应助理模型,提供多样化知识并缩小师生模型之间的能力差距
文献[120]无数据学习利用师生模型的差异生成样本来解决无数据蒸馏中样本质量不理想的问题
文献[79]助理学习通过助理模型来进行跨架构的蒸馏,弥合异构模型之间的巨大特征差异
), ArticleFig(id=1325490760480596113, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=EN, label=Table 7, caption=Summary of Optimization Methods for Knowledge Distillation, figureFileSmall=null, figureFileBig=null, tableContent=
方法来源优化位置优化方法关键技术
文献[150]训练过程提前停止提出提前停止优化方法,缓解蒸馏损失后期对模型的损害
文献[176]损失函数对抗性技术提出利用分块训练损失来从教师模型中提取多样化知识,缓解知识单一的问题
文献[179]训练过程对抗性技术利用对抗攻击发现决策边界样本,传递准确的边界信息来提升性能
文献[151]训练过程正则化约束性能较弱的模型能够提供正则化约束给性能较强的模型来提升性能
文献[38]训练过程辅助模型引入一个助理模型来弥合师生模型的性能差距,缓解师生模型能力差距带来的性能下降
文献[158]知识结构特征筛选提出选择性转移策略,利用样本变换带来的暗知识以提高模型性能
文献[166]温度系数动态温度利用倒数第2层特征的${{L}_2}$范数作为温度系数来缓解标签噪声的影响
文献[177]训练过程对抗性技术设计任务驱动的注意力机制,构造紧凑的知识结构以减少信息丢失
文献[186]模型架构强化学习设计奖励函数寻找教师模型的最佳学生模型,缓解师生模型能力差距带来的性能问题
文献[187]模型架构自动机器学习提出一种基于预言机的搜索算法来缓解师生模型能力差距带来的性能问题
文献[185]模型架构自动机器学习提出一种分块的搜索评估方案以保证候选框架得到公平的训练
文献[167]温度系数动态温度通过退火函数生成动态温度来缓解师生模型能力差距过大带来的性能下降
文献[152]训练过程综合探究影响蒸馏性能的原因,优化不佳是主要原因
文献[178]损失函数对抗性技术提出自我破坏知识蒸馏来缓解蒸馏知识产权暴露的风险
文献[172]损失函数解耦提出将损失函数解耦来缓解经典蒸馏损失本身内部的冲突博弈
文献[153]训练过程非参数化对齐提出通过奇异值分解对齐师生模型来缓解投射层性能效率不佳的问题
文献[154]训练过程推理蒸馏分离通过保存教师模型推理的数据增强及输出来缓解蒸馏过程教师模型推理的开销
文献[159]知识结构特征筛选提出知识凝聚蒸馏,对知识进行分级来处理蒸馏知识冗余的问题
文献[160]知识结构皮尔森系数利用师生模型预测输出类别的相对关系来处理师生模型预测的差异问题
文献[112]知识结构批次结果重用提出小批量自蒸馏算法,利用上一批次的信息缓解难以并行化的问题
文献[155]训练过程特征路由提出点自适应位置蒸馏来处理师生模型蒸馏对齐位置不确定的问题
文献[188]模型架构自动机器学习提出小规模数据集ViT搜索方案来缓解师生模型能力差距带来的性能问题
文献[191]训练设置自动机器学习构造蒙特卡洛搜索空间探寻最佳设置来缓解手工设计蒸馏方案的压力
文献[189]模型架构自动机器学习通过衡量样本间相似性寻找最佳师生模型架构来解决师生模型能力差距带来的性能下降
文献[180]训练过程对抗性技术通过构造给定输入球约束内对抗样本集合挖掘决策边界来提升模型性能
文献[181]训练过程对抗性技术提出一种重加权策略的引导对抗对比蒸馏来帮助学生模型继承对抗鲁棒性
文献[182]训练过程对抗性技术设计了自适应对抗性蒸馏方案来缓解先前模型过度修正的问题
文献[184]训练过程对抗性技术利用多个实例之间的结构化关系来缓解教师模型不稳定预测的影响
文献[173]损失函数解耦通过对解耦后的输出归一化来处理师生模型输出不一致的问题
文献[175]损失函数解耦调整优化器动量值让蒸馏损失占据主导,打破不同损失的冲突博弈
文献[162]知识结构数据筛选通过困难样本挖掘和校准来缓解教师模型软标签置信度可靠性问题
文献[156]训练过程正则化约束提出混合权重和间歇融合寻找更平坦的最优解来提高模型泛化性
文献[161]知识结构困难样本挖掘通过筛选易错样本加强学习来缓解教师模型传播错误知识的风险
文献[168]温度系数对抗温度利用温度系数调整任务难度来更合理地为学生模型设置学习进程
文献[169]温度系数标准化Logits温度将Logits输出标准差作为温度系数来缓解样本知识传递不充分的问题
文献[163]知识结构正交投影层提出在Stiefel流形上构建正交投影层来提高知识传递效率
文献[170]温度系数师生模型差异温度提出通过平衡输出特征的平滑度来解决不同难度样本的知识转移问题
文献[171]温度系数标准化Logits温度提出对Logits输出进行标准化来解除师生模型输出在范围和方差上的耦合
文献[183]训练过程对抗性技术提出公平的鲁棒性蒸馏框架来解决鲁棒性的类别差异性
文献[194]训练过程扩散模型提出扩散模型去噪模块来解决蒸馏特征中的噪声问题
文献[190]模型架构自动机器学习通过遗传算法寻找最佳师生模型架构来缓解师生模型能力差距带来的性能问题
文献[174]损失函数解耦、序列化优化将真实标签和教师模型标签的优化任务序列化进行,缓解不同分布优化任务的冲突
文献[157]训练过程优化路径调整通过将学生模型与教师模型交叉蒸馏来缓解真实标签与教师模型错误分类之间的冲突问题
文献[165]知识结构错误标签掩码利用掩码屏蔽教师模型的错误分类来减少教师模型的错误输出对学生模型的负面影响
), ArticleFig(id=1325490763500494996, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, language=CN, label=表7, caption=

知识蒸馏的优化方法汇总

, figureFileSmall=null, figureFileBig=null, tableContent=
方法来源优化位置优化方法关键技术
文献[150]训练过程提前停止提出提前停止优化方法,缓解蒸馏损失后期对模型的损害
文献[176]损失函数对抗性技术提出利用分块训练损失来从教师模型中提取多样化知识,缓解知识单一的问题
文献[179]训练过程对抗性技术利用对抗攻击发现决策边界样本,传递准确的边界信息来提升性能
文献[151]训练过程正则化约束性能较弱的模型能够提供正则化约束给性能较强的模型来提升性能
文献[38]训练过程辅助模型引入一个助理模型来弥合师生模型的性能差距,缓解师生模型能力差距带来的性能下降
文献[158]知识结构特征筛选提出选择性转移策略,利用样本变换带来的暗知识以提高模型性能
文献[166]温度系数动态温度利用倒数第2层特征的${{L}_2}$范数作为温度系数来缓解标签噪声的影响
文献[177]训练过程对抗性技术设计任务驱动的注意力机制,构造紧凑的知识结构以减少信息丢失
文献[186]模型架构强化学习设计奖励函数寻找教师模型的最佳学生模型,缓解师生模型能力差距带来的性能问题
文献[187]模型架构自动机器学习提出一种基于预言机的搜索算法来缓解师生模型能力差距带来的性能问题
文献[185]模型架构自动机器学习提出一种分块的搜索评估方案以保证候选框架得到公平的训练
文献[167]温度系数动态温度通过退火函数生成动态温度来缓解师生模型能力差距过大带来的性能下降
文献[152]训练过程综合探究影响蒸馏性能的原因,优化不佳是主要原因
文献[178]损失函数对抗性技术提出自我破坏知识蒸馏来缓解蒸馏知识产权暴露的风险
文献[172]损失函数解耦提出将损失函数解耦来缓解经典蒸馏损失本身内部的冲突博弈
文献[153]训练过程非参数化对齐提出通过奇异值分解对齐师生模型来缓解投射层性能效率不佳的问题
文献[154]训练过程推理蒸馏分离通过保存教师模型推理的数据增强及输出来缓解蒸馏过程教师模型推理的开销
文献[159]知识结构特征筛选提出知识凝聚蒸馏,对知识进行分级来处理蒸馏知识冗余的问题
文献[160]知识结构皮尔森系数利用师生模型预测输出类别的相对关系来处理师生模型预测的差异问题
文献[112]知识结构批次结果重用提出小批量自蒸馏算法,利用上一批次的信息缓解难以并行化的问题
文献[155]训练过程特征路由提出点自适应位置蒸馏来处理师生模型蒸馏对齐位置不确定的问题
文献[188]模型架构自动机器学习提出小规模数据集ViT搜索方案来缓解师生模型能力差距带来的性能问题
文献[191]训练设置自动机器学习构造蒙特卡洛搜索空间探寻最佳设置来缓解手工设计蒸馏方案的压力
文献[189]模型架构自动机器学习通过衡量样本间相似性寻找最佳师生模型架构来解决师生模型能力差距带来的性能下降
文献[180]训练过程对抗性技术通过构造给定输入球约束内对抗样本集合挖掘决策边界来提升模型性能
文献[181]训练过程对抗性技术提出一种重加权策略的引导对抗对比蒸馏来帮助学生模型继承对抗鲁棒性
文献[182]训练过程对抗性技术设计了自适应对抗性蒸馏方案来缓解先前模型过度修正的问题
文献[184]训练过程对抗性技术利用多个实例之间的结构化关系来缓解教师模型不稳定预测的影响
文献[173]损失函数解耦通过对解耦后的输出归一化来处理师生模型输出不一致的问题
文献[175]损失函数解耦调整优化器动量值让蒸馏损失占据主导,打破不同损失的冲突博弈
文献[162]知识结构数据筛选通过困难样本挖掘和校准来缓解教师模型软标签置信度可靠性问题
文献[156]训练过程正则化约束提出混合权重和间歇融合寻找更平坦的最优解来提高模型泛化性
文献[161]知识结构困难样本挖掘通过筛选易错样本加强学习来缓解教师模型传播错误知识的风险
文献[168]温度系数对抗温度利用温度系数调整任务难度来更合理地为学生模型设置学习进程
文献[169]温度系数标准化Logits温度将Logits输出标准差作为温度系数来缓解样本知识传递不充分的问题
文献[163]知识结构正交投影层提出在Stiefel流形上构建正交投影层来提高知识传递效率
文献[170]温度系数师生模型差异温度提出通过平衡输出特征的平滑度来解决不同难度样本的知识转移问题
文献[171]温度系数标准化Logits温度提出对Logits输出进行标准化来解除师生模型输出在范围和方差上的耦合
文献[183]训练过程对抗性技术提出公平的鲁棒性蒸馏框架来解决鲁棒性的类别差异性
文献[194]训练过程扩散模型提出扩散模型去噪模块来解决蒸馏特征中的噪声问题
文献[190]模型架构自动机器学习通过遗传算法寻找最佳师生模型架构来缓解师生模型能力差距带来的性能问题
文献[174]损失函数解耦、序列化优化将真实标签和教师模型标签的优化任务序列化进行,缓解不同分布优化任务的冲突
文献[157]训练过程优化路径调整通过将学生模型与教师模型交叉蒸馏来缓解真实标签与教师模型错误分类之间的冲突问题
文献[165]知识结构错误标签掩码利用掩码屏蔽教师模型的错误分类来减少教师模型的错误输出对学生模型的负面影响
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202440694, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202440694, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202440694, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202440694, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1767196800000, fullTextJson=null, articleText=null, reference=null)
收藏切换
面向视觉算法的知识蒸馏研究综述
潘海为 , 于丰铭 , 张可佳 * , 兰海燕 , 孟庆宇 , 李哲
计算机研究与发展 | 人工智能 2026,63(1): 90-122
收起
收藏切换
计算机研究与发展 |人工智能 2026 , 63 (1) : 90 -122
面向视觉算法的知识蒸馏研究综述
全屏
[Author(id=1325490584328216600, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=panhaiwei@hrbeu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325490586010132506, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490584328216600, language=EN, stringName=Haiwei Pan, firstName=Haiwei, middleName=null, lastName=Pan, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001, bio={"content":"

born in 1975. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include remote sensing image processing, medical image processing, data mining, machine learning, and deep learning

"}, bioImg=null, bioContent=

born in 1975. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include remote sensing image processing, medical image processing, data mining, machine learning, and deep learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490588136644635, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490584328216600, language=CN, stringName=潘海为, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001, bio={"img":"RvLRdxKhIw4RYAitV5qsfw==","content":"

1975年生。博士,教授,博士生导师。CCF高级会员。主要研究方向为遥感图像处理、医学图像处理、数据挖掘、机器学习、深度学习

"}, bioImg=RvLRdxKhIw4RYAitV5qsfw==, bioContent=

1975年生。博士,教授,博士生导师。CCF高级会员。主要研究方向为遥感图像处理、医学图像处理、数据挖掘、机器学习、深度学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325490582310756372, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, xref=null, ext=[AuthorCompanyExt(id=1325490582361088021, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001), AuthorCompanyExt(id=1325490582587580438, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001)])]), Author(id=1325490589852114973, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325490593136255008, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490589852114973, language=EN, stringName=Fengming Yu, firstName=Fengming, middleName=null, lastName=Yu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001, bio={"content":"

born in 1998. PhD candidate. Student member of CCF. His main research interests include knowledge distillation, model compression, and machine learning

"}, bioImg=null, bioContent=

born in 1998. PhD candidate. Student member of CCF. His main research interests include knowledge distillation, model compression, and machine learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490594843336737, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490589852114973, language=CN, stringName=于丰铭, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001, bio={"img":"4cLSSbHqrsnl91CyY3GZyA==","content":"

1998年生。博士研究生。CCF学生会员。主要研究方向为知识蒸馏、模型压缩、机器学习

"}, bioImg=4cLSSbHqrsnl91CyY3GZyA==, bioContent=

1998年生。博士研究生。CCF学生会员。主要研究方向为知识蒸馏、模型压缩、机器学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325490582310756372, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, xref=null, ext=[AuthorCompanyExt(id=1325490582361088021, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001), AuthorCompanyExt(id=1325490582587580438, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001)])]), Author(id=1325490597276033062, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=kejiazhang@hrbeu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1325490598555295784, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490597276033062, language=EN, stringName=Kejia Zhang, firstName=Kejia, middleName=null, lastName=Zhang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001, bio={"content":"

born in 1981. PhD, associate professor, PhD supervisor. Member of CCF. His main research interests include machine learning, smart healthcare, and big data

"}, bioImg=null, bioContent=

born in 1981. PhD, associate professor, PhD supervisor. Member of CCF. His main research interests include machine learning, smart healthcare, and big data

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490599914250281, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490597276033062, language=CN, stringName=张可佳, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001, bio={"img":"2DN+pW3Sz4H4IJYowRSC0A==","content":"

1981年生。博士,副教授,博士生导师。CCF会员。主要研究方向为机器学习、智慧医疗、大数据

"}, bioImg=2DN+pW3Sz4H4IJYowRSC0A==, bioContent=

1981年生。博士,副教授,博士生导师。CCF会员。主要研究方向为机器学习、智慧医疗、大数据

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325490582310756372, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, xref=null, ext=[AuthorCompanyExt(id=1325490582361088021, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001), AuthorCompanyExt(id=1325490582587580438, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001)])]), Author(id=1325490602325975083, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325490605404594222, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490602325975083, language=EN, stringName=Haiyan Lan, firstName=Haiyan, middleName=null, lastName=Lan, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001, bio={"content":"

born in 1983. PhD, lecturer, master supervisor. Senior member of CCF. Her main research interests include signal processing and information security

"}, bioImg=null, bioContent=

born in 1983. PhD, lecturer, master supervisor. Senior member of CCF. Her main research interests include signal processing and information security

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490607765987377, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490602325975083, language=CN, stringName=兰海燕, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001, bio={"img":"LcbrCCD2Mr8BKhKqmKb0kw==","content":"

1983年生。博士,讲师,硕士生导师。CCF高级会员。主要研究方向为信号处理、信息安全

"}, bioImg=LcbrCCD2Mr8BKhKqmKb0kw==, bioContent=

1983年生。博士,讲师,硕士生导师。CCF高级会员。主要研究方向为信号处理、信息安全

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325490582310756372, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, xref=null, ext=[AuthorCompanyExt(id=1325490582361088021, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001), AuthorCompanyExt(id=1325490582587580438, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001)])]), Author(id=1325490611121430579, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325490612409081910, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490611121430579, language=EN, stringName=Qingyu Meng, firstName=Qingyu, middleName=null, lastName=Meng, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001, bio={"content":"

born in 1998. Master. His main research interests include model compression, natural language processing, and machine learning

"}, bioImg=null, bioContent=

born in 1998. Master. His main research interests include model compression, natural language processing, and machine learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490614904692792, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490611121430579, language=CN, stringName=孟庆宇, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001, bio={"img":"lbXEooh6sVk23lCVpi7YtQ==","content":"

1998年生。硕士。主要研究方向为模型压缩、自然语言处理、机器学习

"}, bioImg=lbXEooh6sVk23lCVpi7YtQ==, bioContent=

1998年生。硕士。主要研究方向为模型压缩、自然语言处理、机器学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325490582310756372, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, xref=null, ext=[AuthorCompanyExt(id=1325490582361088021, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001), AuthorCompanyExt(id=1325490582587580438, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001)])]), Author(id=1325490616150401082, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, orderNo=5, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325490617878454333, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490616150401082, language=EN, stringName=Zhe Li, firstName=Zhe, middleName=null, lastName=Li, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001, bio={"content":"

born in 1999. PhD candidate. Student member of CCF. His main research interests include computer vision, image generation, and multimodal image processing

"}, bioImg=null, bioContent=

born in 1999. PhD candidate. Student member of CCF. His main research interests include computer vision, image generation, and multimodal image processing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490619988189246, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, authorId=1325490616150401082, language=CN, stringName=李哲, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001, bio={"img":"2EItHUXjrCrylIjyNkGmfQ==","content":"

1999年生。博士研究生。CCF学生会员。主要研究方向为计算机视觉、图像生成、多模态图像处理

"}, bioImg=2EItHUXjrCrylIjyNkGmfQ==, bioContent=

1999年生。博士研究生。CCF学生会员。主要研究方向为计算机视觉、图像生成、多模态图像处理

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325490582310756372, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, xref=null, ext=[AuthorCompanyExt(id=1325490582361088021, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Computer Science and Technology, Harbin Engineering University, Harbin 150001), AuthorCompanyExt(id=1325490582587580438, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490414429545387, companyId=1325490582310756372, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001)])])]
潘海为 , 于丰铭, 张可佳* , 兰海燕, 孟庆宇, 李哲
作者信息
  • 哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001
通讯作者:
作者简介:

1975年生。博士,教授,博士生导师。CCF高级会员。主要研究方向为遥感图像处理、医学图像处理、数据挖掘、机器学习、深度学习

1998年生。博士研究生。CCF学生会员。主要研究方向为知识蒸馏、模型压缩、机器学习

1981年生。博士,副教授,博士生导师。CCF会员。主要研究方向为机器学习、智慧医疗、大数据

1983年生。博士,讲师,硕士生导师。CCF高级会员。主要研究方向为信号处理、信息安全

1998年生。硕士。主要研究方向为模型压缩、自然语言处理、机器学习

1999年生。博士研究生。CCF学生会员。主要研究方向为计算机视觉、图像生成、多模态图像处理

Knowledge Distillation in Visual Algorithms: A Survey
Haiwei Pan , Fengming Yu, Kejia Zhang* , Haiyan Lan, Qingyu Meng, Zhe Li
Affiliations
  • College of Computer Science and Technology, Harbin Engineering University, Harbin 150001
  • born in 1975. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include remote sensing image processing, medical image processing, data mining, machine learning, and deep learning

    born in 1998. PhD candidate. Student member of CCF. His main research interests include knowledge distillation, model compression, and machine learning

    born in 1981. PhD, associate professor, PhD supervisor. Member of CCF. His main research interests include machine learning, smart healthcare, and big data

    born in 1983. PhD, lecturer, master supervisor. Senior member of CCF. Her main research interests include signal processing and information security

    born in 1998. Master. His main research interests include model compression, natural language processing, and machine learning

    born in 1999. PhD candidate. Student member of CCF. His main research interests include computer vision, image generation, and multimodal image processing

出版时间: 2026-01-01 doi: 10.7544/issn1000-1239.202440694
文章导航
收藏切换

知识蒸馏作为深度学习中的关键技术,通过将大型教师模型的知识传递给较小的学生模型,实现了模型的压缩与加速。在保证性能的前提下,显著减少了计算资源和存储需求,促进了高性能模型在资源受限的边缘设备上的部署。围绕知识蒸馏的最新研究进展进行了系统性的综述,从知识类型和师生模型架构2个角度对知识蒸馏进行分类,详细汇总了输出特征知识、中间特征知识、关系特征知识3种典型知识类型的蒸馏方法,以及卷积架构到卷积架构、卷积架构到ViT(vision Transformer)架构、ViT架构到卷积架构和ViT架构到ViT架构的蒸馏方法;探讨了离线蒸馏、在线蒸馏、自蒸馏、无数据蒸馏、多教师蒸馏和助理蒸馏的学习方式;归纳了基于蒸馏过程、知识结构、温度系数及损失函数的蒸馏优化方法,分析了对抗性技术、自动机器学习、强化学习和扩散模型对蒸馏的改进,并总结了蒸馏技术在常见应用中的实现。尽管知识蒸馏取得了显著进展,但在实际应用和理论研究中仍面临诸多挑战。最后,对这些问题进行了深入分析,并对未来发展方向提出了见解。

知识蒸馏  /  模型压缩  /  深度学习  /  卷积神经网络  /  视觉Transformer

Knowledge distillation, as a key technique in deep learning, achieves model compression and acceleration by transferring knowledge from a large teacher model to a smaller student model. Under the premise of maintaining performance, this technology significantly reduces the requirements of computational resources and storage, and facilitates the deployment of high-performance models on resource-constrained edge devices. Firstly, we provide a systematic review of the recent research in knowledge distillation and categorize it from two perspectives: the type of knowledge and teacher-student model architectures. We comprehensively summarize the distillation methods based on three typical types of knowledge: output feature knowledge, intermediate feature knowledge, and relational feature knowledge, as well as distillation methods based on CNN to CNN architecture, CNN to ViT (vision Transformer) architecture, ViT to CNN architecture, and ViT to ViT architecture. Next, we explore various learning paradigms such as offline distillation, online distillation, self-distillation, data-free distillation, multi-teacher distillation, and assistant distillation. Then, we summarize distillation optimization methods based on the distillation process, knowledge structure, temperature coefficient, and loss functions and analyze improvements in distillation brought by adversarial techniques, automated machine learning, reinforcement learning, and diffusion models, and conclude with the implementation of distillation technology in common applications. Despite significant advancements in knowledge distillation, numerous challenges remain in both practical applications and theoretical research. Finally, we provide an in-depth analysis of these issues and offer insights into future development directions.

knowledge distillation  /  model compression  /  deep learning  /  convolutional neural network  /  vision Transformer
潘海为, 于丰铭, 张可佳, 兰海燕, 孟庆宇, 李哲. 面向视觉算法的知识蒸馏研究综述. 计算机研究与发展, 2026 , 63 (1) : 90 -122 . DOI: 10.7544/issn1000-1239.202440694
Haiwei Pan, Fengming Yu, Kejia Zhang, Haiyan Lan, Qingyu Meng, Zhe Li. Knowledge Distillation in Visual Algorithms: A Survey[J]. Journal of Computer Research and Development, 2026 , 63 (1) : 90 -122 . DOI: 10.7544/issn1000-1239.202440694
近年来,随着深度学习技术的迅猛发展,神经网络模型在计算机视觉、自然语言处理等诸多领域取得了显著的突破。1998年,LeNet-5[1]作为最早的卷积神经网络(convolutional neural network,CNN)之一,奠定了现代CNN的基础框架。2012年,AlexNet[2]通过学习到的特征超越手工设计特征,掀起深度学习热潮,并推动了如VGGNet[3]和Inception系列[4-6]网络的出现。随着网络深度增加,梯度消失问题成为瓶颈,ResNet[7]和DenseNet[8]提出残差连接与密集连接,有效解决该问题并提升网络深度上限。与此同时,深度网络带来的高计算开销限制了其在资源受限环境中的部署,促使MobileNet系列[9-11]和EfficientNet系列[12-13]网络聚焦于轻量化和高效性设计。
随着Transformer[14]在自然语言处理(natural language processing,NLP)领域的成功,Dosovitskiy等人[15]提出ViT(vision Transformer),将其架构引入视觉任务,引领视觉Transformer的研究。随后,Swin Transformer[16]通过滑动窗口机制实现高分辨率图像的高效处理;DeiT[17]通过蒸馏技术降低了ViT对大规模数据的依赖,PVT[18]结合金字塔结构提升多尺度建模能力。此外,CLIP[19]和DALL-E[20]等模型结合视觉与语言,将ViT应用于多模态任务中,展现了强大的跨模态理解与生成能力。
然而,这些性能卓越的模型往往具有庞大的参数规模和计算复杂度,导致在资源受限的环境中难以部署。为了解决这一问题,模型压缩技术应运而生。模型压缩方法主要包括权重剪枝、量化、低秩分解以及知识蒸馏等。其中知识蒸馏作为一种有效的模型压缩方法,受到了广泛关注,目前已经成为深度学习领域的一大研究热点。
知识蒸馏最早由Hinton等人[21]提出,其基本思想是通过训练一个较小的学生模型,来逼近一个较大且性能较好的教师模型。具体而言,教师模型通常预先训练完成,并在训练学生模型时提供额外的监督信号,帮助学生模型更好地学习。这样的机制不仅可以显著减少模型的参数数量和计算开销,还能够在保持精度的情况下提升模型的推理速度。
知识蒸馏方法发展至今,已经衍生出多种变体和应用。从最初的经典方法到对教师模型输出分布的探索,再到多教师模型和多任务学习的结合,研究者们在这一领域不断创新,提出了多种改进策略。例如,基于中间特征层的蒸馏方法通过教师模型的中间层特征来指导学生模型的训练;自蒸馏方法则利用同一模型在不同训练阶段的信息进行蒸馏。此外,还有结合对抗训练和强化学习等新兴技术的知识蒸馏方法,这些策略不仅在理论上深化了我们对知识蒸馏的理解,也在实践中推动了模型压缩技术的广泛应用。
知识蒸馏的应用领域也非常广泛。除了在计算机视觉和自然语言处理等传统深度学习领域的应用之外,知识蒸馏还在自动驾驶、医疗影像分析、语音识别等领域展现出了强大的潜力。这些应用不仅验证了知识蒸馏的有效性,也为未来的研究提供了丰富的实践案例。
本综述旨在系统地梳理和总结知识蒸馏在视觉领域的研究进展。首先,我们将回顾知识蒸馏的基本原理和经典方法;然后从知识蒸馏的分类、学习方式、优化方法以及应用的角度介绍近年来出现的各种改进和扩展;最后我们将讨论当前研究中存在的挑战和未来的发展方向,为该领域的进一步研究提供参考。
先前2项比较全面的外文综述[22-23]和3项不错的中文综述[24-26]已经对知识蒸馏有了一个较为系统的介绍,但它们都忽略了对于知识蒸馏学习范式优化的梳理以及关于视觉Transformer架构模型蒸馏的总结,通过这篇综述希望能够为读者提供一个全面而清晰的知识蒸馏研究脉络,帮助相关领域的研究人员和工程师更好地理解和应用这一重要的模型压缩技术。
本文的章节将从知识蒸馏的背景、知识蒸馏的分类、知识蒸馏的学习方式、知识蒸馏的优化和知识蒸馏的应用详细介绍,如图1所示。
随着深度学习技术的快速发展,模型的规模和复杂度不断攀升。从早期的简单卷积网络(如LeNet-5[1])到近年来的深层宽层网络,再到以Transformer为基础的大型预训练模型,这些创新架构在训练时通常能够取得卓越的性能,推动了计算机视觉、自然语言处理等领域的飞跃。然而,这些高性能模型往往面临着庞大的参数量和计算需求,这使得它们在计算资源和存储空间受限的环境中部署变得极为困难,亟需解决效率和资源消耗的问题。
为了解决这一挑战,研究者提出了多种模型压缩技术,旨在减少大规模模型对资源的占用及推理过程中的计算消耗。常见的压缩策略包括权重剪枝、量化、低秩分解以及知识蒸馏等,其中知识蒸馏因其在保持模型性能的同时显著降低计算负担而受到广泛关注,成为近年来深度学习研究的重点之一。
相比于其他模型压缩方法,知识蒸馏具有独特的优势。权重剪枝通过移除冗余权重来减少模型参数,但往往需要精细的剪枝策略以平衡性能与压缩率;量化通过将权重和激活值映射到较低的比特表示以降低存储和计算成本,但可能会带来一定的精度损失;低秩分解通过将权重矩阵分解为低秩矩阵来减少计算复杂度,但实现过程较为复杂;知识蒸馏则通过让学生模型模仿教师模型的行为,直接优化学生模型的输出,使其在较小规模下仍能保持高性能,并且知识蒸馏不仅能够有效地压缩模型,还能保持甚至提高学生模型的泛化能力。知识蒸馏在许多实际应用中表现出色,成为模型压缩领域的重要方法。
数据集输入表示为$ \{ {\boldsymbol x}_i\} _{i = 1}^N = \{ {{\boldsymbol x}_1},{{\boldsymbol x}_2},…,{{\boldsymbol x}_N}\} $,对应的标签集合表示为$ \{ {\boldsymbol y}_i\} _{i = 1}^N = \{ {{\boldsymbol y}_1},{{\boldsymbol y}_2},…,{{\boldsymbol y}_N}\} $。针对2种不同的网络架构分别给出形式化描述:
1)卷积架构。对于具有$l$层的卷积网络,如图2所示,$f( \cdot )$表示整个网络架构,${\boldsymbol F}$表示中间层特征。
$ \{ {\boldsymbol F}_i\} _{i = 1}^l = \{ {{\boldsymbol F}_1},{{\boldsymbol F}_2},…,{{\boldsymbol F}_l}\} \;。$
${\boldsymbol z} = f({\boldsymbol x})$表示Logits输出,Logits输出经过softmax层激活后为概率分布输出,表示为$ {\boldsymbol p} = softma{x_\tau }({\boldsymbol z}) $,其中:
$ softma{x_\tau }({\boldsymbol z}) = \frac{{\exp ({{\boldsymbol z} / \tau })}}{{\displaystyle\sum_j {\exp ({{{{\boldsymbol z}_j}} / \tau })} }} \text{,} $
$\tau $表示温度系数。
对于教师模型,具有${l_{\rm T}}$层的网络表示为${f^{\rm T}}( \cdot )$,$ \{ {{\boldsymbol F}^{\rm T}_i}\} _{i = 1}^{{l_{\rm T}}} = \{ {\boldsymbol F}_1^{\rm T},{\boldsymbol F}_2^{\rm T},…,{\boldsymbol F}_{{l_{\rm T}}}^{\rm T}\} $表示教师模型的特征层,${{\boldsymbol z}^{\rm T}} = {f^{\rm T}}({\boldsymbol x})$,${{\boldsymbol p}^{\rm T}} = softma{x_\tau }({{\boldsymbol z}^{\rm T}})$分别表示教师模型的Logits输出和概率分布输出。对于学生模型同理。
2)视觉Transformer架构。对于具有$L$个Transformer块的ViT[15],如图3所示。
输入图像$ {\boldsymbol x} \in {\mathbb{R}^{H \times W \times C}} $被切分为$N$个图像块并拉伸为1维向量$ \{ {\boldsymbol x}_{\rm p}^j \in {\mathbb{R}^{{P^2}C}}|j \in N,1 \leqslant j \leqslant N\} $,$P$表示图像块大小,$H$和$W$分别表示图像的高和宽,$C$表示图像通道数,$N = {{HW} / {{P^2}}}$,每一个图像块通过$D$维向量模型编码:
$ {{\boldsymbol e}_j} = {\boldsymbol x}_{\rm p}^j \cdot {\boldsymbol E},\;\;{\boldsymbol E} \in {\mathbb{R}^{({P^2}C) \times D}} \;。$
编码后的图像块插入一个可学习的分类token,并对整个序列添加位置编码:
$ Input = {{\boldsymbol v}_0} = [{{\boldsymbol x}_{{{\mathrm{cls}}}}};{{\boldsymbol e}_1},{{\boldsymbol e}_2},…,{{\boldsymbol e}_N}] + {{\boldsymbol E}_{{\rm pos}}},\;\; {{\boldsymbol E}_{{\rm pos}}} \in {\mathbb{R}^{(N + 1) \times D}} \;。$
将编码后的序列输入到Transformer块中,Transformer块由多头自注意力(mutil-head attention,MHA)层、多层感知器(multilayer perceptron,MLP)层和层归一化(layer normalization,LN)层组成,每一个Transformer块使用上一个块的输出${{\boldsymbol v}_{l - 1}}$作为输入,得到当前块的特征:
$ {{\boldsymbol v}_l} = T ransformerBlock({{\boldsymbol v}_{l - 1}}),\;\;l \in \{1,2,…,L\} \;。$
最后一个块的输出${{\boldsymbol v}_L}$称之为输出特征,${\boldsymbol q}$表示由分类token${\boldsymbol v}_L^0$计算得到的类别特征。块内特征可以表示为
$ \begin{aligned}& {{\boldsymbol v}'_l = MHA(LN({{\boldsymbol v}_{l - 1}})) + {{\boldsymbol v}_{l - 1}},}\;\;{l = 1,2,…,L,} \\ & {{{\boldsymbol v}_l} = LN(MLP({\boldsymbol v}'_l )) + {\boldsymbol v}'_l ,}\;\;{l = 1,2,…,L,} \\ & {{\boldsymbol q} = LN({\boldsymbol v}_L^0)\;。}{} \end{aligned} $
对于具有${L_{\rm T}}$个Transformer块的教师模型,$ \{ {{\boldsymbol v}^{\rm T}}\} _{l = 1}^{{L_{\rm T}}} = $$\{ {\boldsymbol v}_1^{\rm T},{\boldsymbol v}_2^{\rm T},…,{\boldsymbol v}_{{L_{\rm T}}}^{\rm T}\} $表示Transformer块的特征,${\boldsymbol v}_{{L_{\rm T}}}^{\rm T}$和$ {{\boldsymbol q}^{\rm T}} $分别表示教师模型的输出特征和类别特征。对于学生模型同理。
针对下文涉及的距离度量,表1给出其符号缩写、英文全称和中文名称。
蒸馏这一概念在2015年首次由Hinton等人[21]提出,核心思想是通过训练一个较小的学生模型来逼近一个较大且性能较好的教师模型。这一类似的思想最早由Bucila等人[28]在2006年提出,他们的工作令人信服地证明了较大的模型获得的知识可以转移到较小的模型中。
知识蒸馏系统通常由3个关键部分组成:知识、蒸馏算法和师生模型架构。图4展示了一个知识蒸馏的基本框架。
Hinton等人[21]提出通过最小化教师模型产生的概率分布输出(${{\boldsymbol p}^{\rm T}}$)和学生模型产生的概率分布输出(${{\boldsymbol p}^{\rm S}}$)之间的差异来进行知识的转移,然而训练有素的教师模型在softmax函数上产生的概率分布输出具有较高的真实类置信度,而其他类的概率都接近于0。因此引入了softmax函数的温度概念来软化标签,称之为温度系数,软化后的标签称为软标签,表示为
$ {{\boldsymbol p}^{\rm T}} = \frac{{\exp ({{{{\boldsymbol z}^{\rm T}}} / \tau })}}{{\displaystyle\sum_j {\exp ({{{\boldsymbol z}_j^{\rm T}} / \tau })} }},\;\;{{\boldsymbol p}^{\rm S}} = \frac{{\exp ({{{{\boldsymbol z}^{\rm S}}} / \tau })}}{{\displaystyle\sum_j {\exp ({{{\boldsymbol z}_j^{\rm S}} / \tau })} }} \text{,} $
其中$\tau $表示温度系数,随着$\tau $的不断增大,概率分布输出分布变得越来越平稳,最终趋于均匀分布,当$\tau = 1$时,退化为普通的softmax函数。更平稳的概率分布输出分布能更好地表达教师模型发现的与真实类相近的类别信息,这部分知识被称为暗知识,众多学者认为暗知识是使知识蒸馏有效的因素之一。
除了教师模型软标签提供的暗知识,数据的真实标签也能提供有效的信息,一个典型的蒸馏损失可以形式化表述为
$ \begin{split}{L_{{\rm KD}}} =& \alpha H({{\boldsymbol p}^{\rm S}};{\boldsymbol y}) + \beta {D_{{\rm KL}}}({{\boldsymbol p}^{\rm T}}\| {{\boldsymbol p}^{\rm S}})= \\ & - \alpha \displaystyle\sum_i {{{\boldsymbol y}_i}\log ({\boldsymbol p}_i^{\rm S})} + \beta \displaystyle\sum_i {{\boldsymbol p}_i^{\rm T}\log \frac{{{\boldsymbol p}_i^{\rm T}}}{{{\boldsymbol q}_i^{\rm S}}}} \end{split} \;。$
知识蒸馏在深度学习领域有多种应用,主要通过将大型、复杂的教师模型的知识传递给较小、较简单的学生模型,达到在推理阶段加速并减少存储需求的目的;其次知识蒸馏还有进行知识迁移和隐私保护等应用,有以下3个方面:
1)模型压缩。知识蒸馏最主要的用途是模型压缩。通过将大型教师模型的知识传递给较小的学生模型,能够在保持模型性能的同时,显著减少模型的参数量和计算复杂度,使其适用于资源受限的环境。
2)知识迁移。知识蒸馏可以利用其训练时传递知识的特性进行跨模态或跨架构知识的迁移,这进一步扩大了蒸馏的灵活性和适用性,能够使学生模型获得更多样化和丰富的知识,从而提升其在不同任务中的性能。
3)隐私保护。知识蒸馏可以在不直接暴露原始数据的情况下传递知识,这在涉及隐私数据的应用中(如医疗数据、金融数据等)具有重要意义。通过知识蒸馏,敏感数据可以用于训练教师模型,而学生模型则可以在不接触这些敏感数据的情况下获取知识。
本文从知识类型和师生模型架构2个角度对知识蒸馏方法进行了分类,如图1中所示的知识蒸馏的分类。知识类型的分类遵循先前的工作[22-26],通常从输出特征、中间特征和关系特征3个层面进行汇总。近年来,随着Transformer模型的出现,师生模型的架构不再局限于卷积模型。因此,本文还从师生模型架构的角度对蒸馏方法进行了分类和汇总。
本节从学生模型获取知识的来源对知识蒸馏方法进行分类,通常包括基于输出特征的蒸馏、基于中间特征的蒸馏和基于关系特征的蒸馏。这些方法分别利用教师模型的输出特征、中间层特征,以及样本间或特征图间的关系作为知识来指导学生模型训练。
基于输出特征的蒸馏通常是指利用教师模型最后一层神经元的输出指导学生模型训练,其主要思想是让学生模型来模仿训练有素的教师模型的输出来获得高可信的结果。基于输出特征的知识蒸馏简单有效,有着广泛的应用。根据输出特征知识的位置,将经过softmax层激活的知识称之为概率分布输出;将未经过softmax层激活的知识称之为Logits输出。
1)概率分布输出。概率分布输出指神经网络经过softmax层之后的输出,如图5所示,基于概率分布输出的蒸馏损失可以描述为
$ {L_{{\rm KD}}} = {L_{{\rm Dis}}}({{\boldsymbol p}^{\rm T}},{{\boldsymbol p}^{\rm S}}) \text{,} $
其中$ {L_{{\rm Dis}}} $表示距离度量函数。
2015年,Hinton等人[21]提出了利用软目标中蕴含的来自教师模型的暗知识来指导学生模型学习,并利用温度系数来加强概率较低的标签概率对模型的影响。Sau等人[29]对教师模型输出的知识添加噪声来模拟同一领域的不同教师模型,证明了添加噪声相当于正则化项的作用,实现了更好的深度模型的压缩。Zhang等人[30]提出了互学习的概念,让多个模型之间互相学习其概率分布输出,证明了具有高熵的概率模型能训练出更鲁棒的结构。Furlanello等人[31]提出了序列化建模方案,令前一个模型指导后一个模型的训练,最终将所有学生模型集成得到最终的学生模型。Yang等人[32]认为视觉上相似的物体同样包含着有效的信息,引入了类别相似性知识,计算主类与前k个类别的置信度差异,加强次要类别的置信度来得到一个更宽容的教师模型。Lee等人[33]认为线性可分的任务经过某些数据增强手段之后可能不再线性可分,提出学习一个原始标签与增强标签的联合分布任务来进行蒸馏学习。Yang等人[34]专注于教师模型分类器的学习,在倒数第2层匹配师生模型特征后通过教师模型的softmax函数计算概率分布输出,计算交叉熵来优化整体的蒸馏损失。
2)Logits输出。Logits输出指神经网络最后1层和softmax层之前的输出,如图6所示,基于Logits输出的蒸馏损失可以描述为
$ {L_{{\rm KD}}} = {L_{{\rm Dis}}}({{\boldsymbol z}^{\rm T}},{{\boldsymbol z}^{\rm S}}) \text{,} $
其中$ {L_{{\rm Dis}}} $表示距离度量函数。
在2014年,Ba等人[35]提出了与Hinton等人[21]类似的观点,他们认为模型压缩的成功表明了较小的网络也能够达到与复杂网络相似的精度,使用简单的网络模仿已经训练好的复杂网络的Logits输出,并在隐藏层的学习中加入了矩阵分解来加快模型的学习。Zhou等人[36]实验发现对Logits输出进行蒸馏要优于对概率分布输出进行蒸馏,提出一个共享低级别参数的师生模型架构,并通过梯度阻断模块来避免学生网络的梯度对教师模型的负优化。Son等人[37]针对Mirzadeh等人[38]提出的助理模型蒸馏来弥合师生模型能力差距的方案,助理模型会对错误的预测进行累计,造成性能的下降,Son等人借鉴DenseNet[8]的思想将上级教师模型的输出知识始终作为下级的指导信息,并借鉴Dropout的思想提出随机丢弃来缓解过拟合。Li等人[39]认为师生模型之间的差距可以作为知识来指导学生模型训练,利用神经架构搜索(neural architecture search,NAS)得到残差学生模型架构,提出能量熵来设定停止条件,在推理阶段提出自适应推理,对不同难度的样本使用不同数量的残差学生模型来加快简单样本的推理速度。
基于输出特征的蒸馏方法汇总于表2。
基于中间特征的蒸馏通常指利用神经网络中间层特征图中的知识指导学生模型训练,如图7所示,通过设计特定的投射层来对齐师生模型特征图的语义信息,基于中间特征的蒸馏损失可以描述为
$ {L_{{\rm KD}}} = {L_{{\rm Dis}}}({{\boldsymbol F}^{\rm T}},r({{\boldsymbol F}^{\rm S}})) \text{,} $
其中${{\boldsymbol F}^{\rm T}}$,${{\boldsymbol F}^{\rm S}}$分别表示教师模型和学生模型的中间层特征图,$r( \cdot )$表示投射层。
Romero等人[40]在2015年提出了迁移中间层的知识,即训练一个比教师模型更深且更细的网络对模型进行压缩,选择教师模型的隐藏层作为提示层,学生模型的某个隐藏层作为引导层,引导层从提示层进行学习。
如何选择中间层特征传递的知识是基于中间特征蒸馏的一个焦点,Zagoruyko等人[41]提出了使用基于组的注意力图转移特征知识,给出了基于绝对值和、绝对值的p次幂的和以及绝对值的p次幂的最大值3种计算注意力图的方式。Lin等人[42]针对特征图语义不匹配的问题提出目标感知对齐模块,利用教师模型特征来对学生模型语义特征进行重建,并且针对大特征图提出分组蒸馏和锚点蒸馏来提取局部特征和全局特征。Lao等人[43]认为基于中间特征的蒸馏方法的重要知识很容易被简单知识所压制,提出使用掩码图像机制来传递更有效的知识,强制学生模型学习教师模型的输出来恢复缺失特征。Guo等人[44]给出了更具解释性的注意力图蒸馏方案,调整类注意力图的生成位置,提出根据类别生成注意力图,给出了更具解释性的可视化知识转移结果。Zhang等人[45]将频率域的知识引入蒸馏学习中,通过Prompt方法[46]和离散小波变换来微调教师模型以引入频率域知识,并利用连续2层之间的相关性矩阵作为频率损失的权重。
教师模型的激活边界被认为是训练学生模型的关键信息,Huang等人[47]通过最小化师生模型特征图分布之间的最大均值差异来匹配神经元的激活分布,并且在计算过程中提出使用核函数进行映射来辅助计算。Heo等人[48]给出了线性层和卷积层的激活信息的传递方案,并给出一个替代函数将离散的激活边界近似为连续函数进行优化。
Shen等人[49]认为重用预训练的模型能够大大降低训练的成本,从多个训练有素的教师模型中提取并合并知识,获得一个能够综合处理分类的轻量级学生模型。Wei等人[50]将量化方法与蒸馏方法结合,对学生模型特征图维度对齐后对师生模型进行均匀量化,对量化后的特征图进行蒸馏。Ahn等人[51]从信息论的角度对知识蒸馏进行了解释,将知识的转移表述为师生模型之间的互信息最大化,通过最大化变分下界来近似分布帮助互信息的计算。Heo等人[52]对先前具有代表性的6项工作(FitNets[40],AT[41],FSP[53],Jacobian[54],FT[55],AB[48])进行对比总结,对ReLU进行修改,提出margin-ReLU来保存教师模型信息,并提出一种新的距离度量用于师生模型的特征蒸馏。Yang等人[56]认为CRD[57]基于对比学习的蒸馏方案会损害模型对原任务的表征能力,将联合标签增强的自蒸馏方案[33]迁移到传统蒸馏中。教师模型的强大性能不仅仅归功于骨干网络的特征提取能力,最后的判别器也同等重要。Yang等人[34]将师生模型输出均输入到教师模型分类器中来计算两者的交叉熵,并引入一项简化的FitNets[40]提示损失来进行师生模型中间层的匹配。Chen等人[58]通过一个简单的投射层对齐师生模型的倒数第2层特征,直接使用教师模型的分类器进行预测输出。
基于中间特征的蒸馏方法汇总如表3所示。
基于关系特征的知识通常指不同层之间或不同输入样本特征之间的关系知识,如图8所示,基于关系的蒸馏可以描述为
$ {L_{{\text{L-KD}}}} = {L_{{\rm Dis}}}\left({\psi _{\rm T}}({\boldsymbol F}_i^{\rm T},{\boldsymbol F}_j^{\rm T}),{\psi _{\rm S}}({\boldsymbol F}_i^{\rm S},{\boldsymbol F}_j^{\rm S})\right) \text{,} $
$ {L_{{\text{I-KD}}}} = {L_{{\rm Dis}}}\left({\varphi _{\rm T}}({\boldsymbol z}_i^{\rm T},{\boldsymbol z}_j^{\rm T}),{\varphi _{\rm S}}({\boldsymbol z}_i^{\rm S},{\boldsymbol z}_j^{\rm S})\right) \text{,} $
其中$ {L_{{\text{L-KD}}}} $,$ {L_{{\text{I-KD}}}} $分别表示不同层之间的关系和不同样本特征之间的关系,$\psi ( \cdot )$,$\varphi ( \cdot )$分别表示层间关系抽取模块和样本特征关系抽取模块,${{\boldsymbol F}^{\rm T}}$,${{\boldsymbol F}^{\rm S}}$表示师生模型的特征图,$ {{\boldsymbol z}^{\rm T}} $,${{\boldsymbol z}^{\rm S}}$表示师生模型样本特征。
对于层间关系的研究,Yim等人[53]将教师模型处理问题的流程抽象为求解程序的流程(flow of solution procedure,FSP)矩阵,学生模型通过学习教师模型的FSP矩阵来获得解决问题的流程。You等人[61]认为不同样本的中间层特征之间的相对差异可以作为教师模型对学生模型更明确更适合的指导知识。Zhang等人[62]使用每个教师模型的Logits输出特征和中间特征作为节点来构成2个图,通过图结构对不同教师模型的重要性和关系进行建模。直接对师生模型特征层进行对齐存在语义不匹配的问题,Chen等人[63]提出使用相似度矩阵和注意力机制来进行中间层的关联,从而获得语义对齐的知识来进行蒸馏学习。Liu等人[64]同样计算特征图通道之间的相似性矩阵用于蒸馏学习,并针对大特征图给出了分组计算聚合方案。Jin等人[65]针对概率分布输出蒸馏,提出多层次对齐方案,分别从样本实例角度、批次角度和类别角度对师生模型的输出特征进行对齐。
对于样本间关系的研究,Chen等人[66]提出了样本交叉相似性,利用${L_2}$范数给出一个样本相似性的打分函数,根据相似性分数来选择不同的蒸馏策略。Liu等人[67]引入了图结构来传递样本间的实例关系,将实例特征和实例关系分别作为关系图的点和边,有效地捕获了整个网络中的知识。Peng等人[68]不仅考虑了样本内容的一致性,还考虑了样本关系的一致性,引入核方法来捕获样本之间的相似性。Tung等人[69]认为相似的样本会引发神经网络中相似的激活模式,提出利用样本相似性向学生模型传递教师模型的激活模式。Tian等人[57]引入了对比损失来表达样本之间的依赖关系,通过最大化教师模型和学生模型特征之间的互信息下界来优化学生模型。Xu等人[70]引入了词袋模型来聚合相似的数据,通过样本内蒸馏损失和样本间蒸馏损失来最小化师生模型距离并传递词袋知识。
基于关系特征的蒸馏方法汇总如表4所示。
本节介绍了3种知识类型。基于输出特征的蒸馏的优势在于结构简单、易于学习且不受模型架构的限制,但这种简单的结构难以为学生模型提供充足的知识。如何从输出中挖掘更有价值的信息是未来研究的一大趋势。一类方法通过调整温度系数来为学生模型带来更多的知识,这将在4.3节中详细介绍。基于中间特征的蒸馏相比于基于输出特征的蒸馏,知识类型更加丰富,能从中间层提取更复杂的知识传递给学生模型。然而,对于架构不同的师生模型,从中间层提取的知识可能存在维度和语义不匹配的问题。现有方法大多通过引入额外的投射层来进行语义对齐,但这会带来额外的计算参数。如何在不引入额外开销的情况下完美对齐师生模型的语义维度特征,是一个值得深入研究的问题。此外,从中间层提取知识的位置以及学生模型接收知识的位置也值得进一步研究。基于关系特征的蒸馏通过特征图或样本特征之间的关系向学生模型传递知识。现有方法大多通过样本之间或层之间的相似性来提取关系知识,然而,如何获得更复杂且包含更多语义知识的关系知识来帮助学生模型学习,仍然是一个具有挑战性的问题。图结构在基于关系特征的蒸馏中是一种常用的数据结构,刘静等人[71]的一项关于图知识的蒸馏综述在这一部分具有一定参考价值。
随着Transformer架构模型的出现,蒸馏方法不再局限于卷积架构模型。本节从教师模型和学生模型的架构角度进行分类,CNN和ViT[15]是2种主流的视觉网络框架,它们分别应用于教师模型和学生模型,共有4种组合:CNN2CNN,CNN2ViT,ViT2CNN,ViT2ViT。具体而言,这4种组合分别表示:卷积架构的教师模型向卷积架构的学生模型进行蒸馏、卷积架构的教师模型向ViT架构的学生模型进行蒸馏、ViT架构的教师模型向卷积架构的学生模型进行蒸馏,以及ViT架构的教师模型向ViT架构的学生模型进行蒸馏。
CNN2CNN是指卷积架构的教师模型向卷积架构的学生模型进行蒸馏,通常为一个较深且性能较好的卷积网络教师模型来指导一个较浅且较为轻量的卷积网络学生模型训练,卷积网络经过近年来的爆炸式发展,已经广泛应用于各个领域并取得显著的成果,因此在知识蒸馏中卷积架构到卷积架构蒸馏的研究占到了绝大多数,2.1节总结的基于输出特征的蒸馏[21,32,36]、基于中间特征的蒸馏[40-41,48]以及基于关系特征的蒸馏[53,69,72]大多数均以卷积架构师生模型为基础。
CNN2ViT是指卷积架构的教师模型向ViT架构的学生模型进行蒸馏,通常利用卷积网络前期的归纳偏差来帮助ViT架构的网络模型训练,如图9所示,蒸馏损失可以描述为
$ {L_{{\rm KD}}} = \alpha {L_{{\rm Dis}}}(\rho ({{\boldsymbol F}^{\rm T}}),{{\boldsymbol v}^{\rm S}}) + \beta {L_{{\rm Dis}}}({{\boldsymbol z}^{\rm T}},{{\boldsymbol q}^{\rm S}}) \text{,} $
其中${{\boldsymbol F}^{\rm T}}$表示卷积架构教师模型的中间层特征,${{\boldsymbol v}^{\rm S}}$表示ViT架构学生模型的Transformer块特征,$\rho ( \cdot )$表示特征转换层,用于对齐教师模型的卷积特征和学生模型的Transformer块特征。${{\boldsymbol z}^{\rm T}}$和${{\boldsymbol q}^{\rm S}}$分别表示卷积架构教师模型的Logits输出特征和ViT架构学生模型的分类标记输出特征,两者的特征维度均为类别数量。$\alpha $和$\beta $为超参数,用于平衡两者的权重。
ViT[15]在没有大规模数据的情况下不能很好地泛化,Touvron等人[17]在尾部额外添加一个蒸馏token来接收来自CNN教师模型的知识。Ren等人[73]对Touvron等人提出的DeiT[17]做出改进,认为不同教师模型具有不同的归纳偏差,提出引入多个token分别接收卷积网络的知识和内卷积网络[74]的知识。Zhao等人[75]认为CNN后期的局部特征直接被使用会损害ViT的优化,将CNN知识通过池化层分为局部知识和全局知识,在训练前期通过局部知识充分利用归纳偏差的益处,在后期利用全局特征继续向ViT输送知识。
ViT2CNN是指ViT架构的教师模型向卷积架构的学生模型进行蒸馏,如图10所示,蒸馏损失可以描述为
$ {L_{{\rm KD}}} = \alpha {L_{{\rm Dis}}}({{\boldsymbol v}^{\rm T}},\phi ({{\boldsymbol F}^{\rm S}})) + \beta {L_{{\rm Dis}}}({{\boldsymbol q}^{\rm T}},{{\boldsymbol z}^{\rm S}}) \text{,} $
其中${{\boldsymbol v}^{\rm T}}$表示ViT架构教师模型的Transformer块特征,${\boldsymbol F^{\rm S}}$表示卷积架构学生模型的中间层特征,$\phi ( \cdot )$表示特征转换层,用于对齐教师模型的Transformer块特征和学生模型的卷积特征。$ {{\boldsymbol q}^{\rm T}} $和${{\boldsymbol z}^{\rm S}}$分别表示ViT架构教师模型的分类标记输出特征和卷积架构学生模型的Logits输出特征,两者的特征维度均为类别数量。$\alpha $和$\beta $为超参数,用于平衡两者的权重。
Liu等人[76]致力于将Transformer的知识传递给CNN模型,主要解决异构模型的特征语义不匹配问题,提出部分交叉注意力投射层和分组线性投射层来映射学生模型特征使之与教师模型特征更相似,并引入多种数据增强手段来提升学生模型的鲁棒性。
Zhao等人[77]通过将像素级特征映射到具有统一感受野的局部特征中来解决跨架构师生模型的特征对齐问题;Hao等人[78]将学生模型中间特征映射到Logits空间,来解决异构模型的特征对齐问题;Li等人[79]通过引入助理模型来进行跨架构的蒸馏,弥合异构模型之间的特征差异;Ni等人[80]引入多个具有不同归纳偏差的学生模型进行相互学习,来缓解跨架构蒸馏的性能问题;Zheng等人[81]通过一个特征适应模块来处理教师特征,让其与异构学生模型特征对齐。
ViT2ViT是指ViT架构的教师模型向ViT架构的学生模型进行蒸馏,将知识从较大的ViT架构教师模型转移到较小的ViT架构学生模型中,如图11所示,蒸馏损失可以描述为
$ {L_{{\rm KD}}} = \alpha {L_{{\rm Dis}}}({{\boldsymbol v}^{\rm T}},r({{\boldsymbol v}^{\rm S}})) + \beta {L_{{\rm Dis}}}({{\boldsymbol q}^{\rm T}},{{\boldsymbol q}^{\rm S}}) \text{,} $
其中${{\boldsymbol v}^{\rm T}}$和${{\boldsymbol v}^{\rm S}}$分别表示ViT架构的教师模型和学生模型的Transformer块特征,$r( \cdot )$表示特征映射层。${{\boldsymbol q}^{\rm T}}$和${{\boldsymbol q}^{\rm S}}$分别表示ViT架构教师模型和学生模型的分类标记输出特征。$\alpha $和$\beta $为超参数,用于平衡两者的权重。
Wu等人[82]将教师模型的推理与学生模型的蒸馏分离,提出稀疏的特征编码和数据增强编码来存储教师模型的推理结果以进行快速蒸馏,从预训练的知识中获得性能提升。Zhang等人[83]发现直接将权重共享技术应用到ViT中会带来训练上的不稳定和性能下降的问题,提出使用权重复用和权重蒸馏,前者通过在多头注意力前后插入线性层和在多层感知器层中插入转换模块来增加参数的多样性,后者利用概率分布输出、自注意力QKV矩阵和多层感知器层的关系矩阵进行蒸馏来保证性能具有竞争性。Yang等人[84]认为不同的ViT层具有不同的特征分布,提出对浅层使用基于模仿的线性特征蒸馏,对于深层特征使用基于掩码的生成式蒸馏。Zhao等人[85]认为对多头注意力的1对1蒸馏忽略了语义的一致性,对于不同数量的注意力头不能完美匹配,提出了注意力头的对齐蒸馏,利用教师模型注意力头与学生模型注意力头之间的余弦相似度,每一个教师模型注意力头指导所有的学生模型注意力头,并通过相似性来控制权重。
掩码图像建模(masked image modeling, MIM)具有强大的特征表征能力,Ren等人[86]通过蒸馏技术将MIM的知识转移到较小的学生模型中,研究发现对自注意力模块的QKV矩阵的Q-K,V-V关系矩阵进行蒸馏效果最好。Lin等人[87]发现ViT在小样本的设定下容易过拟合且缺乏类似CNN模型归纳偏置的机制,有严重的性能下降问题,利用分类token的交叉熵和MIM损失进行自监督训练,并通过余弦相似度进行跨类内图像块蒸馏学习。Bai等人[88]提出了直接从MAE[89]预训练模型中提取知识,将掩码率提升到95%并利用曼哈顿距离匹配中间特征来训练学生模型。Zhao等人[90]认为Bai等人提出的DMAE[88]对教师模型的高掩码率会丢失重要信息,提出让教师模型拥有更多的可见块,学生模型的可见块是教师模型的子集,通过直接对齐和生成式对齐来处理师生模型中均可见的块和仅在教师模型中可见的块。
涉及ViT架构的蒸馏方法总结如表5所示,InfoNCE[27]表示一项对比损失。基于Transformer的视觉模型由于其强大的性能,在多个领域都有重要应用。对于异构架构的蒸馏,大多数工作集中在利用卷积网络的归纳偏差来辅助ViT的训练,但很少关注将ViT中丰富的知识转移到更普适的卷积网络中。将ViT强大的全局建模能力转移到卷积网络中,是一个值得深入研究的课题。
知识蒸馏的学习方式通常可以根据训练过程的不同分为离线蒸馏、在线蒸馏、自蒸馏、无数据蒸馏、多教师蒸馏和助理蒸馏。其中离线蒸馏是最常见的学习方式,通过预先训练好的教师模型来训练学生模型;在线蒸馏允许师生模型同时更新;在自蒸馏方法中,模型本身同时作为教师模型和学生模型,降低了计算和存储的开销;无数据蒸馏可以在不访问原始数据的情况下完成对学生模型的训练;多教师蒸馏打破了单一教师模型的限制,通过多个教师模型共同训练学生模型;助理蒸馏通过引入一个能力介于师生模型之间的助理模型来帮助学生模型更好地优化。各种学习方式之间并非相互独立,某些方法可以同时归属于多种学习方式。以上学习方式示意图如图12所示。
离线蒸馏中,教师模型和学生模型的训练过程是分开的,首先在蒸馏学生模型之前先在数据集上训练教师模型,然后利用训练好的教师模型中提取的知识来指导较小的学生模型训练。离线蒸馏的方式简单易行,师生模型的训练可以分开进行,并且在蒸馏时仅需关注学生模型的学习,如图12 (a)所示;但学生模型只能从教师模型学习到固定的知识,无法适应动态的变化,并且学生模型的能力极大地依赖于教师模型提供的知识。为应对不同任务场景的需求,离线蒸馏逐步发展出标准蒸馏和跨任务及跨模态蒸馏2个研究方向,分别聚焦于单一任务内的知识高效传递与跨任务跨模态知识迁移的优化。
1)标准蒸馏。大多数蒸馏方法采用离线方式进行,其第1阶段通常不被纳入知识蒸馏的讨论范围。离线蒸馏的重点在于知识的有效转移,例如,Hinton等人[21]通过软化预测标签并引入温度系数,增强置信度较低标签的影响,以转移输出特征知识;Romero等人[40]通过线性层对齐教师模型与学生模型的特征层,实现中间特征知识的迁移;Yim等人[53]则提取教师模型的关系矩阵,反映问题求解流程中的关系特征,从而完成关系特征知识的转移。
2)跨任务及跨模态蒸馏。一些工作利用离线蒸馏进行跨任务和跨模态学习,通过其他模态任务的数据来提升下游任务性能。Ye等人[91]提出分步蒸馏,先通过对比蒸馏增强特征提取器的判别性,再利用相似性蒸馏优化分类器,重用跨任务模型知识;Lê等人[92] 探讨有限数据下目标检测与语义分割的跨任务学习,通过蒸馏技术缓解数据不足问题;Li等人[93]通过将教师模型的知识应用于多种任务场景,有效缓解了不同标签空间的约束问题,提升了学生模型的泛化能力;Auty等人[94]利用反向投影,将在不同任务中训练的教师模型知识迁移至目标任务的学生模型中,从而实现知识的有效传递;Yuan等人[95]将图像语义理解任务模型中的知识蒸馏到文本图像生成任务中,确保了生成图像与文本语义的一致性;Bayramli等人[96]利用注意力块实现语义分割教师模型向场景流学生模型的知识迁移;Wang等人[97]提出双向模态重建模块进行跨模态蒸馏,利用标注丰富的模态补充标注稀缺模态的数据。
在没有大容量高性能的教师模型或教师模型与学生模型能力差距很大时,学生模型的性能会受到影响,为了进一步提升学生模型的性能,在线蒸馏被提出。在线蒸馏中教师模型和学生模型同时更新,如图12(b)所示,在线蒸馏的典型模式有互学习、共享学习和协同学习[23,25]。
1)互学习。Zhang等人[30]在2017年提出了互学习的概念,让多个模型之间互相学习其概率分布输出来提高模型性能。为了缓解师生模型能力差距的影响,Qian等人[98]通过自适应切换阈值来让师生模型角色互换和互相学习。Yang等人[99]提出一个对比互学习框架,在一组模型之间进行交互和对比分布的传递,以增强模型的特征表示能力。Jacob等人[100]提出自适应特征蒸馏和在线任务加权方案为每个任务选择训练层来缓解多任务学习中准确率受损的问题。Rao等人[101]通过在教师模型中插入可学习的适配器模块进行更新,来缓解师生模型预测分布的尖锐度差距。
2)共享学习。Anil等人[102]2018年提出了共蒸馏,每个子网在数据的不相交的子集上训练,迁移其他模型的知识来互相学习。Zhou等人[36]提出一个共享低级别参数的师生模型架构的蒸馏框架。
3)协同学习。Song等人[103]提出了协同学习,在相同数据上训练多个分类器,来提高模型的泛化能力。Chen等人[104]通过多个辅助网络和1个主网络进行2级蒸馏来提高模型多样性;而Gong等人[105]提出自适应层级分支融合在线蒸馏框架来提高模型的多样性,缓解多学生模型集成的同质化问题。Wu等人[106]构建一个多分支网络,将分支随机增强后与另一个分支分类器集成构成教师模型相互学习。Li等人[107]将学生模型划分为领导学生模型和普通学生模型,领导学生模型指导普通学生模型训练,并且普通学生模型的知识能够整合提炼给领导学生模型来提高模型的整体能力。
自蒸馏不同于传统的知识蒸馏,其教师模型和学生模型都是同一个模型,如图12(c)所示,自蒸馏通过利用模型自身的预测结果或中间特征来提高其模型性能。
Zhang等人[108]在2019年提出自蒸馏的概念,通过特征相应的KL散度和中间层特征的对齐损失来将模型深层的知识转移到浅层。Lan等人[109]有相似的思想,将训练中模型发现的知识提炼回自身来规范后续学习过程。与之相反,Yang等人[110]提出将网络训练前期的知识转移到网络训练后期,来实现同一网络的监督训练。Mobahi等人[111]对自蒸馏进行理论分析,发现自蒸馏迭代逐渐限制了用于表示解决方案的基函数的数量。自蒸馏方法存在即时信息丢失、额外内存消耗和并行化程度低的问题,Shen等人[112]提出了利用上一批次输出结果进行蒸馏,来保持模型的一致性。Gou等人[113]提出学生模型可以通过自我反思,进行更有效的训练,通过自蒸馏将前几层的知识蒸馏到后几层中。余鹰等人[114]提出一种多层次知识自蒸馏联合多步训练方法,将特征学习和知识自蒸馏分步进行,解决了单步训练网络优化难的问题。
Lopes等人[115]在2017年提出无数据蒸馏的概念,如图12 (d)所示,现有蒸馏方法需要访问原始训练数据,随着数据集规模的增大其分布可能变得越来越复杂,并且许多数据集涉及隐私或安全问题(生物特征、敏感数据)使得数据共享和数据采集变得困难,提出通过利用教师模型的层级或信息重建训练学生模型的数据。与之思想类似的工作,Nayak等人[116]提出通过狄利克雷分布对softmax函数进行建模来构建样本数据。针对无数据蒸馏训练的稳定性和效率问题,Fang等人[117]通过重用训练数据中的共同特征来提高数据合成的效率。Yu等人[118]设计了通道级特征交换和多尺度空间激活来解决合成样本的多样性和效率问题。Li等人[119]提出了一种动态生成由易到难伪样本的方法来解决无数据蒸馏训练不充分及不稳定的问题。Wang等人[120]探究了师生模型之间的性能差距与合成样本质量之间的关系,提出了一种差距敏感的样本生成方法。张晶等人[121]建立教师模型与学生模型生成器并同时优化,实现网络任务与优化目标一致,提升学生模型判别性能,并利用教师模型潜在分布先验信息优化生成器,保证学生模型识别准确率并提升泛化性。
Chen等人[122]在2019年提出利用生成对抗网络合成数据样本来训练学生模型,推动了接下来一系列基于生成对抗网络(generative adversarial nets,GAN)[123]工作的发展[124-127]。Do等人[128]提出动量对抗蒸馏方法来缓解生成器合成数据偏移的问题。Zhao等人[129]提出通过双判别器对抗蒸馏来获得教师模型的内在统计信息和与学生模型的最大差异,从而更好地生成样本。Shao等人[130]认为现有的合成方法无法区分不同类别的样本分布,引入条件GAN来合成特定类别的多样性样本。Patel等人[131]认为伪标签在多个生成器更新下为非平稳分布,准确性受到影响,在知识保持和任务获取之间设置一个对齐因子来缓解该问题。
多教师蒸馏的概念在2017年由You等人[61]提出,旨在通过多个教师模型为学生模型提供更全面的指导,如图12(e)所示,通过平均教师模型的软标签并通过一种投票策略筛选多个教师模型的中间特征,复合多个教师模型的知识来指导学生模型训练。他们的工作引发了后续一系列工作对于多教师蒸馏方法的研究[31,38,102,132-135]。
Son等人[37]借鉴DenseNet[8]的思想将上级教师模型的输出知识始终作为下级教师模型的指导信息来缓解教师模型预测的错误累计,并借鉴Dropout的思想提出随机丢弃来缓解过拟合。Shi等人[136]利用学生模型学习效果的反馈来关注学生模型对不同教师模型知识的偏好,提高知识转移的效率。Pham等人[137]融合量化的方法,鼓励量化教师模型之间的协作学习以及量化教师模型与量化学生模型之间的相互学习。针对教师模型可能出现的低质量预测,Zhang等人[138]利用真实标签为每个教师模型分配预测置信度,提高教师模型软标签的可靠性。Cheng等人[139]将多教师蒸馏损失解耦,通过预测熵为每个教师模型分配权重,缓解低质量预测带来的负面影响。
近期部分工作聚焦于跨模态和跨任务的知识转移,通过不同来源的教师模型协作提升学生模型性能。Klingner等人[140]将基于激光雷达的3D目标检测教师模型的跨模态知识蒸馏用于多相机3D目标检测学生模型,并通过预训练的实例分割教师模型进行跨任务蒸馏。Zhao等人[141]设计4个蒸馏模块,从激光雷达点云和多视角相机检测器到雷达点云和多视角相机检测器,实现高效跨模态蒸馏。
由于教师模型和学生模型能力差距过大,会影响知识蒸馏的性能,其中一种解决方案是引入一个能力介于教师模型和学生模型之间的助理模型来弥合性能鸿沟[38]。该思想由Wang等人[142]提出,利用助理模型作为GAN的判别器,来缩减教师模型和学生模型产生特征之间的差异。我们将此类方法称之为助理蒸馏,如图12(f)所示。
Passalis等人[143]训练了一个助理模型来建立师生模型各层之间1对1的匹配关系;Ganta等人[144]实现了多个助理模型的集成,实现了更好的蒸馏性能;Gao等人[145]引入一个助理模型来学习教师模型和学生模型之间的残差来弥合其学习能力的差异;Son等人[37]引入多级助理模型,并利用高层助理模型和教师模型来同时指导低层助理模型和学生模型训练,缓解错误累计的问题;Mirzadeh等人[38]在训练助理模型时只考虑了输出特征;Morikawa等人[146]结合了输出特征和中间层特征来训练助理模型;He等人[147]引入了在线学习,让助理模型和学生模型相互学习,提升了学生模型的表现;Zhao等人[148]微调教师模型来构造自适应助理模型,提供多样化知识并缩小师生模型之间的能力差距;Li等人[79]通过助理模型来进行跨架构的蒸馏,弥合异构模型之间的特征差异;Huo等人[149] 提出了一种动态知识选择策略,从非目标类别中提取有效的跨模态知识,并通过助理模型逐步传递这些知识,增强跨模态学习效果。
本节介绍了6种常见的学习方式,针对各种学习方式的蒸馏方法汇总于表6所示。离线蒸馏方法简单易行,但仅能学习到教师模型的固定知识,灵活性不强;在线蒸馏中,教师模型也参与参数更新,尽管能带来更大的性能提升,但引入了额外的计算开销;自蒸馏方法中,教师模型和学生模型为同一模型,计算和存储开销较小,但知识结构相对单一;无数据蒸馏能够很好地保护数据隐私,但由于缺乏数据集支持,在复杂场景中性能较差;多教师蒸馏由多个教师模型提供知识,知识结构较为丰富,但特征融合较难且会引入额外的开销;助理蒸馏能够在一定程度上弥合师生模型之间的性能差异,但助理模型的训练也引入了额外的计算开销。每种方式各有优劣,在实际应用中应根据具体场景选择适当的方法。
第2节和第3节归纳的方法大多是基于Hinton等人[21]提出的经典蒸馏范式。近年来,随着知识蒸馏的发展,出现了一些研究打破了传统固定模式,对学习范式进行了改进和优化。本文将这类方法总结为知识蒸馏的优化,如图13所示,分为4个类别:蒸馏过程的优化、知识结构的优化、温度系数的优化和损失结构的优化。蒸馏过程的优化主要针对师生模型特征的对齐和训练过程的调整;知识结构的优化汇总了对知识的调整和筛选的方法,以高效地向学生模型传递知识;温度系数的优化打破了先前固定系数的模式,总结了动态温度的设置方法;损失结构的优化汇总了对损失函数的解耦方法,解决了损失内部的冲突博弈。此外,本节还总结了利用其他技术对知识蒸馏方法进行优化的研究,包括对抗性技术、自动机器学习、强化学习和扩散模型。
知识蒸馏训练过程的优化主要针对于特征对齐模块的改进以及蒸馏学习过程的调整,来获得更好的学生模型。Cho等人[150]发现蒸馏损失后期会损害模型的训练,提出停止方案以阻止后期蒸馏损失对模型性能的削弱。Mirzadeh等人[38]发现师生能力差距过大会导致学生模型的性能下降,提出引入一个介于师生模型之间的助理模型来帮助学生模型学习,弥合性能的差距。Yuan等人[151]证明了用较弱的学生模型指导教师模型或训练不佳的教师模型指导性能更佳的学生模型同样能够带来性能的提升。Stanton等人[152]探究影响蒸馏性能的原因,对学生模型容量、网络架构、数据集规模、复杂性、数据域以及优化方法进行了大量实验,认为优化方法限制了模型的收敛。He等人[153]认为在师生模型之间添加可学习模块会损害表示学习的性能,提出了一种非参数的对齐方式,通过奇异值分解来完成师生模型的对齐。Shen等人[154]认为针对蒸馏训练需要额外对教师模型进行推理,提出FKD,在教师模型推理阶段保存数据增强方法及输出结果,在训练学生模型时直接复用来加快蒸馏的训练。Song等人[155]更关注“从哪里蒸馏”,提出点自适应的蒸馏模块,自适应地确定每个样本在教师模型中的蒸馏位置。Zhang等人[156]提出一个在线蒸馏框架,旨在寻找更平坦的最优解,提出使用混合权重模型配合间歇融合策略来帮助模型更好地收敛。Wang等人[157]将学生模型中间特征转移到教师模型进行推理,与教师模型软标签进行蒸馏,学生模型最后几层只通过真实标签优化,通过这种方式缓解真实标签与教师模型的错误分类之间的冲突。
知识蒸馏的知识结构优化指的是对教师模型提供的知识进行再抽取来减少信息的冗余或进行再调整来提升知识的传递效率,大大提高教师模型知识的有效性,从而能够训练得到性能更好的学生模型。
Xu等人[158]认为学生模型并没有获得教师模型全面的知识,提出通过topK选择性转移策略来更好地利用样本变换带来的暗知识。Li等人[159]认为对所有的知识进行蒸馏是冗余的,将知识点分为强、中、弱3个档次,强价值知识直接蒸馏,中价值知识利用弱价值知识增强后再蒸馏。Huang等人[160]认为学生模型应当关注教师模型的相对排名而不是准确的概率输出,引入类内相似性和类间相似性,并用皮尔森系数取代KL散度作为新的距离度量。Shen等人[112]针对先前自蒸馏方法即时信息丢失、额外内存消耗和并行化程度低的问题,提出了利用上一批次输出结果进行蒸馏的方法来保持模型的一致性。Choi等人[161]认为教师模型可能将错误信息传递给学生模型,提出分组训练的框架,学生模型组先反馈出一组易于出错的样本,经过数据增强后交给教师模型加强学习,利用教师模型和性能最好的学生模型一起指导学生模型组中的所有学生模型训练。Shen提出FerKD[162],该方法是在他们的另一项工作FKD[154]的基础上提出自适应标签校正,丢弃非常简单和非常困难的样本,重点来优化中等难度的样本。Miles等人[163]专注于高效的知识转移,提出在Stiefel流形上构建一个正交投影层,最大限度地将蒸馏知识传递给学生模型。他们还针对蒸馏中的投影层、归一化层和距离函数进行了更深层次的实验探究[164]。Sun等人[165]提出一种基于掩码的方法屏蔽教师模型的错误分类,来减少教师模型的错误输出对学生模型的负面影响。
知识蒸馏的温度系数从被提出时始终作为固定超参数,然而针对不同的样本使用相同的温度系数来软化目标,得到的结果往往是次优的,因此针对不同的特征自适应地设置不同的温度系数,能够获得更好的软标签来指导学生模型训练,进一步提升学生模型的性能。
Xu等人[166]从one-hot标签角度研究了标签分布与真实分布之间的差距,发现网络中倒数第2层特征的${L_2}$范数可以作为标签中的噪声强度来代替温度系数。Jafari等人[167]通过一个退火函数来生成动态温度,并用均方误差取代KL散度作为距离度量。Li等人[168]认为训练学生模型应当遵循由易到难的原则,提出使用温度系数来调整任务的难度,通过GAN的思想来学习调整温度系数。Chi等人[169]将样本的Logits输出视为高斯分布,利用其标准差作为温度系数。Wei等人[170]提出了样本Logits输出平滑度的概念,利用log-sum-exp函数评估Logits输出的尖锐度,动态调整温度系数,对尖锐度高的教师模型设置较高温度,对尖锐度低的学生模型设置较低温度,以平衡师生模型输出一致性。Sun等人[171]从信息论的最大熵理论利用拉格朗日乘子法推导温度系数,根据推导后的结果提出对Logits输出特征进行标准化,即利用均值对Logits输出进行平移后将标准差作为温度系数。
知识蒸馏的损失函数通常包括任务损失和蒸馏损失,有研究发现这2项损失之间以及蒸馏损失内部存在耦合和冲突,会影响蒸馏学习的性能。因此近期一些工作[172-175]专注于打破这种关系来进一步提升蒸馏的性能。
Zhao等人[172]提出将损失解耦为针对目标类的二元蒸馏和针对非目标类的多类别蒸馏;Yang等人[173]发现Zhao等人[172]提出的DKD非目标类别师生模型的概率分布输出和不同,这阻碍了分布的相似,提出对分布进行归一化;Zhao等人[175]认为任务损失与蒸馏损失发生冲突,退化为多任务学习,提出调整优化器动量值让蒸馏损失占据主导地位来打破冲突的博弈;Park等人[174]将DKD[172]中对非目标类暗知识的动态自适应调整学习同样应用到目标类上,并且对任务进行序列化训练,来降低2种不同分布优化任务的冲突。
交叉技术优化旨在利用其他领域的技术,例如对抗性技术、自动机器学习、强化学习、扩散模型等,通过这些技术的优势来改善优化知识蒸馏学习的过程,来获得性能更好的学生模型。
1)对抗性技术。对抗学习在2014年Goodfellow等人[123]提出的GAN取得巨大成功后受到广泛关注,GAN中判别器估计样本来自训练数据分布的概率,生成器试图利用生成的数据来欺骗判别器,两者相互博弈来提高生成数据的质量。而该思想同样可以应用到知识蒸馏中来提升学生模型的性能。Shen等人[176]为了从不同教师模型中提取多样化知识,定义了一个分块训练损失来指导学生模型恢复教师模型中的知识。Li等人[177]在对抗训练框架下引入任务驱动的注意力机制,将知识设计成紧凑的形式传递给学生模型。Ma等人[178]提出了一项利用对抗性思想防止闭源模型通过蒸馏的方式被窃取,通过最大化教师模型与预训练模型的KL散度来尽可能地获得释放错误信号的能力。
在蒸馏中决策边界一直被认为是重要的,Heo等人[179]利用对抗攻击发现支持决策边界的样本来向学生模型传递边界信息。Nguyen-Duc等人[180]构造了一个给定输入球约束内所有对抗样本的集合,来挖掘决策边界。
鲁棒性蒸馏是指通过对抗性训练或其他技术得到一个鲁棒性的教师模型,然后通过蒸馏技术将鲁棒性的知识传递给学生模型的方法。Bai等人[181]发现学生模型很难从教师模型中继承对抗鲁棒性,通过一种重加权的策略有选择地从教师模型学习。Huang等人[182]设计了自适应对抗性蒸馏方案,在内部优化中自适应搜索最优匹配点。Yue等人[183]考虑了鲁棒性的类别平衡性,提出公平对抗鲁棒性蒸馏,增加困难类的权重来增强学生模型的鲁棒公平性。Wang等人[184]认为提供鲁棒性知识的教师模型并不总是能正确预测,通过多个实例之间的结构化关系代替1对1的模仿学习。
2)自动机器学习。自动机器学习(automated machine learning,AutoML)是指通过自动化的方式来完成机器学习模型的设计、训练和优化过程。AutoML的目标是使得机器学习的应用变得更加简单和高效,即使对于没有专业知识的用户,也能使用机器学习技术解决实际问题,其中NAS是其中最流行的一项技术之一,在知识蒸馏中学生模型的性能不仅仅取决于教师模型提供的知识,还取决于学生模型架构的设计,因此有的工作将NAS用于改进知识蒸馏的师生模型架构。
Li等人[185]提出一种分块搜索方案来评估每一块的候选框架,其搜索结果超过了教师模型的性能。Liu等人[186]提出一种基于强化学习的NAS搜索算法来寻找给定教师模型的最佳的学生模型。Kang等人[187]通过提出一种预言机损失来帮助NAS搜索,缓解师生模型能力差距带来的性能下降问题。Wei等人[188]提出了一种在小规模数据集上对ViT架构进行搜索的方案,找到最佳的学生模型架构进行蒸馏学习。Dong等人[189]提出一种基于NAS的学生模型搜索策略,通过衡量类别样本之间的相似性来获得最佳学生模型。Lu等人[190]提出一种基于NSGA-II遗传算法的搜索框架,并提出连续减半的优化加速策略。有的工作通过总结先前工作构造搜索空间,利用搜索算法来获得最佳的蒸馏设置。Li等人[191]通过蒙特卡洛搜索最佳的蒸馏策略,提出离线推理、随机掩码的稀疏蒸馏,提前停止策略来加速蒸馏。
3)强化学习。强化学习(reinforcement learning, RL)是机器学习的一个重要领域,强化学习可以被简单理解为智能体在环境中采取行动并得到奖励来扩充自己的知识储备,使得自己在环境中有更好的表现,得到更多的回报。Liu等人[186]设计了基于知识蒸馏的奖励函数,通过强化学习的方法搜索最佳的师生模型架构。
4)扩散模型。扩散模型(diffusion models)是生成模型的一类,它通过逐步去噪的过程生成新数据。受非平衡态热力学的启发,在2015年扩散模型的概念[192]被提出,并在2020年由Ho等人[193]提出的DDPM推动其进一步发展,扩散模型的基本思想是将数据逐步转化为噪声,然后通过逆过程逐步去噪生成新数据。这一过程可以看作是一种马尔可夫链,它由前向过程(添加噪声)和逆向过程(去除噪声)组成。扩散模型已经在图像生成等领域取得了显著的成功。Huang等人[194]认为蒸馏特征中的噪声是影响蒸馏性能的原因之一,受扩散模型逆过程的启发,提出通过扩散模型逐步地从图像或特征中去除噪声。
针对知识蒸馏的优化方法汇总如表7所示。这类研究是近年来的热点,通过从训练优化的角度改进知识蒸馏的学习范式,取得了显著的性能提升。然而,目前的方法对于师生模型架构的设计仍然依赖于人工设计,尽管通过搜索方法可以找到最佳的蒸馏设置,但这会引入额外的计算负担。因此,如何在低开销的情况下获得针对特定任务场景的最佳师生模型架构设计仍值得深入研究。此外,当前知识筛选的工作主要集中在教师模型上,而对学生模型难以区分的困难样本进行挖掘将是一项有意义的研究方向。并且目前的研究大多基于实验观察结果,缺乏理论上的完整性。
知识蒸馏有着广泛的应用,通常是利用其模型压缩的能力和知识转移的特性,本节总结了5个较为热门的下游任务,例如目标检测、语义分割、人脸识别、异常检测和姿态估计等;不仅仅局限于下游任务,知识蒸馏中知识转移和隐私保护的特性还能够直接应用于某一个领域,例如增量学习和联邦学习等。
目标检测旨在从图像或视频中识别并定位特定类别的目标物体。与图像分类不同,目标检测不仅需要识别图像中的目标类别,还需要确定每个目标在图像中的具体位置。通常通过在图像中绘制边界框来实现,每个边界框标记一个目标物体的空间位置。根据目标检测的维度不同,可以将目标检测分为2D目标检测和3D目标检测。
大量的假阴性样本和较低定位精度样本影响性能,Guo等人[195]采用一致性正则化来处理假阴性样本和低精度定位带来的噪声问题。Zhou等人[196]引导模型从简单样本到困难样本逐步蒸馏显著性知识,并通过预测边界周围纹理来细化噪声标签的边界。
视觉语言模型(vision language model,VLM)中包含着丰富的知识,Wang等人[197]提出一个物体感知的蒸馏金字塔框架,从预训练的VLM提取知识,并引入全局和分块蒸馏进行知识传递。Li等人[198]将VLM获得的知识提取到DERT的检测器中,通过语义蒸馏和关系蒸馏来传递语义知识和利用物体之间的隐式关系。Zhou等人[199]提出了一个通用的跨模态知识蒸馏框架来提高单模态检测器的性能,将不同模态的信息蒸馏到同一个模态来合成。
Wu等人[200]认为提取物体特征的域不变表示具有重要意义,提出循环解耦自蒸馏用于从视觉特征中循环提取域的不变特征表示。针对以上方法无法从异构教师模型中提取知识的问题,Lao等人[201]引入额外可变形的交叉注意力解码头,让不同架构的师生模型互相学习。Zeng等人[202]研究了如何从一个不完美的教师模型中提取知识。
在3D目标检测中Klingner等人[140]介绍了一种跨任务、跨模态、跨阶段的多相机3D目标检测综合知识蒸馏框架。Liu等人[203]提出一种跨模态蒸馏方法,通过Logits输出特征蒸馏来自上级激光雷达点云LiDAR(light detection and ranging)模型的立体探测器,来缩小基于立体和基于LiDAR方法之间的差距。来自估计的深度先验不可避免的误差可能会导致不一致的语义信息和3D定位,Wu等人[204]提出了具有3D感知位置编码的注意力深度知识蒸馏框架来约束学生模型学习。
基于多视角鸟瞰图BEV(bird’s eye view)的3D目标检测与基于LiDAR的方法有明显的性能差距,Wang等人[205]提出基于多视角BEV的学生模型,利用蒸馏训练好的LiDAR教师模型的特征来弥补性能差距。Zhang等人[206]提出一种新视图引导蒸馏目标,可以稳定量化感知训练,同时利用图像特征和BEV特征来增强模型性能。Zhao等人[207]提出了一种用于BEV目标检测的多模态蒸馏方案,将教师模型的LiDAR特征知识转移到多模态学生模型中。
现有方法只有在师生模型共享相似的中间特征表示时才会显示出有效性,点云的内在稀疏性和不规则性导致了表示差异。Li等人[208]提出了一种表征差异蒸馏,减少较小的学生模型与过度参数化的教师模型之间的性能差异。Gambashidze等人[209]提出通过X射线蒸馏解决LiDAR目标检测中的稀疏性和遮挡性。Huang等人[210]将动力学与雨环境理论统一起来,提出晴雨知识蒸馏,增强雨天条件下的LiDAR目标检测。
语义分割旨在将图像中的每一个像素分类到某一个特定的类别中。与图像分类和目标检测不同,语义分割不仅需要识别图像中的物体,还需要准确地定位每一个物体的边界。它在自动驾驶、医学影像分析、遥感影像处理等领域有广泛的应用。近期利用知识蒸馏的语义分割方法主要通过蒸馏算法来对特征和结构信息进行补充,从而构建更好的特征表示来增强模型的泛化性和准确性。
Ji等人[211]充分利用低层的结构纹理知识,通过迭代拉普拉斯金字塔和方向滤波来分解低级特征,挖掘纹理知识。Gao等人[212]提出一种跨域相关蒸馏的域适应框架处理光照影响,提取内容和风格知识特征,计算2幅图像之间的固有差异和光照差异。Yang等人[213]侧重在整个图像之间传递结构化的像素和像素到区域的关系。Peng等人[214]利用自注意力模块建立层次化稠密特征,引入相关性蒸馏来促进细粒度的分割。Liu等人[215]提出通过多个教师模型解码器生成可靠软标签,提出解耦特征集成蒸馏机制,利用重要的特征图进行编码器学习。
在3D点云语义分割方面,由于点云稀疏性和随机性导致直接蒸馏效果欠佳,Hou等人[216]提出从点级和体素级转移隐藏的知识,利用逐点和体素蒸馏来补充稀疏的监督信号。Chen等人[217]尝试对点云进行完全无监督的语义分割,将多视图的视觉特征反向投影到3D空间聚合为统一的点特征,将点特征聚合为超体素进行语义的挖掘。
人脸识别任务是一种生物识别技术,旨在通过计算机视觉和机器学习算法,自动检测和识别图像或视频中的人脸。近期利用知识蒸馏的人脸识别工作主要利用蒸馏技术知识迁移的能力来获得更好的人脸特征表示。
Xu等人[218]提出对人脸的贝叶斯集成平均模型进行蒸馏压缩来提高模型性能。Li等人[219]试图去除学生模型训练时的监督信号,通过反向蒸馏约束教师模型的搜索空间来缩小师生模型的内在差距。学生模型的类内相似度与教师模型有较大差距,Yu等人[220]提出估计师生模型的概率分布,并引入相似性分布一致性损失来提高学生模型的类内紧凑型。轻量级的学生模型由于模型容量较低难以拟合教师模型的Logits输出特征,Zhao等人[221]通过探测Logits输出特征来提取与面孔身份相关的主要知识,并抛弃其他不相关知识。Zhang等人[222]为减轻训练数据集中人脸姿态分布不均匀的影响,将教师模型人脸的角度知识提取到学生模型中,使之能够处理较大姿态变化的人脸。为解决面部标注数据有限的问题,Hu等人[223]提出一个自监督人脸表示学习框架,能够从人脸视频中爬取通用的、多样的人脸表示。
异常检测是一种识别数据中异常模式或异常行为的技术。异常通常指那些显著偏离正常模式的数据点,这些异常数据可能表示某种问题或罕见事件。
反向蒸馏是Deng等人[224]在2022年为解决异常检测问题提出的一项技术,让学生模型将教师模型对于单个类别的编码作为输入来恢复教师模型的多尺度表示。Tien等人[225]通过反向蒸馏的思想提出一个新的异常检测框架,在低延迟的情况下表现良好。
Gu等人[226]通过存储常态信息来强化学生模型生成特征的常态性,作为其进行异常检测的先验知识。Binh等人[227]提出基于注意力机制的深度伪造检测蒸馏模型,应用知识蒸馏领域中的频域学习和最优传输理论,改善低质量压缩深度伪造图像的检测。Ma等人[228]提出一种多教师的蒸馏框架进行异常检测,来融合多种不同神经网络的优势。Ristea等人[229]提出了一种自蒸馏掩码自动编码器的视频异常检测框架,利用师生模型解码器之间的差异来提升检测性能,并通过合成异常事件来增强训练。刑鹏等人[230]提出一种基于特征约束的蒸馏学习方法,充分利用教师模型的特征来指导学生模型高效地识别异常图像。
姿态估计旨在从图像或视频中检测并估计人体、动物或物体的姿态。姿态估计的主要目标是识别图像或视频中的特定关键点(如关节、骨骼点等)的位置和姿态信息。
通过分类的预训练模型来生成视频特征用于动作定位,不同优化任务会使时间定位结果出现不一致的问题,Ju等人[231]提出从预训练的 VLM提取自由动作知识,这类知识与基于分类的模型知识互补,通过蒸馏将两者融合。Wu等人[232]提出一种基于VLM的端到端的零样本人-物交互检测框架,结合2阶段二分匹配算法交互评分模块,以动作无关的方式实现对人-物对的交互区分。
依赖于跟踪过去位置的方法在拥挤环境可能会累计错误,导致跟踪漂移。Monti等人[233]引入时空注意力模块恢复不同时间步之间的依赖关系,迁移至学生模型使其能用较短的序列进行预测。Ma等人[234]提出一个多目标条件变分推荐预测模型来平衡准确性采样和多样性采样。Yang等人[235]提出利用多人线性边缘特征提取来处理复杂姿态和物体遮挡问题,生成一个包含人-物边界和遮挡信息的重叠边缘,并在该边缘上进行特征蒸馏。
Ye等人[236]将基于热力图的模型作为教师模型、基于回归的模型作为学生模型,通过蒸馏集成两者优势,生成兼具高效性和优越性的学生模型。为了使Transformer架构的模型能有效部署到边缘设备上,Chen等人[237]提出一种改进的多重循环Transformer模块架构的自蒸馏算法,来改进小型模型欠拟合的问题。
增量学习又称为持续学习或终身学习,是指模型能够随着时间的推移逐步学习新知识,其中有一项主要的挑战是灾难性遗忘问题,模型在学习新知识的过程中,可能出现对已学知识的遗忘现象。将知识蒸馏技术应用于增量学习是一个有效的方法,利用蒸馏技术知识迁移的能力,可以缓解增量学习的灾难性遗忘问题。
Lu等人[238]提出增强几何蒸馏,在记忆不同的视图上以成对和交叉的方式进行蒸馏,并构建一个通用无数据增量框架来避免隐私泄露。Shang等人[239]在解码器中添加新的类标记来进行新类的学习,并通过对旧类知识的蒸馏来减少灾难性遗忘。Phan等人[240]提出一种基于类相似性的知识蒸馏方法,将先前模型知识提取到与新类相似的旧类上进行联合学习。Gu等人[241]对旧类和新类分别设计了损失函数,并对旧类进行重规范化来淡化对旧知识的丢失。Li等人[242]提出通过重要性采样的对比连续学习来保存知识,提出一种重放缓冲区的策略来恢复先前的数据分布。
增量学习的另一个问题是对新样本的欠拟合和对少数旧样本的过拟合,Zhao等人[243]提出双教师模型结构,一个模型在基类数据上训练有丰富通用知识的模型来缓解新类的过拟合,另一个模型用来获取新类的知识。Yan等人[244]训练了堆叠的子教师模型,集成多级监督和反向自蒸馏,从各个教师模型中收集其优势。对无限数据流进行训练时只能考虑短期信息,Ye等人[245]提出使用变分自动编码器开发一个短时记忆模块和一个参数化可扩展性记忆模块来构成无监督持续学习框架。
增量学习中还存在着训练的不稳定的问题,Wen等人[246]提出多教师蒸馏,通过权重排列、特征扰动和多样性正则化技术确保教师模型知识的多样性和有效性。Yang等人[247]发现背景类和新类之间存在混淆影响增量学习的性能,将旧模型的输出在真实标签的引导下进行扩展移植,使其与新的模型输出形成相匹配的类对应关系。
联邦学习是一种分布式机器学习技术,通过在多个设备(如智能手机、物联网设备等)上本地训练模型,并在不共享原始数据的前提下将模型参数或梯度汇总到中央服务器,从而构建全局模型。这种方法能够在保护数据隐私的同时,实现高效的机器学习模型训练。
数据异构性是联邦学习的一大挑战,Zhang等人[248]提出一种无数据知识蒸馏来微调服务器中的全局模型,引入一个生成器来探索局部模型的输入空间并将其知识迁移到全局模型中。Shen等人[249]提出一种基于信道分配循环蒸馏引导的通道解耦框架。
Kim等人[250]使用联邦学习来提高单分类器的性能,提出基于原型表示蒸馏的无监督联邦学习来增强全局模型的表达能力。为处理具有极端标签噪声的客户端,Lu等人[251]提出负蒸馏,令标签噪声上训练的模型充当“坏老师”来降低提供错误信息的风险。Wang等人[252]将每个客户端中的本地数据视为一个特定的领域,提出一种领域知识感知的联邦蒸馏方法。乐俊青等人[253]设计了一种基于双重知识蒸馏的训练算法,确保模型学习到每个样本的知识,使模型保持较高的准确度,并通过自适应的差分隐私策略来实现客户端级的强隐私保护。
知识蒸馏主要用于模型压缩,但由于其在知识转移中的独特优势,被广泛应用于其他领域,特别是在跨模态知识转移任务中。本节总结了知识蒸馏在5种下游任务(目标检测、语义分割、人脸识别、异常检测和姿态估计)和2种特定领域(增量学习和联邦学习)中的应用。在实际应用中,应根据具体场景和需求选择合适的蒸馏方法,以最大限度地提升模型性能和效率。通过知识蒸馏,不仅能有效压缩模型,还能实现特征融合和知识共享,拓展其在多种复杂任务中的应用潜力。
知识蒸馏作为深度学习模型压缩和加速的重要技术,近年来在学术界和工业界都得到了广泛关注和深入研究。
最初,知识蒸馏主要集中在对教师模型知识来源的探索。Hinton等人[21]提出的蒸馏算法通过利用教师模型的Logits输出特征进行知识传递,奠定了蒸馏的基础。随着研究的深入,发现中间层特征和关系特征同样具有重要的价值,进一步提升了蒸馏效果。在大模型出现后,Transformer架构的系列模型也为知识蒸馏提供了新的视角,探索了如何在不同类型的模型之间进行有效的知识迁移。
随后,知识蒸馏的研究逐步转向了不同学习方式的探索。早期的蒸馏方法主要基于离线蒸馏,通过预先训练好的教师模型传递知识给学生模型。随着需求的多样化,在线蒸馏、自蒸馏以及无数据蒸馏等方法应运而生,允许在没有教师模型或原始数据的情况下进行蒸馏。此外,多教师蒸馏和助理蒸馏等方法也获得广泛关注,通过引入多个教师模型或助理模型的知识,进一步提高了蒸馏的效果和鲁棒性。
近几年的研究更加关注优化方式的探讨,旨在进一步提升蒸馏的效率和效果。研究者们通过知识结构的筛选来确保传递给学生模型的知识更具代表性和有效性,避免冗余和错误信息的传递;优化蒸馏过程的交互方式和改进损失函数以更好地对接教师与学生模型的特征输出,也是近期研究的重点。温度系数的优化作为一个重要方向,通过调节软标签的平滑程度,进一步平衡教师与学生模型之间的知识传递效果。
尽管知识蒸馏在多个领域取得了显著的进展,但在实际应用和理论研究中仍面临诸多挑战:
1)蒸馏知识的结构。教师模型中的知识可以呈现多种形式,包括输出特征知识、中间层特征知识、样本或通道关系知识以及ViT架构模型中的知识等。如何选择合适的知识类型依然是一个值得研究的问题。针对这些多样化的知识类型,选择合适的提取位置也是一个重要的研究方向。此外,从教师模型中抽取的知识可能会包含噪声或不确定性,这些因素可能对学生模型产生不利影响。因此,研究如何处理和过滤教师模型知识中的不利因素是一个具有挑战性的课题。另一方面,如何从教师模型中挖掘出令学生模型难以区分的困难特征知识以加强学习,也是一个值得深入探索的方向。
2)师生模型架构的探索。教师模型的选择对知识蒸馏的效果至关重要。在实际应用中,教师模型的选择通常依赖于任务的性质、数据的特点和模型的计算资源。如何选择合适的教师模型,以确保其能够有效地指导学生模型进行训练,仍然是一个开放性的问题。
已有研究[38]证明,差距过大的师生模型架构会对学生模型产生不利影响。然而,目前对师生模型架构设计的探索仍主要依赖人工设计。尽管一些工作[189-191]通过搜索方法找到了最佳的师生模型架构设置,但这会引入大量额外的计算开销。在较少资源消耗的前提下找到最佳的师生模型架构仍是一个重要的挑战。
目前,大多数知识蒸馏方法主要针对特定类型的模型或特定任务展开研究。然而,对于跨架构的蒸馏,尤其是在ViT架构与CNN架构之间进行蒸馏时存在显著的挑战。这是因为不同架构所提取的特征通常存在语义不匹配的问题,这使得直接将教师模型的知识传递给学生模型变得困难。因此,设计合适的语义对齐模块,以便更有效地将教师模型的知识传递给学生模型,成为一个亟待解决的研究课题。此外,如何设计一种通用的蒸馏方法,能够跨越不同的网络架构和任务类型,适应各种应用场景,依然是知识蒸馏领域面临的重大挑战。
3)蒸馏学习的优化。尽管知识蒸馏能够有效压缩模型并提高学生模型的泛化能力,但在实际应用中,学生模型往往难以完美复制教师模型的能力。如何控制蒸馏过程中的性能损失,确保学生模型的效果尽可能接近教师模型,仍然是一个长期存在的挑战。
现有的蒸馏学习范式主要基于Hinton等人[21]在2015年提出的蒸馏学习算法。但有研究[152]指出,这一范式可能限制了学生模型性能的提升。尽管当前的研究通过优化损失函数、温度系数等方面进行了改进,仍未形成被广泛接受的新范式,且大多数改进基于实验观察,缺乏理论上的系统性。因此,构建一种新的优化学习体系,以突破现有瓶颈,仍是亟待解决的核心问题。
此外,知识蒸馏过程通常需要较长的训练时间和大量的计算资源,尤其是当教师模型规模庞大时,这一问题更为突出。与此同时,随着蒸馏方法应用的扩展,模型的可解释性和透明度逐渐成为关注重点。如何设计蒸馏算法,使得学生模型不仅能接近教师模型的性能,还能够提供清晰的决策过程解释,是当前研究中的一个重要难题。
本节从知识的来源、蒸馏的方式和蒸馏的优化总结了目前知识蒸馏的发展,并从知识蒸馏的关键组成:知识、师生模型架构和蒸馏算法3个方面讨论了目前面临的挑战,未来的研究需要在这些方向上继续探索,以进一步提升蒸馏效果和学生模型的性能。自2015年知识蒸馏提出以来,该技术在学术界和工业界受到了广泛关注。希望本综述能为相关研究人员和工程师提供一些启发和参考,继续推动知识蒸馏的进一步发展。
作者贡献声明:潘海为、于丰铭完成相关文献资料的收集和分析,是论文的主要撰写人;张可佳、兰海燕负责论文整体结构的设计,并对论文进行修改;孟庆宇、李哲参与文献资料的分析、整理。

参考文献 引证文献
排序方式:
1
Lecun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278−2324
2
Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks[C]//Proc of the 26th Annual Conference on Neural Information Processing Systems. Cambridge, MA: MIT, 2012: 1106−1114
3
Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint, arXiv: 1409.1556, 2014
4
Szegedy C, Liu Wei, Jia Yangqing, et al. Going deeper with convolutions[C]//Proc of 2015 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2015: 1−9
5
Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the inception architecture for computer vision[C]//Proc of 2016 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2016: 2818−2826
6
Szegedy C, Ioffe S, Vanhoucke V, et al. Inception-v4, inception-ResNet and the impact of residual connections on learning[C]//Proc of the 31st AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2017: 4278−4284
7
He Kaiming, Zhang Xiangyu, Ren Shaoqing, et al. Deep residual learning for image recognition[C]//Proc of 2016 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2016: 770−778
8
Huang Gao, Liu Zhuang, van der M L, et al. Densely connected convolutional networks[C]//Proc of 2017 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2017: 4700−4708
9
Howard A G, Zhu Menglong, Chen Bo, et al. MobileNets: Efficient convolutional neural networks for mobile vision applications[J]. arXiv preprint, arXiv: 1704.04861, 2017
10
Sandler M, Howard A, Zhu Menglong, et al. MobileNetV2: Inverted residuals and linear bottlenecks[C]//Proc of 2018 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 4510−4520
11
Howard A, Sandler M, Chu G, et al. Searching for MobileNetV3[C]//Proc of 2019 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 1314−1324
12
Tan Mingxing, Le Q. EfficientNet: Rethinking model scaling for convolutional neural networks[C]//Proc of the 36th Int Conf on Machine Learning. New York: ACM, 2019: 6105−6114
13
Tan Mingxing, Le Q. EfficientNetV2: Smaller models and faster training[C]//Proc of the 36th Int Conf on Machine Learning. New York: ACM, 2021: 10096−10106
14
Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proc of the 31st Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2017: 5998−6008
15
Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16×16 words: Transformers for image recognition at scale[J]. arXiv preprint, arXiv: 2010.11929, 2021
16
Liu Ze, Lin Yutong, Cao Yue, et al. Swin Transformer: Hierarchical vision Transformer using shifted windows[C]//Proc of 2021 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2021: 10012−10022
17
Touvron H, Cord M, Douze M, et al. Training data-efficient image transformers & distillation through attention[C]//Proc of the 38th Int Conf on Machine Learning. New York: ACM, 2021: 10347−10357
18
Wang Wenhai, Xie Enze, Li Xiang, et al. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions[C]//Proc of 2021 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2021: 568−578
19
Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//Proc of the 38th Int Conf on Machine Learning. New York: ACM, 2021: 8748−8763
20
Ramesh A, Pavlov M, Goh G, et al. Zero-shot text-to-image generation[C]//Proc of the 38th Int Conf on Machine Learning. New York: ACM, 2021: 8821−8831
21
Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network[J]. arXiv preprint, arXiv: 1503.02531, 2015
22
Gou Jianping, Yu Baosheng, Maybank S J, et al. Knowledge distillation: A survey[J]. International Journal of Computer Vision, 2021, 129(6): 1789−1819
23
Wang Lin, Yoon K J. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(6): 3048−3068
24
黄震华, 杨顺志, 林威, 等. 知识蒸馏研究综述[J]. 计算机学报, 2022, 45(3): 624−653
Huang Zhenhua, Yang Shunzhi, Lin Wei, et al. Knowledge distillation: A survey[J]. Chinese Journal of Computers, 2022, 45(3): 624−653 (in Chinese)
25
邵仁荣, 刘宇昂, 张伟, 等. 深度学习中知识蒸馏研究综述[J]. 计算机学报, 2022, 45(8): 1638−1673
Shao Renrong, Liu Yuang, Zhang Wei, et al. A survey of knowledge distillation in deep learning[J]. Chinese Journal of Computers, 2022, 45(8): 1638−1673 (in Chinese)
26
司兆峰, 齐洪钢. 知识蒸馏方法研究与应用综述[J]. 中国图象图形学报, 2023, 28(9): 2817−2832
Si Zhaofeng, Qi Honggang. Survey on knowledge distillation and its application[J]. Journal of Image and Graphics, 2023, 28(9): 2817−2832 (in Chinese)
27
Oord A, Li Yazhe, Vinyals O. Representation learning with contrastive predictive coding[J]. arXiv preprint, arXiv: 1807.03748, 2019
28
Bucila C, Caruana R, Niculescu-Mizil A. Model compression[C]//Proc of the 12th ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining. New York: ACM, 2006: 535−541
29
Sau B B, Balasubramanian V N. Deep model compression: Distilling knowledge from noisy teachers[J]. arXiv preprint, arXiv: 1610.09650, 2016
30
Zhang Ying, Xiang Tao, Hospedales T M, et al. Deep mutual learning[C]//Proc of 2018 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 4320−4328
31
Furlanello T, Lipton Z, Tschannen M, et al. Born again neural networks[C]//Proc of the 35th Int Conf on Machine Learning. New York: ACM, 2018: 1607−1616
32
Yang Chenglin, Xie Lingxi, Qiao Siyuan, et al. Training deep neural networks in generations: A more tolerant teacher educates better students[C]//Proc of the 33rd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 5628−5635
33
Lee H, Hwang S J, Shin J. Self-supervised label augmentation via input transformations[C]//Proc of the 37th Int Conf on Machine Learning. New York: ACM, 2020: 5714−5724
34
Yang Jing, Martinez B, Bulat A, et al. Knowledge distillation via softmax regression representation learning[C/OL]//Proc of 9th Int Conf on Learning Representations, New York: Ithaca, 2020[2024-08-20]. https://openreview.net/forum?id=ZzwDy_wiWv
35
Ba J, Caruana R. Do deep nets really need to be deep?[C]//Proc of the 28th Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2014: 2654−2662
36
Zhou Guorui, Fan Ying, Cui Runpeng, et al. Rocket launching: A universal and efficient framework for training well-performing light net[C]//Proc of the 32nd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2018: 4580−4587
37
Son W, Na J, Choi J, et al. Densely guided knowledge distillation using multiple teacher assistants[C]//Proc of 2021 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2021: 9375−9384
38
Mirzadeh S I, Farajtabar M, Li Ang, et al. Improved knowledge distillation via teacher assistant[C]//Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 5191−5198
39
Li Xuewei, Li Songyuan, Omar B, et al. ResKD: Residual-guided knowledge distillation[J]. IEEE Transactions on Image Processing, 2021, 30: 4735−4746
40
Romero A, Ballas N, Kahou S E, et al. FitNets: Hints for thin deep nets[J]. arXiv preprint, arXiv: 1412.6550, 2015
41
Zagoruyko S, Komodakis N. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer[J]. arXiv preprint, arXiv: 1612.03928, 2017
42
Lin Sihao, Xie Hongwei, Wang Bing, et al. Knowledge distillation via the target-aware transformer[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 10915−10924
43
Lao Shanshan, Song Guanglu, Liu Boxiao, et al. Masked autoencoders are stronger knowledge distillers[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 6361−6370
44
Guo Ziyao, Yan Haonan, Li Hui, et al. Class attention transfer based knowledge distillation[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 11868−11877
45
Zhang Yuan, Huang Tao, Liu Jiaming, et al. FreeKD: Knowledge distillation via semantic frequency prompt[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 15931−15940
46
Jia Menglin, Tang Luming, Chen B, et al. Visual prompt tuning[C]//Proc of the 17th European Conf on Computer Vision. Berlin: Springer, 2022: 709−727
47
Huang Zehao, Wang Naiyan. Like what you like: Knowledge distill via neuron selectivity transfer[J]. arXiv preprint, arXiv: 1707.01219, 2017
48
Heo B, Lee M, Yun S, et al. Knowledge transfer via distillation of activation boundaries formed by hidden neurons[C]//Proc of the 33rd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 3779−3787
49
Shen Chengchao, Wang Xinchao, Song Jie, et al. Amalgamating knowledge towards comprehensive classification[C]//Proc of the 33rd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 3068−3075
50
Wei Yi, Pan Xinyu, Qin Hongwei, et al. Quantization mimic: Towards very tiny CNN for object detection[C]//Proc of the 15th European Conf on Computer Vision. Berlin: Springer, 2018: 274−290
51
Ahn S, Hu S X, Damianou A, et al. Variational information distillation for knowledge transfer[C]//Proc of 2019 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 9155−9163
52
Heo B, Kim J, Yun S, et al. A comprehensive overhaul of feature distillation[C]//Proc of 2019 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 1921−1930
53
Yim J, Joo D, Bae J, et al. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning[C]//Proc of 2017 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2017: 7130−7138
54
Srinivas S, Fleuret F. Knowledge transfer with Jacobian matching[C]//Proc of the 35th Int Conf on Machine Learning. New York: ACM, 2018: 4730−4738
55
Kim J, Park S, Kwak N. Paraphrasing complex network: Network compression via factor transfer[C]//Proc of the 32nd Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2018: 2765−2774
56
Yang Chuanguang, An Zhulin, Cai Linhang, et al. Hierarchical self-supervised augmented knowledge distillation[C]//Proc of the 30th Int Joint Conf on Artificial Intelligence. San Francisco, CA: Morgan Kaufmann, 2021: 1217−1223
57
Tian Yonglong, Krishnan D, Isola P. Contrastive representation distillation[J]. arXiv preprint, arXiv: 1910.10699, 2022
58
Chen Defang, Mei Jian Ping, Zhang Hailin, et al. Knowledge distillation with the reused teacher classifier[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 11923−11932
59
Wang Xiaobo, Fu Tianyu, Liao Shengcai, et al. Exclusivity-consistency regularized knowledge distillation for face recognition[C]//Proc of the 16th European Conf on Computer Vision. Berlin: Springer, 2020: 325−342
60
Chen Pengguang, Liu Shu, Zhao Hengshuang, et al. Distilling knowledge via knowledge review[C]//Proc of 2021 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2021: 5006−5015
61
You Shan, Xu Chang, Xu Chao, et al. Learning from multiple teacher networks[C]//Proc of the 23rd ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining. New York: ACM, 2017: 1285−1294
62
Zhang Chenrui, Peng Yuxin. Better and faster: Knowledge transfer from multiple self-supervised learning tasks via graph distillation for video classification[C]//Proc of the 27th Int Joint Conf on Artificial Intelligence. San Francisco, CA: Morgan Kaufmann, 2018: 1135−1141
63
Chen Defang, Mei Jianping, Zhang Yuan, et al. Cross-layer distillation with semantic calibration[C]//Proc of the 35th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2021: 7028−7036
64
Liu Li, Huang Qingle, Lin Sihao, et al. Exploring inter-channel correlation for diversity-preserved knowledge distillation[C]//Proc of 2021 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2021: 8251−8260
65
Jin Ying, Wang Jiaqi, Lin Dahua. Multi-level logit distillation[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 24276−24285
66
Chen Yuntao, Wang Naiyan, Zhang Zhaoxiang. DarkRank: Accelerating deep metric learning via cross sample similarities transfer[C]//Proc of the 32nd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2018: 2852−2859
67
Liu Yufan, Cao Jiajiong, Li Bing, et al. Knowledge distillation via instance relationship graph[C]//Proc of 2019 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 7089−7097
68
Peng Baoyun, Jin Xiao, Li Dongsheng, et al. Correlation congruence for knowledge distillation[C]//Proc of 2019 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 5006−5015
69
Tung F, Mori G. Similarity-preserving knowledge distillation[C]//Proc of 2019 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 1365−1374
70
Xu Haohang, Fang Jiemin, Zhang Xiaopeng, et al. Bag of instances aggregation boosts self-supervised distillation[J]. arXiv preprint, arXiv: 2107.01691, 2022
71
刘静, 郑铜亚, 郝沁汾. 图知识蒸馏综述: 算法分类与应用分析[J]. 软件学报, 2023, 35(2): 675−710
Liu Jing, Zheng Tongya, Hao Qinfen. Survey on knowledge distillation with graph: Algorithms classification and application analysis[J]. Journal of Software, 2023, 35(2): 675−710 (in Chinese)
72
Park W, Kim D, Lu Yan, et al. Relational knowledge distillation[C]//Proc of 2019 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 3967−3976
73
Ren Sucheng, Gao Zhengqi, Hua Tianyu, et al. Co-advise: Cross inductive bias distillation[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 16752−16761
74
Li Duo, Hu Jie, Wang Changhu, et al. Involution: Inverting the inherence of convolution for visual recognition[C]//Proc of 2021 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2021: 12316−12325
75
Zhao Borui, Song Renjie, Liang Jiajun. Cumulative spatial knowledge distillation for vision transformers[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 6123−6132
76
Liu Yufan, Cao Jiajiong, Li Bing, et al. Cross-architecture knowledge distillation[C]//Proc of the 16th Asian Conf on Computer Vision. Berlin: Springer, 2022: 179−195
77
Zhao Weisong, Zhu Xiangyu, He Zhixiang, et al. Cross-architecture distillation for face recognition[C]//Proc of the 31st ACM Int Conf on Multimedia. New York: ACM, 2023: 8076−8085
78
Hao Zhiwei, Guo Jianyuan, Han Kai, et al. One-for-all: Bridge the gap between heterogeneous architectures in knowledge distillation[C]//Proc of the 37th Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2023: 79570−79582
79
Li Guopeng, Wang Qiang, Yan Ke, et al. Fuse before transfer: Knowledge fusion for heterogeneous distillation[C]// Proc of 2025 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2025: 3445-3454
80
Ni Jianyuan, Tang Hao, Shang Yuzhang, et al. Adaptive cross-architecture mutual knowledge distillation[C]//Proc of the 18th Int Conf on Automatic Face and Gesture Recognition. Piscataway, NJ: IEEE, 2024: 1−5
81
Zheng Zhimeng, Huang Tao, Li Gongsheng, et al. Promoting CNNs with cross-architecture knowledge distillation for efficient monocular depth estimation[J]. arXiv preprint, arXiv: 2404.16386, 2024
82
Wu Kan, Zhang Jinnian, Peng Houwen, et al. TinyViT: Fast pretraining distillation for small vision transformers[C]//Proc of the 17th European Conf on Computer Vision. Berlin: Springer, 2022: 68−85
83
Zhang Jinnian, Peng Houwen, Wu Kan, et al. MiniViT: Compressing vision transformers with weight multiplexing[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 12135−12144
84
Yang Zhendong, Li Zhe, Zeng Ailing, et al. ViTKD: Practical guidelines for ViT feature knowledge distillation[J]. arXiv preprint, arXiv: 2209.02432, 2022
85
Zhao Tianyang, Singh K Y, Appalaraju S, et al. No head left behind–multi-head alignment distillation for transformers[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 7514−7524
86
Ren Sucheng, Wei Fangyun, Zhang Zheng, et al. TinyMIM: An empirical study of distilling MIM pre-trained models[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 3687−3697
87
Lin Han, Han Guangxing, Ma Jiawei, et al. Supervised masked knowledge distillation for few-shot transformers[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 19649−19659
88
Bai Yutong, Wang Zeyu, Xiao Junfei, et al. Masked autoencoders enable efficient knowledge distillers[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 24256−24265
89
He Kaiming, Chen Xinlei, Xie Saining, et al. Masked autoencoders are scalable vision learners[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 16000−16009
90
Zhao Zhiyu, Huang Bingkun, Xing Sen, et al. Asymmetric masked distillation for pre-training small foundation models[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 18516−18526
91
Ye Hanjia, Lu Su, Zhan Dechuan. Distilling cross-task knowledge via relationship matching[C]//Proc of 2020 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 12396−12405
92
Lê H Â, Pham M T. Self-training and multi-task learning for limited data: Evaluation study on object detection[C]//Proc of 2019 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 1003−1009
93
Li Deng, Wu Aming, Han Yahong, et al. Prototype-guided cross-task knowledge distillation for large-scale models[J]. arXiv preprint, arXiv: 2212.13180, 2022
94
Auty D, Miles R, Kolbeinsson B, et al. Learning to project for cross-task knowledge distillation[J]. arXiv preprint, arXiv: 2403.14494, 2024
95
Yuan Mingkuan, Peng Yuxin. CKD: Cross-task knowledge distillation for text-to-image synthesis[J]. IEEE Transactions on Multimedia, 2020, 22(8): 1955−1968
96
Bayramli B, Ding Yue, Lu Hongtao. Integrating semantic segmentation model for self-supervised scene flow estimation via cross task distillation[C]//Proc of 2024 Int Joint Conf on Neural Networks. Piscataway, NJ: IEEE, 2024: 1−8
97
Wang Lin, Chae Y, Yoon S H, et al. EvDistill: Asynchronous events to end-task learning via bidirectional reconstruction-guided cross-modal knowledge distillation[C]//Proc of 2021 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2021: 608−619
98
Qian Biao, Wang Yang, Yin Hongzhi, et al. Switchable online knowledge distillation[C]//Proc of the 17th European Conf on Computer Vision. Berlin: Springer, 2022: 449−466
99
Yang Chuanguang, An Zhulin, Zhou Helong, et al. Online knowledge distillation via mutual contrastive learning for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(8): 10212−10227
100
Jacob G M, Agarwal V, Stenger B. Online knowledge distillation for multi-task learning[C]//Proc of 2023 IEEE/CVF Winter Conf on Applications of Computer Vision. Piscataway, NJ: IEEE, 2023: 2358−2367
101
Rao Jun, Meng Xv, Ding Liang, et al. Parameter-efficient and student-friendly knowledge distillation[J]. IEEE Transactions on Multimedia, 2024, 26: 4230−4241
102
Anil R, Pereyra G, Passos A, et al. Large scale distributed neural network training through online distillation[J]. arXiv preprint, arXiv: 1804.03235, 2020
103
Song Guocong, Chai Wei. Collaborative learning for deep neural networks[C]//Proc of the 32nd Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2018: 1837−1846
104
Chen Defang, Mei Jianping, Wang Can, et al. Online knowledge distillation with diverse peers[C]//Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 3430−3437
105
Gong Linrui, Lin Shaohui, Zhang Baochang, et al. Adaptive hierarchy-branch fusion for online knowledge distillation[C]//Proc of the 37th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023: 7731−7739
106
Wu Guile, Gong Shaogang. Peer collaborative learning for online knowledge distillation[C]//Proc of the 35th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2021: 10302−10310
107
Li Shaojie, Lin Mingbao, Wang Yan, et al. Distilling a powerful student model via online knowledge distillation[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 34(11): 8743−8752
108
Zhang Linfeng, Song Jiebo, Gao Anni, et al. Be your own teacher: Improve the performance of convolutional neural networks via self distillation[C]//Proc of 2019 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 3712−3721
109
Lan Xu, Zhu Xiatian, Gong Shaogang. Self-referenced deep learning[C]//Proc of the 14th Asian Conf on Computer Vision. Berlin: Springer, 2018: 284−300
110
Yang Chenglin, Xie Lingxi, Su Chi, et al. Snapshot distillation: Teacher-student optimization in one generation[C]//Proc of 2019 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 2854−2863
111
Mobahi H, Farajtabar M, Bartlett P L. Self-distillation amplifies regularization in Hilbert space[C]//Proc of the 34th Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2020: 3351−3361
112
Shen Yiqing, Xu Liwu, Yang Yuzhe, et al. Self-distillation from the last mini-batch for consistency regularization[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 11933−11942
113
Gou Jianping, Xiong Xiangshuo, Yu Baosheng, et al. Multi-target knowledge distillation via student self-reflection[J]. International Journal of Computer Vision, 2023, 131(7): 1857−1874
114
余鹰, 危伟, 汤洪, 等. 多层次知识自蒸馏联合多步骤训练的细粒度图像识别[J]. 计算机研究与发展, 2023, 60(8): 1834−1845
Yu Ying, Wei Wei, Tang Hong, et al. Multi-stage training with multi-level knowledge self-distillation for fine-grained image recognition[J]. Journal of Computer Research and Development, 2023, 60(8): 1834−1845 (in Chinese)
115
Lopes R G, Fenu S, Starner T. Data-free knowledge distillation for deep neural networks[J]. arXiv preprint, arXiv: 1710.07535, 2017
116
Nayak G K, Mopuri K R, Shaj V, et al. Zero-shot knowledge distillation in deep networks[C]//Proc of the 36th Int Conf on Machine Learning. New York: ACM, 2019: 4743−4751
117
Fang Gongfan, Mo Kanya, Wang Xinchao, et al. Up to 100× faster data-free knowledge distillation[C]//Proc of the 36th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2022: 6597−6604
118
Yu Shikang, Chen Jiachen, Han Hu, et al. Data-free knowledge distillation via feature exchange and activation region constraint[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 24266−24275
119
Li Jingru, Zhou Sheng, Li Liangcheng, et al. Dynamic data-free knowledge distillation by easy-to-hard learning strategy[J]. Information Sciences, 2023, 642: 119202
120
Wang Yang, Qian Biao, Liu Haipeng, et al. Unpacking the gap box against data-free knowledge distillation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 46(9): 6280−6291
121
张晶, 鞠佳良, 任永功. 基于双生成器网络的Data-Free知识蒸馏[J]. 计算机研究与发展, 2023, 60(7): 1615−1627
Zhang Jing, Ju Jialiang, Ren Yonggong. Double-generators network for data-free knowledge distillation[J]. Journal of Computer Research and Development, 2023, 60(7): 1615−1627 (in Chinese)
122
Chen Hanting, Wang Yunhe, Xu Chang, et al. Data-free learning of student networks[C]//Proc of 2019 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 3513−3521
123
Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets[C]//Proc of the 28th Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2014: 2672−2680
124
Fang Gongfan, Song Jie, Shen Chengchao, et al. Data-free adversarial distillation[J]. arXiv preprint, arXiv: 1912.11006, 2020
125
Luo Liangchen, Sandler M, Lin Zi, et al. Large-scale generative data-free distillation[J]. arXiv preprint, arXiv: 2012.05578, 2020
126
Qu Xiaoyang, Wang Jianzong, Xiao Jing. Enhancing data-free adversarial distillation with activation regularization and virtual interpolation[C]//Proc of 2021 IEEE Int Conf on Acoustics, Speech and Signal Processing. Piscataway, NJ: IEEE, 2021: 3340−3344
127
Micaelli P, Storkey A J. Zero-shot knowledge transfer via adversarial belief matching[C]//Proc of the 33rd Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2019: 9547−9557
128
Do K, Le T H, Nguyen D, et al. Momentum adversarial distillation: Handling large distribution shifts in data-free knowledge distillation[C]//Proc of the 36th Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2022: 10055−10067
129
Zhao Haoran, Sun Xin, Dong Junyu, et al. Dual discriminator adversarial distillation for data-free model compression[J]. International Journal of Machine Learning and Cybernetics, 2021, 13(5): 1213−1230
130
Shao Renrong, Zhang Wei, Wang Jun. Conditional pseudo-supervised contrast for data-free knowledge distillation[J]. Pattern Recognition, 2023, 143: 109781
131
Patel G, Mopuri K R, Qiu Qiang. Learning to retain while acquiring: Combating distribution-shift in adversarial data-free knowledge distillation[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 7786−7794
132
Fukuda T, Suzuki M, Kurata G, et al. Efficient knowledge distillation from an ensemble of teachers[C]//Proc of the 18th Annual Conf of the Int Speech Communication Association. Grenoble, FR: ISCA, 2017: 3697−3701
133
Tarvainen A, Valpola H. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[C]//Proc of the 31st Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2017: 1195−1204
134
Xiang Liuyu, Ding Guiguang, Han Jungong. Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification[C]//Proc of the 16th European Conf on Computer Vision. Berlin: Springer, 2020: 247−263
135
Nguyen L T, Lee K, Shim B. Stochasticity and skip connection improve knowledge transfer[C]//Proc of the 28th European Signal Processing Conf. Piscataway, NJ: IEEE, 2021: 1537−1541
136
Shi Luyao, Jiang Ning, Tang Jialiang, et al. Correlation guided multi-teacher knowledge distillation[C]//Proc of the 30th Int Conf on Neural Information Processing. Berlin: Springer, 2023: 562−574
137
Pham C, Hoang T, Do T T. Collaborative multi-teacher knowledge distillation for learning low bit-width deep neural networks[C]//Proc of 2023 IEEE/CVF Winter Conf on Applications of Computer Vision. Piscataway, NJ: IEEE, 2023: 6424−6432
138
Zhang Hailin, Chen Defang, Wang Can. Confidence-aware multi-teacher knowledge distillation[C]//Proc of 2022 IEEE Int Conf on Acoustics, Speech and Signal Processing. Piscataway, NJ: IEEE, 2022: 4498−4502
139
Cheng Xin, Zhang Zhiqiang, Weng Wei, et al. DE-MKD: Decoupled multi-teacher knowledge distillation based on entropy[J]. Mathematics, 2024, 12(11): 1672
140
Klingner M, Borse S, Kumar V R, et al. X3KD: Knowledge distillation across modalities, tasks and stages for multi-camera 3D object detection[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 13343−13353
141
Zhao Lingjun, Song Jingyu, Skinner Katherine A. CRKD: Enhanced camera-radar object detection with cross-modality knowledge distillation[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 15470−15480
142
Wang Yunhe, Xu Chang, Xu Chao, et al. Adversarial learning of portable student networks[C]//Proc of the 32nd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2018: 4260−4267
143
Passalis N, Tzelepi M, Tefas A. Heterogeneous knowledge distillation using information flow modeling[C]//Proc of 2020 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 2339−2348
144
Ganta D P, Gupta H D, Sheng V S. Knowledge distillation via weighted ensemble of teaching assistants[C]//Proc of 2021 IEEE Int Conf on Big Knowledge. Piscataway, NJ: IEEE, 2021: 30−37
145
Gao Mengya, Wang Yujun, Wan Liang. Residual error based knowledge distillation[J]. Neurocomputing, 2021, 433: 154−161
146
Morikawa T, Kameyama K. Multi-stage model compression using teacher assistant and distillation with hint-based training[C]//Proc of 2022 IEEE Int Conf on Pervasive Computing and Communications Workshops and other Affiliated Events. Piscataway, NJ: IEEE, 2022: 484−490
147
He Limuxuan, Hua Bei. Semi-supervised knowledge distillation via teaching assistant[J]. Highlights in Science, Engineering and Technology, 2023, 72: 429−436
148
Zhao Lei, Ng W W Y, Zhang Jianjun. Distillation of ensemble adaptive teacher assistant knowledge[J]. International Journal of Machine Learning and Cybernetics, 2025, 16(10): 8063-8078
149
Huo Fushuo, Xu Wenchao, Guo Jingcai, et al. C2KD: Bridging the modality gap for cross-modal knowledge distillation[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 16006−16015
150
Cho J H, Hariharan B. On the efficacy of knowledge distillation[C]//Proc of 2019 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 4793−4801
151
Yuan Li, Tay F E, Li Guilin, et al. Revisiting knowledge distillation via label smoothing regularization[C]//Proc of 2020 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 3902−3910
152
Stanton S, Izmailov P, Kirichenko P, et al. Does knowledge distillation really work?[C]//Proc of the 35th Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2021: 6906−6919
153
He Ruifei, Sun Shuyang, Yang Jihan, et al. Knowledge distillation as efficient pre-training: Faster convergence, higher data-efficiency, and better transferability[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 9151−9161
154
Shen Zhiqiang, Xing E. A fast knowledge distillation framework for visual recognition[C]//Proc of the 17th European Conf on Computer Vision. Berlin: Springer, 2022: 673−690
155
Song Jie, Chen Ying, Ye Jingwen, et al. Spot-adaptive knowledge distillation[J]. IEEE Transactions on Image Processing, 2022, 31: 3359−3370
156
Zhang Tianli, Xue Mengqi, Zhang Jiangtao, et al. Generalization matters: Loss minima flattening via parameter hybridization for efficient online knowledge distillation[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 20176−20185
157
Wang Jiabao, Chen Yuming, Zheng Zhaohui, et al. CrossKD: Cross-head knowledge distillation for object detection[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 16520−16530
158
Xu Guodong, Liu Ziwei, Li Xiaoxiao, et al. Knowledge distillation meets self-supervision[C]//Proc of the 16th European Conf on Computer Vision. Berlin: Springer, 2020: 588−604
159
Li Chenxin, Lin Mingbao, Ding Zhiyuan, et al. Knowledge condensation distillation[C]//Proc of the 17th European Conf on Computer Vision. Berlin: Springer, 2022: 19−35
160
Huang Tao, You Shan, Wang Fei, et al. Knowledge distillation from a stronger teacher[C]//Proc of the 36th Annual Conference on Neural Information Processing Systems. Cambridge, MA: MIT, 2022: 33716−33727
161
Choi J, Cho H, Cheung S, et al. ORC: Network group-based knowledge distillation using online role change[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 17335−17344
162
Shen Zhiqiang. FerKD: Surgical label adaptation for efficient distillation[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 1666−1675
163
Miles R, Elezi I, Deng Jiankang. VkD: Improving knowledge distillation using orthogonal projections[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 15720−15730
164
Miles R, Mikolajczyk K. Understanding the role of the projector in knowledge distillation[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 4233−4241
165
Sun Wujie, Chen Defang, Lyu Siwei, et al. Knowledge distillation with refined logits[J]. arXiv preprint, arXiv: 2408.07703, 2024
166
Xu Kunran, Rui Lai, Li Yishi, et al. Feature normalized knowledge distillation for image classification[C]//Proc of the 16th European Conf on Computer Vision. Berlin: Springer, 2020: 664−680
167
Jafari A, Rezagholizadeh M, Sharma P, et al. Annealing knowledge distillation[C]//Proc of the 16th Conf of the European Chapter of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2021: 2493−2504
168
Li Zheng, Li Xiang, Yang Lingfeng, et al. Curriculum temperature for knowledge distillation[C]//Proc of the 37th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023: 1504−1512
169
Chi Zhihao, Zheng Tu, Li Hengjia, et al. NormKD: Normalized logits for knowledge distillation[J]. arXiv preprint, arXiv: 2308.00520, 2023
170
Wei Yukang, Bai Yu. Dynamic temperature knowledge distillation[J]. arXiv preprint, arXiv: 2404.12711, 2024
171
Sun Shangquan, Ren Wenqi, Li Jingzhi, et al. Logit standardization in knowledge distillation[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 15731−15740
172
Zhao Borui, Cui Quan, Song Renjie, et al. Decoupled knowledge distillation[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 11943−11952
173
Yang Zhendong, Zeng Ailing, Li Zhe, et al. From knowledge distillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 17139−17148
174
Park H, Lee J S. Adaptive explicit knowledge transfer for knowledge distillation[J]. arXiv preprint, arXiv: 2409.01679, 2024
175
Zhao Borui, Cui Quan, Song Renjie, et al. DOT: A distillation-oriented trainer[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 6166−6175
176
Shen Zhiqiang, He Zhankui, Xue Xiangyang. MEAL: Multi-model ensemble via adversarial learning[C]//Proc of the 33rd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 4886−4893
177
Li Peng, Shu Chang, Xie Yuan, et al. Hierarchical knowledge squeezed adversarial network compression[C]//Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 11370−11377
178
Ma Haoyu, Chen Tianlong, Hu T K, et al. Undistillable: Making a nasty teacher that cannot teach students[J]. arXiv preprint, arXiv: 2105.07381, 2021
179
Heo B, Lee M, Yun S, et al. Knowledge distillation with adversarial samples supporting decision boundary[C]//Proc of the 33rd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 3771−3778
180
Nguyen-Duc T, Le T, Zhao He, et al. Adversarial local distribution regularization for knowledge distillation[C]//Proc of 2023 IEEE/CVF Winter Conf on Applications of Computer Vision. Piscataway, NJ: IEEE, 2023: 4670−4679
181
Bai Tao, Zhao Jun, Wen Bihan. Guided adversarial contrastive distillation for robust students[J]. IEEE Transactions on Information Forensics and Security, 2024, 19: 9643−9655
182
Huang Bo, Chen Mingyang, Wang Yi, et al. Boosting accuracy and robustness of student models via adaptive adversarial distillation[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 24668−24677
183
Yue Xinli, Mou Ningping, Wang Qian, et al. Revisiting adversarial robustness distillation from the perspective of robust fairness[C]//Proc of the 37th Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2023: 30390−30401
184
Wang Yuzheng, Chen Zhaoyu, Yang Dingkang, et al. Adversarial contrastive distillation with adaptive denoising[C]//Proc of 2023 IEEE Int Conf on Acoustics, Speech and Signal Processing, Piscataway, NJ: IEEE, 2023: 1−5
185
Li Changlin, Peng Jiefeng, Yuan Liuchun, et al. Block-wisely supervised neural architecture search with knowledge distillation[C]//Proc of 2020 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 1986−1995
186
Liu Yu, Jia Xuhui, Tan Mingxing, et al. Search to distill: Pearls are everywhere but not the eyes[C]//Proc of 2020 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 7536−7545
187
Kang M, Mun J, Han B. Towards oracle knowledge distillation with neural architecture search[C]//Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 4404−4411
188
Wei Zimian, Pan Hengyue, Li Lujun, et al. TVT: Training-free vision transformer search on tiny datasets[J]. arXiv preprint, arXiv: 2311.14337, 2023
189
Dong Peijie, Li Lujun, Wei Zimian. DisWOT: Student architecture search for distillation without training[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 11898−11908
190
Lu Liming, Chen Zhenghan, Lu Xiaoyu, et al. UniADS: Universal architecture-distiller search for distillation gap[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 14167−14174
191
Li Lujun, Dong Peijie, Wei Zimian, et al. Automated knowledge distillation via monte carlo tree search[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 17367−17378
192
Sohl-Dickstein J, Weiss E A, Maheswaranathan N, et al. Deep unsupervised learning using nonequilibrium thermodynamics[C]//Proc of the 32nd Int Conf on Machine Learning. New York: ACM, 2015: 2256−2265
193
Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models[C]//Proc of the 34th Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2020: 6840−6851
194
Huang Tao, Zhang Yuan, Zheng Mingkai, et al. Knowledge diffusion for distillation[C]//Proc of the 37th Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2023, 36: 65299−65316
195
Guo Qiushan, Mu Yao, Chen Jianyu, et al. Scale-equivalent distillation for semi-supervised object detection[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 14502−14511
196
Zhou Huajun, Qiao Bo, Yang Lingxiao, et al. Texture-guided saliency distilling for unsupervised salient object detection[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 7257−7267
197
Wang Luting, Liu Yi, Du Penghui, et al. Object-aware distillation pyramid for open-vocabulary object detection[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 11186−11196
198
Li Liangqi, Miao Jiaxu, Shi Dahu, et al. Distilling DETR with visual-linguistic knowledge for open-vocabulary object detection[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 6478−6487
199
Zhou Shengchao, Liu Weizhou, Hu Chen, et al. UniDistill: A universal cross-modality knowledge distillation framework for 3D object detection in bird’s-eye view[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 5116−5125
200
Wu Aming, Deng Cheng. Single-domain generalized object detection in urban scene via cyclic-disentangled self-distillation[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 837−846
201
Lao Shanshan, Song Guanglu, Liu Boxiao, et al. UniKD: Universal knowledge distillation for mimicking homogeneous or heterogeneous object detectors[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 6362−6372
202
Zeng Jia, Chen Li, Deng Hanming, et al. Distilling focal knowledge from imperfect expert for 3D object detection[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 992−1001
203
Liu Zhe, Ye Xiaoqing, Tan Xiao, et al. StereoDistill: Pick the cream from LiDAR for distilling stereo-based 3D object detection[C]//Proc of the 37th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023: 1790−1798
204
Wu Zizhang, Wu Yunzhe, Pu Jian, et al. Attention-based depth distillation with 3D-aware positional encoding for monocular 3D object detection[C]//Proc of the 37th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023: 2892−2900
205
Wang Zeyu, Li Dingwen, Luo Chenxu, et al. DistillBEV: Boosting multi-camera 3D object detection with cross-modal knowledge distillation[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 8603−8612
206
Zhang Yifan, Dong Zhen, Yang Huanrui, et al. QD-BEV: Quantization-aware view-guided distillation for multi-view 3D object detection[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 3802−3812
207
Zhao Haimei, Zhang Qiming, Zhao Shanshan, et al. SimDistill: Simulated multi-modal distillation for BEV 3D object detection[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 7460−7468
208
Li Yanjing, Xu Sheng, Lin Mingbao, et al. Representation disparity-aware distillation for 3D object detection[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 6692−6701
209
Gambashidze A, Dadukin A, Golyadkin M, et al. Weak-to-strong 3D object detection with X-ray distillation[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 15055−15064
210
Huang Xun, Wu Hai, Li Xin, et al. Sunshine to rainstorm: Cross-weather knowledge distillation for robust 3D object detection[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 2409−2416
211
Ji Deyi, Wang Haoran, Tao Mingyuan, et al. Structural and statistical texture knowledge distillation for semantic segmentation[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 16855−16864
212
Gao Huan, Guo Jichang, Wang Guoli, et al. Cross-domain correlation distillation for unsupervised domain adaptation in nighttime semantic segmentation[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 9903−9913
213
Yang Chuanguang, Zhou Helong, An Zhulin, et al. Cross-image relational knowledge distillation for semantic segmentation[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 12309−12318
214
Peng Bohao, Tian Zhuotao, Wu Xiaoyang, et al. Hierarchical dense correlation distillation for few-shot segmentation[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 23641−23651
215
Liu Yuang, Zhang Wei, Wang Jun. Self-decoupling and ensemble distillation for efficient segmentation[C]//Proc of the 37th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023: 1772−1780
216
Hou Yuenan, Zhu Xinge, Ma Yuexin, et al. Point-to-voxel knowledge distillation for LiDAR semantic segmentation[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 8469−8478
217
Chen Zisheng, Xu Hongbin, Chen Weitao, et al. PointDC: Unsupervised semantic segmentation of 3D point clouds via cross-modal distillation and super-voxel clustering[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 14244−14253
218
Xu Jianqing, Li Shen, Deng Ailin, et al. Probabilistic knowledge distillation of face ensembles[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 3489−3498
219
Li Jingzhi, Guo Zidong, Li Hui, et al. Rethinking feature-based knowledge distillation for face recognition[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 20156−20165
220
Yu Zhipeng, Liu Jiaheng, Qin Haoyu, et al. ICD-face: Intra-class compactness distillation for face recognition[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 20985−20995
221
Zhao Weisong, Zhu Xiangyu, Guo Kaiwen, et al. Grouped knowledge distillation for deep face recognition[C]//Proc of the 37th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023: 3615−3623
222
Zhang Zhenduo, Chen Yongru, Yang Wenming, et al. Pose-invariant face recognition via adaptive angular distillation[C]//Proc of the 36th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2022: 3390−3398
223
Hu Siyuan, Wang Zheng, Hu Peng, et al. PrefAce: Face-centric pretraining with self-structure aware distillation[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 12538−12546
224
Deng Hanqiu, Li Xingyu. Anomaly detection via reverse distillation from one-class embedding[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 9727−9736
225
Tien T D, Nguyen A T, Tran N H, et al. Revisiting reverse distillation for anomaly detection[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 24511−24520
226
Gu Zhihao, Liu Liang, Chen Xu, et al. Remembering normality: Memory-guided knowledge distillation for unsupervised anomaly detection[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 16355−16363
227
Binh L M, Woo S. ADD: Frequency attention and multi-view based knowledge distillation to detect low-quality compressed deepfake images[C]//Proc of the 36th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2022: 122−130
228
Ma Ye, Jiang Xu, Guan Nan, et al. Anomaly detection based on multi-teacher knowledge distillation[J]. Journal of Systems Architecture, 2023, 138: 102861
229
Ristea N C, Croitoru F A, Ionescu R T, et al. Self-distilled masked auto-encoders are efficient video anomaly detectors[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 15984−15995
230
邢鹏, 蒋鑫, 潘永华, 等. 基于特征约束蒸馏学习的视觉异常检测[J]. 软件学报, 2022, 34(9): 4378−4391
Xing Peng, Jiang Xin, Pan Yonghua, et al. Feature constrained restricted distillation learning for visual anomaly detection[J]. Journal of Software, 2022, 34(9): 4378−4391 (in Chinese)
231
Ju Chen, Zheng Kunhao, Liu Jinxiang, et al. Distilling vision-language pre-training to collaborate with weakly-supervised temporal action localization[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 14751−14762
232
Wu Mingrui, Gu Jiaxin, Shen Yunhang, et al. End-to-end zero-shot HOI detection via vision and language knowledge distillation[C]//Proc of the 37th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023: 2839−2846
233
Monti A, Porrello A, Calderara S, et al. How many observations are enough? Knowledge distillation for trajectory forecasting[C]//Proc of 2018 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 6543−6552
234
Ma Hengbo, Li Jiachen, Hosseini R, et al. Multi-objective diverse human motion prediction with knowledge distillation[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 8151−8161
235
Yang C, Kong K, Min S, et al. SEFD: Learning to distill complex pose and occlusion[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 14895−14906
236
Ye Suhang, Zhang Yingyi, Hu Jie, et al. DistilPose: Tokenized pose regression with heatmap distillation[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 2163−2172
237
Chen Sichen, Zhang Yingyi, Huang Siming, et al. SDPose: Tokenized pose estimation via circulation-guide self-distillation[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 1082−1090
238
Lu Yichen, Wang Mei, Deng Weihong. Augmented geometric distillation for data-free incremental person ReID[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 7319−7328
239
Shang Chao, Li Hongliang, Meng Fanman, et al. Incrementer: Transformer for class-incremental semantic segmentation with knowledge distillation focusing on old class[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 7214−7224
240
Phan M H, Ta T A, Phung S L, et al. Class similarity weighted knowledge distillation for continual semantic segmentation[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 16845−16854
241
Gu Peiyan, Zhang Chuyu, Xu Ruijie, et al. Class-relation knowledge distillation for novel class discovery[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 16428−16437
242
Li Jiyong, Azizov D, Li Yang, et al. Contrastive continual learning with importance sampling and prototype-instance relation distillation[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 13554−13562
243
Zhao Linglan, Lu Jing, Xu Yunlu, et al. Few-shot class-incremental learning via class-aware bilateral distillation[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 11838−11847
244
Yan Hongwei, Wang Liyuan, Ma Kaisheng, et al. Orchestrate latent expertise: Advancing online continual learning with multi-level supervision and reverse self-distillation[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 23670−23680
245
Ye Fei, Bors A G. Continual variational autoencoder via continual generative knowledge distillation[C]//Proc of the 37th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023: 10918−10926
246
Wen Haitao, Pan Lili, Dai Yu, et al. Class incremental learning with multi-teacher distillation[C]//Proc of 2024 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 28443−28452
247
Yang Ze, Li Ruibo, Ling Evan, et al. Label-guided knowledge distillation for continual semantic segmentation on 2D images and 3D point clouds[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 18555−18566
248
Zhang Lin, Shen Li, Ding Liang, et al. Fine-tuning global model via data-free knowledge distillation for non-IID federated learning[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 10164−10173
249
Shen Yiqing, Zhou Yuyin, Yu Lequan. CD2-pFed: Cyclic distillation-guided channel decoupling for model personalization in federated learning[C]//Proc of 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 10031−10040
250
Kim H, Kwak Y, Jung M, et al. ProtoFL: Unsupervised federated learning via prototypical distillation[C]//Proc of 2023 IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 6447−6456
251
Lu Yang, Chen Lin, Zhang Yonggang, et al. Federated learning with extremely noisy clients via negative distillation[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 14184−14192
252
Wang Haozhao, Li Yichen, Xu Wenchao, et al. DaFKD: Domain-aware federated knowledge distillation[C]//Proc of 2023 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 20412−20421
253
乐俊青, 谭州勇, 张迪, 等. 面向车联网数据持续共享的安全高效联邦学习[J]. 计算机研究与发展, 2024, 61(9): 2199−2212
Le Junqing, Tan Zhouyong, Zhang Di, et al. Secure and efficient federated learning for continuous IoV data sharing[J]. Journal of Computer Research and Development, 2024, 61(9): 2199−2212 (in Chinese)
2026年第63卷第1期
255
112
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202440694
  • 接收时间:2024-08-29
  • 首发时间:2026-11-06
  • 出版时间:2026-01-01
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2024-08-29
  • 修回日期:2025-04-29
  • 录用日期:2025-06-11
基金
作者信息
    哈尔滨工程大学计算机科学与技术学院 哈尔滨 150001

通讯作者:

参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202440694
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏