Article(id=1327018586058633959, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1327018536993665336, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202440522, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=research-article, receivedDate=1718553600000, receivedDateStr=2024-06-17, revisedDate=1733932800000, revisedDateStr=2024-12-12, acceptedDate=1736352000000, acceptedDateStr=2025-01-09, onlineDate=1794315485728, onlineDateStr=2026-11-10, pubDate=1753977600000, pubDateStr=2025-08-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1794315485728, onlineIssueDateStr=2026-11-10, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1794315485728, creator=13701087609, updateTime=1794315485728, updator=13701087609, issue=Issue{id=1327018536993665336, tenantId=1146029695717560320, journalId=1324759914146529478, year='2025', volume='62', issue='8', pageStart='1833', pageEnd='2124', issueExtLink='null', onlineDate='null', pubDate='1753977600000', pubDateStr='2025-08-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1794315474031, creator='13701087609', updateTime=1794376881014, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext={EN=IssueExt(id=1327276097470837643, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1327018536993665336, language=EN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=), CN=IssueExt(id=1327276097470837644, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1327018536993665336, language=CN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=)}, issueFiles=null, downloadFileDto=null}, startPage=1875, endPage=1883, ext={EN=ArticleExt(id=1327018586272543464, articleId=1327018586058633959, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=Open-Vocabulary Multi-Label Action Recognition Guided by LLM Knowledge, columnId=1325489891789910943, journalTitle=Journal of Computer Research and Development, columnName=Artificial Intelligence, runingTitle=null, highlight=null, articleAbstract=

Open-vocabulary multi-label action recognition tasks aim to identify various human actions in videos that are not seen during the training phase. Compared with traditional action recognition, this task is more practical as it closely mirrors real-world scenarios and has broader application prospects. However, it poses significant challenges in effectively generalizing models to unseen action categories. To address this issue, we propose an open-vocabulary multi-label action recognition method enhanced by the knowledge of large language models knowledge. This method extracts rich co-occurrence knowledge of action categories implicit in large language models and incorporates this co-occurrence knowledge into prompt learning of visual-language models, facilitating information transfer between base action classes and novel action classes to improve the recognition performance of novel action classes. We set up two ratios of base action classes to novel action classes in experiments, namely 3∶1 and 1∶1, represented as “75% seen” and “50% seen” respectively. Experimental results on AVA and MovieNet datasets show that compared with existing methods, when the base action classes are “75% seen”, our method improves the mAP metric for novel action recognition by 1.95% and 1.21% on AVA and MovieNet datasets, respectively. When faced with the more challenging scenario of “50% seen”, our method improves the mAP metric for novel action recognition by 2.59% and 1.06% on the two datasets, respectively.

, authors=Rongjiang Zhu1, 2, Yuheng Shi1, 2, Shuo Yang3, 4, Ziyi Wang1, 2, Xinxiao Wu1, 2, *, authorsList=Rongjiang Zhu, Yuheng Shi, Shuo Yang, Ziyi Wang, Xinxiao Wu, authorCompany=null, correspAuthors=Xinxiao Wu, authorNote=

born in 2001. Master candidate. His main research interest includes action recognition

born in 2000. Master. His main research interest includes action recognition

born in 1992. PhD, associate professor. His main research interest includes video grounding

born in 2001. Master candidate. His main research interests include domain adaptation and domain generalization

born in 1982. Professor, PhD supervisor. Member of CCF. Her main research interests include vision and language, machine learning, and video understanding

, correspAuthorsNote=null, copyrightStatement=Copyright © 2025 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1327018589686706941, articleId=1327018586058633959, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=大语言模型知识引导的开放域多标签动作识别, columnId=1325489892104483745, journalTitle=计算机研究与发展, columnName=人工智能, runingTitle=null, highlight=null, articleAbstract=

开放域多标签动作识别任务旨在对视频中训练阶段未见的人的多类动作进行识别. 相较于传统动作识别,该任务更适应实际场景,具有广泛的应用前景. 然而,开放域多标签动作识别具有很大的挑战性,需要将模型有效泛化到未见过的新动作类别. 为了解决此问题,提出大语言模型知识引导的开放域多标签动作识别方法. 该方法挖掘大语言模型蕴含的丰富的动作类别共现知识,并将共现知识嵌入视觉-语言模型的提示学习,实现基本动作类别(base action classes)与新动作类别(novel action classes)之间的信息传递,从而提升新类别的识别性能. 在实验中将基本动作类别和新动作类别的比例设置为3∶1和1∶1,分别表示为“75% 可见”和“50% 可见”. 在AVA和MovieNet数据集上的实验结果表明,相较于现有方法,当基本动作类别为“75%”时,该方法在2个数据集的新动作类别识别指标mAP上分别提升了1.95个百分点和1.21个百分点;当面临基本动作类别为“50%”的更困难场景时,提出的方法在这2个数据集上新动作类别识别指标mAP上分别提升了2.59个百分点和1.06个百分点.

, authors=朱荣江1, 2, 石语珩1, 2, 杨硕3, 4, 王子奕1, 2, 吴心筱1, 2, *, authorsList=朱荣江, 石语珩, 杨硕, 王子奕, 吴心筱, authorCompany=null, correspAuthors=吴心筱, authorNote=

2001年生. 硕士研究生. 主要研究方向为动作识别

2000年生. 硕士. 主要研究方向为动作识别

1992年生. 博士,副教授. 主要研究方向为时序视频定位

2001年生. 硕士研究生. 主要研究方向为领域适应、领域泛化

1982年生. 教授,博士生导师. CCF会员. 主要研究方向为视觉与语言、机器学习、图像视频理解

, correspAuthorsNote=
吴心筱()
, copyrightStatement=版权所有 © 2025 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=PHKoJxy7km8yopnL2LG3KQ==, magXml=ypb0V7Nn9DzEWiq+O6VXrQ==, pdfUrl=null, pdf=gRFfEkJPoW0iUVy0AS1xGw==, pdfFileSize=2442107, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=1igjcQGDdqX2phtB8EqV8Q==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=S3OZh90/5pw69a8CvD+h0g==, mapNumber=null, fund=null)}, authors=[Author(id=1327018590236160779, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=3220241447@bit.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327018590332629774, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018590236160779, language=EN, stringName=Rongjiang Zhu, firstName=Rongjiang, middleName=null, lastName=Zhu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081
2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081, bio={"content":"

born in 2001. Master candidate. His main research interest includes action recognition

"}, bioImg=null, bioContent=

born in 2001. Master candidate. His main research interest includes action recognition

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327018590408127247, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018590236160779, language=CN, stringName=朱荣江, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1北京理工大学计算机学院 北京 100081
2智能信息技术北京市重点实验室(北京理工大学) 北京 100081, bio={"img":"CAuv3TYRlY6aoiu0+dUrmw==","content":"

2001年生. 硕士研究生. 主要研究方向为动作识别

"}, bioImg=CAuv3TYRlY6aoiu0+dUrmw==, bioContent=

2001年生. 硕士研究生. 主要研究方向为动作识别

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327018589888033534, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=1, ext=[AuthorCompanyExt(id=1327018589896422143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589888033534, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081), AuthorCompanyExt(id=1327018589904810752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589888033534, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京理工大学计算机学院 北京 100081)]), AuthorCompany(id=1327018589971919617, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=2, ext=[AuthorCompanyExt(id=1327018589980308226, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589971919617, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081), AuthorCompanyExt(id=1327018589984502531, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589971919617, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2智能信息技术北京市重点实验室(北京理工大学) 北京 100081)])]), Author(id=1327018590492013329, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327018590575899412, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018590492013329, language=EN, stringName=Yuheng Shi, firstName=Yuheng, middleName=null, lastName=Shi, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081
2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081, bio={"content":"

born in 2000. Master. His main research interest includes action recognition

"}, bioImg=null, bioContent=

born in 2000. Master. His main research interest includes action recognition

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327018590651396885, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018590492013329, language=CN, stringName=石语珩, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1北京理工大学计算机学院 北京 100081
2智能信息技术北京市重点实验室(北京理工大学) 北京 100081, bio={"img":"J6Ugn7spo2cdXZie/VI7Bg==","content":"

2000年生. 硕士. 主要研究方向为动作识别

"}, bioImg=J6Ugn7spo2cdXZie/VI7Bg==, bioContent=

2000年生. 硕士. 主要研究方向为动作识别

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327018589888033534, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=1, ext=[AuthorCompanyExt(id=1327018589896422143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589888033534, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081), AuthorCompanyExt(id=1327018589904810752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589888033534, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京理工大学计算机学院 北京 100081)]), AuthorCompany(id=1327018589971919617, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=2, ext=[AuthorCompanyExt(id=1327018589980308226, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589971919617, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081), AuthorCompanyExt(id=1327018589984502531, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589971919617, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2智能信息技术北京市重点实验室(北京理工大学) 北京 100081)])]), Author(id=1327018590718505751, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327018590802391834, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018590718505751, language=EN, stringName=Shuo Yang, firstName=Shuo, middleName=null, lastName=Yang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, 4, address=3Shenzhen MSU-BIT University, Shenzhen, Guangdong 518172
4Guangdong Key Laboratory of Machine Perception and Intelligent Computing (Shenzhen MSU-BIT University), Shenzhen, Guangdong 518172, bio={"content":"

born in 1992. PhD, associate professor. His main research interest includes video grounding

"}, bioImg=null, bioContent=

born in 1992. PhD, associate professor. His main research interest includes video grounding

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327018590869500699, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018590718505751, language=CN, stringName=杨硕, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, 4, address=3深圳北理莫斯科大学 广东深圳 518172
4广东省智能感知与计算重点实验室(深圳北理莫斯科大学) 广东深圳 518172, bio={"img":"nsuG3Qr0VzHQ7Rs9Q1US9Q==","content":"

1992年生. 博士,副教授. 主要研究方向为时序视频定位

"}, bioImg=nsuG3Qr0VzHQ7Rs9Q1US9Q==, bioContent=

1992年生. 博士,副教授. 主要研究方向为时序视频定位

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327018590055805700, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=3, ext=[AuthorCompanyExt(id=1327018590060000005, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018590055805700, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3Shenzhen MSU-BIT University, Shenzhen, Guangdong 518172), AuthorCompanyExt(id=1327018590068388614, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018590055805700, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3深圳北理莫斯科大学 广东深圳 518172)]), AuthorCompany(id=1327018590131303175, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=4, ext=[AuthorCompanyExt(id=1327018590143886088, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018590131303175, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=4Guangdong Key Laboratory of Machine Perception and Intelligent Computing (Shenzhen MSU-BIT University), Shenzhen, Guangdong 518172), AuthorCompanyExt(id=1327018590152274697, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018590131303175, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=4广东省智能感知与计算重点实验室(深圳北理莫斯科大学) 广东深圳 518172)])]), Author(id=1327018590936609565, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327018591020495648, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018590936609565, language=EN, stringName=Ziyi Wang, firstName=Ziyi, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081
2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081, bio={"content":"

born in 2001. Master candidate. His main research interests include domain adaptation and domain generalization

"}, bioImg=null, bioContent=

born in 2001. Master candidate. His main research interests include domain adaptation and domain generalization

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327018591087604513, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018590936609565, language=CN, stringName=王子奕, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1北京理工大学计算机学院 北京 100081
2智能信息技术北京市重点实验室(北京理工大学) 北京 100081, bio={"img":"LPfnSKiLa+CP4J7v7f6glw==","content":"

2001年生. 硕士研究生. 主要研究方向为领域适应、领域泛化

"}, bioImg=LPfnSKiLa+CP4J7v7f6glw==, bioContent=

2001年生. 硕士研究生. 主要研究方向为领域适应、领域泛化

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327018589888033534, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=1, ext=[AuthorCompanyExt(id=1327018589896422143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589888033534, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081), AuthorCompanyExt(id=1327018589904810752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589888033534, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京理工大学计算机学院 北京 100081)]), AuthorCompany(id=1327018589971919617, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=2, ext=[AuthorCompanyExt(id=1327018589980308226, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589971919617, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081), AuthorCompanyExt(id=1327018589984502531, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589971919617, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2智能信息技术北京市重点实验室(北京理工大学) 北京 100081)])]), Author(id=1327018591171490595, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=wuxinxiao@bit.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1327018591246988070, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018591171490595, language=EN, stringName=Xinxiao Wu, firstName=Xinxiao, middleName=null, lastName=Wu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, *, address=1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081
2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081, bio={"content":"

born in 1982. Professor, PhD supervisor. Member of CCF. Her main research interests include vision and language, machine learning, and video understanding

"}, bioImg=null, bioContent=

born in 1982. Professor, PhD supervisor. Member of CCF. Her main research interests include vision and language, machine learning, and video understanding

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327018591347651367, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018591171490595, language=CN, stringName=吴心筱, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, *, address=1北京理工大学计算机学院 北京 100081
2智能信息技术北京市重点实验室(北京理工大学) 北京 100081, bio={"img":"K5uy5e3HlK09B5bCbXSi+Q==","content":"

1982年生. 教授,博士生导师. CCF会员. 主要研究方向为视觉与语言、机器学习、图像视频理解

"}, bioImg=K5uy5e3HlK09B5bCbXSi+Q==, bioContent=

1982年生. 教授,博士生导师. CCF会员. 主要研究方向为视觉与语言、机器学习、图像视频理解

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327018589888033534, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=1, ext=[AuthorCompanyExt(id=1327018589896422143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589888033534, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081), AuthorCompanyExt(id=1327018589904810752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589888033534, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京理工大学计算机学院 北京 100081)]), AuthorCompany(id=1327018589971919617, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=2, ext=[AuthorCompanyExt(id=1327018589980308226, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589971919617, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081), AuthorCompanyExt(id=1327018589984502531, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589971919617, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2智能信息技术北京市重点实验室(北京理工大学) 北京 100081)])])], keywords=[Keyword(id=1327018592618525481, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=EN, orderNo=1, keyword=open-vocabulary action recognition), Keyword(id=1327018592694022954, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=EN, orderNo=2, keyword=multi-label classification), Keyword(id=1327018592761131819, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=EN, orderNo=3, keyword=prompt learning), Keyword(id=1327018592815657772, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=EN, orderNo=4, keyword=large language model), Keyword(id=1327018592874378029, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=EN, orderNo=5, keyword=CLIP model), Keyword(id=1327018592937292590, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=CN, orderNo=1, keyword=开放域动作识别), Keyword(id=1327018592991818543, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=CN, orderNo=2, keyword=多标签分类), Keyword(id=1327018593084093232, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=CN, orderNo=3, keyword=提示学习), Keyword(id=1327018593151202097, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=CN, orderNo=4, keyword=大语言模型), Keyword(id=1327018593214116658, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=CN, orderNo=5, keyword=CLIP模型)], refs=[Reference(id=1327018594656957251, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Limin, Tong Zhan, Ji Bin, et al. TDN: Temporal difference networks for efficient action recognition[C]//Proc of the 30th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2021: 1895−1904, articleTitle=null, refAbstract=null), Reference(id=1327018594740843332, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Mengmeng, Xing Jiazheng, Liu Yong. Action-CLIP: A new paradigm for video action recognition[J]. arXiv preprint, arXiv: 2109.08472, 2021, articleTitle=null, refAbstract=null), Reference(id=1327018594816340805, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Munro J, Damen D. Multi-modal domain adaptation for fine-grained action recognition[C]//Proc of the 29th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 122−132, articleTitle=null, refAbstract=null), Reference(id=1327018594891838278, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Xiang, Zhang Shiwei, Qing Zhiwu, et al. Molo: Motion-augmented long-short contrastive learning for few-shot action recognition[C]//Proc of the 32nd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 18011−18021, articleTitle=null, refAbstract=null), Reference(id=1327018594958947143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=王冲,魏子令,陈曙晖. 基于自注意力机制的无边界应用动作识别方法[J]. 计算机研究与发展,2022,59(5):1092−1104, articleTitle=null, refAbstract=null), Reference(id=1327018595030250312, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Chong, Wei Ziling, Chen Shuhui. Action identification without bounds on applications based on self-attention mechanism[J]. Journal of Computer Research and Development, 2022, 59(5): 1092−1104 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1327018595097359177, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Mondal A, Nag S, Prada J M, et al. Actor-agnostic multi-label action recognition with multi-modal query[C]//Proc of the 19th IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 784−794, articleTitle=null, refAbstract=null), Reference(id=1327018595160273738, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Xie Wentao, Ren Guanghui, Liu Si. Video relation detection with trajectory-aware multi-modal features[C]//Proc of the 28th ACM Int Conf on Multimedia. New York: ACM, 2020: 4590−4594, articleTitle=null, refAbstract=null), Reference(id=1327018595235771211, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Nandwani P, Verma R. A review on sentiment analysis and emotion detection from text[J]. Social Network Analysis and Mining, 2021, 11(1): 81, articleTitle=null, refAbstract=null), Reference(id=1327018595307074380, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Lin Ji, Gan Chuang, Han Song. TSM: Temporal shift module for efficient video understanding[C]//Proc of the 28th IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 7083−7093, articleTitle=null, refAbstract=null), Reference(id=1327018595365794637, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Wu Chaoyuan, Feichtenhofer C, Fan Haoqi, et al. Long-term feature banks for detailed video understanding[C]//Proc of the 28th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 284−293, articleTitle=null, refAbstract=null), Reference(id=1327018595420320590, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//Proc of the 38th Int Conf on Machine Learning. New York: ACM, 2021: 8748−8763, articleTitle=null, refAbstract=null), Reference(id=1327018595491623759, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Gao Peng, Geng Shijie, Zhang Rrenrui, et al. Clip-adapter: Better vision-language models with feature adapters[J]. International Journal of Computer Vision, 2023, 12(6): 1−15, articleTitle=null, refAbstract=null), Reference(id=1327018595546149712, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Taojiannan, Zhu Yi, Xie Yusheng, et al. AIM: Adapting image models for efficient video action recognition[J]. arXiv preprint, arXiv: 2302.03024, 2023, articleTitle=null, refAbstract=null), Reference(id=1327018595613258577, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Tirupattur P, Duarte K, Rawat Y S, et al. Modeling multi-label action dependencies for temporal action localization[C]//Proc of the 30th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2021: 1460−1470, articleTitle=null, refAbstract=null), Reference(id=1327018595684561746, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Dai Xiyang, Singh B, Ng Jy H, et al. TAN: Temporal aggregation network for dense multi-label action recognition[C]///Proc of the 19th IEEE Winter Conf on Applications of Computer Vision (WACV). Piscataway, NJ: IEEE, 2019: 151−160, articleTitle=null, refAbstract=null), Reference(id=1327018595755864915, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Sozykin K, Protasov S, Khan A, et al. Multi-label class-imbalanced action recognition in hockey videos via 3D convolutional neural networks[C]//Proc of the 19th IEEE/ACIS Int Conf on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing. Piscataway, NJ: IEEE, 2018: 146−151, articleTitle=null, refAbstract=null), Reference(id=1327018595827168084, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Ji Shuiwang, Xu Wei, Yang Ming, et al. 3D convolutional neural networks for human action recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 35(1): 221−231, articleTitle=null, refAbstract=null), Reference(id=1327018595881694037, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Yao Guangle, Lei Tao, Zhong Jiandan. A review of convolutional-neural-network-based action recognition[J]. Pattern Recognition Letters, 2019, 18(6): 14−22, articleTitle=null, refAbstract=null), Reference(id=1327018595940414294, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Yanyi, Li Xinyu, Marsic I. Multi-label activity recognition using activity-specific features and activity correlations[C]//Proc of the 30th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2021: 14625−14635, articleTitle=null, refAbstract=null), Reference(id=1327018596007523159, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Ni Bolin, Peng Houwen, Chen Minghao, et al. Expanding language-image pretrained models for general video recognition[C]//Proc of the 17th IEEE/CVF European Conf on Computer Vision. Berlin: Springer, 2022: 1−18, articleTitle=null, refAbstract=null), Reference(id=1327018596062049112, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proc of the 31st Int Conf on Neural Information Processing Systems. Berlin: Springer, 2017, 6000−6010, articleTitle=null, refAbstract=null), Reference(id=1327018596120769369, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Gu Chunhui, Sun Chen, Ross D A, et al. AVA: A video dataset of spatio-temporally localized atomic visual actions[C]//Proc of the 27th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 6047−6056, articleTitle=null, refAbstract=null), Reference(id=1327018597785908058, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Qingqiu, Xiong Yu, Rao Anyi, et al. MovieNet: A holistic dataset for movie understanding[C]//Proc of the 16th IEEE/CVF European Conf on Computer Vision. Berlin: Springer, 2020: 709−727, articleTitle=null, refAbstract=null), Reference(id=1327018597840434011, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv preprint, arXiv: 2010.11929, 2020, articleTitle=null, refAbstract=null), Reference(id=1327018597907542876, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=Kingma D, Ba J. Adam: A method for stochastic optimization[J]. arXiv preprint, arXiv: 1412.6980, 2014, articleTitle=null, refAbstract=null), Reference(id=1327018597970457437, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=26, authorNames=null, journalName=null, refType=null, unstructuredReference=Su Wanhua, Yuan Yan, Zhu Mu. A relationship between the average precision and the area under the roc curve[C]//Proc of the 5th Int Conf on the Theory of Information Retrieval. New York: ACM, 2015: 349−352, articleTitle=null, refAbstract=null), Reference(id=1327018598058537822, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=27, authorNames=null, journalName=null, refType=null, unstructuredReference=Sasaki Y. The truth of the F-measure[J/OL]. Teach Tutor Mater, 2007[2024-09-19]. https://www.researchgate.net/publication/268185911_The_truth_of_the_f-measure, articleTitle=null, refAbstract=null), Reference(id=1327018598159201119, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=28, rfOrder=28, authorNames=null, journalName=null, refType=null, unstructuredReference=Narayan S, Gupta A, Khan S, et al. Discriminative region-based multi-label zero-shot learning[C]//Proc of the 30th IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2021: 8731−8740, articleTitle=null, refAbstract=null), Reference(id=1327018598238892896, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=29, rfOrder=29, authorNames=null, journalName=null, refType=null, unstructuredReference=Kerrigan A, Duarte K, Rawat Y, et al. Reformulating zero-shot action recognition for multi-label actions[C]//Proc of the 35th IEEE/CVF Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2021: 25566−25577, articleTitle=null, refAbstract=null), Reference(id=1327018598314390369, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=30, rfOrder=30, authorNames=null, journalName=null, refType=null, unstructuredReference=Sun Ximeng, Hu Ping, Saenko K. DualCoOP: Fast adaptation to multi-label recognition with limited annotations[C]//Proc of the 36th IEEE/CVF Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2022: 30569−30582, articleTitle=null, refAbstract=null), Reference(id=1327018598385693538, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=31, rfOrder=31, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhou Kaiyang, Yang Jingkang, Loy C C, et al. Learning to prompt for vision-language models[J]. International Journal of Computer Vision, 2022, 130(9): 2337−2348, articleTitle=null, refAbstract=null), Reference(id=1327018598456996707, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=32, rfOrder=32, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhou Kaiyang, Yang Jingkang, Loy C C, et al. Conditional learning for vision-language models[C]//Proc of the 31st IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 16816−16825, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1327018589888033534, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=1, ext=[AuthorCompanyExt(id=1327018589896422143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589888033534, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081), AuthorCompanyExt(id=1327018589904810752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589888033534, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京理工大学计算机学院 北京 100081)]), AuthorCompany(id=1327018589971919617, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=2, ext=[AuthorCompanyExt(id=1327018589980308226, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589971919617, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081), AuthorCompanyExt(id=1327018589984502531, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589971919617, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2智能信息技术北京市重点实验室(北京理工大学) 北京 100081)]), AuthorCompany(id=1327018590055805700, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=3, ext=[AuthorCompanyExt(id=1327018590060000005, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018590055805700, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3Shenzhen MSU-BIT University, Shenzhen, Guangdong 518172), AuthorCompanyExt(id=1327018590068388614, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018590055805700, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3深圳北理莫斯科大学 广东深圳 518172)]), AuthorCompany(id=1327018590131303175, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=4, ext=[AuthorCompanyExt(id=1327018590143886088, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018590131303175, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=4Guangdong Key Laboratory of Machine Perception and Intelligent Computing (Shenzhen MSU-BIT University), Shenzhen, Guangdong 518172), AuthorCompanyExt(id=1327018590152274697, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018590131303175, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=4广东省智能感知与计算重点实验室(深圳北理莫斯科大学) 广东深圳 518172)])], figs=[ArticleFig(id=1327018593323168563, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=EN, label=Figure 1, caption=Open-vocabulary multi-label action recognition architecture guided by LLM knowledge, figureFileSmall=L/9115Mm3HAyKCDeLJ5erg==, figureFileBig=1igjcQGDdqX2phtB8EqV8Q==, tableContent=null), ArticleFig(id=1327018593386083124, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=CN, label=图1, caption=大语言模型知识引导的开放域多标签动作识别架构, figureFileSmall=L/9115Mm3HAyKCDeLJ5erg==, figureFileBig=1igjcQGDdqX2phtB8EqV8Q==, tableContent=null), ArticleFig(id=1327018593633547061, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=EN, label=Figure 2, caption=Example of querying LLM, figureFileSmall=CKctDPb68Met03cNHPtrLA==, figureFileBig=csEAIGt3kAnSAONs1I0KZA==, tableContent=null), ArticleFig(id=1327018593704850230, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=CN, label=图2, caption=向大语言模型提问的示例, figureFileSmall=CKctDPb68Met03cNHPtrLA==, figureFileBig=csEAIGt3kAnSAONs1I0KZA==, tableContent=null), ArticleFig(id=1327018593776153399, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=EN, label=Figure 3, caption=Comparison of different categories with and without knowledge on AVA dataset, figureFileSmall=l/EbDf9YgjulJXJuOOtnSQ==, figureFileBig=7XHXwTD9M21hWTD+KVWDIw==, tableContent=null), ArticleFig(id=1327018593839067960, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=CN, label=图3, caption=在AVA数据集上有无知识在不同类别上的比较, figureFileSmall=l/EbDf9YgjulJXJuOOtnSQ==, figureFileBig=7XHXwTD9M21hWTD+KVWDIw==, tableContent=null), ArticleFig(id=1327018593901982521, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=EN, label=Figure 4, caption=Visualization of the action category prediction results with and without adding action co-occurrence knowledge, figureFileSmall=Cbh1CxWSE/fFfOdWV6PuFg==, figureFileBig=6SG8nbPmiIdGAmxd+GDCyw==, tableContent=null), ArticleFig(id=1327018593973285690, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=CN, label=图4, caption=添加和未添加动作共现知识动作分类预测结果可视化, figureFileSmall=Cbh1CxWSE/fFfOdWV6PuFg==, figureFileBig=6SG8nbPmiIdGAmxd+GDCyw==, tableContent=null), ArticleFig(id=1327018594052977467, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=EN, label=Table 1, caption=Compared with the State-of-the-Art Methods on AVA and MovieNet Datasets %, figureFileSmall=null, figureFileBig=null, tableContent=
类别划分方法AVA 数据集MovieNet 数据集
50% 可见75% 可见50% 可见75% 可见
mAPF1mAPF1mAPF1mAPF1
注:黑体数值表示最优结果.
基本动作BiAM42.2425.6736.2027.1435.1823.9630.8723.51
Action-CLIP40.1129.6836.1426.3437.3224.7137.0726.87
PS-ZSAR45.4824.5639.8321.5239.5118.9736.7023.18
DualCoOp45.8429.8739.3428.4841.6224.6636.6323.83
CoOp47.2825.4742.5420.7843.5522.6443.4419.34
CoCoOp51.4226.6147.1225.5147.3826.1545.8227.86
MSQ-Net46.8726.9946.7725.7751.2225.8441.0324.03
本文方法52.7731.1452.6129.4251.9027.9147.3128.36
新动作BiAM35.9820.4938.2428.8326.8322.8932.5019.53
Action-CLIP33.3723.5529.6724.5433.9421.2734.3918.81
PS-ZSAR35.8418.3638.6123.8930.4518.3832.6620.41
DualCoOp35.4925.5139.0129.1330.0721.8233.8919.25
CoOp36.4321.5240.4422.7337.2516.6140.7323.71
CoCoOp38.6822.5141.7926.3739.4720.6241.6421.74
MSQ-Net37.3424.1940.3326.6033.7021.2938.0322.87
本文方法41.2726.8043.7430.7540.5323.8942.8524.11
), ArticleFig(id=1327018594132669244, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=CN, label=表1, caption=

在AVA数据集和MovieNet数据集上与现有的方法作对比

, figureFileSmall=null, figureFileBig=null, tableContent=
类别划分方法AVA 数据集MovieNet 数据集
50% 可见75% 可见50% 可见75% 可见
mAPF1mAPF1mAPF1mAPF1
注:黑体数值表示最优结果.
基本动作BiAM42.2425.6736.2027.1435.1823.9630.8723.51
Action-CLIP40.1129.6836.1426.3437.3224.7137.0726.87
PS-ZSAR45.4824.5639.8321.5239.5118.9736.7023.18
DualCoOp45.8429.8739.3428.4841.6224.6636.6323.83
CoOp47.2825.4742.5420.7843.5522.6443.4419.34
CoCoOp51.4226.6147.1225.5147.3826.1545.8227.86
MSQ-Net46.8726.9946.7725.7751.2225.8441.0324.03
本文方法52.7731.1452.6129.4251.9027.9147.3128.36
新动作BiAM35.9820.4938.2428.8326.8322.8932.5019.53
Action-CLIP33.3723.5529.6724.5433.9421.2734.3918.81
PS-ZSAR35.8418.3638.6123.8930.4518.3832.6620.41
DualCoOp35.4925.5139.0129.1330.0721.8233.8919.25
CoOp36.4321.5240.4422.7337.2516.6140.7323.71
CoCoOp38.6822.5141.7926.3739.4720.6241.6421.74
MSQ-Net37.3424.1940.3326.6033.7021.2938.0322.87
本文方法41.2726.8043.7430.7540.5323.8942.8524.11
), ArticleFig(id=1327018594208166717, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=EN, label=Table 2, caption=Effectiveness Verification Results of Relational Prompts and Standard Prompts on AVA Dataset %, figureFileSmall=null, figureFileBig=null, tableContent=
类别
划分
CS50% 可见75% 可见
mAPF1mAPF1
注:黑体数值表示最优结果. “C”代表关系提示,“S”代表标准提示. “√”表示含有相应模块,“×”表示去掉相应模块.
基本动作×√48.8227.6147.7425.43
√×51.7730.1648.3925.17
√√52.7731.1452.6129.42
新动作×√36.9224.3638.2427.59
√×37.3225.9440.1127.07
√√41.2726.8043.7430.75
), ArticleFig(id=1327018594283664190, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=CN, label=表2, caption=

在AVA数据集上关系提示和标准提示的有效性验证结果

, figureFileSmall=null, figureFileBig=null, tableContent=
类别
划分
CS50% 可见75% 可见
mAPF1mAPF1
注:黑体数值表示最优结果. “C”代表关系提示,“S”代表标准提示. “√”表示含有相应模块,“×”表示去掉相应模块.
基本动作×√48.8227.6147.7425.43
√×51.7730.1648.3925.17
√√52.7731.1452.6129.42
新动作×√36.9224.3638.2427.59
√×37.3225.9440.1127.07
√√41.2726.8043.7430.75
), ArticleFig(id=1327018594350773055, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=EN, label=Table 3, caption=Effectiveness Verification Results of KL Divergence %, figureFileSmall=null, figureFileBig=null, tableContent=
类别划分$ {\mathcal{L}_{{\text{KL}}}} $50% 可见75% 可见
mAPF1mAPF1
注:黑体数值表示最优结果;$ {\mathcal{L}_{{\text{KL}}}} $表示KL损失. “√”表示含有使用KL损失,“×”表示去掉KL损失.
基本动作×52.7130.8752.0726.58
√52.7731.1452.6129.42
新动作×39.8325.7442.9826.02
√41.2726.8043.7430.75
), ArticleFig(id=1327018594417881920, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=CN, label=表3, caption=

KL散度的有效性验证结果

, figureFileSmall=null, figureFileBig=null, tableContent=
类别划分$ {\mathcal{L}_{{\text{KL}}}} $50% 可见75% 可见
mAPF1mAPF1
注:黑体数值表示最优结果;$ {\mathcal{L}_{{\text{KL}}}} $表示KL损失. “√”表示含有使用KL损失,“×”表示去掉KL损失.
基本动作×52.7130.8752.0726.58
√52.7731.1452.6129.42
新动作×39.8325.7442.9826.02
√41.2726.8043.7430.75
), ArticleFig(id=1327018594484990785, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=EN, label=Table 4, caption=Experimental Results of Multiple Learned Marker Numbers on the Different Datasets %, figureFileSmall=null, figureFileBig=null, tableContent=
数据集类别划分K50% 可见75% 可见
mAPF1mAPF1
注:黑体数值表示最优结果.
AVA基本动作852.5730.8850.9529.05
1652.7731.1452.6129.42
3252.1129.5754.4330.74
新动作838.5623.4440.1426.87
1641.2726.8043.7430.75
3240.5226.5139.1627.30
MovieNet基本动作851.5726.4146.9223.53
1651.9027.9147.3128.36
3252.3428.0947.5328.66
新动作840.5622.3840.1423.57
1640.5323.8942.8524.11
3238.7121.0641.8923.90
), ArticleFig(id=1327018594573071170, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, language=CN, label=表4, caption=

不同数据集上多种学习标记数量的实验结果

, figureFileSmall=null, figureFileBig=null, tableContent=
数据集类别划分K50% 可见75% 可见
mAPF1mAPF1
注:黑体数值表示最优结果.
AVA基本动作852.5730.8850.9529.05
1652.7731.1452.6129.42
3252.1129.5754.4330.74
新动作838.5623.4440.1426.87
1641.2726.8043.7430.75
3240.5226.5139.1627.30
MovieNet基本动作851.5726.4146.9223.53
1651.9027.9147.3128.36
3252.3428.0947.5328.66
新动作840.5622.3840.1423.57
1640.5323.8942.8524.11
3238.7121.0641.8923.90
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202440522, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202440522, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202440522, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202440522, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1753977600000, fullTextJson=null, articleText=null, reference=null)
收藏切换
大语言模型知识引导的开放域多标签动作识别
朱荣江 1, 2 , 石语珩 1, 2 , 杨硕 3, 4 , 王子奕 1, 2 , 吴心筱 1, 2, *
计算机研究与发展 | 人工智能 2025,62(8): 1875-1883
收起
收藏切换
计算机研究与发展 |人工智能 2025 , 62 (8) : 1875 -1883
大语言模型知识引导的开放域多标签动作识别
全屏
[Author(id=1327018590236160779, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=3220241447@bit.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327018590332629774, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018590236160779, language=EN, stringName=Rongjiang Zhu, firstName=Rongjiang, middleName=null, lastName=Zhu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081
2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081, bio={"content":"

born in 2001. Master candidate. His main research interest includes action recognition

"}, bioImg=null, bioContent=

born in 2001. Master candidate. His main research interest includes action recognition

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327018590408127247, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018590236160779, language=CN, stringName=朱荣江, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1北京理工大学计算机学院 北京 100081
2智能信息技术北京市重点实验室(北京理工大学) 北京 100081, bio={"img":"CAuv3TYRlY6aoiu0+dUrmw==","content":"

2001年生. 硕士研究生. 主要研究方向为动作识别

"}, bioImg=CAuv3TYRlY6aoiu0+dUrmw==, bioContent=

2001年生. 硕士研究生. 主要研究方向为动作识别

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327018589888033534, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=1, ext=[AuthorCompanyExt(id=1327018589896422143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589888033534, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081), AuthorCompanyExt(id=1327018589904810752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589888033534, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京理工大学计算机学院 北京 100081)]), AuthorCompany(id=1327018589971919617, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=2, ext=[AuthorCompanyExt(id=1327018589980308226, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589971919617, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081), AuthorCompanyExt(id=1327018589984502531, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589971919617, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2智能信息技术北京市重点实验室(北京理工大学) 北京 100081)])]), Author(id=1327018590492013329, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327018590575899412, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018590492013329, language=EN, stringName=Yuheng Shi, firstName=Yuheng, middleName=null, lastName=Shi, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081
2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081, bio={"content":"

born in 2000. Master. His main research interest includes action recognition

"}, bioImg=null, bioContent=

born in 2000. Master. His main research interest includes action recognition

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327018590651396885, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018590492013329, language=CN, stringName=石语珩, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1北京理工大学计算机学院 北京 100081
2智能信息技术北京市重点实验室(北京理工大学) 北京 100081, bio={"img":"J6Ugn7spo2cdXZie/VI7Bg==","content":"

2000年生. 硕士. 主要研究方向为动作识别

"}, bioImg=J6Ugn7spo2cdXZie/VI7Bg==, bioContent=

2000年生. 硕士. 主要研究方向为动作识别

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327018589888033534, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=1, ext=[AuthorCompanyExt(id=1327018589896422143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589888033534, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081), AuthorCompanyExt(id=1327018589904810752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589888033534, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京理工大学计算机学院 北京 100081)]), AuthorCompany(id=1327018589971919617, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=2, ext=[AuthorCompanyExt(id=1327018589980308226, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589971919617, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081), AuthorCompanyExt(id=1327018589984502531, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589971919617, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2智能信息技术北京市重点实验室(北京理工大学) 北京 100081)])]), Author(id=1327018590718505751, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327018590802391834, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018590718505751, language=EN, stringName=Shuo Yang, firstName=Shuo, middleName=null, lastName=Yang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, 4, address=3Shenzhen MSU-BIT University, Shenzhen, Guangdong 518172
4Guangdong Key Laboratory of Machine Perception and Intelligent Computing (Shenzhen MSU-BIT University), Shenzhen, Guangdong 518172, bio={"content":"

born in 1992. PhD, associate professor. His main research interest includes video grounding

"}, bioImg=null, bioContent=

born in 1992. PhD, associate professor. His main research interest includes video grounding

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327018590869500699, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018590718505751, language=CN, stringName=杨硕, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=3, 4, address=3深圳北理莫斯科大学 广东深圳 518172
4广东省智能感知与计算重点实验室(深圳北理莫斯科大学) 广东深圳 518172, bio={"img":"nsuG3Qr0VzHQ7Rs9Q1US9Q==","content":"

1992年生. 博士,副教授. 主要研究方向为时序视频定位

"}, bioImg=nsuG3Qr0VzHQ7Rs9Q1US9Q==, bioContent=

1992年生. 博士,副教授. 主要研究方向为时序视频定位

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327018590055805700, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=3, ext=[AuthorCompanyExt(id=1327018590060000005, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018590055805700, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3Shenzhen MSU-BIT University, Shenzhen, Guangdong 518172), AuthorCompanyExt(id=1327018590068388614, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018590055805700, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=3深圳北理莫斯科大学 广东深圳 518172)]), AuthorCompany(id=1327018590131303175, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=4, ext=[AuthorCompanyExt(id=1327018590143886088, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018590131303175, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=4Guangdong Key Laboratory of Machine Perception and Intelligent Computing (Shenzhen MSU-BIT University), Shenzhen, Guangdong 518172), AuthorCompanyExt(id=1327018590152274697, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018590131303175, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=4广东省智能感知与计算重点实验室(深圳北理莫斯科大学) 广东深圳 518172)])]), Author(id=1327018590936609565, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327018591020495648, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018590936609565, language=EN, stringName=Ziyi Wang, firstName=Ziyi, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081
2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081, bio={"content":"

born in 2001. Master candidate. His main research interests include domain adaptation and domain generalization

"}, bioImg=null, bioContent=

born in 2001. Master candidate. His main research interests include domain adaptation and domain generalization

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327018591087604513, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018590936609565, language=CN, stringName=王子奕, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, address=1北京理工大学计算机学院 北京 100081
2智能信息技术北京市重点实验室(北京理工大学) 北京 100081, bio={"img":"LPfnSKiLa+CP4J7v7f6glw==","content":"

2001年生. 硕士研究生. 主要研究方向为领域适应、领域泛化

"}, bioImg=LPfnSKiLa+CP4J7v7f6glw==, bioContent=

2001年生. 硕士研究生. 主要研究方向为领域适应、领域泛化

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327018589888033534, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=1, ext=[AuthorCompanyExt(id=1327018589896422143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589888033534, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081), AuthorCompanyExt(id=1327018589904810752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589888033534, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京理工大学计算机学院 北京 100081)]), AuthorCompany(id=1327018589971919617, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=2, ext=[AuthorCompanyExt(id=1327018589980308226, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589971919617, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081), AuthorCompanyExt(id=1327018589984502531, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589971919617, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2智能信息技术北京市重点实验室(北京理工大学) 北京 100081)])]), Author(id=1327018591171490595, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=wuxinxiao@bit.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1327018591246988070, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018591171490595, language=EN, stringName=Xinxiao Wu, firstName=Xinxiao, middleName=null, lastName=Wu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, *, address=1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081
2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081, bio={"content":"

born in 1982. Professor, PhD supervisor. Member of CCF. Her main research interests include vision and language, machine learning, and video understanding

"}, bioImg=null, bioContent=

born in 1982. Professor, PhD supervisor. Member of CCF. Her main research interests include vision and language, machine learning, and video understanding

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327018591347651367, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, authorId=1327018591171490595, language=CN, stringName=吴心筱, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, *, address=1北京理工大学计算机学院 北京 100081
2智能信息技术北京市重点实验室(北京理工大学) 北京 100081, bio={"img":"K5uy5e3HlK09B5bCbXSi+Q==","content":"

1982年生. 教授,博士生导师. CCF会员. 主要研究方向为视觉与语言、机器学习、图像视频理解

"}, bioImg=K5uy5e3HlK09B5bCbXSi+Q==, bioContent=

1982年生. 教授,博士生导师. CCF会员. 主要研究方向为视觉与语言、机器学习、图像视频理解

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327018589888033534, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=1, ext=[AuthorCompanyExt(id=1327018589896422143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589888033534, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081), AuthorCompanyExt(id=1327018589904810752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589888033534, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京理工大学计算机学院 北京 100081)]), AuthorCompany(id=1327018589971919617, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, xref=2, ext=[AuthorCompanyExt(id=1327018589980308226, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589971919617, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081), AuthorCompanyExt(id=1327018589984502531, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018586058633959, companyId=1327018589971919617, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2智能信息技术北京市重点实验室(北京理工大学) 北京 100081)])])]
朱荣江1, 2 , 石语珩1, 2, 杨硕3, 4, 王子奕1, 2, 吴心筱1, 2, *
作者信息
  • 1北京理工大学计算机学院 北京 100081
  • 2智能信息技术北京市重点实验室(北京理工大学) 北京 100081
  • 3深圳北理莫斯科大学 广东深圳 518172
  • 4广东省智能感知与计算重点实验室(深圳北理莫斯科大学) 广东深圳 518172
通讯作者:
吴心筱()
作者简介:

2001年生. 硕士研究生. 主要研究方向为动作识别

2000年生. 硕士. 主要研究方向为动作识别

1992年生. 博士,副教授. 主要研究方向为时序视频定位

2001年生. 硕士研究生. 主要研究方向为领域适应、领域泛化

1982年生. 教授,博士生导师. CCF会员. 主要研究方向为视觉与语言、机器学习、图像视频理解

Open-Vocabulary Multi-Label Action Recognition Guided by LLM Knowledge
Rongjiang Zhu1, 2 , Yuheng Shi1, 2, Shuo Yang3, 4, Ziyi Wang1, 2, Xinxiao Wu1, 2, *
Affiliations
  • 1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081
  • 2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081
  • 3Shenzhen MSU-BIT University, Shenzhen, Guangdong 518172
  • 4Guangdong Key Laboratory of Machine Perception and Intelligent Computing (Shenzhen MSU-BIT University), Shenzhen, Guangdong 518172
  • born in 2001. Master candidate. His main research interest includes action recognition

    born in 2000. Master. His main research interest includes action recognition

    born in 1992. PhD, associate professor. His main research interest includes video grounding

    born in 2001. Master candidate. His main research interests include domain adaptation and domain generalization

    born in 1982. Professor, PhD supervisor. Member of CCF. Her main research interests include vision and language, machine learning, and video understanding

出版时间: 2025-08-01 doi: 10.7544/issn1000-1239.202440522
文章导航
收藏切换

开放域多标签动作识别任务旨在对视频中训练阶段未见的人的多类动作进行识别. 相较于传统动作识别,该任务更适应实际场景,具有广泛的应用前景. 然而,开放域多标签动作识别具有很大的挑战性,需要将模型有效泛化到未见过的新动作类别. 为了解决此问题,提出大语言模型知识引导的开放域多标签动作识别方法. 该方法挖掘大语言模型蕴含的丰富的动作类别共现知识,并将共现知识嵌入视觉-语言模型的提示学习,实现基本动作类别(base action classes)与新动作类别(novel action classes)之间的信息传递,从而提升新类别的识别性能. 在实验中将基本动作类别和新动作类别的比例设置为3∶1和1∶1,分别表示为“75% 可见”和“50% 可见”. 在AVA和MovieNet数据集上的实验结果表明,相较于现有方法,当基本动作类别为“75%”时,该方法在2个数据集的新动作类别识别指标mAP上分别提升了1.95个百分点和1.21个百分点;当面临基本动作类别为“50%”的更困难场景时,提出的方法在这2个数据集上新动作类别识别指标mAP上分别提升了2.59个百分点和1.06个百分点.

开放域动作识别  /  多标签分类  /  提示学习  /  大语言模型  /  CLIP模型

Open-vocabulary multi-label action recognition tasks aim to identify various human actions in videos that are not seen during the training phase. Compared with traditional action recognition, this task is more practical as it closely mirrors real-world scenarios and has broader application prospects. However, it poses significant challenges in effectively generalizing models to unseen action categories. To address this issue, we propose an open-vocabulary multi-label action recognition method enhanced by the knowledge of large language models knowledge. This method extracts rich co-occurrence knowledge of action categories implicit in large language models and incorporates this co-occurrence knowledge into prompt learning of visual-language models, facilitating information transfer between base action classes and novel action classes to improve the recognition performance of novel action classes. We set up two ratios of base action classes to novel action classes in experiments, namely 3∶1 and 1∶1, represented as “75% seen” and “50% seen” respectively. Experimental results on AVA and MovieNet datasets show that compared with existing methods, when the base action classes are “75% seen”, our method improves the mAP metric for novel action recognition by 1.95% and 1.21% on AVA and MovieNet datasets, respectively. When faced with the more challenging scenario of “50% seen”, our method improves the mAP metric for novel action recognition by 2.59% and 1.06% on the two datasets, respectively.

open-vocabulary action recognition  /  multi-label classification  /  prompt learning  /  large language model  /  CLIP model
朱荣江, 石语珩, 杨硕, 王子奕, 吴心筱. 大语言模型知识引导的开放域多标签动作识别. 计算机研究与发展, 2025 , 62 (8) : 1875 -1883 . DOI: 10.7544/issn1000-1239.202440522
Rongjiang Zhu, Yuheng Shi, Shuo Yang, Ziyi Wang, Xinxiao Wu. Open-Vocabulary Multi-Label Action Recognition Guided by LLM Knowledge[J]. Journal of Computer Research and Development, 2025 , 62 (8) : 1875 -1883 . DOI: 10.7544/issn1000-1239.202440522
视频中人的动作识别[1-5]是计算机视觉和人工智能领域备受关注的研究方向. 目前多数研究局限于单一动作分类、有限动作集的动作识别. 但在实际应用中,视频中的动作不受预定义标签的限制,随着场景和任务的动态变化,会出现新的多个动作类别,并可能在时间和空间上相互交叉和重叠. 本文研究开放域多标签动作识别[6],旨在识别视频中出现的以及包括训练时未见的各种动作,在关系检测[7]、情绪检测[8]、视频理解[9-10]中拥有广泛的应用前景.
近年来,具备零样本(zero-shot)学习能力的视觉-语言模型(例如CLIP[11])显著缩小了视觉和文本之间的语义差距,对齐了图像和文本的特征空间. 因其出色的泛化能力,越来越多的研究者将视觉-语言模型应用于动作识别领域. 目前大多数工作[2,12-13]通过学习提示或微调适配器等方式,匹配视频片段与文本描述以实现动作分类. 针对有多个动作同时发生的复杂视频,Mondal等人[6]则尝试将CLIP模型应用于多标签动作识别任务,编码每个动作类别标签并在视频特征中查询相应动作. 虽然这些方法通过应用CLIP,能将动作识别模型泛化至新的动作类别,但它们忽视了多类动作之间的共现关系,即不同动作在同一时间或同一场景中同时发生的情况. 当动作发生在复杂场景时,动作的共现关系对多标签动作识别具有更加重要的作用.
本文提出了一种大语言模型(large language model,LLM)引导的开放域多标签动作识别方法. 首先提问大语言模型以获取动作间的共现关系知识,并设计利用大语言模型知识的提示学习模块,然后通过结合动作共现关系的文本提示与视觉特征交互,在基本动作类别(base action classes)与新动作类别(novel action classes)间传递信息,进而实现开放域多标签动作识别. 具体来说,首先根据动作交互方式将所有动作分为人与物交互、人与人交互、人自身状态3个大类. 然后设计问题,如“please select {$ num $} actions that may occur simultaneously with {${a_i}$} in [action_list]j”,${a_i}$表示在动作集合中的第$i$个动作,[action_list]j为第j大类的所有动作,从大语言模型中获取动作共现关系知识. 为了避免大语言模型的不稳定所带来的知识噪声,设计多轮询问的策略减少噪声以提升知识的准确性. 最后,借助获得的知识,构建动作共现知识矩阵$\boldsymbol C \in {{(0,1)}^{{N} \times {N}}}$,其中$N$是动作类别的数量,$C{\text{(}}i,j) = 1$表示第$i$个动作与第$j$个动作之间有共现性,反之$C{\text{(}}i,j{{) = 0}}$则表示没有共现性.
在获得动作共现知识矩阵之后,本文设计了利用动作共现关系知识引导的提示学习模块,实现对训练时不可见的新动作类别的识别. 具体来说,本文设计2种本文提示:一种是基于大语言模型知识的关系提示,通过动作共现知识矩阵使模型在基本动作类别中获得的信息传递到新动作类别上,提升模型的泛化能力;另一种是基于固定模板,形如“a photo of [CLASS]”的标准提示,通过标准提示来约束关系提示,维持训练稳定的同时使关系提示更符合CLIP模型的嵌入空间,提升模型的特征提取能力.
本文的主要贡献包括3个方面:
1)提出了一种新颖的大语言模型知识引导文本提示学习的开放域多标签动作识别方法,设计提问从大语言模型获取动作共现关系知识,并利用动作共现知识引导提示学习,将基本动作类别信息传递到新动作类别,提高了模型的泛化能力.
2)提出了结合共现关系知识的提示学习模型,通过对基于大语言模型知识的关系提示和基于固定模板的标准提示的学习,提升了模型的特征提取能力.
3)在AVA和MovieNet这2个数据集上进行了大量实验. 实验结果表明,相较于现有方法,当基本动作类别为“75%”时,本文方法在AVA和 MovieNet数据集上的新动作识别指标mAP分别提升了1.95个百分点和1.21个百分点;当基本动作类别为“50%”的更困难设置时,本文方法在2个数据集上的新动作识别指标mAP分别提升了2.59 个百分点和1.06个百分点.
多标签动作识别[3,14-16]旨在识别视频中出现的多个动作. 早期工作[15-18]为每个动作类别训练一个二分类器. Dai等人[15]和Sozykin等人[16]使用3D卷积神经网络处理视频的时序信息,并使用多个二分类器集成神经网络作为分类器头来处理每个动作的分类. Tirupattur等人[14]构建一个基于注意力的多标签动作依赖性(multi-label action dependency,MLAD)层中建模动作之间的共现依赖性和时序依赖性. Zhang等人[19]为每个活动提取时空独立并且特定于活动的特征,并学习不同类别之间的活动关联图,以用于多标签识别.
与这些方法不同,本文关注开放域的多标签动作识别. 训练时,仅使用划分的基本动作类别,在测试时还需要识别出训练未见过的新动作类别. 相较于大多数闭集分类的研究工作,本文的工作更具有现实意义,因为在日常生活中,人们常常会面对更加错综复杂的情境和更加繁琐多样的动作.
视觉-语言预训练模型(如CLIP)在开放域图像识别领域展现出了卓越的性能. 近年来,研究人员开始将视觉-语言预训练模型引入开放域动作识别领域[2,12]. Wang等人[2]利用CLIP丰富的模型知识,通过添加时序融合层,在视频中达到零样本动作识别能力. Ni等人[20]提出帧间的注意力机制以获得视频帧之间的时序依赖信息,并将信息融入CLIP模型,使得视频特征和文本特征对齐. 然而,上述工作解决的是单标签识别任务,本文工作关注的是多标签任务,更加贴合现实场景中的复杂动作识别需求.
Mondal等人[6]使用CLIP构建多模态语义查询网络并将多模态融合信息作为Transformer解码器[21]中的查询部分,生成用于多标签分类的表示. 本文使用从大语言模型提取的动作间共现知识融入提示学习,有助于提升CLIP在开放域多标签动作识别上的识别能力.
开放域多标签动作识别任务旨在识别包括训练时未见过的视频中多种动作类别. 具体而言,所有动作类别被分为训练时可见的基本动作类别和训练时不可见的新动作类别. 模型在基本动作类别上进行训练,但在测试时需要识别包含新动作类别在内的所有动作类别.
本文提出了一种基于大语言模型知识引导的开放域多标签动作识别方法. 首先,通过设计问题向大语言模型询问动作与其他动作之间的共现关系,以引导关系提示学习的训练,其中关系提示通过联合共现动作与视觉特征交互,确保提示学习能够泛化到新的动作类别. 同时,引入标准提示来约束关系提示学习,进一步增强模型的特征提取能力. 如图1所示,本文提出的方法可以分为大语言模型知识获取模块和提示学习模块.
本文通过设计问题询问大语言模型获取动作共现知识. 首先根据动作的出现及动作交互方式不同,本文将动作分为了3个大类:人与物交互(例如“work on a computer”等)、人与人交互(例如“hand shake”等)、人自身状态(例如“walk”等). 基于这3个大类,本文设置了“50%可见”场景和“75%可见”场景进行实验,详细内容将在实验部分进行介绍. 在通过大语言模型获取动作共现知识时,为了使得大语言模型的回答不会偏向某一交互方式,本文按照大类针对每个动作分别向大语言模型提问,获取每个动作的共现动作集合. 这一过程表示为
$ \begin{split}&\{ A\} = \{ \{ {A_1}\} ,\{ {A_2}\} ,\{ {A_3}\} \} ,\\&{r_i} = \mathcal{Q}({a_i},{A_1}) + \mathcal{Q}({a_i},{A_2}) + \mathcal{Q}({a_i},{A_3}), \end{split} $
其中$A$表示数量为N的动作集合,${A_k}$表示第$k$个动作大类($k = 1,2,3$). $\mathcal{Q}$代表提问的模板,形如“please select {$num$} actions that may occur simultaneously with {${a_i}$} in ${A_k}$”,${a_i}$表示第$i$个动作,${r_i}$表示大语言模型对第$i$个动作类别输出的共现动作集合,图2展示了使用提问模板获取动作共现知识的示例.
基于从大语言模型中获取的共现关系知识$ r = \{ {r_1},\;{r_2},\;…,\;{r_N}\} $,本文提出了动作共现知识矩阵$ \boldsymbol C \in (0, 1)^{{N} \times {N}} $,其中$N$是动作类别的数量,$C{\text{(}}i,j{{) = 1}}$表示第$i$个动作与第$j$个动作之间有共现性,反之$C{\text{(}}i,j{{) = 0}}$则表示没有共现性. 所有动作类别包含基本动作类别和新动作类别. 对于新动作类别$C{\text{(}}i,j{\text{)}}$的值取决于动作$j$是否存在于$i$动作的共现关系知识${r_i}$中. 对于基本动作类别,本文从训练集真值中提取动作共现关系,并对每个动作选出共现频率最高的$k$个动作. 为了与大语言模型输出的动作数保持一致,$k=$10.
本文提出了一个结合动作共现关系知识的提示学习模块以适应开放域多标签动作识别. 该模块引入基于知识共现矩阵的可学习关系提示和固定的标准提示作为查询,并通过基于注意力机制的解码器,与视觉特征交互以实现多标签分类. 在训练过程中,将动作共现矩阵作为基本动作类别和新动作类别之间的关联,实现从基本动作类别到新动作类别的信息传递. 因此,利用共现的基本动作特征提高了对新动作的识别能力. 更具体地说,提示学习模块包括CLIP的文本和图像编码器、语义解码器、关系解码器和多标签分类器. 文本-图像编码器分别对文本提示和视频帧编码. 语义解码器将动作标签语义融合到视觉特征中,生成动作特定(action-specific)的语义特征. 关系解码器将动作共现知识融合到视觉特征中生成关系特征. 多标签分类器连接语义和关系特征进行预测,最后通过加入标准提示约束关系提示学习,提升模型特征提取能力.
1)图像-文本编码器. 本文利用CLIP的文本编码器$ {\varPsi _{t}}( \cdot ) $对可学习的关系提示和固定模板的标准提示编码. 本文所设置的关系提示为每一个动作设计数量为$ K $的学习标记(token)$ [{\boldsymbol{v}}] $. 固定提示模板${{\boldsymbol{T}}_{\text{l}}}$形如“a photo of [CLASS]”,通过计算KL散度约束关系提示,使其更符合CLIP模型的嵌入空间,并维持模型训练的稳定性. 给定固定模板的标准提示${{\boldsymbol{T}}_{\text{l}}}$和可学习的关系提示$ {{\boldsymbol{T}}_{\text{c}}} $,动作特定的标签嵌入${{\boldsymbol{E}}_{\text{l}}}$和关系嵌入${{\boldsymbol{E}}_{\text{c}}}$计算为:
$ \begin{split}&{{\boldsymbol{t}}_{i}} = {[{\boldsymbol{v}}]_1}{[{\boldsymbol{v}}]_2}…{[{\boldsymbol{v}}]_{K}}{[{\text{CLASS}}]_{i}}, \\&{{\boldsymbol{T}}_{\rm c}} = [{{\boldsymbol{t}}_1},{{\boldsymbol{t}}_2},…,{{\boldsymbol{t}}_{N}}], \\&{{\boldsymbol{E}}_{\rm l}} = {\varPsi _{t}}({{\boldsymbol{T}}_{\rm l}}), \\&{{\boldsymbol{E}}_{\rm c}} = {\varPsi _{t}}({{\boldsymbol{T}}_{\rm c}}).\end{split} $
给定视频$ \boldsymbol V \in {\mathbb{R}^{T \times 3 \times H \times W}} $,其中$H \times W$表示空间维度,$T$表示帧数,本文利用CLIP图像编码器$ {\varPsi _{\rm v}}( \cdot ) $将其编码为特征$ {{\boldsymbol{F}}_{\text{v}}} $:
$ {{\boldsymbol{F}}_{\text{v}}} = {\varPsi _{\rm v}}(V) \in {\mathbb{R}^{T \times d}} , $
其中$d = 512$表示嵌入维度. 另外,本文使用KL散度损失来约束关系提示和标准提示,表示为
$ {\mathcal{L}_{{\text{KL}}}} = {\phi _{{\text{KL}}}}({{\boldsymbol{E}}_{\rm l}},{{\boldsymbol{E}}_{\rm c}}) . $
2)语义解码器. 本文设计了语义解码器,以标签嵌入${{\boldsymbol{E}}_{\text{l}}}$作为查询(query),视觉特征$ {{\boldsymbol{F}}_{\text{v}}} $作为键(key)和值(value),利用注意力机制从动作帧中提取时空信息,获得用于分类的动作特定语义特征$ {{\boldsymbol{F}}_{\text{s}}} $. 语义解码器是一个掩码Transformer结构,包括多头自注意力(multi-head self-attention,MHSA)模块、多头交叉注意力(multi-head cross attention,MHCA)模块和前馈神经网络(feed-forward neural networds,FFN). FFN输出的语义特征$ {{\boldsymbol{F}}_{\text{s}}} $与标签嵌入在维度上保持一致,表示为
$ \begin{split}&{{\hat {\boldsymbol{Q}}}_{{\mathrm{s}}}} = {\mathcal{M}_{{\text{MHSA}}}}({{\boldsymbol{E}}_{\rm l}}), \\&{{\boldsymbol{Q}}_{{\mathrm{s}}}} = {\mathcal{M}_{{\text{MHCA}}}}({{\hat {\boldsymbol{Q}}}_{\rm s}},{{\boldsymbol{F}}_{\rm v}}), \\&{{\boldsymbol{F}}_{{\mathrm{s}}}} = {\mathcal{M}_{{\text{FFN}}}}({{\boldsymbol{Q}}_{\rm s}}). \end{split} $
3)关系解码器. 语义解码器有效地利用了CLIP的泛化能力,将视觉特征与其相应的动作类别标签对齐. 然而,在开放域动作识别任务中,我们认为还需要考虑到多标签动作依赖性,这种依赖性在共同出现的动作中至关重要. 为此,本文设计了一个关系解码器,它将动作共现知识矩阵${\boldsymbol{C}}$和关系嵌入${{\boldsymbol{E}}_{\text{c}}}$融合并作为查询,将视觉特征$ {{\boldsymbol{F}}_{\text{v}}} $作为键和值,通过交叉注意力机制,将动作共现知识整合到视觉特征中,以生成关系特征${{\boldsymbol{F}}_{\text{c}}}$. 通过矩阵乘法将动作共现知识矩阵${\boldsymbol{C}}$和关系嵌入${{\boldsymbol{E}}_{\text{c}}}$融合,使每个动作的嵌入与其共现动作的嵌入相加,形成一个复合查询. 这种方式能够学习共现动作之间的共享特征,并在反向传播的过程中共同更新共现动作提示的参数,实现这些动作之间的信息传递. 关系解码器包括一个多头交叉注意力模块和一个前馈网络,输出用于分类的关系特征${{\boldsymbol{F}}_{\text{c}}}$,表示为
$ \begin{split}&{{\boldsymbol{Q}}_{\text{c}}} = {\boldsymbol{C}} \times {{\boldsymbol{E}}_{\text{c}}}, \\&{{\boldsymbol{Q}}_{\text{c}}} = {\mathcal{M}_{{\text{MHCA}}}}({{\boldsymbol{Q}}_{\text{c}}},{{\boldsymbol{F}}_{\text{v}}}), \\&{{\boldsymbol{F}}_{\text{c}}} = {\mathcal{M}_{{\text{FFN}}}}({{\boldsymbol{Q}}_{\text{c}}}). \end{split} $
4)多标签分类器. 为了实现多标签动作分类,本文将语义特征$ {{\boldsymbol{F}}_{\text{s}}} $和关系特征${{\boldsymbol{F}}_{\rm c}}$连接起来形成最终动作特定的特征$ {\boldsymbol{F}} = [{{\boldsymbol{F}}_{\rm s}}|{{\boldsymbol{F}}_{\rm c}}] \in {\mathbb{R}^{{M} \times {2}{d}}} $,其中$[ \cdot | \cdot ]$表示连接操作. 本文使用sigmoid函数$\sigma $和全连接(full connected,FC)层在训练过程中,对$M$个基本类别进行预测,采用二元交叉熵(binary cross entropy,BCE)损失和KL损失作为训练目标,表示为
$ \begin{split}&{p_i} = \sigma ({\mathcal{M}_{{\text{FC}}}}({{{\boldsymbol{F}}}_i})),i \in \{1,2, …,M\}, \\&\mathcal{L} = \sum\limits_{i = 1}^M {{\phi _{{\text{BCE}}}}} ({y_i},{p_i}) + {\mathcal{L}_{{\text{KL}}}},\end{split} $
其中,${p_i}$表示被分类为第$i$个类别的预测概率,${y_i} \in \{ 0,1\} $表示真实类别标签.
本文在AVA[22]和MovieNet[23]两个电影动作数据集上对模型进行了评估,其中的电影场景和动作分布与日常生活更接近. AVA数据集包含了430个15 min的视频片段,标注了80种原始动作,如“sit”“stand”“walk”等. 本文使用180000个视频片段进行训练,使用50000个视频片段进行测试.
MovieNet数据集提供了大量的电影动作片段,并为视频片段中的每个人物标注了边界框,用于视频理解和目标检测任务. 为了将数据集应用到多标签动作识别任务上,本文定义了43个动作类别,标注了大约30部电影,并使用了其中15000个动作片段用于训练,3000个动作片段用于测试.
本文将动作分为人与物交互、人与人交互以及人自身状态三大类. 根据训练时是否可见,将所有的动作分为基本动作类别和新动作类别,并将它们的比例分别设置为3∶1和1∶1,表示为“75%可见”和“50%可见”. 在“75%可见”场景下,每个大类中排名前75%的动作为基本动作类别,后25%的动作为新动作类别;在“50%可见”场景下,前50%为基本动作类别,后50%为新动作类别. 通过这样划分基本动作类别和新动作类别是为了确保模型在训练过程中不会偏向某一个动作大类,并能够提取更为有效的动作共现知识. 这种处理方式有助于提升模型在开放域多标签动作识别任务中的泛化能力和准确性. 在训练时,模型使用基本动作类别训练. 在测试时,模型需要识别所有的动作类别,即计算在基本动作类别和新动作类别的识别性能. 本文使用CLIP ViT/B-16[24]中的视觉和文本编码器. 视频帧的分辨率设置为224$ \times $224,模型使用Adam[25]优化器,将学习率设定为0.000 003,总共进行50轮的训练. 本文采用的是与开放域多标签动作识别算法MSQ-Net[6]相同的评价指标,在每个数据批次计算mAP[26]和F1[27]分数,最终计算所有批次分数的平均值.
本文与目前适用开放域多标签动作识别的多种方法进行对比,包括Action-CLIP[2],BiAM[28], PS-ZSAR[29],DualCoOp[30],CoOp[31],CoCoOp[32],MSQ-Net[6].
表1展示了在AVA和MovieNet数据集上的对比实验结果. 从结果可以看出:本文方法在AVA和MovieNet数据集上的新动作类别都取得了最好的效果,这主要得益于动作共现知识引导的提示学习能够将基本动作类别学到的多标签动作共现信息有效传递给有共现关系的新动作类别. 此外,本文方法在基本动作类别上也取得了不错的结果,表明动作共现知识提示学习在促进多个类别间信息交互方面的有效性.
为验证本文方法中各个模块的有效性,以及在AVA数据集上验证关系提示和标准提示的有效性,设置了相应的消融实验. 结果如表2所示,可以看到,无论是去除关系提示还是去除标准提示,本文方法在基本动作类别和新动作类别上的性能都有所下降,这验证了关系提示和标准提示有效增强了模型在开放域下的动作识别能力.
本文还在AVA数据集上验证关系提示和标准提示之间的KL损失的作用,结果如表3所示. 当去除KL损失之后在基本动作类别和新动作类别上的性能都有所下降,这验证了KL损失的有效性. 图3展示了本文方法在去掉关系提示的情况下在AVA数据集上不同动作类别的性能比较. 可以看到,本文方法利用关系提示改善了在大多数类别上的识别性能,体现了共现关系的有效性. 对于一些动作类别,比如“take from”和“sail boat”,本文方法并没有达到预期的效果,这可能因为大语言模型中的动作共现知识和数据集中电影片段中的动作共现分布存在差异导致的.
为了探究关系提示中可学习标记数量$K$对于模型的影响,本文设计可学习标记数量$K=$8,16,32的参数实验.
表4展示了在2个数据集上不同可学习标记数量的实验结果. 可以看到,在给定的可学习标记下,当关系提示中的K=16时,模型能够达到相对较优的效果. 可能的原因是当学习的标记数量过少时,会限制模型学习到提示的表达能力,无法捕捉更细致的共现知识. 较多的可学习标记则增加了过拟合的风险,限制了模型的泛化能力.
图4展示了没有引入共现知识方法和本文方法排名前三的动作类别预测结果. 正如我们所预料,通过添加动作共现知识,模型实现了更好的性能,如图4(a)~(c)所示. 在图4 (c)中,没有引入动作共现知识的方法错误地将人物动作分类为“take a photo”,而本文方法成功估计出了“watch”这一动作,这是因为本文方法从“shoot”这一动作传递相关信息,提高了“watch”的预测分数,结果展示了动作共现知识的优势.
在图4(d)中展示了本文方法预测的一个失败示例,其中打架场景中的“fight/hit”“jump/leap”动作被错误地分类为“push”“take from”“bend/bow”,我们推测这是因为从大语言模型获取的知识会在引导关系提示学习时将“fight/hit”更多地和手部动作关联,使模型更倾向预测“take from”和“push”等动作,误导了关系提示学习. 在未来的工作中,需要持续关注大语言模型动作共现知识与数据集动作分布存在差异以及知识去噪等问题.
本文提出了大语言模型知识引导文本提示学习的开放域多标签动作识别方法. 本文通过设计提问从大语言模型获取动作共现知识,随后利用动作共现知识引导对CLIP模型的关系提示学习,在多标签开放域动作识别任务上取得了有效的进展. 在AVA和MovieNet数据集上大量的对比和消融实验验证了本文方法对动作识别的准确性和对各模块的有效性. 在未来工作中,我们将继续探究如何获得更加有效的动作关系知识以及如何处理知识中的噪声,探索不同的知识来源,提升模型对开放域动作的识别性能.
作者贡献声明:朱荣江负责设计与方法开发,监督整个研究进程,撰写论文及收集实验数据;石语珩负责方法分析与总结以及数据分析和结果解释,协助修订论文;杨硕实施实验与校对,协助修改论文;王子奕参与实验,协助数据处理;吴心筱监督整个实验过程,提供关键科学指导,全面修订和审阅论文.

参考文献 引证文献
排序方式:
1
Wang Limin, Tong Zhan, Ji Bin, et al. TDN: Temporal difference networks for efficient action recognition[C]//Proc of the 30th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2021: 1895−1904
2
Wang Mengmeng, Xing Jiazheng, Liu Yong. Action-CLIP: A new paradigm for video action recognition[J]. arXiv preprint, arXiv: 2109.08472, 2021
3
Munro J, Damen D. Multi-modal domain adaptation for fine-grained action recognition[C]//Proc of the 29th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 122−132
4
Wang Xiang, Zhang Shiwei, Qing Zhiwu, et al. Molo: Motion-augmented long-short contrastive learning for few-shot action recognition[C]//Proc of the 32nd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 18011−18021
5
王冲,魏子令,陈曙晖. 基于自注意力机制的无边界应用动作识别方法[J]. 计算机研究与发展,2022,59(5):1092−1104
Wang Chong, Wei Ziling, Chen Shuhui. Action identification without bounds on applications based on self-attention mechanism[J]. Journal of Computer Research and Development, 2022, 59(5): 1092−1104 (in Chinese)
6
Mondal A, Nag S, Prada J M, et al. Actor-agnostic multi-label action recognition with multi-modal query[C]//Proc of the 19th IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 784−794
7
Xie Wentao, Ren Guanghui, Liu Si. Video relation detection with trajectory-aware multi-modal features[C]//Proc of the 28th ACM Int Conf on Multimedia. New York: ACM, 2020: 4590−4594
8
Nandwani P, Verma R. A review on sentiment analysis and emotion detection from text[J]. Social Network Analysis and Mining, 2021, 11(1): 81
9
Lin Ji, Gan Chuang, Han Song. TSM: Temporal shift module for efficient video understanding[C]//Proc of the 28th IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 7083−7093
10
Wu Chaoyuan, Feichtenhofer C, Fan Haoqi, et al. Long-term feature banks for detailed video understanding[C]//Proc of the 28th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 284−293
11
Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//Proc of the 38th Int Conf on Machine Learning. New York: ACM, 2021: 8748−8763
12
Gao Peng, Geng Shijie, Zhang Rrenrui, et al. Clip-adapter: Better vision-language models with feature adapters[J]. International Journal of Computer Vision, 2023, 12(6): 1−15
13
Yang Taojiannan, Zhu Yi, Xie Yusheng, et al. AIM: Adapting image models for efficient video action recognition[J]. arXiv preprint, arXiv: 2302.03024, 2023
14
Tirupattur P, Duarte K, Rawat Y S, et al. Modeling multi-label action dependencies for temporal action localization[C]//Proc of the 30th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2021: 1460−1470
15
Dai Xiyang, Singh B, Ng Jy H, et al. TAN: Temporal aggregation network for dense multi-label action recognition[C]///Proc of the 19th IEEE Winter Conf on Applications of Computer Vision (WACV). Piscataway, NJ: IEEE, 2019: 151−160
16
Sozykin K, Protasov S, Khan A, et al. Multi-label class-imbalanced action recognition in hockey videos via 3D convolutional neural networks[C]//Proc of the 19th IEEE/ACIS Int Conf on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing. Piscataway, NJ: IEEE, 2018: 146−151
17
Ji Shuiwang, Xu Wei, Yang Ming, et al. 3D convolutional neural networks for human action recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 35(1): 221−231
18
Yao Guangle, Lei Tao, Zhong Jiandan. A review of convolutional-neural-network-based action recognition[J]. Pattern Recognition Letters, 2019, 18(6): 14−22
19
Zhang Yanyi, Li Xinyu, Marsic I. Multi-label activity recognition using activity-specific features and activity correlations[C]//Proc of the 30th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2021: 14625−14635
20
Ni Bolin, Peng Houwen, Chen Minghao, et al. Expanding language-image pretrained models for general video recognition[C]//Proc of the 17th IEEE/CVF European Conf on Computer Vision. Berlin: Springer, 2022: 1−18
21
Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proc of the 31st Int Conf on Neural Information Processing Systems. Berlin: Springer, 2017, 6000−6010
22
Gu Chunhui, Sun Chen, Ross D A, et al. AVA: A video dataset of spatio-temporally localized atomic visual actions[C]//Proc of the 27th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 6047−6056
23
Huang Qingqiu, Xiong Yu, Rao Anyi, et al. MovieNet: A holistic dataset for movie understanding[C]//Proc of the 16th IEEE/CVF European Conf on Computer Vision. Berlin: Springer, 2020: 709−727
24
Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv preprint, arXiv: 2010.11929, 2020
25
Kingma D, Ba J. Adam: A method for stochastic optimization[J]. arXiv preprint, arXiv: 1412.6980, 2014
26
Su Wanhua, Yuan Yan, Zhu Mu. A relationship between the average precision and the area under the roc curve[C]//Proc of the 5th Int Conf on the Theory of Information Retrieval. New York: ACM, 2015: 349−352
27
Sasaki Y. The truth of the F-measure[J/OL]. Teach Tutor Mater, 2007[2024-09-19]. https://www.researchgate.net/publication/268185911_The_truth_of_the_f-measure
28
Narayan S, Gupta A, Khan S, et al. Discriminative region-based multi-label zero-shot learning[C]//Proc of the 30th IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2021: 8731−8740
29
Kerrigan A, Duarte K, Rawat Y, et al. Reformulating zero-shot action recognition for multi-label actions[C]//Proc of the 35th IEEE/CVF Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2021: 25566−25577
30
Sun Ximeng, Hu Ping, Saenko K. DualCoOP: Fast adaptation to multi-label recognition with limited annotations[C]//Proc of the 36th IEEE/CVF Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2022: 30569−30582
31
Zhou Kaiyang, Yang Jingkang, Loy C C, et al. Learning to prompt for vision-language models[J]. International Journal of Computer Vision, 2022, 130(9): 2337−2348
32
Zhou Kaiyang, Yang Jingkang, Loy C C, et al. Conditional learning for vision-language models[C]//Proc of the 31st IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 16816−16825
2025年第62卷第8期
89
40
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202440522
  • 接收时间:2024-06-17
  • 首发时间:2026-11-10
  • 出版时间:2025-08-01
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2024-06-17
  • 修回日期:2024-12-12
  • 录用日期:2025-01-09
基金
作者信息
    1北京理工大学计算机学院 北京 100081
    2智能信息技术北京市重点实验室(北京理工大学) 北京 100081
    3深圳北理莫斯科大学 广东深圳 518172
    4广东省智能感知与计算重点实验室(深圳北理莫斯科大学) 广东深圳 518172

通讯作者:

吴心筱()
参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202440522
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏