收藏切换
Open-Vocabulary Multi-Label Action Recognition Guided by LLM Knowledge
收藏切换
Rongjiang Zhu1, 2, Yuheng Shi1, 2, Shuo Yang3, 4, Ziyi Wang1, 2, Xinxiao Wu1, 2, *
Journal of Computer Research and Development | 2025, 62(8) : 1875 - 1883
Less
收藏切换
Journal of Computer Research and Development | 2025, 62(8): 1875-1883
• Artificial Intelligence •
Open-Vocabulary Multi-Label Action Recognition Guided by LLM Knowledge
Full
Rongjiang Zhu1, 2, Yuheng Shi1, 2, Shuo Yang3, 4, Ziyi Wang1, 2, Xinxiao Wu1, 2, *
Affiliations
  • 1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081
  • 2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081
  • 3Shenzhen MSU-BIT University, Shenzhen, Guangdong 518172
  • 4Guangdong Key Laboratory of Machine Perception and Intelligent Computing (Shenzhen MSU-BIT University), Shenzhen, Guangdong 518172
  • born in 2001. Master candidate. His main research interest includes action recognition

    born in 2000. Master. His main research interest includes action recognition

    born in 1992. PhD, associate professor. His main research interest includes video grounding

    born in 2001. Master candidate. His main research interests include domain adaptation and domain generalization

    born in 1982. Professor, PhD supervisor. Member of CCF. Her main research interests include vision and language, machine learning, and video understanding

About Author:

born in 2001. Master candidate. His main research interest includes action recognition

born in 2000. Master. His main research interest includes action recognition

born in 1992. PhD, associate professor. His main research interest includes video grounding

born in 2001. Master candidate. His main research interests include domain adaptation and domain generalization

born in 1982. Professor, PhD supervisor. Member of CCF. Her main research interests include vision and language, machine learning, and video understanding

Published: 2025-08-01 doi: 10.7544/issn1000-1239.202440522
Outline
收藏切换

Open-vocabulary multi-label action recognition tasks aim to identify various human actions in videos that are not seen during the training phase. Compared with traditional action recognition, this task is more practical as it closely mirrors real-world scenarios and has broader application prospects. However, it poses significant challenges in effectively generalizing models to unseen action categories. To address this issue, we propose an open-vocabulary multi-label action recognition method enhanced by the knowledge of large language models knowledge. This method extracts rich co-occurrence knowledge of action categories implicit in large language models and incorporates this co-occurrence knowledge into prompt learning of visual-language models, facilitating information transfer between base action classes and novel action classes to improve the recognition performance of novel action classes. We set up two ratios of base action classes to novel action classes in experiments, namely 3∶1 and 1∶1, represented as “75% seen” and “50% seen” respectively. Experimental results on AVA and MovieNet datasets show that compared with existing methods, when the base action classes are “75% seen”, our method improves the mAP metric for novel action recognition by 1.95% and 1.21% on AVA and MovieNet datasets, respectively. When faced with the more challenging scenario of “50% seen”, our method improves the mAP metric for novel action recognition by 2.59% and 1.06% on the two datasets, respectively.

open-vocabulary action recognition  /  multi-label classification  /  prompt learning  /  large language model  /  CLIP model
朱荣江, 石语珩, 杨硕, 王子奕, 吴心筱. 大语言模型知识引导的开放域多标签动作识别. 计算机研究与发展, 2025 , 62 (8) : 1875 -1883 . DOI: 10.7544/issn1000-1239.202440522
Rongjiang Zhu, Yuheng Shi, Shuo Yang, Ziyi Wang, Xinxiao Wu. Open-Vocabulary Multi-Label Action Recognition Guided by LLM Knowledge[J]. Journal of Computer Research and Development, 2025 , 62 (8) : 1875 -1883 . DOI: 10.7544/issn1000-1239.202440522
Year 2025 volume 62 Issue 8
91
41
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202440522
  • Receive Date:2024-06-17
  • Online Date:2026-11-10
  • Published:2025-08-01
Article Data
Affiliations
History
  • Received:2024-06-17
  • Revised:2024-12-12
  • Accepted:2025-01-09
Affiliations
    1School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081
    2Beijing Key Laboratory of Intelligent Information Technology (Beijing Institute of Technology), Beijing 100081
    3Shenzhen MSU-BIT University, Shenzhen, Guangdong 518172
    4Guangdong Key Laboratory of Machine Perception and Intelligent Computing (Shenzhen MSU-BIT University), Shenzhen, Guangdong 518172
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202440522
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT