收藏切换
Video Question Answering Scheme Base on Multimodal Knowledge Active Learning
收藏切换
Mingyang Liu, Ruomei Wang, Fan Zhou, Ge Lin*
Journal of Computer Research and Development | 2024, 61(4) : 889 - 902
Less
收藏切换
Journal of Computer Research and Development | 2024, 61(4): 889-902
• Artificial Intelligence •
Video Question Answering Scheme Base on Multimodal Knowledge Active Learning
Full
Mingyang Liu, Ruomei Wang, Fan Zhou, Ge Lin*
Affiliations
  • 1National Engineering Research Center of Digital Life, School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou 510006
  • born in 1994. PhD. His main research interests include video comprehension, video question answering, and machine learning

    born in 1961. PhD, professor, PhD supervisor. Her main research interests include computer graphics, computer-aided design, and multimedia processing

    born in 1976. PhD, professor, PhD supervisor. His main research interests include intelligent home systems, intelligent question answering, and the processing of cross-media data

    born in 1980. PhD, associate professor. His main research interests include intelligent home systems and artificial intelligence

About Author:

born in 1994. PhD. His main research interests include video comprehension, video question answering, and machine learning

born in 1961. PhD, professor, PhD supervisor. Her main research interests include computer graphics, computer-aided design, and multimedia processing

born in 1976. PhD, professor, PhD supervisor. His main research interests include intelligent home systems, intelligent question answering, and the processing of cross-media data

born in 1980. PhD, associate professor. His main research interests include intelligent home systems and artificial intelligence

Published: 2024-04-01 doi: 10.7544/issn1000-1239.202221008
Outline
收藏切换

Video question answering requires models to understand, fuse, and reason about the multimodal data in videos to assist people in quickly retrieving, analyzing, and summarizing complex scenes in videos, becoming a hot research topic in artificial intelligence. However, existing methods lack abilities of obtaining the motion details of visual objects in feature extraction, which may lead to false causality. In addition, in data fusion and reasoning, existing methods lack effective active learning ability, making it difficult to obtain prior knowledge beyond feature extraction, which affects the model’s deep understanding of multimodal content. To address these issues, we propose a multimodal knowledge-based active learning video question answering solution. The solution acquires the semantic correlation of visual targets in image sequences and the dynamic relationship with the surrounding environment to establish the motion trajectory of each visual target. Further, static content is supplemented with dynamic content to provide more accurate video feature expression for data fusion and reasoning. Then, the solution achieves self-improvement and logical thinking focus of multimodal information understanding through knowledge auto-enhancement multimodal data fusion and reasoning model, filling the gap in deep understanding of multimodal content. Experimental results show that the performance of our scheme is better than the most advanced video question answering algorithm, and a large number of ablation and visualization experiments also verify the rationality of this solution.

video question answering  /  data fusion and reasoning  /  multimodal active learning  /  video details description extraction  /  deep learning
刘明阳, 王若梅, 周凡, 林格. 基于多模态知识主动学习的视频问答方案. 计算机研究与发展, 2024 , 61 (4) : 889 -902 . DOI: 10.7544/issn1000-1239.202221008
Mingyang Liu, Ruomei Wang, Fan Zhou, Ge Lin. Video Question Answering Scheme Base on Multimodal Knowledge Active Learning[J]. Journal of Computer Research and Development, 2024 , 61 (4) : 889 -902 . DOI: 10.7544/issn1000-1239.202221008
Year 2024 volume 61 Issue 4
83
39
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202221008
  • Receive Date:2022-12-16
  • Online Date:2026-11-17
  • Published:2024-04-01
Article Data
Affiliations
History
  • Received:2022-12-16
  • Revised:2023-06-26
Affiliations
    1National Engineering Research Center of Digital Life, School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou 510006
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202221008
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT