收藏切换
Visual Relationship Reasoning via Commonsense Guide and Multimodal Feature Fusion
收藏切换
Xuhan Wu, Shiliang Sun*
Journal of Computer Research and Development | 2026, 63(9) : 2285 - 2301
Less
收藏切换
Journal of Computer Research and Development | 2026, 63(9): 2285-2301
• Special Issue on Frontier Technologies and Applications of Knowledge-Enhanced Artificial Intelligence •
Visual Relationship Reasoning via Commonsense Guide and Multimodal Feature Fusion
Full
Xuhan Wu, Shiliang Sun*
Affiliations
  • School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai 200240
  • born in 2003. Master candidate. Her main research interests include scene graph generation and machine learning

    born in 1979. Professor. His main research interests include machine learning and artificial intelligence

About Author:

born in 2003. Master candidate. Her main research interests include scene graph generation and machine learning

born in 1979. Professor. His main research interests include machine learning and artificial intelligence

Published: 2026-09-01 doi: 10.7544/issn1000-1239.202660459
Outline
收藏切换

Efficient scene graph generation (SGG) serves as a bridge connecting low-level visual perception and high-level semantic reasoning, playing an important role in various vision-language tasks. However, existing dedicated SGG models often produce unreasonable relation predictions due to the lack of semantic constraint mechanisms. Although multimodal large language models (MLLMs) possess rich commonsense knowledge, their inherent black-box characteristics and severe hallucination issues lead to unreliable relationship reasoning results when directly applied to SGG tasks. To address these challenges, we propose KETR (commonsense knowledge embedding Transformer), an end-to-end scene graph generation model that incorporates commonsense knowledge embeddings. The model extracts global and local image features through a multi-scale visual feature fusion encoder and employs a parallel decoder based on learnable queries for entity detection. The core innovation lies in a soft alignment knowledge fusion mechanism that dynamically aggregates knowledge information through probabilistic mapping between visual features and external commonsense knowledge graph embedding spaces, and a commonsense-aware subject-object indicator and knowledge-driven scoring constraints are introduced, integrating the TransE scoring function as a regularization term into the joint training loss to significantly improve the logical consistency of relationship reasoning. Extensive experiments on Visual Genome and Open Images V6 demonstrate that KETR achieves competitive performance on multiple standard SGG metrics. Further ablation studies show that soft-aligned knowledge fusion improves the recognition of infrequent predicates, while the commonsense-aware subject-object indicator and knowledge-driven scoring constraint reduce relation predictions that conflict with entity roles, object affordances, and spatial semantics.

scene graph generation  /  commonsense knowledge graph  /  machine learning  /  relationship reasoning  /  Transformer
武栩含, 孙仕亮. 基于常识引导与多模态特征融合的视觉关系推理. 计算机研究与发展, 2026 , 63 (9) : 2285 -2301 . DOI: 10.7544/issn1000-1239.202660459
Xuhan Wu, Shiliang Sun. Visual Relationship Reasoning via Commonsense Guide and Multimodal Feature Fusion[J]. Journal of Computer Research and Development, 2026 , 63 (9) : 2285 -2301 . DOI: 10.7544/issn1000-1239.202660459
Year 2026 volume 63 Issue 9
88
43
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202660459
  • Receive Date:2026-06-06
  • Online Date:2026-11-17
  • Published:2026-09-01
Article Data
Affiliations
History
  • Received:2026-06-06
  • Revised:2026-07-17
Affiliations
    School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai 200240
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202660459
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT