收藏切换
A Multi-Feature Image Captioning Model with Common-Specific Collaborative Modeling and Mixture-of-Experts Fusion
收藏切换
Jianfeng Yang1, Dongpo Zhu2, Rui Cao1, *, Xuanhao Cui1, Ping Wang1, Jie Zheng1
Journal of Computer Research and Development | 2026, 63(11) : 3002 - 3016
Less
收藏切换
Journal of Computer Research and Development | 2026, 63(11): 3002-3016
• Artificial Intelligence •
A Multi-Feature Image Captioning Model with Common-Specific Collaborative Modeling and Mixture-of-Experts Fusion
Full
Jianfeng Yang1, Dongpo Zhu2, Rui Cao1, *, Xuanhao Cui1, Ping Wang1, Jie Zheng1
Affiliations
  • 1School of Computer Science, Northwest University, Xi’an 710127
  • 2Beijing Sankuai Technology Co., Ltd., Beijing 100190
  • born in 1981. Master, senior engineer, master supervisor. His main research interests include software development methodologies and data science, with a focus on applications in higher education

    born in 1999. Master, engineer. His main research interest includes multi-modal fusion and application

    born in 1991. PhD, lecturer, master supervisor. Member of CCF. His main research interests include text representation learning, applications and optimization of generative large language models, and art generation using diffusion models

    born in 2003. Master candidate. His main research interest includes large model development and application

    born in 2002. Master candidate. His main research interest includes large model visual reasoning

    born in 1985. PhD, professor, master supervisor. His main research interest includes edge general intelligence

About Author:

born in 1981. Master, senior engineer, master supervisor. His main research interests include software development methodologies and data science, with a focus on applications in higher education

born in 1999. Master, engineer. His main research interest includes multi-modal fusion and application

born in 1991. PhD, lecturer, master supervisor. Member of CCF. His main research interests include text representation learning, applications and optimization of generative large language models, and art generation using diffusion models

born in 2003. Master candidate. His main research interest includes large model development and application

born in 2002. Master candidate. His main research interest includes large model visual reasoning

born in 1985. PhD, professor, master supervisor. His main research interest includes edge general intelligence

Published: 2026-11-01 doi: 10.7544/issn1000-1239.202550936
Outline
收藏切换

Existing image captioning methods often struggle to simultaneously capture global semantics and fine-grained details, leading to overly generic descriptions or missing key information. To address this issue, we propose EAFCC, short for expert-aware fusion with common-specific collaboration, a multi-feature image captioning model with mixture-of-experts fusion. The model takes grid features, region features, and segmentation features as parallel inputs. In the encoding stage, a pre-fusion module first performs implicit alignment of heterogeneous visual features and models cross-feature common information within a shared semantic space, while preserving the basis for feature-specific representations. A shared Transformer encoder is then employed to learn the common representations, whose consistency and discriminability are further enhanced by a contrastive learning constraint. Meanwhile, three independent encoding branches are used to model feature-specific representations, so as to retain fine-grained complementary information associated with different feature types. A gated routing mechanism inspired by mixture-of-experts is further introduced to dynamically weight and fuse the specific information. Finally, the enhanced common features and the fused specific features are collaboratively integrated, processed by a skip-connected encoder, and fed into a Transformer decoder to generate image captions. Experimental results on MS-COCO dataset show that EAFCC achieves competitive performance on the CIDEr-D metric and outperforms existing single-feature and dual-feature methods, demonstrating the effectiveness of common-specific collaborative modeling and dynamic multi-feature fusion for image captioning.

image captioning  /  multi-feature fusion  /  common-specific collaborative modeling  /  contrastive learning  /  Transformer
杨建锋, 朱东坡, 曹瑞, 崔选皓, 王平, 郑杰. 基于共性-特性协同建模与多专家融合的多特征图像描述模型. 计算机研究与发展, 2026 , 63 (11) : 3002 -3016 . DOI: 10.7544/issn1000-1239.202550936
Jianfeng Yang, Dongpo Zhu, Rui Cao, Xuanhao Cui, Ping Wang, Jie Zheng. A Multi-Feature Image Captioning Model with Common-Specific Collaborative Modeling and Mixture-of-Experts Fusion[J]. Journal of Computer Research and Development, 2026 , 63 (11) : 3002 -3016 . DOI: 10.7544/issn1000-1239.202550936
Year 2026 volume 63 Issue 11
102
49
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202550936
  • Receive Date:2025-12-24
  • Online Date:2026-11-17
  • Published:2026-11-01
Article Data
Affiliations
History
  • Received:2025-12-24
  • Revised:2026-07-23
Affiliations
    1School of Computer Science, Northwest University, Xi’an 710127
    2Beijing Sankuai Technology Co., Ltd., Beijing 100190
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202550936
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT