收藏切换
An Image Captioning Method Based on DSC-Net
收藏切换
Zetao Jiang, Wencai Zhu*, Xin Jin, Peiqi Liao, Jingfan Huang
Journal of Computer Research and Development | 2024, 61(11) : 2897 - 2908
Less
收藏切换
Journal of Computer Research and Development | 2024, 61(11): 2897-2908
• Graphics and Image Processing •
An Image Captioning Method Based on DSC-Net
Full
Zetao Jiang, Wencai Zhu*, Xin Jin, Peiqi Liao, Jingfan Huang
Affiliations
  • Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004
  • born in 1961. PhD, professor. His main research interests include image processing, computer vision, and artificial intelligence

    born in 1999. Master. His main research interests include image processing and computer vision

    born in 1998. Master. His main research interests include image processing and computer vision

    born in 1995. Master. His main research interests include image processing and computer vision

    born in 1999. Master. His main research interests include image processing and computer vision

About Author:

born in 1961. PhD, professor. His main research interests include image processing, computer vision, and artificial intelligence

born in 1999. Master. His main research interests include image processing and computer vision

born in 1998. Master. His main research interests include image processing and computer vision

born in 1995. Master. His main research interests include image processing and computer vision

born in 1999. Master. His main research interests include image processing and computer vision

Published: 2024-11-01 doi: 10.7544/issn1000-1239.202330523
Outline
收藏切换

As visual features closer to the text domain, the grid features extracted by the CLIP (contrastive language-image pre-training) image encoder are easy to convert into the corresponding semantic natural language, which can alleviate the semantic gap problem, so it may become an important source of visual features in the image captioning in the future. However, this method does not consider that the division of image content may cause a complete object to be divided into several grids. The segmentation of the objects will inevitably lead to the lack of a complete expression of the object information in the feature extraction results, and further lead to the lack of an accurate expression of the object and the relationship between the objects in the generated sentence. Aiming at the phenomenon of grid features extracted by CLIP image encoder, we propose dual semantic collaborative network (DSC-Net) for image captioning. Specifically, dual semantic collaborative self-attention (DSCS) module is first proposed to enhance the expression of object information by CLIP grid features. Then dual semantic collaborative cross-attention (DSCC) module is proposed to integrate semantic information between grid and object to generate visual features, and to be used to predict sentences. Finally, dual semantic fusion (DSF) module is proposed to provide region-oriented fusion features for the above two semantic cooperation modules, and to solve the problem of correlation conflicts that may arise in the process of semantic cooperation. After a large number of experiments on the COCO dataset, the proposed model achieves a CIDEr score of 138.5% on the offline test set divided by Karpathy et al., and a CIDEr score of 137.6% in the official online test. Compared with the current mainstream image captioning methods, this result has obvious advantages.

image captioning  /  grid features  /  attention mechanism  /  dual semantic collaborative attention  /  dual semantic collaborative feature fusion
江泽涛, 朱文才, 金鑫, 廖培期, 黄景帆. 一种基于双重语义协作网络的图像描述方法. 计算机研究与发展, 2024 , 61 (11) : 2897 -2908 . DOI: 10.7544/issn1000-1239.202330523
Zetao Jiang, Wencai Zhu, Xin Jin, Peiqi Liao, Jingfan Huang. An Image Captioning Method Based on DSC-Net[J]. Journal of Computer Research and Development, 2024 , 61 (11) : 2897 -2908 . DOI: 10.7544/issn1000-1239.202330523
Year 2024 volume 61 Issue 11
77
38
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202330523
  • Receive Date:2023-06-19
  • Online Date:2026-11-17
  • Published:2024-11-01
Article Data
Affiliations
History
  • Received:2023-06-19
  • Revised:2023-12-19
Affiliations
    Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202330523
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT