Article(id=1329460761270792884, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1329460564176253901, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202330523, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=null, receivedDate=1687104000000, receivedDateStr=2023-06-19, revisedDate=1702915200000, revisedDateStr=2023-12-19, acceptedDate=null, acceptedDateStr=null, onlineDate=1794897745672, onlineDateStr=2026-11-17, pubDate=1730390400000, pubDateStr=2024-11-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1794897745672, onlineIssueDateStr=2026-11-17, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1794897745672, creator=13701087609, updateTime=1794897745672, updator=13701087609, issue=Issue{id=1329460564176253901, tenantId=1146029695717560320, journalId=1324759914146529478, year='2024', volume='61', issue='11', pageStart='2663', pageEnd='2936', issueExtLink='null', onlineDate='null', pubDate='1730390400000', pubDateStr='2024-11-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1794897698682, creator='13701087609', updateTime=1794897698682, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext=null, issueFiles=null, downloadFileDto=null}, startPage=2897, endPage=2908, ext={EN=ArticleExt(id=1329460761472119477, articleId=1329460761270792884, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=An Image Captioning Method Based on DSC-Net, columnId=1327013252946572224, journalTitle=Journal of Computer Research and Development, columnName=Graphics and Image Processing, runingTitle=null, highlight=null, articleAbstract=

As visual features closer to the text domain, the grid features extracted by the CLIP (contrastive language-image pre-training) image encoder are easy to convert into the corresponding semantic natural language, which can alleviate the semantic gap problem, so it may become an important source of visual features in the image captioning in the future. However, this method does not consider that the division of image content may cause a complete object to be divided into several grids. The segmentation of the objects will inevitably lead to the lack of a complete expression of the object information in the feature extraction results, and further lead to the lack of an accurate expression of the object and the relationship between the objects in the generated sentence. Aiming at the phenomenon of grid features extracted by CLIP image encoder, we propose dual semantic collaborative network (DSC-Net) for image captioning. Specifically, dual semantic collaborative self-attention (DSCS) module is first proposed to enhance the expression of object information by CLIP grid features. Then dual semantic collaborative cross-attention (DSCC) module is proposed to integrate semantic information between grid and object to generate visual features, and to be used to predict sentences. Finally, dual semantic fusion (DSF) module is proposed to provide region-oriented fusion features for the above two semantic cooperation modules, and to solve the problem of correlation conflicts that may arise in the process of semantic cooperation. After a large number of experiments on the COCO dataset, the proposed model achieves a CIDEr score of 138.5% on the offline test set divided by Karpathy et al., and a CIDEr score of 137.6% in the official online test. Compared with the current mainstream image captioning methods, this result has obvious advantages.

, authors=Zetao Jiang, Wencai Zhu*, Xin Jin, Peiqi Liao, Jingfan Huang, authorsList=Zetao Jiang, Wencai Zhu, Xin Jin, Peiqi Liao, Jingfan Huang, authorCompany=null, correspAuthors=Wencai Zhu, authorNote=

born in 1961. PhD, professor. His main research interests include image processing, computer vision, and artificial intelligence

born in 1999. Master. His main research interests include image processing and computer vision

born in 1998. Master. His main research interests include image processing and computer vision

born in 1995. Master. His main research interests include image processing and computer vision

born in 1999. Master. His main research interests include image processing and computer vision

, correspAuthorsNote=null, copyrightStatement=Copyright © 2024 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1329460763506356941, articleId=1329460761270792884, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=一种基于双重语义协作网络的图像描述方法, columnId=1329460761551811254, journalTitle=计算机研究与发展, columnName=图形与图像处理, runingTitle=null, highlight=null, articleAbstract=

CLIP(contrastive language-image pre-training)视觉编码器提取的网格特征作为一种更加靠近文本域的视觉特征,具有易转化为对应语义自然语言的特点,可以缓解语义鸿沟问题,因而未来可能成为图像描述任务中视觉特征的重要来源. 但该方法中未考虑图像内容的划分,可能使一个完整的目标被划分到若干个网格中,目标被切割势必会导致特征提取结果中缺少对目标信息的完整表达,进而导致生成的描述语句中缺少对目标及目标间关系的准确表述. 针对CLIP视觉编码器提取网格特征这一现象,提出一种基于双重语义协作网络(dual semantic collaborative network,DSC-Net)的图像描述方法. 具体来说:首先提出双重语义协作自注意力(dual semantic collaborative self-attention,DSCS)模块增强CLIP网格特征对目标信息的表达能力;接着提出双重语义协作交叉注意力(dual semantic collaborative cross-attention,DSCC)模块,综合网格和目标2个层面的语义构造与文本相关的视觉特征,进行描述语句预测;最后提出双重语义融合(dual semantic fusion,DSF)模块,为上述的2个语义协作模块提供以区域为主导的融合特征,解决在语义协作过程中可能出现的相关性冲突问题. 经过在COCO数据集上的大量实验,提出的模型在Karpathy等人划分的离线测试集上取得了138.5%的CIDEr分数,在官方在线测试中取得了137.6%的CIDEr分数,与目前主流的图像描述方法相比具有显著优势.

, authors=江泽涛, 朱文才*, 金鑫, 廖培期, 黄景帆, authorsList=江泽涛, 朱文才, 金鑫, 廖培期, 黄景帆, authorCompany=null, correspAuthors=朱文才, authorNote=

1961年生. 博士,教授. 主要研究方向为图像处理、计算机视觉、人工智能

1999年生. 硕士. 主要研究方向为图像处理、计算机视觉

1998年生. 硕士. 主要研究方向为图像处理、计算机视觉

1995年生. 硕士. 主要研究方向为图像处理、计算机视觉

1999年生. 硕士. 主要研究方向为图像处理、计算机视觉

, correspAuthorsNote=
朱文才()
, copyrightStatement=版权所有 © 2024 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=czeHS/HWM2i5bWgZPPSUKg==, magXml=jDMAV2tmGnN45EzHSx95Kw==, pdfUrl=null, pdf=kVk4SlAj2YP5aCqe+zvmww==, pdfFileSize=1176303, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=yuHn60h1HZVsn/H+MZoVSg==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=KRzLwOuDYs+xEFHCl2kVfA==, mapNumber=null, fund=null)}, authors=[Author(id=1329460763783181010, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=zetaojiang@126.com, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329460763883844308, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460763783181010, language=EN, stringName=Zetao Jiang, firstName=Zetao, middleName=null, lastName=Jiang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004, bio={"content":"

born in 1961. PhD, professor. His main research interests include image processing, computer vision, and artificial intelligence

"}, bioImg=null, bioContent=

born in 1961. PhD, professor. His main research interests include image processing, computer vision, and artificial intelligence

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329460763950953173, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460763783181010, language=CN, stringName=江泽涛, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004, bio={"img":"K0PyxtGhzZbmPXwP+lNCXw==","content":"

1961年生. 博士,教授. 主要研究方向为图像处理、计算机视觉、人工智能

"}, bioImg=K0PyxtGhzZbmPXwP+lNCXw==, bioContent=

1961年生. 博士,教授. 主要研究方向为图像处理、计算机视觉、人工智能

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329460763711877838, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, xref=null, ext=[AuthorCompanyExt(id=1329460763716072143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004), AuthorCompanyExt(id=1329460763724460752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004)])]), Author(id=1329460764043227863, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=zhuwencai00@qq.com, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329460764110336729, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764043227863, language=EN, stringName=Wencai Zhu, firstName=Wencai, middleName=null, lastName=Zhu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004, bio={"content":"

born in 1999. Master. His main research interests include image processing and computer vision

"}, bioImg=null, bioContent=

born in 1999. Master. His main research interests include image processing and computer vision

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329460764177445594, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764043227863, language=CN, stringName=朱文才, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004, bio={"img":"RHbqdykSWeY1WKf2q2Ottw==","content":"

1999年生. 硕士. 主要研究方向为图像处理、计算机视觉

"}, bioImg=RHbqdykSWeY1WKf2q2Ottw==, bioContent=

1999年生. 硕士. 主要研究方向为图像处理、计算机视觉

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329460763711877838, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, xref=null, ext=[AuthorCompanyExt(id=1329460763716072143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004), AuthorCompanyExt(id=1329460763724460752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004)])]), Author(id=1329460764257137372, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329460764361994974, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764257137372, language=EN, stringName=Xin Jin, firstName=Xin, middleName=null, lastName=Jin, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004, bio={"content":"

born in 1998. Master. His main research interests include image processing and computer vision

"}, bioImg=null, bioContent=

born in 1998. Master. His main research interests include image processing and computer vision

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329460764424909535, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764257137372, language=CN, stringName=金鑫, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004, bio={"img":"jXy+WjkR0BEIeJJ5oTIg+g==","content":"

1998年生. 硕士. 主要研究方向为图像处理、计算机视觉

"}, bioImg=jXy+WjkR0BEIeJJ5oTIg+g==, bioContent=

1998年生. 硕士. 主要研究方向为图像处理、计算机视觉

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329460763711877838, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, xref=null, ext=[AuthorCompanyExt(id=1329460763716072143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004), AuthorCompanyExt(id=1329460763724460752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004)])]), Author(id=1329460764487824097, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329460764550738659, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764487824097, language=EN, stringName=Peiqi Liao, firstName=Peiqi, middleName=null, lastName=Liao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004, bio={"content":"

born in 1995. Master. His main research interests include image processing and computer vision

"}, bioImg=null, bioContent=

born in 1995. Master. His main research interests include image processing and computer vision

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329460764626236132, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764487824097, language=CN, stringName=廖培期, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004, bio={"img":"1Df8KRPAFXMKVJLjoVLcyg==","content":"

1995年生. 硕士. 主要研究方向为图像处理、计算机视觉

"}, bioImg=1Df8KRPAFXMKVJLjoVLcyg==, bioContent=

1995年生. 硕士. 主要研究方向为图像处理、计算机视觉

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329460763711877838, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, xref=null, ext=[AuthorCompanyExt(id=1329460763716072143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004), AuthorCompanyExt(id=1329460763724460752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004)])]), Author(id=1329460764689150694, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329460764777231080, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764689150694, language=EN, stringName=Jingfan Huang, firstName=Jingfan, middleName=null, lastName=Huang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004, bio={"content":"

born in 1999. Master. His main research interests include image processing and computer vision

"}, bioImg=null, bioContent=

born in 1999. Master. His main research interests include image processing and computer vision

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329460764856922857, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764689150694, language=CN, stringName=黄景帆, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004, bio={"img":"JML2tZfo7ovlsMlvYY7jqg==","content":"

1999年生. 硕士. 主要研究方向为图像处理、计算机视觉

"}, bioImg=JML2tZfo7ovlsMlvYY7jqg==, bioContent=

1999年生. 硕士. 主要研究方向为图像处理、计算机视觉

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329460763711877838, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, xref=null, ext=[AuthorCompanyExt(id=1329460763716072143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004), AuthorCompanyExt(id=1329460763724460752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004)])])], keywords=[Keyword(id=1329460764953391850, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, orderNo=1, keyword=image captioning), Keyword(id=1329460765028889323, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, orderNo=2, keyword=grid features), Keyword(id=1329460765100192492, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, orderNo=3, keyword=attention mechanism), Keyword(id=1329460765158912749, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, orderNo=4, keyword=dual semantic collaborative attention), Keyword(id=1329460765230215918, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, orderNo=5, keyword=dual semantic collaborative feature fusion), Keyword(id=1329460765309907695, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, orderNo=1, keyword=图像描述), Keyword(id=1329460765372822256, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, orderNo=2, keyword=网格特征), Keyword(id=1329460765431542513, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, orderNo=3, keyword=注意力机制), Keyword(id=1329460765494457074, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, orderNo=4, keyword=双重语义协作注意力), Keyword(id=1329460765548983027, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, orderNo=5, keyword=双重语义协作特征融合)], refs=[Reference(id=1329460767444808458, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=李志欣,魏海洋,张灿龙,等. 图像描述生成研究进展[J]. 计算机研究与发展,2021,58(9):1951−1974, articleTitle=null, refAbstract=null), Reference(id=1329460767507723019, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Zhixin, Wei Haiyang, Zhang Canlong, et al. Research progress on image captioning[J]. Journal of Computer Research and Development, 2021, 58(9): 1951−1974 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1329460767574831884, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proc of the 31st Advances in Neural Information Processing Systems. Cambridge, MA: MIT, 2017 : 5998−6008, articleTitle=null, refAbstract=null), Reference(id=1329460767646135053, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Vinyals O, Toshev A, Bengio S, et al. Show and tell: A neural image caption generator[C]//Proc of the 28th IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2015: 3156−3164, articleTitle=null, refAbstract=null), Reference(id=1329460767709049614, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Xu K, Ba J, Kiros R, et al. Show, attend and tell: Neural image caption generation with visual attention[C]//Proc of the 32nd Int Conf on Machine Learning. New York: ACM, 2015: 2048−2057, articleTitle=null, refAbstract=null), Reference(id=1329460767776158479, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Anderson P, He Xiaodong, Buehler C, et al. Bottom-up and top-down attention for image captioning and visual question answering[C]//Proc of the 31st IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 6077−6086, articleTitle=null, refAbstract=null), Reference(id=1329460767851655952, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//Proc of the 38th Int Conf on Machine Learning. New York: ACM, 2021: 8748−8763, articleTitle=null, refAbstract=null), Reference(id=1329460767935542033, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Guo Longteng, Liu Jing, Zhu Xinxin, et al. Normalized and geometry-aware self-attention network for image captioning[C]//Proc of the 33rd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 10327−10336, articleTitle=null, refAbstract=null), Reference(id=1329460768011039506, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Pan Yingwei, Yao Ting, Li Yehao, et al. X-linear attention networks for image captioning[C]//Proc of the 33rd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 10971−10980, articleTitle=null, refAbstract=null), Reference(id=1329460768099119891, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Shen Sheng, Li L H, Tan Hao, et al. How much can CLIP benefit vision-and-language tasks[C/OL]//Proc of the 10th Int Conf on Learning Representations. 2022[2022-01-29].https://openreview.net/forum?id=zf_Ll3HZWgy, articleTitle=null, refAbstract=null), Reference(id=1329460768178811668, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Yehao, Pan Yingwei, Yao Ting, et al. Comprehending and ordering semantics for image captioning[C]//Proc of the 35th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 17990−17999, articleTitle=null, refAbstract=null), Reference(id=1329460768241726229, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Karpathy A, Li Feifei. Deep visual-semantic alignments for generating image descriptions[C]//Proc of the 28th IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2015: 3128−3137, articleTitle=null, refAbstract=null), Reference(id=1329460768313029398, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Xu, Tang Kaihua, Zhang Hanwang, et al. Auto-encoding scene graphs for image captioning[C]//Proc of the 32nd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 10685−10694, articleTitle=null, refAbstract=null), Reference(id=1329460768375943959, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Cornia M, Stefanini M, Baraldi L, et al. Meshed-memory transformer for image captioning[C]//Proc of the 33rd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 10578−10587, articleTitle=null, refAbstract=null), Reference(id=1329460768472412952, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Ji Jiayi, Luo Yunpeng, Sun Xiaoshuai, et al. Improving image captioning by leveraging intra-and inter-layer global representation in transformer network[C]//Proc of the 35th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2021: 1655−1663, articleTitle=null, refAbstract=null), Reference(id=1329460768543716121, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Xu, Gao Chongyang, Zhang Hanwang, et al. Auto-parsing network for image captioning and visual question answering[C]//Proc of the 18th IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2021: 2197−2207, articleTitle=null, refAbstract=null), Reference(id=1329460768656962330, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Rennie S J, Marcheret E, Mroueh Y, et al. Self-critical sequence training for image captioning[C]//Proc of the 30th IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2017: 7008−7024, articleTitle=null, refAbstract=null), Reference(id=1329460768732459803, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Sharma P, Ding Nan, Goodman S, et al. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning[C]//Proc of the 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2018: 2556−2565, articleTitle=null, refAbstract=null), Reference(id=1329460768833123100, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Xuying, Sun Xiaoshuai, Luo Yunpeng, et al. RSTNet: Captioning with adaptive attention on visual and non-visual words[C]//Proc of the 34th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2021: 15465−15474, articleTitle=null, refAbstract=null), Reference(id=1329460768904426269, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Xuewen, Liu Yingru, Wang Xin. ReFormer: The relational transformer for image captioning[C]//Proc of the 30th ACM Int Conf on Multimedia. New York: ACM, 2022: 5398−5406, articleTitle=null, refAbstract=null), Reference(id=1329460768992506654, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Chi, Shen Yulin, Ji Luping. Geometry attention Transformer with position-aware LSTMs for image captioning[J]. Expert Systems with Applications, 2022, 201: 117174, articleTitle=null, refAbstract=null), Reference(id=1329460769076392735, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Lun, Wang Wenmin, Chen Jie, et al. Attention on attention for image captioning[C]//Proc of the 17th IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 4634−4643, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1329460763711877838, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, xref=null, ext=[AuthorCompanyExt(id=1329460763716072143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004), AuthorCompanyExt(id=1329460763724460752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004)])], figs=[ArticleFig(id=1329460765670617844, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Figure 1, caption=DSC-Net structure diagram, figureFileSmall=zr3zlPdue6/8op9up1axBQ==, figureFileBig=yuHn60h1HZVsn/H+MZoVSg==, tableContent=null), ArticleFig(id=1329460765762892533, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=图1, caption=DSC-Net结构图, figureFileSmall=zr3zlPdue6/8op9up1axBQ==, figureFileBig=yuHn60h1HZVsn/H+MZoVSg==, tableContent=null), ArticleFig(id=1329460765930664694, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Figure 2, caption=DSCS structure diagram, figureFileSmall=f+NsZvAxwwjI+URXJt1MlA==, figureFileBig=iCtrHsk/USSqgWXujTngaQ==, tableContent=null), ArticleFig(id=1329460765993579255, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=图2, caption=DSCS结构图, figureFileSmall=f+NsZvAxwwjI+URXJt1MlA==, figureFileBig=iCtrHsk/USSqgWXujTngaQ==, tableContent=null), ArticleFig(id=1329460766060688120, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Figure 3, caption=DSF structure diagram, figureFileSmall=Ga1qFQPOk50S6XoSQwxORA==, figureFileBig=HwqIxC/eqk/BSVCOKPHORg==, tableContent=null), ArticleFig(id=1329460766127796985, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=图3, caption=DSF结构图, figureFileSmall=Ga1qFQPOk50S6XoSQwxORA==, figureFileBig=HwqIxC/eqk/BSVCOKPHORg==, tableContent=null), ArticleFig(id=1329460766203294458, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Figure 4, caption=DSCC structure diagram, figureFileSmall=jOqRj60p2un9Y1ahSZpIiQ==, figureFileBig=LzdnFurNhE/xX8qSyo/CYg==, tableContent=null), ArticleFig(id=1329460766266209019, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=图4, caption=DSCC结构图, figureFileSmall=jOqRj60p2un9Y1ahSZpIiQ==, figureFileBig=LzdnFurNhE/xX8qSyo/CYg==, tableContent=null), ArticleFig(id=1329460766337512188, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Figure 5, caption=Impact of hyperparameter on performance of model, figureFileSmall=uyuofFzA/3S679gWNjJWMg==, figureFileBig=aWXBPbj9uvCQ3hDLjBP+6Q==, tableContent=null), ArticleFig(id=1329460766396232445, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=图5, caption=超参数对模型性能的影响, figureFileSmall=uyuofFzA/3S679gWNjJWMg==, figureFileBig=aWXBPbj9uvCQ3hDLjBP+6Q==, tableContent=null), ArticleFig(id=1329460766484312830, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Figure 6, caption=Comparison of prediction results between DSC-Net and baseline, figureFileSmall=MU5OLyxb1c8vts5SBc1XFw==, figureFileBig=mGG1iSQVK2yuv7xHuSLKmQ==, tableContent=null), ArticleFig(id=1329460766551421695, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=图6, caption=DSC-Net与基线的预测结果对比, figureFileSmall=MU5OLyxb1c8vts5SBc1XFw==, figureFileBig=mGG1iSQVK2yuv7xHuSLKmQ==, tableContent=null), ArticleFig(id=1329460766618530560, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Table 1, caption=

Performance Comparison of Our Method and Various Typical Methods in Offline Test %

, figureFileSmall=null, figureFileBig=null, tableContent=
方法BLEU-
1
BLEU-
4
METEORROUGECIDErSPICE
up-down[5](CVPR’18)79.836.327.756.9120.121.4
Transformer[17](ACL’18)80.539.229.158.7130.023.0
SGAE[12](CVPR’19)80.838.427.458.6127.822.1
M2Transformer[13]
(CVPR’20)
80.839.129.258.6131.222.6
X-Transformer[8]
(CVPR’20)
80.939.729.559.1132.823.4
GET[14](AAAI’21)81.539.529.358.9131.622.8
APN[15] (ICCV’21)39.629.259.1131.823.0
RSTNet[18](CVPR’21)81.139.329.458.8133.323.0
ReFormer[19](MM’22)82.339.829.759.8131.923.0
GAT[20](Expert Syst.
Appl.’22)
80.839.729.159.0130.522.9
CLIP-ViL[9](ICLR’22)40.229.7134.223.8
up-down*81.339.429.259.3131.922.8
Transformer*81.640.629.959.8136.223.9
RSTNet*82.040.430.059.7137.023.7
DSC-Net(本文)82.941.130.159.9138.523.7
), ArticleFig(id=1329460766740165377, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=表1, caption=

本文方法在离线测试中与各种典型方法的性能比较

, figureFileSmall=null, figureFileBig=null, tableContent=
方法BLEU-
1
BLEU-
4
METEORROUGECIDErSPICE
up-down[5](CVPR’18)79.836.327.756.9120.121.4
Transformer[17](ACL’18)80.539.229.158.7130.023.0
SGAE[12](CVPR’19)80.838.427.458.6127.822.1
M2Transformer[13]
(CVPR’20)
80.839.129.258.6131.222.6
X-Transformer[8]
(CVPR’20)
80.939.729.559.1132.823.4
GET[14](AAAI’21)81.539.529.358.9131.622.8
APN[15] (ICCV’21)39.629.259.1131.823.0
RSTNet[18](CVPR’21)81.139.329.458.8133.323.0
ReFormer[19](MM’22)82.339.829.759.8131.923.0
GAT[20](Expert Syst.
Appl.’22)
80.839.729.159.0130.522.9
CLIP-ViL[9](ICLR’22)40.229.7134.223.8
up-down*81.339.429.259.3131.922.8
Transformer*81.640.629.959.8136.223.9
RSTNet*82.040.430.059.7137.023.7
DSC-Net(本文)82.941.130.159.9138.523.7
), ArticleFig(id=1329460766832440066, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Table 2, caption=

Performance Comparison of Our Method and Various Typical Methods in Online Test %

, figureFileSmall=null, figureFileBig=null, tableContent=
方法BLEU-1BLEU-2BLEU-3BLEU-4METEORROUGECIDEr
c5c40c5c40c5c40c5c40c5c40c5c40c5c40
up-down[5](CVPR’18)80.295.264.188.849.179.436.968.527.636.757.172.4117.9120.5
SGAE[12](CVPR’19)81.095.365.689.550.780.438.569.728.237.258.673.6123.8126.5
AoANet[21](ICCV’19)81.095.065.889.651.481.339.471.229.138.558.974.5126.9129.6
X-Transformer[8](CVPR’20)81.995.766.990.552.482.540.372.429.639.259.575.0131.1133.5
M2Transformer[13](CVPR’20)81.696.066.490.851.882.739.772.829.439.059.274.8129.3132.1
GET[14](AAAI’21)81.696.166.590.951.982.839.772.929.438.859.174.4130.3132.5
RSTNet[18](CVPR’21)82.196.467.091.352.283.040.073.129.639.159.574.6131.9134.0
ReFormer[19](MM’22)82.096.740.173.229.839.559.975.2129.9132.8
GAT[20](Expert Syst.Appl’22)81.195.166.189.751.881.539.971.429.138.459.174.4127.8129.8
RSTNet[18](单一模型)81.896.266.691.051.982.739.772.529.739.059.474.5131.4134.4
DSC-Net(本文单一模型)81.996.366.691.151.982.839.772.829.739.259.374.4132.7135.3
DSC-Net(本文集成模型)82.896.967.892.353.284.540.974.730.139.660.175.3135.8137.6
), ArticleFig(id=1329460766924714755, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=表2, caption=

本文方法在在线测试中与各种典型方法的性能比较

, figureFileSmall=null, figureFileBig=null, tableContent=
方法BLEU-1BLEU-2BLEU-3BLEU-4METEORROUGECIDEr
c5c40c5c40c5c40c5c40c5c40c5c40c5c40
up-down[5](CVPR’18)80.295.264.188.849.179.436.968.527.636.757.172.4117.9120.5
SGAE[12](CVPR’19)81.095.365.689.550.780.438.569.728.237.258.673.6123.8126.5
AoANet[21](ICCV’19)81.095.065.889.651.481.339.471.229.138.558.974.5126.9129.6
X-Transformer[8](CVPR’20)81.995.766.990.552.482.540.372.429.639.259.575.0131.1133.5
M2Transformer[13](CVPR’20)81.696.066.490.851.882.739.772.829.439.059.274.8129.3132.1
GET[14](AAAI’21)81.696.166.590.951.982.839.772.929.438.859.174.4130.3132.5
RSTNet[18](CVPR’21)82.196.467.091.352.283.040.073.129.639.159.574.6131.9134.0
ReFormer[19](MM’22)82.096.740.173.229.839.559.975.2129.9132.8
GAT[20](Expert Syst.Appl’22)81.195.166.189.751.881.539.971.429.138.459.174.4127.8129.8
RSTNet[18](单一模型)81.896.266.691.051.982.739.772.529.739.059.474.5131.4134.4
DSC-Net(本文单一模型)81.996.366.691.151.982.839.772.829.739.259.374.4132.7135.3
DSC-Net(本文集成模型)82.896.967.892.353.284.540.974.730.139.660.175.3135.8137.6
), ArticleFig(id=1329460766996017924, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Table 3, caption=

Ablation Results on Modules %

, figureFileSmall=null, figureFileBig=null, tableContent=
网络结构CIDEr
BaselineDSCSDSCCDSF
✔136.3
✔✔137.4
✔✔136.6
✔✔✔137.1
✔✔✔137.7
✔✔✔137.5
✔✔✔✔138.5
), ArticleFig(id=1329460767058932485, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=表3, caption=

关于模块的消融结果

, figureFileSmall=null, figureFileBig=null, tableContent=
网络结构CIDEr
BaselineDSCSDSCCDSF
✔136.3
✔✔137.4
✔✔136.6
✔✔✔137.1
✔✔✔137.7
✔✔✔137.5
✔✔✔✔138.5
), ArticleFig(id=1329460767138624262, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Table 4, caption=

Ablation Experiment Results of DSCC and DSCS Modules %

, figureFileSmall=null, figureFileBig=null, tableContent=
方法BLEU-1BLEU-4METEORROUGECIDErSPICE
w/o Region82.040.329.859.8136.323.5
w/o DSC82.440.629.959.7136.823.4
w/o DSCC82.240.529.959.6137.723.5
w/o DSCS82.540.730.059.9137.523.6
DSC-Net82.941.130.159.9138.523.7
), ArticleFig(id=1329460767205733127, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=表4, caption=

关于双重语义协作机制的消融实验结果

, figureFileSmall=null, figureFileBig=null, tableContent=
方法BLEU-1BLEU-4METEORROUGECIDErSPICE
w/o Region82.040.329.859.8136.323.5
w/o DSC82.440.629.959.7136.823.4
w/o DSCC82.240.529.959.6137.723.5
w/o DSCS82.540.730.059.9137.523.6
DSC-Net82.941.130.159.9138.523.7
), ArticleFig(id=1329460767277036296, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Table 5, caption=

Ablation Experiment Results of Feature Fusion %

, figureFileSmall=null, figureFileBig=null, tableContent=
方法BLEU-1BLEU-4METEORROUGECIDErSPICE
w/o DSF82.240.630.059.7137.123.6
w/o RFD82.140.529.959.7137.323.6
DSC-Net82.941.130.159.9138.523.7
), ArticleFig(id=1329460767344145161, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=表5, caption=

特征融合的消融实验结果

, figureFileSmall=null, figureFileBig=null, tableContent=
方法BLEU-1BLEU-4METEORROUGECIDErSPICE
w/o DSF82.240.630.059.7137.123.6
w/o RFD82.140.529.959.7137.323.6
DSC-Net82.941.130.159.9138.523.7
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202330523, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202330523, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202330523, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202330523, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1730390400000, fullTextJson=null, articleText=null, reference=null)
收藏切换
一种基于双重语义协作网络的图像描述方法
江泽涛 , 朱文才 * , 金鑫 , 廖培期 , 黄景帆
计算机研究与发展 | 图形与图像处理 2024,61(11): 2897-2908
收起
收藏切换
计算机研究与发展 |图形与图像处理 2024 , 61 (11) : 2897 -2908
一种基于双重语义协作网络的图像描述方法
全屏
[Author(id=1329460763783181010, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=zetaojiang@126.com, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329460763883844308, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460763783181010, language=EN, stringName=Zetao Jiang, firstName=Zetao, middleName=null, lastName=Jiang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004, bio={"content":"

born in 1961. PhD, professor. His main research interests include image processing, computer vision, and artificial intelligence

"}, bioImg=null, bioContent=

born in 1961. PhD, professor. His main research interests include image processing, computer vision, and artificial intelligence

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329460763950953173, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460763783181010, language=CN, stringName=江泽涛, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004, bio={"img":"K0PyxtGhzZbmPXwP+lNCXw==","content":"

1961年生. 博士,教授. 主要研究方向为图像处理、计算机视觉、人工智能

"}, bioImg=K0PyxtGhzZbmPXwP+lNCXw==, bioContent=

1961年生. 博士,教授. 主要研究方向为图像处理、计算机视觉、人工智能

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329460763711877838, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, xref=null, ext=[AuthorCompanyExt(id=1329460763716072143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004), AuthorCompanyExt(id=1329460763724460752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004)])]), Author(id=1329460764043227863, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=zhuwencai00@qq.com, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329460764110336729, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764043227863, language=EN, stringName=Wencai Zhu, firstName=Wencai, middleName=null, lastName=Zhu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004, bio={"content":"

born in 1999. Master. His main research interests include image processing and computer vision

"}, bioImg=null, bioContent=

born in 1999. Master. His main research interests include image processing and computer vision

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329460764177445594, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764043227863, language=CN, stringName=朱文才, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004, bio={"img":"RHbqdykSWeY1WKf2q2Ottw==","content":"

1999年生. 硕士. 主要研究方向为图像处理、计算机视觉

"}, bioImg=RHbqdykSWeY1WKf2q2Ottw==, bioContent=

1999年生. 硕士. 主要研究方向为图像处理、计算机视觉

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329460763711877838, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, xref=null, ext=[AuthorCompanyExt(id=1329460763716072143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004), AuthorCompanyExt(id=1329460763724460752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004)])]), Author(id=1329460764257137372, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329460764361994974, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764257137372, language=EN, stringName=Xin Jin, firstName=Xin, middleName=null, lastName=Jin, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004, bio={"content":"

born in 1998. Master. His main research interests include image processing and computer vision

"}, bioImg=null, bioContent=

born in 1998. Master. His main research interests include image processing and computer vision

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329460764424909535, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764257137372, language=CN, stringName=金鑫, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004, bio={"img":"jXy+WjkR0BEIeJJ5oTIg+g==","content":"

1998年生. 硕士. 主要研究方向为图像处理、计算机视觉

"}, bioImg=jXy+WjkR0BEIeJJ5oTIg+g==, bioContent=

1998年生. 硕士. 主要研究方向为图像处理、计算机视觉

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329460763711877838, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, xref=null, ext=[AuthorCompanyExt(id=1329460763716072143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004), AuthorCompanyExt(id=1329460763724460752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004)])]), Author(id=1329460764487824097, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329460764550738659, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764487824097, language=EN, stringName=Peiqi Liao, firstName=Peiqi, middleName=null, lastName=Liao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004, bio={"content":"

born in 1995. Master. His main research interests include image processing and computer vision

"}, bioImg=null, bioContent=

born in 1995. Master. His main research interests include image processing and computer vision

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329460764626236132, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764487824097, language=CN, stringName=廖培期, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004, bio={"img":"1Df8KRPAFXMKVJLjoVLcyg==","content":"

1995年生. 硕士. 主要研究方向为图像处理、计算机视觉

"}, bioImg=1Df8KRPAFXMKVJLjoVLcyg==, bioContent=

1995年生. 硕士. 主要研究方向为图像处理、计算机视觉

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329460763711877838, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, xref=null, ext=[AuthorCompanyExt(id=1329460763716072143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004), AuthorCompanyExt(id=1329460763724460752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004)])]), Author(id=1329460764689150694, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329460764777231080, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764689150694, language=EN, stringName=Jingfan Huang, firstName=Jingfan, middleName=null, lastName=Huang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004, bio={"content":"

born in 1999. Master. His main research interests include image processing and computer vision

"}, bioImg=null, bioContent=

born in 1999. Master. His main research interests include image processing and computer vision

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329460764856922857, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764689150694, language=CN, stringName=黄景帆, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004, bio={"img":"JML2tZfo7ovlsMlvYY7jqg==","content":"

1999年生. 硕士. 主要研究方向为图像处理、计算机视觉

"}, bioImg=JML2tZfo7ovlsMlvYY7jqg==, bioContent=

1999年生. 硕士. 主要研究方向为图像处理、计算机视觉

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329460763711877838, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, xref=null, ext=[AuthorCompanyExt(id=1329460763716072143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004), AuthorCompanyExt(id=1329460763724460752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004)])])]
江泽涛 , 朱文才* , 金鑫, 廖培期, 黄景帆
作者信息
  • 广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004
通讯作者:
朱文才()
作者简介:

1961年生. 博士,教授. 主要研究方向为图像处理、计算机视觉、人工智能

1999年生. 硕士. 主要研究方向为图像处理、计算机视觉

1998年生. 硕士. 主要研究方向为图像处理、计算机视觉

1995年生. 硕士. 主要研究方向为图像处理、计算机视觉

1999年生. 硕士. 主要研究方向为图像处理、计算机视觉

An Image Captioning Method Based on DSC-Net
Zetao Jiang , Wencai Zhu* , Xin Jin, Peiqi Liao, Jingfan Huang
Affiliations
  • Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004
  • born in 1961. PhD, professor. His main research interests include image processing, computer vision, and artificial intelligence

    born in 1999. Master. His main research interests include image processing and computer vision

    born in 1998. Master. His main research interests include image processing and computer vision

    born in 1995. Master. His main research interests include image processing and computer vision

    born in 1999. Master. His main research interests include image processing and computer vision

出版时间: 2024-11-01 doi: 10.7544/issn1000-1239.202330523
文章导航
收藏切换

CLIP(contrastive language-image pre-training)视觉编码器提取的网格特征作为一种更加靠近文本域的视觉特征,具有易转化为对应语义自然语言的特点,可以缓解语义鸿沟问题,因而未来可能成为图像描述任务中视觉特征的重要来源. 但该方法中未考虑图像内容的划分,可能使一个完整的目标被划分到若干个网格中,目标被切割势必会导致特征提取结果中缺少对目标信息的完整表达,进而导致生成的描述语句中缺少对目标及目标间关系的准确表述. 针对CLIP视觉编码器提取网格特征这一现象,提出一种基于双重语义协作网络(dual semantic collaborative network,DSC-Net)的图像描述方法. 具体来说:首先提出双重语义协作自注意力(dual semantic collaborative self-attention,DSCS)模块增强CLIP网格特征对目标信息的表达能力;接着提出双重语义协作交叉注意力(dual semantic collaborative cross-attention,DSCC)模块,综合网格和目标2个层面的语义构造与文本相关的视觉特征,进行描述语句预测;最后提出双重语义融合(dual semantic fusion,DSF)模块,为上述的2个语义协作模块提供以区域为主导的融合特征,解决在语义协作过程中可能出现的相关性冲突问题. 经过在COCO数据集上的大量实验,提出的模型在Karpathy等人划分的离线测试集上取得了138.5%的CIDEr分数,在官方在线测试中取得了137.6%的CIDEr分数,与目前主流的图像描述方法相比具有显著优势.

图像描述  /  网格特征  /  注意力机制  /  双重语义协作注意力  /  双重语义协作特征融合

As visual features closer to the text domain, the grid features extracted by the CLIP (contrastive language-image pre-training) image encoder are easy to convert into the corresponding semantic natural language, which can alleviate the semantic gap problem, so it may become an important source of visual features in the image captioning in the future. However, this method does not consider that the division of image content may cause a complete object to be divided into several grids. The segmentation of the objects will inevitably lead to the lack of a complete expression of the object information in the feature extraction results, and further lead to the lack of an accurate expression of the object and the relationship between the objects in the generated sentence. Aiming at the phenomenon of grid features extracted by CLIP image encoder, we propose dual semantic collaborative network (DSC-Net) for image captioning. Specifically, dual semantic collaborative self-attention (DSCS) module is first proposed to enhance the expression of object information by CLIP grid features. Then dual semantic collaborative cross-attention (DSCC) module is proposed to integrate semantic information between grid and object to generate visual features, and to be used to predict sentences. Finally, dual semantic fusion (DSF) module is proposed to provide region-oriented fusion features for the above two semantic cooperation modules, and to solve the problem of correlation conflicts that may arise in the process of semantic cooperation. After a large number of experiments on the COCO dataset, the proposed model achieves a CIDEr score of 138.5% on the offline test set divided by Karpathy et al., and a CIDEr score of 137.6% in the official online test. Compared with the current mainstream image captioning methods, this result has obvious advantages.

image captioning  /  grid features  /  attention mechanism  /  dual semantic collaborative attention  /  dual semantic collaborative feature fusion
江泽涛, 朱文才, 金鑫, 廖培期, 黄景帆. 一种基于双重语义协作网络的图像描述方法. 计算机研究与发展, 2024 , 61 (11) : 2897 -2908 . DOI: 10.7544/issn1000-1239.202330523
Zetao Jiang, Wencai Zhu, Xin Jin, Peiqi Liao, Jingfan Huang. An Image Captioning Method Based on DSC-Net[J]. Journal of Computer Research and Development, 2024 , 61 (11) : 2897 -2908 . DOI: 10.7544/issn1000-1239.202330523
图像描述(image captioning)是一项计算机视觉领域与自然语言处理领域的交叉任务,是图像与视频转化为文字语音的重要桥梁,在图像与视频智能化服务中扮演重要角色,在机器人视觉、智能交通、智慧农业等诸多领域具有重要的潜在应用前景. 由于图像描述研究属于起步阶段,具有非常大的挑战性. 随着深度学习技术的发展,基于深度学习的方法在图像描述中成为主流方向[1]. 目前典型的方法是利用目标检测器提取图像的区域特征,然后在特征间进行目标语义关系建模,利用建模结果提高特征的语义表示能力,最后将特征送入文本生成模块,产生描述图像内容的自然语言. 但是,该方法中使用的目标检测器未与文本进行过联合训练,使得提取到的区域特征与文本域间存在较大差距,继而导致将区域特征转化为自然语言时往往不够准确.
近年来,受到自然语言处理领域中下游任务的启发,研究者们提出使用编码-解码结构进行图像描述任务. 早期的研究中使用卷积神经网络(convolutional neural network,CNN)提取视觉特征,使用循环神经网络(recurrent neural network,RNN)预测描述文本,并将其分别称为编码阶段和解码阶段. 随着Transformer[2]结构的兴起,一些人开始尝试利用Transformer代替RNN进行文本生成,为了避免混淆,将CNN对图像的处理过程称为视觉特征提取,而将编码和解码特指为Transformer中的编码器和解码器,这种新的结构被称为基于Transformer的图像描述方法. 基于RNN的图像描述方法参数量少,但描述精度较低、并行性差、训练周期长;基于Transformer的图像描述方法描述精度高、并行性好、训练周期短,但参数量和运算量都较高. 随着硬件的发展,Transformer的参数和运算规模能够被越来越多的人接受,基于Transformer的图像描述方法逐渐成为了图像描述领域中的主流,被越来越多的研究者们使用.
在视觉特征提取方面,早期的研究中受到分类任务的启发,使用单个向量表示图像的视觉特征[3],但是这种方法会导致描述语句中的全部单词都由1个向量预测,忽略了不同单词在语义上的差异. 后来,使用一组向量作为图像的视觉特征[4],每个向量对应原图中的1个网格,同时引入注意力机制关注单词间的语义差异,以便根据原图中不同位置的特征向量生成不同的单词,但是这种方法的缺点是很难感知到目标层面的语义信息,难以对目标及目标间的关系形成有效建模. 再后来,使用一组通过目标检测器取得的特征向量作为图像的视觉特征[5],这种特征向量基于目标检测器对目标的精准定位能力,从目标附近的区域进行提取,是目前主流的视觉特征提取方式,提取到的特征被称为区域特征,利用区域特征可以对目标及目标间的关系进行有效感知,但是由于图像与文本间存在语义鸿沟,区域特征中丰富的语义信息很难转化为与之对应的自然语言. 最近一些研究表明,利用经过视觉文本联合训练的视觉特征提取模型,即CLIP(contrastive language-image pre-training)[6]的视觉编码器,可以提取到更加靠近文本域的视觉特征,这种视觉特征更容易转化为对应语义的自然语言,缓解语义鸿沟问题.
由于CLIP视觉编码器无法像目标检测器一般对目标进行定位,故只能提取到网格形式的视觉特征,记为CLIP网格特征,特征提取过程中未考虑图像内容的网格划分行为可能破坏目标的完整性,导致CLIP网格特征中缺少对目标信息的完整表达,进而导致生成的描述语句中缺少对目标及目标间关系的准确表述,这是该特征提取方法的不足之处.
针对区域特征生成的描述语句不够准确的问题,Guo等人[7]提出将区域之间的相对位置关系添加到目标语义关系建模中,利用相对位置信息增强对目标间语义关联的表达能力;Pan等人[8]提出在将Transformer中原本的单一维度建模改为对二阶特征交互建模,从多个维度提高对语义的表达能力. 这些方法基于区域特征的特点进行了设计,但目标检测器自身的缺陷限制了模型性能的进一步提高. 为此一些研究者尝试使用经过图像文本联合训练的视觉特征提取模型代替目标检测器,Shen等人[9]提出利用CLIP的视觉编码器提取网格特征,使用这种网格特征替换区域特征,在相同的网络结构下取得了显著的性能提升,但是该团队仅仅证明了CLIP提取网格特征的有效性,却没有提出新的模型结构. Li等人[10]提出利用CLIP的视觉编码器和文本编码器提取网格特征和相关的文本特征,通过视觉和文本2种特征来生成描述图像内容的语句,该团队虽然提出了新的模型,但将研究重心放在了对文本特征的处理上,未基于网格特征的特点进行设计. 综上所述,当前的研究中虽然证明了CLIP网格特征与传统的区域特征相比所具有的优势,但也要注意到其仍然存在缺陷,无法完全代替区域特征.
本文针对提取网格特征时可能将一个完整的目标划分到多个网格中,继而丢失完整目标语义信息的问题,提出了一种基于双重语义协作网络(dual semantic collaborative network,DSC-Net)的图像描述方法. 该方法设计了DSC-Net网络结构,其主要贡献有3个方面:
1) 设计了双重语义协作自注意力(dual semantic collaborative self-attention,DSCS)模块,借助区域特征对目标的准确表示能力进行目标间的语义关联感知;然后利用重叠矩阵将感知结果映射到对应位置的网格上,融入网格特征的语义建模过程,使各网格特征向量得以同时汇聚整合网格间、目标间、目标内的相关联信息,进行基于网格内容、目标内容、目标间关系的多层次建模,完成信息的更新迭代,增强对目标层面语义的表达能力.
2) 设计了双重语义协作交叉注意力(dual semantic collaborative cross-attention,DSCC)模块,借助区域特征对目标位置的捕捉能力,计算文本与目标的语义相关性;然后利用重叠矩阵将计算结果映射到对应位置的网格上,从而在视觉与文本的跨模态建模中综合目标和网格2个层面的信息构建上下文感知的视觉特征表示,进而利用特征表示中蕴含的视觉语义信息对下一个单词进行预测.
3) 设计了双重语义融合(dual semantic fusion,DSF)模块,内置了3种确保区域特征主导地位的融合机制,为上述2个语义协作模块提供以区域为主导的融合特征,融合特征在保留区域特色的同时融入了网格信息,利用这种融合特征可以在不与网格关联冲突的情况下进行目标关联感知,从而解决语义协作过程中可能出现的相关性冲突问题.
本文方法在广泛使用的COCO数据集上进行大量实验,遵循Karpathy等人[11]的划分方式进行离线测试,取得了138.5%的CIDEr分数,在COCO图像描述在线测试平台上进行在线测试,取得了137.6%的CIDEr分数,与目前典型的图像描述方法相比具有明显优势.
CLIP[6]是一个用于视觉语言任务的预训练模型,由一个视觉编码器和一个文本编码器组成,2个编码器均直接使用目前主流的特征提取网络,随意组合图像和文本作为训练样本,其中图像和文本语义一致的作为正样本,否则作为负样本,通过对比学习的方式展开视觉文本联合训练. 具体来说,首先利用2个编码器将样本中的图像和文本编码为向量的形式,然后计算2个向量的相似度得分,并在训练过程中通过对比损失函数控制正样本的相似度得分尽可能高和负样本的相似度得分尽可能低.
显然,利用CLIP取得的视觉特征和文本特征在表达相近的语义时将具有很高的相似度,所以与传统的区域特征相比,这种新的视觉特征更加接近文本域,可以更准确地转化为对应语义的自然语言,非常适合用于图像描述任务.
得益于Transformer[2]的强大性能,近年来研究者们大多遵循Transformer结构对图像描述任务展开研究[7-10,12-15]. 具体来说,首先使用经过预训练的特征提取模型提取图像的视觉特征;接着使用Transformer的编码器通过多头注意力模块在视觉特征内部进行建模,建模过程中基于各视觉特征向量的语义进行关联感知,再利用感知结果汇聚整合与自身相关联的信息,实现更加精细化的视觉语义表达;最后将经过编码器处理的视觉特征送入Transformer解码器,在解码器中逐字生成描述图像内容的自然语言,每个时间步下通过掩码注意力运算提取已生成单词的文本特征,与视觉特征进行跨模态建模,构建上下文感知的视觉特征表示,并根据该特征表示预测下一个单词. 同时,为了提高视觉语义的精细化程度和预测描述语句的准确程度,对编码器和解码器中的建模过程进行若干次堆叠,达到逐步求精的效果,并在建模后使用前馈神经网络进行处理,增强模型的泛化能力.
在基于Transformer的图像描述模型中进行精细化设计是目前进行图像描述相关研究的主流方法之一,而Transformer结构的核心是多头注意力(multi-head attention,MHA)模块,该模块可以有效地对输入之间的关系进行建模,并用建模结果对输入进行更新迭代,得到更加丰富和准确的表示. 多头注意力的运算过程可以用公式表示为
$ {F_{{\text{MHA}}}}({\boldsymbol {Q,K,V}}) = concat(hea{d_1},hea{d_2},…,hea{d_h}){{\boldsymbol {W}}^{\text{o}}},$
$ hea{d_i} = Attention({\boldsymbol {QW}}_i^{\text{q}},{\boldsymbol {KW}}_i^{\text{k}},{\boldsymbol {VW}}_i^{\text{v}}),$
其中$ {{\boldsymbol {W}}^{\text{o}}},{\boldsymbol {W}}_i^{\text{q}},{\boldsymbol {W}}_i^{\text{k}},{\boldsymbol {W}}_i^{\text{v}} $是可学习的参数矩阵,Attention是缩放点积运算函数,用公式可以表示为
$ \begin{split} & Attention({\boldsymbol {QW}}_i^{\text{q}},{\boldsymbol {KW}}_i^{\text{k}},{\boldsymbol {VW}}_i^{\text{v}}) =\\& \quad {{softmax}}\left( {\frac{{({\boldsymbol {QW}}_i^{\text{q}}){{({\boldsymbol {KW}}_i^{\text{k}})}^{\text{T}}}}}{{\sqrt d }}} \right){\boldsymbol {VW}}_i^{\text{v}} .\end{split} $
CLIP视觉编码器提取的网格特征在图像描述任务中展现出潜在的优势,但特征提取过程中未考虑图像内容的网格划分行为可能破坏目标的完整性,将一个完整的目标划分到多个网格中,这将导致特征提取结果中目标层面语义信息的缺失,继而增加后续处理中感知目标及目标间关系的难度. 为此,本文提出DSC-Net网络借助区域特征进行目标语义关联感知,再将感知结果注入对应位置的网格上,实现网格语义和目标语义协作的双重语义关联感知与建模. 该网络遵循目前主流的基于Transformer的图像描述框架进行设计,提出了DSCS,DSCC,DSF这3个全新的模块,如图1所示.
给定一张图片,首先使用CLIP视觉编码器和目标检测器提取网格特征和区域特征,并根据二者在原图中的位置关系构建重叠矩阵;接着在DSCS模块中利用重叠矩阵将区域特征中目标内和目标间的关系映射到对应位置的网格上,为网格特征进行基于网格内容、目标内容、目标间关系的多层次建模,增强对目标层面语义信息的表达能力;然后利用DSF对视觉特征进行处理,生成以区域为主导的融合特征,避免语义协作过程中可能出现的相关性冲突问题;再将经过处理的视觉特征送入DSCC模块进行视觉与文本的跨模态建模,从网格和目标2个层面感知视觉特征与文本特征的语义关联,构建上下文感知的视觉特征表示,最后利用该特征表示预测下一个单词,即将视觉特征中的视觉语义信息转化为对应的自然语言.
在一系列的视觉特征中,利用CLIP视觉编码器提取的网格特征具有靠近文本域,易转化为对应语义自然语言的优点,也有缺少目标层面语义信息的缺点,而这恰好是由目标检测器提取的区域特征所擅长的,为了增强网格特征对目标层面语义的表达能力,本文设计了DSCS模块. 目前,已有的自注意力模块都是为区域特征设计的,未考虑过目标语义不足的情况,所以无法解决网格特征的问题. 本文设计的DSCS模块如图2所示,首先通过相似度运算在2种特征内部分别进行网格和目标层面的语义关联感知;接着利用重叠矩阵将目标层面的语义关联感知结果映射到对应位置的网格上,在网格间形成对目标内容和目标间关系的语义关联感知结果,与原本基于网格内容的语义关联感知结果融合;最后利用融合后得到的双重语义关联感知矩阵进行基于网格内容、目标内容、目标间关系的多层次建模,达到增强网格特征目标语义表达能力的目的.
在上述语义关联感知过程中,网格层面的语义关联感知可以直接通过对网格特征的计算得到,而目标层面的语义关联感知则需要在DSF模块对区域特征处理后才能进行计算,这是为了避免在关联融合中出现“相关性冲突”,DSF模块的结构和“相关性冲突”的细节将会在2.3节中进行详细介绍.
进行语义关联感知时,首先利用线性层将输入映射到2个不同的维度上作为查询和键,接着采用点积的方式计算相似度,最后通过多头的方式关注更多的语义相关信息. 语义关联感知的结果一般表现为矩阵形式,这个计算过程用公式可以表示为
$ S im({\boldsymbol {Q}},{\boldsymbol {K}}) = \frac{{({\boldsymbol {Q}}{{\boldsymbol {W}}^{\text{q}}}){{({\boldsymbol {K}}{{\boldsymbol {W}}^{\text{k}}})}^{\text{T}}}}}{{\sqrt d }} \text{,} $
$\begin{split} & {{\boldsymbol {M}}_{{\text{GSS}}}}({\boldsymbol {G}},{\boldsymbol {G}}) = concat(head_{\text{1}}^{\boldsymbol {g}},head_{\text{2}}^{\boldsymbol {g}},…,head_h^{\boldsymbol {g}}),\\&\quad head_i^{\boldsymbol {g}} = {{softmax}}\left( {S im_i^{\boldsymbol {g}}({\boldsymbol {G}},{\boldsymbol {G}})} \right) \text{,}\\[-1pt]\end{split} $
$ \begin{split} &{{\boldsymbol {M}}_{{\text{OSS}}}}({{\boldsymbol {R}}^{\text{f}}},{{\boldsymbol {R}}^{\text{f}}}) = concat(head_1^{\boldsymbol {r}},head_{\text{2}}^{\boldsymbol {r}},…,head_h^{\boldsymbol {r}}),\\& head_i^{\boldsymbol {r}} = {{softmax}}\left( {S im_i^{\boldsymbol {r}}({{\boldsymbol {R}}^{\text{f}}},{{\boldsymbol {R}}^{\text{f}}})} \right),{{\boldsymbol {R}}^{\text{f}}} = {F_{{\text{DSF}}}}({\boldsymbol {R}},{\boldsymbol {G}},{\boldsymbol {G}}) ,\;\;\;\;\\[-1pt]\end{split} $
其中Sim表示点积相似度运算函数,MGSS表示网格语义关联矩阵,MOSS表示目标语义关联矩阵,DSF表示双重语义融合模块,G表示网格特征,R表示区域特征,${{\boldsymbol {W}}^{\text{q}}}$和${{\boldsymbol {W}}^{\text{k}}}$表示可学习的参数矩阵. 为了将MOSS映射到网格间,需要使用重叠矩阵中记录的区域和网格在原图中的位置关系. 在构建重叠矩阵时,将网格与目标的数量作为矩阵的行和列,使用数字1和0表示对应编号的特征在原图中是否存在重叠,用公式可以将重叠矩阵的构建过程表示为
$ {E}_{ij}=\left\{\begin{aligned}& 1\text{,}目标和网格存在重叠, \\&0\text{,}目标和网格不存在重叠. \end{aligned} \right.$
利用重叠矩阵进行映射的过程如图2所示,其中${{\boldsymbol {r}}_i}$和${{\boldsymbol {g}}_i}$分别表示区域特征向量和网格特征向量,${a_{ij}}$表示目标间的语义关联情况,通过重叠矩阵确认区域与网格的对应关系,然后将区域间的语义关联映射到对应位置的网格上,将目标间的关联感知结果转化为网格间基于目标内容和目标间关系的语义关联感知结果. 上述过程用公式可以表示为重叠矩阵与目标语义关联矩阵的2次矩阵乘法运算,这个映射过程用公式可以表示为
$ \boldsymbol {M}'_{{\text{OSS}}} = {{softmax}}\left( {{{\boldsymbol {E}}^{\text{T}}}({{\boldsymbol {M}}_{{\text{OSS}}}}){\boldsymbol {E}}} \right) . $
最后,将上述结果与网格间原本存在的基于网格内容的语义关联感知结果以求和的方式进行融合,再利用融合后的关联矩阵进行基于网格内容、目标内容、目标间关系的多层次建模,对网格间、目标间、目标内的相关信息进行汇聚与整合,得到基于双重语义协作的更准确、更丰富的视觉表示,用公式可以表示为
$ {\boldsymbol {G}}' = {{softmax}}({{\boldsymbol {M}}_{{\text{GSS}}}} + \boldsymbol {M}'_{{\text{OSS}}}){\boldsymbol {G}}{{\boldsymbol {W}}^{\text{v}}}. $
综上所述,在该模块中将双重语义协作的思想融入网格特征内部的建模过程,利用区域特征对目标的敏锐感知力增强了网格特征对目标信息的表达能力,缓解了网格特征中目标层面语义信息缺失的问题.
使用区域特征辅助网格进行语义关联感知,虽然可以增加对目标间关系的感知能力,但也可能对原本基于网格内容的语义关联感知结果造成破坏. 试想一下,对于原图中的2个网格,当基于网格内容计算出的语义相关性与基于所属目标计算出的语义相关性产生冲突时,使用其中任何一方粗暴地改变另一方都是不合理的. 同时,本文的消融实验结果也证明,在双重语义协作过程中直接使用区域特征进行目标语义关联感知,并不能显著提高模型对图像内容的表述能力,本文将这种现象称为“相关性冲突”.
为了解决这一问题,本文提出在双重语义关联感知过程中使用以区域为主导的融合特征进行目标层面的语义关联计算,以产生不与网格关联冲突的目标语义关联感知结果. 这就要求融合过程中一方面要保留区域特征对目标敏感的特点,另一方面要融入网格信息. 基于双重语义协作的需要,其中保留区域对网格敏感的特点又显得尤其重要,因此本文在生成融合特征的DSF模块内精心设计了3种机制来保证区域特征的主导地位. DSF模块结构如图3所示.
首先,以区域特征为查询,网格特征为键和值进行多头注意力运算,运算过程中计算查询和键的相似度,再以相似度为权重对值加权求和,通过这种方式计算出的结果由网格特征构成,却以接近区域特征为目标,这是确保区域特征主导地位的第1种机制. 接着,将区域特征和多头注意力的运算结果送入线性层引入可学习参数,再通过函数Sigmoid将区域特征中的元素转化为0和1之间的小数,与多头注意力的运算结果计算哈达玛积,在这个过程中,与区域特征相近的部分将被鼓励,反之被抑制,这是确保区域特征主导地位的第2种机制. 最后,将上述运算结果与区域特征求和,使原始的区域特征在最终的结果中固定占有一半的比重,这是确保区域特征主导地位的第3种机制. 上述过程用公式可以表示为
$ {F_{{\text{DSF}}}}({\boldsymbol {R}},{\boldsymbol {G}},{\boldsymbol {G}}) = {F_{{\text{MHA}}}}({\boldsymbol {R}},{\boldsymbol {G}},{\boldsymbol {G}}){{\boldsymbol {W}}^{\text{f}}} \odot Sigmoid({\boldsymbol {R}}{{\boldsymbol {W}}^{\boldsymbol {r}}}) + {\boldsymbol {R}} \text{,} $
其中MHA表示多头注意力模块,$ {{\boldsymbol {W}}^{\text{f}}} $和$ {{\boldsymbol {W}}^{\boldsymbol {r}}} $表示可学习的参数矩阵,$ \odot $表示哈达玛积运算. 综上所述,该模块中通过精心设计的结构进行特征融合,使得双重语义协作过程中可以利用其中占据主导地位的区域特征进行目标语义关联感知,以及利用其中基于网格内容的语义信息避免冲突的产生,从而解决双重语义协作过程中可能出现的相关性冲突问题.
在解码器中,需要在视觉特征和已生成文本间进行跨模态建模,构建上下文感知的视觉特征表示,然后利用该特征表示预测下一个单词,而网格特征的缺陷可能导致建模过程中丢失目标语义. 为此,本文提出DSCC模块,通过双重语义协作的方式进行视觉文本跨模态建模,综合网格和目标2个层面的语义信息构建上下文感知的视觉特征表示. 模块结构如图4所示.
在Transformer的解码器结构中,每个解码层接收相同的视觉特征,在堆叠过程中仅对描述语句逐步精细化地预测,因此,将DSF对区域特征的处理过程安排在解码层外,以保证各层中接收到的视觉信息相同. 解码层内部的DSCC模块中,首先计算文本特征与网格特征、融合特征间的语义关联情况;然后借助重叠矩阵将文本与目标的语义关联感知结果映射到文本与网格间,与原本基于网格内容的感知结果融合,并引入超参数对映射过程进行控制,以避免注意力机制失效;最后利用融合后的关联矩阵整合视觉特征,综合目标和网格2个层面的语义信息构建上下文感知的视觉特征表示,用来进行下一个单词的预测. 公式为
$ \begin{split} {{\boldsymbol {M}}_{{\text{TGS}}}}({\boldsymbol {T}},{{\boldsymbol {G}'}}) = &S im({\boldsymbol {T}},{{\boldsymbol {G}'}}){\text{,}}{{\boldsymbol {M}}_{{\text{TOS}}}}{\text{(}}{\boldsymbol {T}}{\text{,}}{{\boldsymbol {R}}_{\text{f}}}{\text{) = }}S im({\boldsymbol {T}},{{\boldsymbol {R}}_{\text{f}}}),\\&{{\boldsymbol {R}}_{\text{f}}} = {F_{{\text{DSF}}}}({\boldsymbol {R}},{\boldsymbol {G}},{\boldsymbol {G}}) \text{,}\\[-1pt] \end{split}$
$ \boldsymbol {M}'_{{\text{TGS}}} = \lambda \left( {{{\boldsymbol {E}}^{\text{T}}}({{\boldsymbol {M}}_{{\text{TOS}}}}){\boldsymbol {E}}} \right) + {{\boldsymbol {M}}_{{\text{TGS}}}} \text{,} $
$ {\boldsymbol {T}}' = {{softmax}}(\boldsymbol {M}'_{{\text{TGS}}}){\boldsymbol {G}}{{\boldsymbol {W}}^{\boldsymbol {g}}}\text{,} $
其中T表示文本特征,MTGS表示文本特征与网格特征的语义关联矩阵,MTOS表示文本特征与融合特征的语义关联矩阵,$ \boldsymbol {M}'_{{\text{TGS}}} $表示基于网格和目标的双重语义关联矩阵,${{\boldsymbol {W}}^{\boldsymbol {g}}}$表示可学习的参数矩阵,$ \lambda $表示超参数. 该过程的运算中引入了超参数,这是因为文本特征与视觉特征间不存在几何层面的相对位置关系,若此处使用与编码器中相同的处理方式,则会导致$ \boldsymbol {M}'_{{\text{TGS}}} $中的元素相近,无法发挥出注意力机制应有的效果,继而导致描述精度下降.
综上所述,该模块中将双重语义协作的思想融入视觉文本的跨模态建模过程,综合目标和网格2个层面的语义信息构建上下文感知的视觉特征表示,利用该特征表示进行描述语句预测,避免结果中缺少目标相关的词汇.
使用与主流方法相同的两阶段策略训练本文提出的模型:1)通过交叉熵损失进行预训练;2)通过强化学习[16]进行训练,以CIDEr分数作为奖励. 在第1阶段中,给定各时间步的真实标签${Y_{1:T}} = \{ {y_1},{y_2},…,{y_T}\} $,将各时间步的交叉熵损失之和作为描述语句的交叉熵损失,公式为
$ {L_{{\text{XE}}}} = - \sum\limits_{t = 1}^T {\ln ({p_\theta }({y_t}|{y_{1:t - 1}}))} \text{,} $
其中$\theta $表示模型的参数,${y_{1:t - 1}}$表示前t–1个时间步的预测结果,式(14)从分类任务的角度进行设计,但是描述任务中各时间步均依赖已有的结果进行预测,容易造成误差积累,同时交叉熵损失与评估指标不相关,所以该式仅能用于预训练而无法进一步提高描述的准确度. 在第2阶段中,将CIDEr分数作为奖励进行优化,目标是最大限度地减少负奖励期望,公式为
$\begin{aligned} {\nabla _\theta }{L_{{\text{RL}}}}(\theta ) =- \frac{1}{k}\sum\limits_{i = 1}^k {\left( {r(y_{1:T}^i) - b} \right)} {\nabla _\theta }\ln \left( {{p_\theta }(y_{1:T}^i)} \right),\end{aligned} $
$\begin{aligned} b = \frac{1}{k}\sum\limits_{i = 1}^k {r(y_{1:T}^{i})} \text{,}\end{aligned} $
其中k是束大小,r是计算CIDEr分数的函数,b是基线,通常使用均值作为基线. 式(15)从强化学习的角度进行设计,直接利用评估指标对模型进行优化,能够进一步提高描述准确度,同时避免了只用1个交叉熵损失函数带来的可能陷入局部最优的问题.
实验在COCO数据集上进行,数据集共有164 262张图片,每张图片对应5条人工标注的描述语句,其中123 287张图片公开标注信息,供研究者下载使用,40 775张图片用作线上测试,不公开标注信息. 本文与主流方法一样使用Karpathy等人[11]提供的划分方式,将COCO数据集可供下载的图片划分为训练集、验证集和测试集,其中训练集中包含113 287张图片,验证集和测试集各为5 000张图片. 此外,通过COCO提供的在线测试平台对未公开标注的图片进行在线测试,进一步衡量模型的性能. 采用5种在图像描述任务中广泛使用的评价指标来分析生成描述的质量,衡量模型性能,包括BLEU(bilingual evaluation understudy),METEOR(metric for evaluation of translation with explicit ordering),ROUGE(recall-oriented understudy for gisting evaluation),CIDEr(consensus-based image description evaluation),SPICE(semantic propositional image caption evaluation). 基于图像描述任务的特点,一般将CIDEr得分作为衡量描述语句质量的主要依据.
为了公平对比,本文在设置实验时遵循主流方法中的超参数设置,将编码器中的编码层数量和解码器中的解码层数量均设置为3,多头注意力的头数设置为8,Transformer内部特征的维度设置为512.在交叉熵损失训练阶段,通过4轮迭代完成对模型的预热,在预热时令学习率线性增长到$1 \times {10^{ - 4}}$,在第5~10轮迭代学习率固定为$1 \times {10^{ - 4}}$,在第11~12轮迭代学习率减小到$1 \times {10^{ - 5}}$,从第13轮迭代开始学习率设置为$1 \times {10^{ - 6}}$,并不再改变. 在完成17轮迭代后进入强化学习阶段,强化学习阶段的学习率固定为$5 \times {10^{ - 6}}$. 批次大小在交叉熵训练阶段设置为50,在强化学习阶段设置为20,这2个阶段都使用Adam优化器进行参数调整,强化学习和测试阶段通过束搜索的方式生成描述,束大小设置为5.
表1展示了本文方法与当前主流方法在离线测试中的性能对比,其中第1部分是主流方法使用传统视觉特征时的描述性能;第2部分是将传统视觉特征替换为由CLIP视觉编码器(主干网络为ResNet101)提取网格特征后的性能. 由于本文方法主要针对CLIP网格特征的不足进行设计,所以为了保证公平,主要与第2部分进行比较. 可以看到,本文方法在离线测试中展现出优异的性能,尤其在图像描述领域中最看重的CIDEr方面,取得了138.5%的得分,与传统视觉特征下表现最出色的RSTNet相比提高了5.2个百分点,与利用CLIP网格特征复现后的RSTNet相比提高了1.5个百分点. 此外,在BLEU-1,BLEU-4,METEOR,ROUGE方面取得了超越RSTNet的性能,在SPICE方面与之相当.
本文方法在CLIP网格特征下超越了RSTNet,同时RSTNet在传统视觉特征下表现最佳,这表明本文方法在结构设计方面超越了以往所有方法,具有显著优势.
通过将模型对官方测试集的预测结果提交到在线服务器,进一步对本文方法进行客观公正的评估. 表2展示了在线测试中本文方法与当前主流方法的表现,其中每个指标都展示了在5个参考字幕(c5)和40个参考字幕(c40)下的得分情况. 由于排行榜上展示的大多是集成模型的预测结果,因此本文也参照之前的方法使用4个模型的集成进行在线测试. 可以看到,本文提出的DSC-Net在图像描述领域中最看重的是在CIDEr上对于c5和c40分别取得了135.8%和137.6%的成绩,与表现优秀的RSTNet相比在CIDEr的c5和c40上分别增长了3.9个百分点和3.6个百分点.
由于在线测试榜单上的其他方法没有使用CLIP网格特征,为了保证公平,本文对使用CLIP网格特征复现的RSTNet模型同样进行了在线测试,令其与本文方法的单一模型进行在线测试结果比较,可以看到本文提出的DSC-Net在单一模型的大部分指标上都取得了最优的性能,特别是在CIDEr的c5上取得了132.7个百分点的成绩,与RSTNet相比提高了1.3个百分点. 在线测试的结果充分证明了本文提出模块的有效性和本文方法的先进性.
为了进一步评估本文方法的性能,验证本文提出各模块的有效性,对模型进行消融实验,表3展示了针对本文提出模块进行消融实验的结果. 基于图像任务的需要,遵循Guo等人[7]的方法对视觉特征进行相对位置编码,并以此作为本次实验的基线模型,以图像描述中最为看重的CIDEr得分作为评价指标. 可以看到,仅使用基线模型可以取得136.3%的CIDEr得分,增加DSCS模块后取得137.4%的CIDEr得分,增加DSCC模块后可以取得136.6%的CIDEr得分,同时增加DSCS和DSCC时可以取得137.1%的CIDEr得分,同时增加DSCS和DSF时可以取得137.7%的CIDEr得分,同时增加DSCC和DSF时可以取得137.5%的CIDEr得分,同时使用所有模块时可以取得138.5%的CIDEr得分.
在本文提出的DSCC模块中,引入了超参数$\lambda $来对矩阵中元素的值进行限制,为了得到合适的超参数取值,通过为$\lambda $赋予不同的值进行实验,同时进行敏感性分析,同样以CIDEr作为评价指标,实验结果如图5所示. 可以看到,当$\lambda $=0.04时取得了最高的CIDEr分数,所以采用0.04作为超参数的取值,且$\lambda $的变化对CIDEr分数的影响不大,所以认为模型对超参数的变化不敏感.
为了评估双重语义协作机制对提升描述性能的影响,设置实验进行验证,实验结果如表4所示. 其中第1行展示了仅使用网格特征进行描述的结果,可以取得136.3%的CIDEr分数,以此作为基线;第2行展示了同时使用区域特征和网格特征,但不使用双重语义协作机制的结果,可以取得136.8%的CIDEr分数,与基线相比提升0.5个百分点;第3行展示了同时使用区域特征和网格特征,但仅使用双重语义协作自注意力模块的结果,可以取得137.7%的CIDEr分数;第4行展示了同时使用区域特征和网格特征,但仅使用双重语义协作交叉注意力模块的结果,可以取得137.5%的CIDEr分数;第5行展示了使用区域特征和网格特征,且同时使用2个模块进行处理的结果,可以取得138.5%的CIDEr分数,与基线相比提升2.2个百分点. 这表明本文提出DSC-Net对描述性能的提升主要得益于对区域和网格特点的准确分析和对双重语义协作结构的巧妙设计.
为了评估以区域为主导的特征融合机制对提升描述性能的影响,设置实验进行验证,实验结果如表5所示. 其中第1行展示禁用DSF模块后DSC-Net的性能,可以取得137.1%的CIDEr分数,与表4中w/o DSC一栏展示的结果相比在性能方面提升不大;第2行展示了在特征融合时将以区域特征为主导改为以网格特征为主导的结果,记为w/o RFD,可以取得137.3%的CIDEr分数,与禁用DSF模块相比提升0.2个百分点;第3行展示了特征融合时以区域为主导的结果,可以取得138.5%的CIDEr分数,与禁用DSF模块相比提升1.4个百分点. 这证明了以区域为主导的特征融合有助于解决相关性冲突问题,更证明了在双重语义融合模块中设置3种机制保证区域特征主导地位的合理性.
表5表明相关性冲突确实存在于双重语义协作过程中,且会影响描述性能的提高. 进行特征融合可以缓解相关性冲突,且融合过程中以区域为主导能够取得更大的性能提升.
为了更直观地展示本文提出的模型在描述图像内容方面的出色性能,特别是展示经过本文提出的双重语义协作机制处理后模型对待描述图像中重要目标的准确表述能力,图6展示了本文提出的模型和基线模型生成的描述语句,并与人工标注的语句进行对比. 如图6(a)所示,基线模型认为包含2头大象(two elephants),本文提出的模型则认为是1头大象和1头牛(a elephant and a cow);如图6(b)所示,基线模型认为包含1个年轻的男孩(a young boy),本文提出的模型则判断出是1个棒球运动员(a baseball player);如图6(c)所示,基线模型认为女人和孩子在行李旁边(next to bunch of luggage),本文提出的模型则关注到孩子在拿着行李(child with her luggage);如图6(d)所示,基线模型认为一群大象在动物园,本文提出的模型则关注到有人出现(with people). 综上所述,与基线模型相比,本文提出的模型更容易关注到待描述图片中的目标和目标之间的关系,所以对图像的描述结果往往更加符合其内容,同时,这也表明了本文提出的双重语义协作机制可以有效地解决网格特征对目标缺乏关注的问题.
针对网格特征中缺少目标层面语义信息的问题,本文提出一种基于DSC-Net的图像描述方法. 通过DSCS进行目标间的语义关联感知,然后将其映射到对应位置的网格上,融入网格特征的语义建模过程,增强网格特征对目标层面语义的表达能力;通过DSCC计算文本特征与区域特征之间的语义相关性,然后映射到对应位置的网格上,使视觉文本的跨模态建模过程可以综合网格和目标2个层面的语义信息构建上下文感知的视觉特征表示,最后将特征表示中蕴含的视觉语义信息精准地转化为自然语言;通过DSF进行以区域为主导的特征融合,在保留目标感知能力的同时融入网格信息,避免双重语义协作过程中可能出现的相关性冲突问题. 实验结果表明,本文提出的方法有效提高了图像描述模型预测描述语句的准确度,与目前的主流方法相比具有明显优势. 在下一步工作中,将探索更多为网格特征补充目标语义信息的策略,减少对区域特征的依赖,提高描述速度以便应用到轻量化的移动智能终端设备上.
作者贡献声明:江泽涛提出了论文整体思路并负责撰写和修改论文;朱文才完成算法设计与实验并撰写与修改论文;金鑫提出指导意见并修改论文;廖培期负责图表绘制;黄景帆参与了论文审阅与格式校正.

参考文献 引证文献
排序方式:
1
李志欣,魏海洋,张灿龙,等. 图像描述生成研究进展[J]. 计算机研究与发展,2021,58(9):1951−1974
Li Zhixin, Wei Haiyang, Zhang Canlong, et al. Research progress on image captioning[J]. Journal of Computer Research and Development, 2021, 58(9): 1951−1974 (in Chinese)
2
Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proc of the 31st Advances in Neural Information Processing Systems. Cambridge, MA: MIT, 2017 : 5998−6008
3
Vinyals O, Toshev A, Bengio S, et al. Show and tell: A neural image caption generator[C]//Proc of the 28th IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2015: 3156−3164
4
Xu K, Ba J, Kiros R, et al. Show, attend and tell: Neural image caption generation with visual attention[C]//Proc of the 32nd Int Conf on Machine Learning. New York: ACM, 2015: 2048−2057
5
Anderson P, He Xiaodong, Buehler C, et al. Bottom-up and top-down attention for image captioning and visual question answering[C]//Proc of the 31st IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 6077−6086
6
Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//Proc of the 38th Int Conf on Machine Learning. New York: ACM, 2021: 8748−8763
7
Guo Longteng, Liu Jing, Zhu Xinxin, et al. Normalized and geometry-aware self-attention network for image captioning[C]//Proc of the 33rd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 10327−10336
8
Pan Yingwei, Yao Ting, Li Yehao, et al. X-linear attention networks for image captioning[C]//Proc of the 33rd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 10971−10980
9
Shen Sheng, Li L H, Tan Hao, et al. How much can CLIP benefit vision-and-language tasks[C/OL]//Proc of the 10th Int Conf on Learning Representations. 2022[2022-01-29].https://openreview.net/forum?id=zf_Ll3HZWgy
10
Li Yehao, Pan Yingwei, Yao Ting, et al. Comprehending and ordering semantics for image captioning[C]//Proc of the 35th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 17990−17999
11
Karpathy A, Li Feifei. Deep visual-semantic alignments for generating image descriptions[C]//Proc of the 28th IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2015: 3128−3137
12
Yang Xu, Tang Kaihua, Zhang Hanwang, et al. Auto-encoding scene graphs for image captioning[C]//Proc of the 32nd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 10685−10694
13
Cornia M, Stefanini M, Baraldi L, et al. Meshed-memory transformer for image captioning[C]//Proc of the 33rd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 10578−10587
14
Ji Jiayi, Luo Yunpeng, Sun Xiaoshuai, et al. Improving image captioning by leveraging intra-and inter-layer global representation in transformer network[C]//Proc of the 35th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2021: 1655−1663
15
Yang Xu, Gao Chongyang, Zhang Hanwang, et al. Auto-parsing network for image captioning and visual question answering[C]//Proc of the 18th IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2021: 2197−2207
16
Rennie S J, Marcheret E, Mroueh Y, et al. Self-critical sequence training for image captioning[C]//Proc of the 30th IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2017: 7008−7024
17
Sharma P, Ding Nan, Goodman S, et al. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning[C]//Proc of the 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2018: 2556−2565
18
Zhang Xuying, Sun Xiaoshuai, Luo Yunpeng, et al. RSTNet: Captioning with adaptive attention on visual and non-visual words[C]//Proc of the 34th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2021: 15465−15474
19
Yang Xuewen, Liu Yingru, Wang Xin. ReFormer: The relational transformer for image captioning[C]//Proc of the 30th ACM Int Conf on Multimedia. New York: ACM, 2022: 5398−5406
20
Wang Chi, Shen Yulin, Ji Luping. Geometry attention Transformer with position-aware LSTMs for image captioning[J]. Expert Systems with Applications, 2022, 201: 117174
21
Huang Lun, Wang Wenmin, Chen Jie, et al. Attention on attention for image captioning[C]//Proc of the 17th IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 4634−4643
2024年第61卷第11期
83
41
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202330523
  • 接收时间:2023-06-19
  • 首发时间:2026-11-17
  • 出版时间:2024-11-01
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2023-06-19
  • 修回日期:2023-12-19
基金
作者信息
    广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004

通讯作者:

朱文才()
参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202330523
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏