Article(id=1329460761270792884, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1329460564176253901, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202330523, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=null, receivedDate=1687104000000, receivedDateStr=2023-06-19, revisedDate=1702915200000, revisedDateStr=2023-12-19, acceptedDate=null, acceptedDateStr=null, onlineDate=1794897745672, onlineDateStr=2026-11-17, pubDate=1730390400000, pubDateStr=2024-11-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1794897745672, onlineIssueDateStr=2026-11-17, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1794897745672, creator=13701087609, updateTime=1794897745672, updator=13701087609, issue=Issue{id=1329460564176253901, tenantId=1146029695717560320, journalId=1324759914146529478, year='2024', volume='61', issue='11', pageStart='2663', pageEnd='2936', issueExtLink='null', onlineDate='null', pubDate='1730390400000', pubDateStr='2024-11-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1794897698682, creator='13701087609', updateTime=1794897698682, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext=null, issueFiles=null, downloadFileDto=null}, startPage=2897, endPage=2908, ext={EN=ArticleExt(id=1329460761472119477, articleId=1329460761270792884, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=An Image Captioning Method Based on DSC-Net, columnId=1327013252946572224, journalTitle=Journal of Computer Research and Development, columnName=Graphics and Image Processing, runingTitle=null, highlight=null, articleAbstract=
As visual features closer to the text domain, the grid features extracted by the CLIP (contrastive language-image pre-training) image encoder are easy to convert into the corresponding semantic natural language, which can alleviate the semantic gap problem, so it may become an important source of visual features in the image captioning in the future. However, this method does not consider that the division of image content may cause a complete object to be divided into several grids. The segmentation of the objects will inevitably lead to the lack of a complete expression of the object information in the feature extraction results, and further lead to the lack of an accurate expression of the object and the relationship between the objects in the generated sentence. Aiming at the phenomenon of grid features extracted by CLIP image encoder, we propose dual semantic collaborative network (DSC-Net) for image captioning. Specifically, dual semantic collaborative self-attention (DSCS) module is first proposed to enhance the expression of object information by CLIP grid features. Then dual semantic collaborative cross-attention (DSCC) module is proposed to integrate semantic information between grid and object to generate visual features, and to be used to predict sentences. Finally, dual semantic fusion (DSF) module is proposed to provide region-oriented fusion features for the above two semantic cooperation modules, and to solve the problem of correlation conflicts that may arise in the process of semantic cooperation. After a large number of experiments on the COCO dataset, the proposed model achieves a CIDEr score of 138.5% on the offline test set divided by Karpathy et al., and a CIDEr score of 137.6% in the official online test. Compared with the current mainstream image captioning methods, this result has obvious advantages.
, authors=Zetao Jiang, Wencai Zhu*, Xin Jin, Peiqi Liao, Jingfan Huang, authorsList=Zetao Jiang, Wencai Zhu, Xin Jin, Peiqi Liao, Jingfan Huang, authorCompany=null, correspAuthors=Wencai Zhu, authorNote=
born in 1961. PhD, professor. His main research interests include image processing, computer vision, and artificial intelligence
born in 1999. Master. His main research interests include image processing and computer vision
born in 1998. Master. His main research interests include image processing and computer vision
born in 1995. Master. His main research interests include image processing and computer vision
born in 1999. Master. His main research interests include image processing and computer vision
, correspAuthorsNote=null, copyrightStatement=Copyright © 2024 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1329460763506356941, articleId=1329460761270792884, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=一种基于双重语义协作网络的图像描述方法, columnId=1329460761551811254, journalTitle=计算机研究与发展, columnName=图形与图像处理, runingTitle=null, highlight=null, articleAbstract=
CLIP(contrastive language-image pre-training)视觉编码器提取的网格特征作为一种更加靠近文本域的视觉特征,具有易转化为对应语义自然语言的特点,可以缓解语义鸿沟问题,因而未来可能成为图像描述任务中视觉特征的重要来源. 但该方法中未考虑图像内容的划分,可能使一个完整的目标被划分到若干个网格中,目标被切割势必会导致特征提取结果中缺少对目标信息的完整表达,进而导致生成的描述语句中缺少对目标及目标间关系的准确表述. 针对CLIP视觉编码器提取网格特征这一现象,提出一种基于双重语义协作网络(dual semantic collaborative network,DSC-Net)的图像描述方法. 具体来说:首先提出双重语义协作自注意力(dual semantic collaborative self-attention,DSCS)模块增强CLIP网格特征对目标信息的表达能力;接着提出双重语义协作交叉注意力(dual semantic collaborative cross-attention,DSCC)模块,综合网格和目标2个层面的语义构造与文本相关的视觉特征,进行描述语句预测;最后提出双重语义融合(dual semantic fusion,DSF)模块,为上述的2个语义协作模块提供以区域为主导的融合特征,解决在语义协作过程中可能出现的相关性冲突问题. 经过在COCO数据集上的大量实验,提出的模型在Karpathy等人划分的离线测试集上取得了138.5%的CIDEr分数,在官方在线测试中取得了137.6%的CIDEr分数,与目前主流的图像描述方法相比具有显著优势.
, authors=江泽涛, 朱文才*, 金鑫, 廖培期, 黄景帆, authorsList=江泽涛, 朱文才, 金鑫, 廖培期, 黄景帆, authorCompany=null, correspAuthors=朱文才, authorNote=
 |
1961年生. 博士,教授. 主要研究方向为图像处理、计算机视觉、人工智能 |
 |
1999年生. 硕士. 主要研究方向为图像处理、计算机视觉 |
 |
1998年生. 硕士. 主要研究方向为图像处理、计算机视觉 |
 |
1995年生. 硕士. 主要研究方向为图像处理、计算机视觉 |
 |
1999年生. 硕士. 主要研究方向为图像处理、计算机视觉 |
, correspAuthorsNote=
, copyrightStatement=版权所有 © 2024 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=czeHS/HWM2i5bWgZPPSUKg==, magXml=jDMAV2tmGnN45EzHSx95Kw==, pdfUrl=null, pdf=kVk4SlAj2YP5aCqe+zvmww==, pdfFileSize=1176303, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=yuHn60h1HZVsn/H+MZoVSg==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=KRzLwOuDYs+xEFHCl2kVfA==, mapNumber=null, fund=null)}, authors=[Author(id=1329460763783181010, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=zetaojiang@126.com, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329460763883844308, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460763783181010, language=EN, stringName=Zetao Jiang, firstName=Zetao, middleName=null, lastName=Jiang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004, bio={"content":"
born in 1961. PhD, professor. His main research interests include image processing, computer vision, and artificial intelligence
"}, bioImg=null, bioContent=
born in 1961. PhD, professor. His main research interests include image processing, computer vision, and artificial intelligence
, aboutCorrespAuthor=null), CN=AuthorExt(id=1329460763950953173, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460763783181010, language=CN, stringName=江泽涛, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004, bio={"img":"K0PyxtGhzZbmPXwP+lNCXw==","content":"
1961年生. 博士,教授. 主要研究方向为图像处理、计算机视觉、人工智能
"}, bioImg=K0PyxtGhzZbmPXwP+lNCXw==, bioContent=
1961年生. 博士,教授. 主要研究方向为图像处理、计算机视觉、人工智能
, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329460763711877838, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, xref=null, ext=[AuthorCompanyExt(id=1329460763716072143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004), AuthorCompanyExt(id=1329460763724460752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004)])]), Author(id=1329460764043227863, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=zhuwencai00@qq.com, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329460764110336729, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764043227863, language=EN, stringName=Wencai Zhu, firstName=Wencai, middleName=null, lastName=Zhu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004, bio={"content":"
born in 1999. Master. His main research interests include image processing and computer vision
"}, bioImg=null, bioContent=
born in 1999. Master. His main research interests include image processing and computer vision
, aboutCorrespAuthor=null), CN=AuthorExt(id=1329460764177445594, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764043227863, language=CN, stringName=朱文才, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004, bio={"img":"RHbqdykSWeY1WKf2q2Ottw==","content":"
1999年生. 硕士. 主要研究方向为图像处理、计算机视觉
"}, bioImg=RHbqdykSWeY1WKf2q2Ottw==, bioContent=
1999年生. 硕士. 主要研究方向为图像处理、计算机视觉
, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329460763711877838, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, xref=null, ext=[AuthorCompanyExt(id=1329460763716072143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004), AuthorCompanyExt(id=1329460763724460752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004)])]), Author(id=1329460764257137372, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329460764361994974, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764257137372, language=EN, stringName=Xin Jin, firstName=Xin, middleName=null, lastName=Jin, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004, bio={"content":"
born in 1998. Master. His main research interests include image processing and computer vision
"}, bioImg=null, bioContent=
born in 1998. Master. His main research interests include image processing and computer vision
, aboutCorrespAuthor=null), CN=AuthorExt(id=1329460764424909535, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764257137372, language=CN, stringName=金鑫, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004, bio={"img":"jXy+WjkR0BEIeJJ5oTIg+g==","content":"
1998年生. 硕士. 主要研究方向为图像处理、计算机视觉
"}, bioImg=jXy+WjkR0BEIeJJ5oTIg+g==, bioContent=
1998年生. 硕士. 主要研究方向为图像处理、计算机视觉
, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329460763711877838, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, xref=null, ext=[AuthorCompanyExt(id=1329460763716072143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004), AuthorCompanyExt(id=1329460763724460752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004)])]), Author(id=1329460764487824097, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329460764550738659, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764487824097, language=EN, stringName=Peiqi Liao, firstName=Peiqi, middleName=null, lastName=Liao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004, bio={"content":"
born in 1995. Master. His main research interests include image processing and computer vision
"}, bioImg=null, bioContent=
born in 1995. Master. His main research interests include image processing and computer vision
, aboutCorrespAuthor=null), CN=AuthorExt(id=1329460764626236132, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764487824097, language=CN, stringName=廖培期, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004, bio={"img":"1Df8KRPAFXMKVJLjoVLcyg==","content":"
1995年生. 硕士. 主要研究方向为图像处理、计算机视觉
"}, bioImg=1Df8KRPAFXMKVJLjoVLcyg==, bioContent=
1995年生. 硕士. 主要研究方向为图像处理、计算机视觉
, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329460763711877838, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, xref=null, ext=[AuthorCompanyExt(id=1329460763716072143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004), AuthorCompanyExt(id=1329460763724460752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004)])]), Author(id=1329460764689150694, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329460764777231080, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764689150694, language=EN, stringName=Jingfan Huang, firstName=Jingfan, middleName=null, lastName=Huang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004, bio={"content":"
born in 1999. Master. His main research interests include image processing and computer vision
"}, bioImg=null, bioContent=
born in 1999. Master. His main research interests include image processing and computer vision
, aboutCorrespAuthor=null), CN=AuthorExt(id=1329460764856922857, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, authorId=1329460764689150694, language=CN, stringName=黄景帆, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004, bio={"img":"JML2tZfo7ovlsMlvYY7jqg==","content":"
1999年生. 硕士. 主要研究方向为图像处理、计算机视觉
"}, bioImg=JML2tZfo7ovlsMlvYY7jqg==, bioContent=
1999年生. 硕士. 主要研究方向为图像处理、计算机视觉
, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329460763711877838, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, xref=null, ext=[AuthorCompanyExt(id=1329460763716072143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004), AuthorCompanyExt(id=1329460763724460752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004)])])], keywords=[Keyword(id=1329460764953391850, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, orderNo=1, keyword=image captioning), Keyword(id=1329460765028889323, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, orderNo=2, keyword=grid features), Keyword(id=1329460765100192492, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, orderNo=3, keyword=attention mechanism), Keyword(id=1329460765158912749, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, orderNo=4, keyword=dual semantic collaborative attention), Keyword(id=1329460765230215918, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, orderNo=5, keyword=dual semantic collaborative feature fusion), Keyword(id=1329460765309907695, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, orderNo=1, keyword=图像描述), Keyword(id=1329460765372822256, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, orderNo=2, keyword=网格特征), Keyword(id=1329460765431542513, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, orderNo=3, keyword=注意力机制), Keyword(id=1329460765494457074, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, orderNo=4, keyword=双重语义协作注意力), Keyword(id=1329460765548983027, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, orderNo=5, keyword=双重语义协作特征融合)], refs=[Reference(id=1329460767444808458, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=李志欣,魏海洋,张灿龙,等. 图像描述生成研究进展[J]. 计算机研究与发展,2021,58(9):1951−1974, articleTitle=null, refAbstract=null), Reference(id=1329460767507723019, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Zhixin, Wei Haiyang, Zhang Canlong, et al. Research progress on image captioning[J]. Journal of Computer Research and Development, 2021, 58(9): 1951−1974 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1329460767574831884, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proc of the 31st Advances in Neural Information Processing Systems. Cambridge, MA: MIT, 2017 : 5998−6008, articleTitle=null, refAbstract=null), Reference(id=1329460767646135053, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Vinyals O, Toshev A, Bengio S, et al. Show and tell: A neural image caption generator[C]//Proc of the 28th IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2015: 3156−3164, articleTitle=null, refAbstract=null), Reference(id=1329460767709049614, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Xu K, Ba J, Kiros R, et al. Show, attend and tell: Neural image caption generation with visual attention[C]//Proc of the 32nd Int Conf on Machine Learning. New York: ACM, 2015: 2048−2057, articleTitle=null, refAbstract=null), Reference(id=1329460767776158479, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Anderson P, He Xiaodong, Buehler C, et al. Bottom-up and top-down attention for image captioning and visual question answering[C]//Proc of the 31st IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 6077−6086, articleTitle=null, refAbstract=null), Reference(id=1329460767851655952, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//Proc of the 38th Int Conf on Machine Learning. New York: ACM, 2021: 8748−8763, articleTitle=null, refAbstract=null), Reference(id=1329460767935542033, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Guo Longteng, Liu Jing, Zhu Xinxin, et al. Normalized and geometry-aware self-attention network for image captioning[C]//Proc of the 33rd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 10327−10336, articleTitle=null, refAbstract=null), Reference(id=1329460768011039506, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Pan Yingwei, Yao Ting, Li Yehao, et al. X-linear attention networks for image captioning[C]//Proc of the 33rd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 10971−10980, articleTitle=null, refAbstract=null), Reference(id=1329460768099119891, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Shen Sheng, Li L H, Tan Hao, et al. How much can CLIP benefit vision-and-language tasks[C/OL]//Proc of the 10th Int Conf on Learning Representations. 2022[2022-01-29].https://openreview.net/forum?id=zf_Ll3HZWgy, articleTitle=null, refAbstract=null), Reference(id=1329460768178811668, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Yehao, Pan Yingwei, Yao Ting, et al. Comprehending and ordering semantics for image captioning[C]//Proc of the 35th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 17990−17999, articleTitle=null, refAbstract=null), Reference(id=1329460768241726229, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Karpathy A, Li Feifei. Deep visual-semantic alignments for generating image descriptions[C]//Proc of the 28th IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2015: 3128−3137, articleTitle=null, refAbstract=null), Reference(id=1329460768313029398, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Xu, Tang Kaihua, Zhang Hanwang, et al. Auto-encoding scene graphs for image captioning[C]//Proc of the 32nd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 10685−10694, articleTitle=null, refAbstract=null), Reference(id=1329460768375943959, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Cornia M, Stefanini M, Baraldi L, et al. Meshed-memory transformer for image captioning[C]//Proc of the 33rd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 10578−10587, articleTitle=null, refAbstract=null), Reference(id=1329460768472412952, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Ji Jiayi, Luo Yunpeng, Sun Xiaoshuai, et al. Improving image captioning by leveraging intra-and inter-layer global representation in transformer network[C]//Proc of the 35th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2021: 1655−1663, articleTitle=null, refAbstract=null), Reference(id=1329460768543716121, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Xu, Gao Chongyang, Zhang Hanwang, et al. Auto-parsing network for image captioning and visual question answering[C]//Proc of the 18th IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2021: 2197−2207, articleTitle=null, refAbstract=null), Reference(id=1329460768656962330, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Rennie S J, Marcheret E, Mroueh Y, et al. Self-critical sequence training for image captioning[C]//Proc of the 30th IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2017: 7008−7024, articleTitle=null, refAbstract=null), Reference(id=1329460768732459803, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Sharma P, Ding Nan, Goodman S, et al. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning[C]//Proc of the 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2018: 2556−2565, articleTitle=null, refAbstract=null), Reference(id=1329460768833123100, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Xuying, Sun Xiaoshuai, Luo Yunpeng, et al. RSTNet: Captioning with adaptive attention on visual and non-visual words[C]//Proc of the 34th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2021: 15465−15474, articleTitle=null, refAbstract=null), Reference(id=1329460768904426269, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Xuewen, Liu Yingru, Wang Xin. ReFormer: The relational transformer for image captioning[C]//Proc of the 30th ACM Int Conf on Multimedia. New York: ACM, 2022: 5398−5406, articleTitle=null, refAbstract=null), Reference(id=1329460768992506654, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Chi, Shen Yulin, Ji Luping. Geometry attention Transformer with position-aware LSTMs for image captioning[J]. Expert Systems with Applications, 2022, 201: 117174, articleTitle=null, refAbstract=null), Reference(id=1329460769076392735, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Lun, Wang Wenmin, Chen Jie, et al. Attention on attention for image captioning[C]//Proc of the 17th IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 4634−4643, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1329460763711877838, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, xref=null, ext=[AuthorCompanyExt(id=1329460763716072143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Guangxi Key Laboratory of Image and Graphic Intelligent Processing (Guilin University of Electronic Technology), Guilin, Guangxi 541004), AuthorCompanyExt(id=1329460763724460752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, companyId=1329460763711877838, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=广西图像图形智能处理重点实验室(桂林电子科技大学) 广西桂林 541004)])], figs=[ArticleFig(id=1329460765670617844, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Figure 1, caption=
DSC-Net structure diagram, figureFileSmall=zr3zlPdue6/8op9up1axBQ==, figureFileBig=yuHn60h1HZVsn/H+MZoVSg==, tableContent=null), ArticleFig(id=1329460765762892533, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=图1, caption=
DSC-Net结构图, figureFileSmall=zr3zlPdue6/8op9up1axBQ==, figureFileBig=yuHn60h1HZVsn/H+MZoVSg==, tableContent=null), ArticleFig(id=1329460765930664694, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Figure 2, caption=
DSCS structure diagram, figureFileSmall=f+NsZvAxwwjI+URXJt1MlA==, figureFileBig=iCtrHsk/USSqgWXujTngaQ==, tableContent=null), ArticleFig(id=1329460765993579255, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=图2, caption=
DSCS结构图, figureFileSmall=f+NsZvAxwwjI+URXJt1MlA==, figureFileBig=iCtrHsk/USSqgWXujTngaQ==, tableContent=null), ArticleFig(id=1329460766060688120, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Figure 3, caption=
DSF structure diagram, figureFileSmall=Ga1qFQPOk50S6XoSQwxORA==, figureFileBig=HwqIxC/eqk/BSVCOKPHORg==, tableContent=null), ArticleFig(id=1329460766127796985, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=图3, caption=
DSF结构图, figureFileSmall=Ga1qFQPOk50S6XoSQwxORA==, figureFileBig=HwqIxC/eqk/BSVCOKPHORg==, tableContent=null), ArticleFig(id=1329460766203294458, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Figure 4, caption=
DSCC structure diagram, figureFileSmall=jOqRj60p2un9Y1ahSZpIiQ==, figureFileBig=LzdnFurNhE/xX8qSyo/CYg==, tableContent=null), ArticleFig(id=1329460766266209019, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=图4, caption=
DSCC结构图, figureFileSmall=jOqRj60p2un9Y1ahSZpIiQ==, figureFileBig=LzdnFurNhE/xX8qSyo/CYg==, tableContent=null), ArticleFig(id=1329460766337512188, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Figure 5, caption=
Impact of hyperparameter on performance of model, figureFileSmall=uyuofFzA/3S679gWNjJWMg==, figureFileBig=aWXBPbj9uvCQ3hDLjBP+6Q==, tableContent=null), ArticleFig(id=1329460766396232445, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=图5, caption=
超参数对模型性能的影响, figureFileSmall=uyuofFzA/3S679gWNjJWMg==, figureFileBig=aWXBPbj9uvCQ3hDLjBP+6Q==, tableContent=null), ArticleFig(id=1329460766484312830, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Figure 6, caption=
Comparison of prediction results between DSC-Net and baseline, figureFileSmall=MU5OLyxb1c8vts5SBc1XFw==, figureFileBig=mGG1iSQVK2yuv7xHuSLKmQ==, tableContent=null), ArticleFig(id=1329460766551421695, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=图6, caption=
DSC-Net与基线的预测结果对比, figureFileSmall=MU5OLyxb1c8vts5SBc1XFw==, figureFileBig=mGG1iSQVK2yuv7xHuSLKmQ==, tableContent=null), ArticleFig(id=1329460766618530560, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Table 1, caption=
Performance Comparison of Our Method and Various Typical Methods in Offline Test %
, figureFileSmall=null, figureFileBig=null, tableContent=
| 方法 | BLEU- 1 | BLEU- 4 | METEOR | ROUGE | CIDEr | SPICE |
|---|
| up-down[5](CVPR’18) | 79.8 | 36.3 | 27.7 | 56.9 | 120.1 | 21.4 |
| Transformer[17](ACL’18) | 80.5 | 39.2 | 29.1 | 58.7 | 130.0 | 23.0 |
| SGAE[12](CVPR’19) | 80.8 | 38.4 | 27.4 | 58.6 | 127.8 | 22.1 |
M2Transformer[13] (CVPR’20) | 80.8 | 39.1 | 29.2 | 58.6 | 131.2 | 22.6 |
X-Transformer[8] (CVPR’20) | 80.9 | 39.7 | 29.5 | 59.1 | 132.8 | 23.4 |
| GET[14](AAAI’21) | 81.5 | 39.5 | 29.3 | 58.9 | 131.6 | 22.8 |
| APN[15] (ICCV’21) | | 39.6 | 29.2 | 59.1 | 131.8 | 23.0 |
| RSTNet[18](CVPR’21) | 81.1 | 39.3 | 29.4 | 58.8 | 133.3 | 23.0 |
| ReFormer[19](MM’22) | 82.3 | 39.8 | 29.7 | 59.8 | 131.9 | 23.0 |
GAT[20](Expert Syst. Appl.’22) | 80.8 | 39.7 | 29.1 | 59.0 | 130.5 | 22.9 |
| CLIP-ViL[9](ICLR’22) | | 40.2 | 29.7 | | 134.2 | 23.8 |
| up-down* | 81.3 | 39.4 | 29.2 | 59.3 | 131.9 | 22.8 |
| Transformer* | 81.6 | 40.6 | 29.9 | 59.8 | 136.2 | 23.9 |
| RSTNet* | 82.0 | 40.4 | 30.0 | 59.7 | 137.0 | 23.7 |
| DSC-Net(本文) | 82.9 | 41.1 | 30.1 | 59.9 | 138.5 | 23.7 |
), ArticleFig(id=1329460766740165377, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=表1, caption=
本文方法在离线测试中与各种典型方法的性能比较
, figureFileSmall=null, figureFileBig=null, tableContent=
| 方法 | BLEU- 1 | BLEU- 4 | METEOR | ROUGE | CIDEr | SPICE |
|---|
| up-down[5](CVPR’18) | 79.8 | 36.3 | 27.7 | 56.9 | 120.1 | 21.4 |
| Transformer[17](ACL’18) | 80.5 | 39.2 | 29.1 | 58.7 | 130.0 | 23.0 |
| SGAE[12](CVPR’19) | 80.8 | 38.4 | 27.4 | 58.6 | 127.8 | 22.1 |
M2Transformer[13] (CVPR’20) | 80.8 | 39.1 | 29.2 | 58.6 | 131.2 | 22.6 |
X-Transformer[8] (CVPR’20) | 80.9 | 39.7 | 29.5 | 59.1 | 132.8 | 23.4 |
| GET[14](AAAI’21) | 81.5 | 39.5 | 29.3 | 58.9 | 131.6 | 22.8 |
| APN[15] (ICCV’21) | | 39.6 | 29.2 | 59.1 | 131.8 | 23.0 |
| RSTNet[18](CVPR’21) | 81.1 | 39.3 | 29.4 | 58.8 | 133.3 | 23.0 |
| ReFormer[19](MM’22) | 82.3 | 39.8 | 29.7 | 59.8 | 131.9 | 23.0 |
GAT[20](Expert Syst. Appl.’22) | 80.8 | 39.7 | 29.1 | 59.0 | 130.5 | 22.9 |
| CLIP-ViL[9](ICLR’22) | | 40.2 | 29.7 | | 134.2 | 23.8 |
| up-down* | 81.3 | 39.4 | 29.2 | 59.3 | 131.9 | 22.8 |
| Transformer* | 81.6 | 40.6 | 29.9 | 59.8 | 136.2 | 23.9 |
| RSTNet* | 82.0 | 40.4 | 30.0 | 59.7 | 137.0 | 23.7 |
| DSC-Net(本文) | 82.9 | 41.1 | 30.1 | 59.9 | 138.5 | 23.7 |
), ArticleFig(id=1329460766832440066, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Table 2, caption=
Performance Comparison of Our Method and Various Typical Methods in Online Test %
, figureFileSmall=null, figureFileBig=null, tableContent=
| 方法 | BLEU-1 | | BLEU-2 | | BLEU-3 | | BLEU-4 | | METEOR | | ROUGE | | CIDEr |
|---|
| c5 | c40 | | c5 | c40 | | c5 | c40 | | c5 | c40 | | c5 | c40 | | c5 | c40 | | c5 | c40 |
|---|
| up-down[5](CVPR’18) | 80.2 | 95.2 | | 64.1 | 88.8 | | 49.1 | 79.4 | | 36.9 | 68.5 | | 27.6 | 36.7 | | 57.1 | 72.4 | | 117.9 | 120.5 |
| SGAE[12](CVPR’19) | 81.0 | 95.3 | | 65.6 | 89.5 | | 50.7 | 80.4 | | 38.5 | 69.7 | | 28.2 | 37.2 | | 58.6 | 73.6 | | 123.8 | 126.5 |
| AoANet[21](ICCV’19) | 81.0 | 95.0 | | 65.8 | 89.6 | | 51.4 | 81.3 | | 39.4 | 71.2 | | 29.1 | 38.5 | | 58.9 | 74.5 | | 126.9 | 129.6 |
| X-Transformer[8](CVPR’20) | 81.9 | 95.7 | | 66.9 | 90.5 | | 52.4 | 82.5 | | 40.3 | 72.4 | | 29.6 | 39.2 | | 59.5 | 75.0 | | 131.1 | 133.5 |
| M2Transformer[13](CVPR’20) | 81.6 | 96.0 | | 66.4 | 90.8 | | 51.8 | 82.7 | | 39.7 | 72.8 | | 29.4 | 39.0 | | 59.2 | 74.8 | | 129.3 | 132.1 |
| GET[14](AAAI’21) | 81.6 | 96.1 | | 66.5 | 90.9 | | 51.9 | 82.8 | | 39.7 | 72.9 | | 29.4 | 38.8 | | 59.1 | 74.4 | | 130.3 | 132.5 |
| RSTNet[18](CVPR’21) | 82.1 | 96.4 | | 67.0 | 91.3 | | 52.2 | 83.0 | | 40.0 | 73.1 | | 29.6 | 39.1 | | 59.5 | 74.6 | | 131.9 | 134.0 |
| ReFormer[19](MM’22) | 82.0 | 96.7 | | | | | | | | 40.1 | 73.2 | | 29.8 | 39.5 | | 59.9 | 75.2 | | 129.9 | 132.8 |
| GAT[20](Expert Syst.Appl’22) | 81.1 | 95.1 | | 66.1 | 89.7 | | 51.8 | 81.5 | | 39.9 | 71.4 | | 29.1 | 38.4 | | 59.1 | 74.4 | | 127.8 | 129.8 |
| RSTNet[18](单一模型) | 81.8 | 96.2 | | 66.6 | 91.0 | | 51.9 | 82.7 | | 39.7 | 72.5 | | 29.7 | 39.0 | | 59.4 | 74.5 | | 131.4 | 134.4 |
| DSC-Net(本文单一模型) | 81.9 | 96.3 | | 66.6 | 91.1 | | 51.9 | 82.8 | | 39.7 | 72.8 | | 29.7 | 39.2 | | 59.3 | 74.4 | | 132.7 | 135.3 |
| DSC-Net(本文集成模型) | 82.8 | 96.9 | | 67.8 | 92.3 | | 53.2 | 84.5 | | 40.9 | 74.7 | | 30.1 | 39.6 | | 60.1 | 75.3 | | 135.8 | 137.6 |
), ArticleFig(id=1329460766924714755, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=表2, caption=
本文方法在在线测试中与各种典型方法的性能比较
, figureFileSmall=null, figureFileBig=null, tableContent=
| 方法 | BLEU-1 | | BLEU-2 | | BLEU-3 | | BLEU-4 | | METEOR | | ROUGE | | CIDEr |
|---|
| c5 | c40 | | c5 | c40 | | c5 | c40 | | c5 | c40 | | c5 | c40 | | c5 | c40 | | c5 | c40 |
|---|
| up-down[5](CVPR’18) | 80.2 | 95.2 | | 64.1 | 88.8 | | 49.1 | 79.4 | | 36.9 | 68.5 | | 27.6 | 36.7 | | 57.1 | 72.4 | | 117.9 | 120.5 |
| SGAE[12](CVPR’19) | 81.0 | 95.3 | | 65.6 | 89.5 | | 50.7 | 80.4 | | 38.5 | 69.7 | | 28.2 | 37.2 | | 58.6 | 73.6 | | 123.8 | 126.5 |
| AoANet[21](ICCV’19) | 81.0 | 95.0 | | 65.8 | 89.6 | | 51.4 | 81.3 | | 39.4 | 71.2 | | 29.1 | 38.5 | | 58.9 | 74.5 | | 126.9 | 129.6 |
| X-Transformer[8](CVPR’20) | 81.9 | 95.7 | | 66.9 | 90.5 | | 52.4 | 82.5 | | 40.3 | 72.4 | | 29.6 | 39.2 | | 59.5 | 75.0 | | 131.1 | 133.5 |
| M2Transformer[13](CVPR’20) | 81.6 | 96.0 | | 66.4 | 90.8 | | 51.8 | 82.7 | | 39.7 | 72.8 | | 29.4 | 39.0 | | 59.2 | 74.8 | | 129.3 | 132.1 |
| GET[14](AAAI’21) | 81.6 | 96.1 | | 66.5 | 90.9 | | 51.9 | 82.8 | | 39.7 | 72.9 | | 29.4 | 38.8 | | 59.1 | 74.4 | | 130.3 | 132.5 |
| RSTNet[18](CVPR’21) | 82.1 | 96.4 | | 67.0 | 91.3 | | 52.2 | 83.0 | | 40.0 | 73.1 | | 29.6 | 39.1 | | 59.5 | 74.6 | | 131.9 | 134.0 |
| ReFormer[19](MM’22) | 82.0 | 96.7 | | | | | | | | 40.1 | 73.2 | | 29.8 | 39.5 | | 59.9 | 75.2 | | 129.9 | 132.8 |
| GAT[20](Expert Syst.Appl’22) | 81.1 | 95.1 | | 66.1 | 89.7 | | 51.8 | 81.5 | | 39.9 | 71.4 | | 29.1 | 38.4 | | 59.1 | 74.4 | | 127.8 | 129.8 |
| RSTNet[18](单一模型) | 81.8 | 96.2 | | 66.6 | 91.0 | | 51.9 | 82.7 | | 39.7 | 72.5 | | 29.7 | 39.0 | | 59.4 | 74.5 | | 131.4 | 134.4 |
| DSC-Net(本文单一模型) | 81.9 | 96.3 | | 66.6 | 91.1 | | 51.9 | 82.8 | | 39.7 | 72.8 | | 29.7 | 39.2 | | 59.3 | 74.4 | | 132.7 | 135.3 |
| DSC-Net(本文集成模型) | 82.8 | 96.9 | | 67.8 | 92.3 | | 53.2 | 84.5 | | 40.9 | 74.7 | | 30.1 | 39.6 | | 60.1 | 75.3 | | 135.8 | 137.6 |
), ArticleFig(id=1329460766996017924, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Table 3, caption=
Ablation Results on Modules %
, figureFileSmall=null, figureFileBig=null, tableContent=
| 网络结构 | CIDEr |
|---|
| Baseline | DSCS | DSCC | DSF |
| ✔ | | | | 136.3 |
| ✔ | ✔ | | | 137.4 |
| ✔ | | ✔ | | 136.6 |
| ✔ | ✔ | ✔ | | 137.1 |
| ✔ | ✔ | | ✔ | 137.7 |
| ✔ | | ✔ | ✔ | 137.5 |
| ✔ | ✔ | ✔ | ✔ | 138.5 |
), ArticleFig(id=1329460767058932485, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=表3, caption=
关于模块的消融结果
, figureFileSmall=null, figureFileBig=null, tableContent=
| 网络结构 | CIDEr |
|---|
| Baseline | DSCS | DSCC | DSF |
| ✔ | | | | 136.3 |
| ✔ | ✔ | | | 137.4 |
| ✔ | | ✔ | | 136.6 |
| ✔ | ✔ | ✔ | | 137.1 |
| ✔ | ✔ | | ✔ | 137.7 |
| ✔ | | ✔ | ✔ | 137.5 |
| ✔ | ✔ | ✔ | ✔ | 138.5 |
), ArticleFig(id=1329460767138624262, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Table 4, caption=
Ablation Experiment Results of DSCC and DSCS Modules %
, figureFileSmall=null, figureFileBig=null, tableContent=
| 方法 | BLEU-1 | BLEU-4 | METEOR | ROUGE | CIDEr | SPICE |
|---|
| w/o Region | 82.0 | 40.3 | 29.8 | 59.8 | 136.3 | 23.5 |
| w/o DSC | 82.4 | 40.6 | 29.9 | 59.7 | 136.8 | 23.4 |
| w/o DSCC | 82.2 | 40.5 | 29.9 | 59.6 | 137.7 | 23.5 |
| w/o DSCS | 82.5 | 40.7 | 30.0 | 59.9 | 137.5 | 23.6 |
| DSC-Net | 82.9 | 41.1 | 30.1 | 59.9 | 138.5 | 23.7 |
), ArticleFig(id=1329460767205733127, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=表4, caption=
关于双重语义协作机制的消融实验结果
, figureFileSmall=null, figureFileBig=null, tableContent=
| 方法 | BLEU-1 | BLEU-4 | METEOR | ROUGE | CIDEr | SPICE |
|---|
| w/o Region | 82.0 | 40.3 | 29.8 | 59.8 | 136.3 | 23.5 |
| w/o DSC | 82.4 | 40.6 | 29.9 | 59.7 | 136.8 | 23.4 |
| w/o DSCC | 82.2 | 40.5 | 29.9 | 59.6 | 137.7 | 23.5 |
| w/o DSCS | 82.5 | 40.7 | 30.0 | 59.9 | 137.5 | 23.6 |
| DSC-Net | 82.9 | 41.1 | 30.1 | 59.9 | 138.5 | 23.7 |
), ArticleFig(id=1329460767277036296, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=EN, label=Table 5, caption=
Ablation Experiment Results of Feature Fusion %
, figureFileSmall=null, figureFileBig=null, tableContent=
| 方法 | BLEU-1 | BLEU-4 | METEOR | ROUGE | CIDEr | SPICE |
|---|
| w/o DSF | 82.2 | 40.6 | 30.0 | 59.7 | 137.1 | 23.6 |
| w/o RFD | 82.1 | 40.5 | 29.9 | 59.7 | 137.3 | 23.6 |
| DSC-Net | 82.9 | 41.1 | 30.1 | 59.9 | 138.5 | 23.7 |
), ArticleFig(id=1329460767344145161, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329460761270792884, language=CN, label=表5, caption=
特征融合的消融实验结果
, figureFileSmall=null, figureFileBig=null, tableContent=
| 方法 | BLEU-1 | BLEU-4 | METEOR | ROUGE | CIDEr | SPICE |
|---|
| w/o DSF | 82.2 | 40.6 | 30.0 | 59.7 | 137.1 | 23.6 |
| w/o RFD | 82.1 | 40.5 | 29.9 | 59.7 | 137.3 | 23.6 |
| DSC-Net | 82.9 | 41.1 | 30.1 | 59.9 | 138.5 | 23.7 |
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202330523, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202330523, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202330523, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202330523, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1730390400000, fullTextJson=null, articleText=null, reference=null)