Article(id=1329435702628037129, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1329435571803501430, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202550936, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=null, receivedDate=1766505600000, receivedDateStr=2025-12-24, revisedDate=1784736000000, revisedDateStr=2026-07-23, acceptedDate=null, acceptedDateStr=null, onlineDate=1794891771227, onlineDateStr=2026-11-17, pubDate=1793462400000, pubDateStr=2026-11-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1794891771227, onlineIssueDateStr=2026-11-17, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1794891771227, creator=13701087609, updateTime=1794891771227, updator=13701087609, issue=Issue{id=1329435571803501430, tenantId=1146029695717560320, journalId=1324759914146529478, year='2026', volume='63', issue='11', pageStart='2767', pageEnd='3031', issueExtLink='null', onlineDate='null', pubDate='1793462400000', pubDateStr='2026-11-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=1, specialIssue=null, createTime=1794891740036, creator='13701087609', updateTime=1794891740036, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext=null, issueFiles=null, downloadFileDto=null}, startPage=3002, endPage=3016, ext={EN=ArticleExt(id=1329435702812586506, articleId=1329435702628037129, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=A Multi-Feature Image Captioning Model with Common-Specific Collaborative Modeling and Mixture-of-Experts Fusion, columnId=1325489891789910943, journalTitle=Journal of Computer Research and Development, columnName=Artificial Intelligence, runingTitle=null, highlight=null, articleAbstract=

Existing image captioning methods often struggle to simultaneously capture global semantics and fine-grained details, leading to overly generic descriptions or missing key information. To address this issue, we propose EAFCC, short for expert-aware fusion with common-specific collaboration, a multi-feature image captioning model with mixture-of-experts fusion. The model takes grid features, region features, and segmentation features as parallel inputs. In the encoding stage, a pre-fusion module first performs implicit alignment of heterogeneous visual features and models cross-feature common information within a shared semantic space, while preserving the basis for feature-specific representations. A shared Transformer encoder is then employed to learn the common representations, whose consistency and discriminability are further enhanced by a contrastive learning constraint. Meanwhile, three independent encoding branches are used to model feature-specific representations, so as to retain fine-grained complementary information associated with different feature types. A gated routing mechanism inspired by mixture-of-experts is further introduced to dynamically weight and fuse the specific information. Finally, the enhanced common features and the fused specific features are collaboratively integrated, processed by a skip-connected encoder, and fed into a Transformer decoder to generate image captions. Experimental results on MS-COCO dataset show that EAFCC achieves competitive performance on the CIDEr-D metric and outperforms existing single-feature and dual-feature methods, demonstrating the effectiveness of common-specific collaborative modeling and dynamic multi-feature fusion for image captioning.

, authors=Jianfeng Yang1, Dongpo Zhu2, Rui Cao1, *, Xuanhao Cui1, Ping Wang1, Jie Zheng1, authorsList=Jianfeng Yang, Dongpo Zhu, Rui Cao, Xuanhao Cui, Ping Wang, Jie Zheng, authorCompany=null, correspAuthors=Rui Cao, authorNote=

born in 1981. Master, senior engineer, master supervisor. His main research interests include software development methodologies and data science, with a focus on applications in higher education

born in 1999. Master, engineer. His main research interest includes multi-modal fusion and application

born in 1991. PhD, lecturer, master supervisor. Member of CCF. His main research interests include text representation learning, applications and optimization of generative large language models, and art generation using diffusion models

born in 2003. Master candidate. His main research interest includes large model development and application

born in 2002. Master candidate. His main research interest includes large model visual reasoning

born in 1985. PhD, professor, master supervisor. His main research interest includes edge general intelligence

, correspAuthorsNote=null, copyrightStatement=Copyright © 2026 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1329435706017034797, articleId=1329435702628037129, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=基于共性-特性协同建模与多专家融合的多特征图像描述模型, columnId=1325489892104483745, journalTitle=计算机研究与发展, columnName=人工智能, runingTitle=null, highlight=null, articleAbstract=

针对现有图像描述方法难以同时兼顾全局语义与细粒度细节、生成结果易偏泛化或关键信息遗漏的问题,提出一种基于共性-特性协同建模与多专家融合的多特征图像描述生成模型EAFCC(expert-aware fusion with common-specific collaboration)。该模型并行接收网格特征、区域特征与分割特征,在编码阶段首先通过预融合模块对多源异构特征进行隐式对齐。在共享语义空间中提取跨特征一致的共性信息,同时为各特征流保留差异化表达基础。随后利用共享Transformer编码器建模共性表征,并通过对比学习约束进一步增强其一致性与判别性。同时,采用3条独立编码分支对不同特征流进行特性建模,以保留与特征类型相关的细粒度补充信息,并借助多专家门控路由机制实现特性信息的动态加权融合。最后,将增强后的共性特征与融合后的特性特征进行协同整合,经跳跃连接编码器后送入Transformer解码器生成图像描述。在MS-COCO数据集上的实验表明,EAFCC在CIDEr-D指标上取得了较优性能,并优于现有单特征或双特征方法,验证了共性与特性联合建模的有效性。

, authors=杨建锋1, 朱东坡2, 曹瑞1, *, 崔选皓1, 王平1, 郑杰1, authorsList=杨建锋, 朱东坡, 曹瑞, 崔选皓, 王平, 郑杰, authorCompany=null, correspAuthors=曹瑞, authorNote=

作者贡献声明

杨建锋提出研究构思并指导研究开展,审阅和修改论文;朱东坡负责方法实现、模型搭建、实验与数据分析,并撰写论文;曹瑞审阅和修改论文,统筹审稿意见回复并负责最终定稿;崔选皓参与论文审阅与修改;王平参与论文审阅与修改;郑杰提出改进思路,并提供方法优化建议与分析支持。

1981年生。硕士,高级工程师,硕士生导师。主要研究方向为软件开发方法与数据科学及其在高校中的应用

1999年生。硕士,工程师。主要研究方向为多模态融合应用

1991年生。博士,讲师,硕士生导师。CCF会员。主要研究方向为文本表示学习、生成式大语言模型应用、优化及扩散模型艺术生成

2003年生。硕士研究生。主要研究方向为大模型开发应用

2002年生。硕士研究生。主要研究方向为大模型视觉推理

1985年生。博士,教授,硕士生导师。主要研究方向为边缘通用智能

, correspAuthorsNote=
曹瑞()
, copyrightStatement=版权所有 © 2026 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=t0VOgGI1AYfGSHRtwYAHFg==, magXml=k0okzEd90rsU7Kw3ZDSt0w==, pdfUrl=null, pdf=DmIv8o5O6F23TSnQKzhCrA==, pdfFileSize=8662548, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=NbfDvL4pNIPbfAKkLw50og==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=El29xaM36g1elpa5Ah2Jjw==, mapNumber=null, fund=null)}, authors=[Author(id=1329435706369356341, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=yjf@nwu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329435706449048119, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435706369356341, language=EN, stringName=Jianfeng Yang, firstName=Jianfeng, middleName=null, lastName=Yang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Computer Science, Northwest University, Xi’an 710127, bio={"content":"

born in 1981. Master, senior engineer, master supervisor. His main research interests include software development methodologies and data science, with a focus on applications in higher education

"}, bioImg=null, bioContent=

born in 1981. Master, senior engineer, master supervisor. His main research interests include software development methodologies and data science, with a focus on applications in higher education

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329435706511962680, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435706369356341, language=CN, stringName=杨建锋, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1西北大学计算机学院 西安 710127, bio={"img":"rl+Yix6xaTprWHjG8wlINg==","content":"

1981年生。硕士,高级工程师,硕士生导师。主要研究方向为软件开发方法与数据科学及其在高校中的应用

"}, bioImg=rl+Yix6xaTprWHjG8wlINg==, bioContent=

1981年生。硕士,高级工程师,硕士生导师。主要研究方向为软件开发方法与数据科学及其在高校中的应用

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329435706209972782, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, xref=1, ext=[AuthorCompanyExt(id=1329435706222555695, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science, Northwest University, Xi’an 710127), AuthorCompanyExt(id=1329435706230944304, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1西北大学计算机学院 西安 710127)])]), Author(id=1329435706570682938, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329435706641986108, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435706570682938, language=EN, stringName=Dongpo Zhu, firstName=Dongpo, middleName=null, lastName=Zhu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2Beijing Sankuai Technology Co., Ltd., Beijing 100190, bio={"content":"

born in 1999. Master, engineer. His main research interest includes multi-modal fusion and application

"}, bioImg=null, bioContent=

born in 1999. Master, engineer. His main research interest includes multi-modal fusion and application

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329435706717483581, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435706570682938, language=CN, stringName=朱东坡, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2北京三快科技有限公司 北京 100190, bio={"img":"x2S1WMCgYVTZqsPil0V8Dw==","content":"

1999年生。硕士,工程师。主要研究方向为多模态融合应用

"}, bioImg=x2S1WMCgYVTZqsPil0V8Dw==, bioContent=

1999年生。硕士,工程师。主要研究方向为多模态融合应用

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329435706293858865, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, xref=2, ext=[AuthorCompanyExt(id=1329435706302247474, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706293858865, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Beijing Sankuai Technology Co., Ltd., Beijing 100190), AuthorCompanyExt(id=1329435706310636083, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706293858865, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2北京三快科技有限公司 北京 100190)])]), Author(id=1329435706776203839, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=cr@nwu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329435706851701313, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435706776203839, language=EN, stringName=Rui Cao, firstName=Rui, middleName=null, lastName=Cao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, *, address=1School of Computer Science, Northwest University, Xi’an 710127, bio={"content":"

born in 1991. PhD, lecturer, master supervisor. Member of CCF. His main research interests include text representation learning, applications and optimization of generative large language models, and art generation using diffusion models

"}, bioImg=null, bioContent=

born in 1991. PhD, lecturer, master supervisor. Member of CCF. His main research interests include text representation learning, applications and optimization of generative large language models, and art generation using diffusion models

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329435706918810178, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435706776203839, language=CN, stringName=曹瑞, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, *, address=1西北大学计算机学院 西安 710127, bio={"img":"3mjX5y/1tZNO/WqqXyMI4g==","content":"

1991年生。博士,讲师,硕士生导师。CCF会员。主要研究方向为文本表示学习、生成式大语言模型应用、优化及扩散模型艺术生成

"}, bioImg=3mjX5y/1tZNO/WqqXyMI4g==, bioContent=

1991年生。博士,讲师,硕士生导师。CCF会员。主要研究方向为文本表示学习、生成式大语言模型应用、优化及扩散模型艺术生成

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329435706209972782, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, xref=1, ext=[AuthorCompanyExt(id=1329435706222555695, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science, Northwest University, Xi’an 710127), AuthorCompanyExt(id=1329435706230944304, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1西北大学计算机学院 西安 710127)])]), Author(id=1329435706985919044, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329435707061416518, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435706985919044, language=EN, stringName=Xuanhao Cui, firstName=Xuanhao, middleName=null, lastName=Cui, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Computer Science, Northwest University, Xi’an 710127, bio={"content":"

born in 2003. Master candidate. His main research interest includes large model development and application

"}, bioImg=null, bioContent=

born in 2003. Master candidate. His main research interest includes large model development and application

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329435707162079815, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435706985919044, language=CN, stringName=崔选皓, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1西北大学计算机学院 西安 710127, bio={"img":"KCF21Vtpk9buAWULZ5uTzA==","content":"

2003年生。硕士研究生。主要研究方向为大模型开发应用

"}, bioImg=KCF21Vtpk9buAWULZ5uTzA==, bioContent=

2003年生。硕士研究生。主要研究方向为大模型开发应用

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329435706209972782, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, xref=1, ext=[AuthorCompanyExt(id=1329435706222555695, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science, Northwest University, Xi’an 710127), AuthorCompanyExt(id=1329435706230944304, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1西北大学计算机学院 西安 710127)])]), Author(id=1329435707233382985, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329435707308880459, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435707233382985, language=EN, stringName=Ping Wang, firstName=Ping, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Computer Science, Northwest University, Xi’an 710127, bio={"content":"

born in 2002. Master candidate. His main research interest includes large model visual reasoning

"}, bioImg=null, bioContent=

born in 2002. Master candidate. His main research interest includes large model visual reasoning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329435707380183628, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435707233382985, language=CN, stringName=王平, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1西北大学计算机学院 西安 710127, bio={"img":"1jrykixFy//FWewzY0cOTw==","content":"

2002年生。硕士研究生。主要研究方向为大模型视觉推理

"}, bioImg=1jrykixFy//FWewzY0cOTw==, bioContent=

2002年生。硕士研究生。主要研究方向为大模型视觉推理

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329435706209972782, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, xref=1, ext=[AuthorCompanyExt(id=1329435706222555695, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science, Northwest University, Xi’an 710127), AuthorCompanyExt(id=1329435706230944304, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1西北大学计算机学院 西安 710127)])]), Author(id=1329435707468264014, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, orderNo=5, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329435707568927312, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435707468264014, language=EN, stringName=Jie Zheng, firstName=Jie, middleName=null, lastName=Zheng, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Computer Science, Northwest University, Xi’an 710127, bio={"content":"

born in 1985. PhD, professor, master supervisor. His main research interest includes edge general intelligence

"}, bioImg=null, bioContent=

born in 1985. PhD, professor, master supervisor. His main research interest includes edge general intelligence

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329435707648619089, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435707468264014, language=CN, stringName=郑杰, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1西北大学计算机学院 西安 710127, bio={"img":"9Vnp7bpLM1DolD2CFV6lXQ==","content":"

1985年生。博士,教授,硕士生导师。主要研究方向为边缘通用智能

"}, bioImg=9Vnp7bpLM1DolD2CFV6lXQ==, bioContent=

1985年生。博士,教授,硕士生导师。主要研究方向为边缘通用智能

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329435706209972782, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, xref=1, ext=[AuthorCompanyExt(id=1329435706222555695, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science, Northwest University, Xi’an 710127), AuthorCompanyExt(id=1329435706230944304, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1西北大学计算机学院 西安 710127)])])], keywords=[Keyword(id=1329435707740893778, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, orderNo=1, keyword=image captioning), Keyword(id=1329435707799614035, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, orderNo=2, keyword=multi-feature fusion), Keyword(id=1329435707858334292, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, orderNo=3, keyword=common-specific collaborative modeling), Keyword(id=1329435707921248853, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, orderNo=4, keyword=contrastive learning), Keyword(id=1329435707996746326, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, orderNo=5, keyword=Transformer), Keyword(id=1329435708055466583, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, orderNo=1, keyword=图像描述), Keyword(id=1329435708118381144, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, orderNo=2, keyword=多特征融合), Keyword(id=1329435708185490009, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, orderNo=3, keyword=共性-特性协同建模), Keyword(id=1329435708240015962, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, orderNo=4, keyword=对比学习), Keyword(id=1329435708302930523, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, orderNo=5, keyword=Transformer)], refs=[Reference(id=1329435711129891452, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=Gurari D, Li Qing, Stangl A J, et al. VizWiz grand challenge: Answering visual questions from blind people[C]//Proc of the 31st IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 3608−3617, articleTitle=null, refAbstract=null), Reference(id=1329435711209583229, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Aneja J, Deshpande A, Schwing A G. Convolutional image captioning[C]//Proc of the 31st IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 5561−5570, articleTitle=null, refAbstract=null), Reference(id=1329435711272497790, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Yu Jun, Li Jing, Yu Zhou, et al. Multimodal Transformer with multi-view visual representation for image captioning[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 30(12): 4467−4480, articleTitle=null, refAbstract=null), Reference(id=1329435711339606655, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Lou Liangshan, Lu Ke, Xue Jian. A novel cross-fusion method of different types of features for image captioning[C]//Proc of the 33rd Int Joint Conf on Neural Networks. Piscataway, NJ: IEEE, 2023: 1−8, articleTitle=null, refAbstract=null), Reference(id=1329435711410909824, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Mathur N, Baldwin T, Cohn T. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics[C]//Proc of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 4984−4997, articleTitle=null, refAbstract=null), Reference(id=1329435711486407297, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Cui Yin, Yang Guandao, Veit A, et al. Learning to evaluate image captioning[C]//Proc of the 31st IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 5804−5812, articleTitle=null, refAbstract=null), Reference(id=1329435711553516162, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Bernardi R, Cakici R, Elliott D, et al. Automatic description generation from images: A survey of models, datasets, and evaluation measures[J]. Journal of Artificial Intelligence Research, 2016, 55: 409−442, articleTitle=null, refAbstract=null), Reference(id=1329435711633207939, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Al-Shamayleh A S, Adwan O, Alsharaiah M A, et al. A comprehensive literature review on image captioning methods and metrics based on deep learning technique[J]. Multimedia Tools and Applications, 2024, 83(12): 34219−34268, articleTitle=null, refAbstract=null), Reference(id=1329435711700316804, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Ghandi T, Pourreza H, Mahyar H. Deep learning approaches on image captioning: A review[J]. ACM Computing Surveys, 2023, 56(3): 1−39, articleTitle=null, refAbstract=null), Reference(id=1329435711771619973, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Lun, Wang Wenmin, Chen Jie, et al. Attention on attention for image captioning[C]//Proc of the 17th IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 4634−4643, articleTitle=null, refAbstract=null), Reference(id=1329435711872283270, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Guang, Zhu Linchao, Liu Ping, et al. Entangled Transformer for image captioning[C]//Proc of the 17th IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 8928−8937, articleTitle=null, refAbstract=null), Reference(id=1329435711939392135, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Cornia M, Stefanini M, Baraldi L, et al. Meshed-memory Transformer for image captioning[C]//Proc of the 33rd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 10578−10587, articleTitle=null, refAbstract=null), Reference(id=1329435712002306696, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Pan Yingwei, Yao Ting, Li Yehao, et al. X-linear attention networks for image captioning[C]//Proc of the 33rd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 10971−10980, articleTitle=null, refAbstract=null), Reference(id=1329435712069415561, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Luo Yunpeng, Ji Jiayi, Sun Xiaoshuai, et al. Dual-level collaborative Transformer for image captioning[C]//Proc of the 35th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2021: 2286−2293, articleTitle=null, refAbstract=null), Reference(id=1329435712136524426, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Nguyen V Q, Suganuma M, Okatani T. GRIT: Faster and better image captioning transformer using dual visual features[G]//LNCS 13696: Proc of the 17th European Conf on Computer Vision. Berlin: Springer, 2022: 167−184, articleTitle=null, refAbstract=null), Reference(id=1329435712195244683, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Wu Mingrui, Zhang Xuying, Sun Xiaoshuai, et al. DIFNet: Boosting visual information flow for image captioning[C]//Proc of the 35th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 18020−18029, articleTitle=null, refAbstract=null), Reference(id=1329435712249770636, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Junnan, Li Dongxu, Xiong Caiming, et al. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation[C]//Proc of the 39th Int Conf on Machine Learning. New York: PMLR, 2022: 12888−12900, articleTitle=null, refAbstract=null), Reference(id=1329435712321073805, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Peng, Yang An, Men Rui, et al. OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework[C]//Proc of the 39th Int Conf on Machine Learning. New York: PMLR, 2022: 23318−23340, articleTitle=null, refAbstract=null), Reference(id=1329435712383988366, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Bai Jinze, Men Rui, Yang Hao, et al. OFASys: A multi-modal multi-task learning system for building generalist models[J]. arXiv preprint, arXiv: 2212.04408, 2022, articleTitle=null, refAbstract=null), Reference(id=1329435712451097231, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Zirui, Yu Jiahui, Yu Adams Wei, et al. SimVLM: Simple visual language model pretraining with weak supervision[J]. arXiv preprint, arXiv: 2108.10904, 2021, articleTitle=null, refAbstract=null), Reference(id=1329435712526594704, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Wei, Gao Can, Niu Guocheng, et al. UNIMO: Towards unified-modal understanding and generation via cross-modal contrastive learning[J]. arXiv preprint, arXiv: 2012.15409, 2020, articleTitle=null, refAbstract=null), Reference(id=1329435712585314961, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhou Luowei, Palangi H, Zhang Lei, et al. Unified vision-language pre-training for image captioning and VQA[C]//Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 13041−13049, articleTitle=null, refAbstract=null), Reference(id=1329435712644035218, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Jiaxuan, Vo D M, Sugimoto A, et al. EVCap: Retrieval-augmented image captioning with external visual-name memory for open-world comprehension[C]//Proc of the 37th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 13733−13742, articleTitle=null, refAbstract=null), Reference(id=1329435712715338387, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=Hua Hang, Liu Qing, Zhang Lingzhi, et al. FineCaption: Compositional image captioning focusing on wherever you want at any granularity[C]//Proc of the 38th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2025: 24763−24773, articleTitle=null, refAbstract=null), Reference(id=1329435712774058644, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=Anderson P, He Xiaodong, Buehler C, et al. Bottom-up and top-down attention for image captioning and visual question answering[C]//Proc of the 31st IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 6077−6086, articleTitle=null, refAbstract=null), Reference(id=1329435712832778901, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=Ren Shaoqing, He Kaiming, Girshick R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137−1149, articleTitle=null, refAbstract=null), Reference(id=1329435712887304854, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=26, authorNames=null, journalName=null, refType=null, unstructuredReference=Xiong Yuwen, Liao Renjie, Zhao Hengshuang, et al. UPSNet: A unified panoptic segmentation network[C]//Proc of the 32nd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 8818−8826, articleTitle=null, refAbstract=null), Reference(id=1329435712946025111, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=28, rfOrder=27, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Xu, Tang Kaihua, Zhang Hanwang, et al. Auto-encoding scene graphs for image captioning[C]//Proc of the 32nd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 10685−10694, articleTitle=null, refAbstract=null), Reference(id=1329435713008939672, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=29, rfOrder=28, authorNames=null, journalName=null, refType=null, unstructuredReference=Yao Ting, Pan Yingwei, Li Yehao, et al. Exploring visual relationship for image captioning[G]//LNCS 11218: Proc of the 15th European Conf on Computer Vision. Berlin: Springer, 2018: 711−727, articleTitle=null, refAbstract=null), Reference(id=1329435713080242841, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=30, rfOrder=29, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Jing, Xie Yingshuai, Ding Weichao, et al. Cross on cross attention: Deep fusion transformer for image captioning[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(8): 4257−4268, articleTitle=null, refAbstract=null), Reference(id=1329435713143157402, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=31, rfOrder=30, authorNames=null, journalName=null, refType=null, unstructuredReference=Jiang Wenhao, Ma Lin, Jiang Yugang, et al. Recurrent fusion network for image captioning[G]//LNCS 11206: Proc of the 15th European Conf on Computer Vision. Berlin: Springer, 2018: 499−515, articleTitle=null, refAbstract=null), Reference(id=1329435713206071963, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=32, rfOrder=31, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Weixuan, Chen Zhihong, Hu Haifeng. Hierarchical attention network for image captioning[C]//Proc of the 33rd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 8957−8964, articleTitle=null, refAbstract=null), Reference(id=1329435713264792220, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=33, rfOrder=32, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Shanshan, Li Lixiang, Peng Haipeng. Incorporating retrieval-based method for feature enhanced image captioning[J]. Applied Intelligence, 2023, 53(8): 9731−9743, articleTitle=null, refAbstract=null), Reference(id=1329435713331901085, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=34, rfOrder=33, authorNames=null, journalName=null, refType=null, unstructuredReference=Ramos R, Martins B, Elliott D, et al. SmallCap: Lightweight image captioning prompted with retrieval augmentation[C]//Proc of the 36th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 2840−2849, articleTitle=null, refAbstract=null), Reference(id=1329435713390621342, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=35, rfOrder=34, authorNames=null, journalName=null, refType=null, unstructuredReference=Song Jiayan, Pan Renjie, Zhou Jun, et al. M-RAT: A multi-grained retrieval augmentation Transformer for image captioning[C]//Proc of the 17th Asian Conf on Computer Vision. Berlin: Springer, 2024: 185−203, articleTitle=null, refAbstract=null), Reference(id=1329435713457730207, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=36, rfOrder=35, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Ning, Xie Jiahao, Wu Jihao, et al. Controllable image captioning via prompting[C]//Proc of the 37th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023: 2617−2625, articleTitle=null, refAbstract=null), Reference(id=1329435713520644768, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=37, rfOrder=36, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Kun, Mao Zhendong, Wang Quan, et al. Negative-aware attention framework for image-text matching[C]//Proc of the 35th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 15661−15670, articleTitle=null, refAbstract=null), Reference(id=1329435713579365025, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=38, rfOrder=37, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Junyi, Guo Longteng, Sun Jia, et al. EVE: Efficient vision-language pre-training with masked prediction and modality-aware MoE[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 1110−1119, articleTitle=null, refAbstract=null), Reference(id=1329435713646473890, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=39, rfOrder=38, authorNames=null, journalName=null, refType=null, unstructuredReference=Rennie S J, Marcheret E, Mroueh Y, et al. Self-critical sequence training for image captioning[C]//Proc of the 30th IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2017: 7008−7024, articleTitle=null, refAbstract=null), Reference(id=1329435713713582755, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=40, rfOrder=39, authorNames=null, journalName=null, refType=null, unstructuredReference=Lin C Y. ROUGE: A package for automatic evaluation of summaries[C]//Proc of the Text Summarization Branches Out. Stroudsburg, PA: ACL, 2004: 74−81, articleTitle=null, refAbstract=null), Reference(id=1329435713776497316, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=41, rfOrder=40, authorNames=null, journalName=null, refType=null, unstructuredReference=Vedantam R, Lawrence Z C, Parikh D. CIDEr: Consensus-based image description evaluation[C]//Proc of the 28th IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2015: 4566−4575, articleTitle=null, refAbstract=null), Reference(id=1329435713843606181, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=42, rfOrder=41, authorNames=null, journalName=null, refType=null, unstructuredReference=Anderson P, Fernando B, Johnson M, et al. SPICE: Semantic propositional image caption evaluation[G]//LNCS 9909: Proc of the 14th European Conf on Computer Vision. Berlin: Springer, 2016: 382−398, articleTitle=null, refAbstract=null), Reference(id=1329435713906520742, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=43, rfOrder=42, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Bing, Wang Dong, Yang Xu, et al. Show, deconfound and tell: Image captioning with causal inference[C]//Proc of the 35th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 18041−18050, articleTitle=null, refAbstract=null), Reference(id=1329435713973629607, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=44, rfOrder=43, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Xuying, Sun Xiaoshuai, Luo Yunpeng, et al. RSTNet: Captioning with adaptive attention on visual and non-visual words[C]//Proc of the 34th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2021: 15465−15474, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1329435706209972782, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, xref=1, ext=[AuthorCompanyExt(id=1329435706222555695, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science, Northwest University, Xi’an 710127), AuthorCompanyExt(id=1329435706230944304, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1西北大学计算机学院 西安 710127)]), AuthorCompany(id=1329435706293858865, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, xref=2, ext=[AuthorCompanyExt(id=1329435706302247474, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706293858865, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Beijing Sankuai Technology Co., Ltd., Beijing 100190), AuthorCompanyExt(id=1329435706310636083, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706293858865, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2北京三快科技有限公司 北京 100190)])], figs=[ArticleFig(id=1329435708437148252, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, label=Figure 1, caption=The solutions for key issues in multi-feature image captioning, figureFileSmall=pkkUxlE3dF1p7l/OSNolOA==, figureFileBig=NbfDvL4pNIPbfAKkLw50og==, tableContent=null), ArticleFig(id=1329435708512645725, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, label=图1, caption=针对多特征图像描述关键问题的解决思路, figureFileSmall=pkkUxlE3dF1p7l/OSNolOA==, figureFileBig=NbfDvL4pNIPbfAKkLw50og==, tableContent=null), ArticleFig(id=1329435708693000798, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, label=Figure 2, caption=Structure diagram of EAFCC model, figureFileSmall=istkOz3vLSkTrqrPCGbIcQ==, figureFileBig=a316XBzibxAQw6E4qqiq5w==, tableContent=null), ArticleFig(id=1329435708751721055, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, label=图2, caption=EAFCC模型结构图, figureFileSmall=istkOz3vLSkTrqrPCGbIcQ==, figureFileBig=a316XBzibxAQw6E4qqiq5w==, tableContent=null), ArticleFig(id=1329435708823024224, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, label=Figure 3, caption=Structure diagram of pre-fusion module network, figureFileSmall=buJtpX58LJjM697CSZpnYQ==, figureFileBig=fQfeD509NeRoY3Bu1KBb8g==, tableContent=null), ArticleFig(id=1329435708890133089, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, label=图3, caption=预融合模块网络结构图, figureFileSmall=buJtpX58LJjM697CSZpnYQ==, figureFileBig=fQfeD509NeRoY3Bu1KBb8g==, tableContent=null), ArticleFig(id=1329435708948853346, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, label=Figure 4, caption=Skip connection encoder module, figureFileSmall=nROAy+MKUfD2pd8Hy3Es5Q==, figureFileBig=LKxn+HiLiHXWDoZzmw3odw==, tableContent=null), ArticleFig(id=1329435709032739427, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, label=图4, caption=跳跃连接编码器模块, figureFileSmall=nROAy+MKUfD2pd8Hy3Es5Q==, figureFileBig=LKxn+HiLiHXWDoZzmw3odw==, tableContent=null), ArticleFig(id=1329435709091459684, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, label=Figure 5, caption=Attention maps for generating descriptions in different scenarios, figureFileSmall=9FHpoNX6kg7TsGw+yHIrvA==, figureFileBig=W+gTSKYJNcB+aeEuxHV9RA==, tableContent=null), ArticleFig(id=1329435709158568549, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, label=图5, caption=不同场景下生成描述的注意力图, figureFileSmall=9FHpoNX6kg7TsGw+yHIrvA==, figureFileBig=W+gTSKYJNcB+aeEuxHV9RA==, tableContent=null), ArticleFig(id=1329435709225677414, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, label=Figure 6, caption=Attention visualization comparative diagrams generated by different models, figureFileSmall=QN+DRtwIAklYeUXrmbMsYg==, figureFileBig=DpzUaQ0XhZ3jVI4hP15w2g==, tableContent=null), ArticleFig(id=1329435709322146407, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, label=图6, caption=不同模型生成的注意力可视化对比图, figureFileSmall=QN+DRtwIAklYeUXrmbMsYg==, figureFileBig=DpzUaQ0XhZ3jVI4hP15w2g==, tableContent=null), ArticleFig(id=1329435709393449576, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, label=Figure 7, caption=Average cosine similarity heatmap of common branch and specific branch output representations, figureFileSmall=jSaQidImjRdqsGC9DuEIZA==, figureFileBig=n+ZzWwsfUspholq7/5Fkog==, tableContent=null), ArticleFig(id=1329435709485724265, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, label=图7, caption=共性分支与特性分支输出表示的平均余弦相似度热力图, figureFileSmall=jSaQidImjRdqsGC9DuEIZA==, figureFileBig=n+ZzWwsfUspholq7/5Fkog==, tableContent=null), ArticleFig(id=1329435709561221738, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, label=Table 1, caption=

Experimental Environment Settings

, figureFileSmall=null, figureFileBig=null, tableContent=
类别参数项参数
实验
环境
PyTorch1.11.0
CUDA11.3
Python3.8
GPURTX 3090(24 GB)
输入预处理尺寸短边600px(保持长宽比)
再随机水平翻转、标准化
网格特征图尺寸7×7=49个网格,每个网格2048维
(ResNet-101conv5_x)
分割特征图尺寸7×7=49个空间位置,每个位置133维
(COCO133类,UPSNet语义头)
区域特征数量36个RoI/图(Faster R-CNN,
IoU>0.7取前36个)
文本词汇量10201词,最大句子长度20
优化器类型Adam
批次大小50
), ArticleFig(id=1329435709649302123, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, label=表1, caption=

实验环境设置

, figureFileSmall=null, figureFileBig=null, tableContent=
类别参数项参数
实验
环境
PyTorch1.11.0
CUDA11.3
Python3.8
GPURTX 3090(24 GB)
输入预处理尺寸短边600px(保持长宽比)
再随机水平翻转、标准化
网格特征图尺寸7×7=49个网格,每个网格2048维
(ResNet-101conv5_x)
分割特征图尺寸7×7=49个空间位置,每个位置133维
(COCO133类,UPSNet语义头)
区域特征数量36个RoI/图(Faster R-CNN,
IoU>0.7取前36个)
文本词汇量10201词,最大句子长度20
优化器类型Adam
批次大小50
), ArticleFig(id=1329435709724799596, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, label=Table 2, caption=

Comparative Experimental Results of EAFCC Model on COCO2014 Dataset

, figureFileSmall=null, figureFileBig=null, tableContent=
模型B4↑M↑R↑C↑S↑
SCST [39]34.326.755.7114.0
Up-Down[25]36.327.756.9120.121.4
GCN-LSTM[29]38.228.558.5128.322.0
AoANet[10]38.929.258.8129.822.4
EAT[11]39.928.959.0127.622.6
$ {\text{M}}^{2} $Transformer[12]39.129.258.6131.222.6
X-Transformer[13]39.729.559.1132.823.4
DLCT[14]39.829.559.1133.823.0
CIIC[43]40.229.559.4133.123.2
RSTNet[44]39.329.458.8133.323.0
ConCap[36]40.530.9133.723.8
EAFCC(本文)40.629.859.8135.223.2
), ArticleFig(id=1329435709796102765, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, label=表2, caption=

EAFCC模型在COCO2014数据集上对比实验结果

, figureFileSmall=null, figureFileBig=null, tableContent=
模型B4↑M↑R↑C↑S↑
SCST [39]34.326.755.7114.0
Up-Down[25]36.327.756.9120.121.4
GCN-LSTM[29]38.228.558.5128.322.0
AoANet[10]38.929.258.8129.822.4
EAT[11]39.928.959.0127.622.6
$ {\text{M}}^{2} $Transformer[12]39.129.258.6131.222.6
X-Transformer[13]39.729.559.1132.823.4
DLCT[14]39.829.559.1133.823.0
CIIC[43]40.229.559.4133.123.2
RSTNet[44]39.329.458.8133.323.0
ConCap[36]40.530.9133.723.8
EAFCC(本文)40.629.859.8135.223.2
), ArticleFig(id=1329435709879988846, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, label=Table 3, caption=

Comparative Results of Cross-Entropy Loss Training Across Different Models in Stage 1

, figureFileSmall=null, figureFileBig=null, tableContent=
模型B1↑B4↑M↑R↑C↑S↑
Up-Down[25]77.236.227.056.4113.520.3
GCN-LSTM[29]77.336.827.957.0116.320.9
SGAE [28]77.636.927.757.2116.720.9
AoANet[10]77.437.228.457.5119.821.3
X-Transformer[13]77.337.028.757.5120.021.8
SMALLCAP[34]37.027.9119.721.3
EAFCC(本文)77.637.528.857.6120.221.8
), ArticleFig(id=1329435709947097711, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, label=表3, caption=

不同模型在阶段1的交叉熵损失训练对比结果

, figureFileSmall=null, figureFileBig=null, tableContent=
模型B1↑B4↑M↑R↑C↑S↑
Up-Down[25]77.236.227.056.4113.520.3
GCN-LSTM[29]77.336.827.957.0116.320.9
SGAE [28]77.636.927.757.2116.720.9
AoANet[10]77.437.228.457.5119.821.3
X-Transformer[13]77.337.028.757.5120.021.8
SMALLCAP[34]37.027.9119.721.3
EAFCC(本文)77.637.528.857.6120.221.8
), ArticleFig(id=1329435710093898352, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, label=Table 4, caption=

Comparative Results of Pre-Trained Vision-Language Models

, figureFileSmall=null, figureFileBig=null, tableContent=
模型预训练图像数量参数量B4↑C↑
BLIPbase[17]14×106223×10638.6129.7
BLIPCapFilt-L[17]129×10639.7133.3
OFASys[19]20×106270×106134.8
OFATiny[18]20×10633×10638.1128.7
SimVLMbase[20]1.8×10986×10639.0134.8
UNIMO[21]39.6127.7
Unified VLP[22]3×10639.5129.3
EAFCC(本文)31.9×10640.6135.2
), ArticleFig(id=1329435710165201521, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, label=表4, caption=

预训练视觉语言模型对比结果

, figureFileSmall=null, figureFileBig=null, tableContent=
模型预训练图像数量参数量B4↑C↑
BLIPbase[17]14×106223×10638.6129.7
BLIPCapFilt-L[17]129×10639.7133.3
OFASys[19]20×106270×106134.8
OFATiny[18]20×10633×10638.1128.7
SimVLMbase[20]1.8×10986×10639.0134.8
UNIMO[21]39.6127.7
Unified VLP[22]3×10639.5129.3
EAFCC(本文)31.9×10640.6135.2
), ArticleFig(id=1329435710232310386, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, label=Table 5, caption=

Ablation Experiment Results of EAFCC Model on COCO2014 Dataset

, figureFileSmall=null, figureFileBig=null, tableContent=
模型增加模块B1↑B2↑B3↑B4↑M↑R↑C↑S↑
$ {\text{M}}^{2} $Transformer[12]80.839.129.258.6131.222.6
DLCT[14]81.439.829.559.1133.823.0
DIFNet[16]81.540.029.559.2134.923.1
EAFCC-base81.666.452.240.329.659.2134.023.0
EAFCC-basePF81.466.452.039.929.659.1133.422.9
EAFCC-baseFSC81.566.552.040.029.759.3133.323.0
EAFCC-basePF+FSC81.866.752.240.529.759.5134.323.1
EAFCC-basePF+FSC+
CL
82.066.752.340.529.659.5134.723.0
EAFCC-basePF+FSC+
MMF
82.166.852.440.429.859.4134.923.1
EAFCC-base全部模块82.467.352.640.629.859.8135.223.2
), ArticleFig(id=1329435710312002163, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, label=表5, caption=

EAFCC模型在COCO2014数据集上的消融实验结果

, figureFileSmall=null, figureFileBig=null, tableContent=
模型增加模块B1↑B2↑B3↑B4↑M↑R↑C↑S↑
$ {\text{M}}^{2} $Transformer[12]80.839.129.258.6131.222.6
DLCT[14]81.439.829.559.1133.823.0
DIFNet[16]81.540.029.559.2134.923.1
EAFCC-base81.666.452.240.329.659.2134.023.0
EAFCC-basePF81.466.452.039.929.659.1133.422.9
EAFCC-baseFSC81.566.552.040.029.759.3133.323.0
EAFCC-basePF+FSC81.866.752.240.529.759.5134.323.1
EAFCC-basePF+FSC+
CL
82.066.752.340.529.659.5134.723.0
EAFCC-basePF+FSC+
MMF
82.166.852.440.429.859.4134.923.1
EAFCC-base全部模块82.467.352.640.629.859.8135.223.2
), ArticleFig(id=1329435710391693940, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, label=Table 6, caption=

Ablation Experimental Results of Iteration Numbers in PF Module of EAFCC-base

, figureFileSmall=null, figureFileBig=null, tableContent=
迭代次
数N1
B1↑B2↑B3↑B4↑M↑R↑C↑S↑
181.566.552.040.229.659.2133.622.9
281.666.752.340.329.659.5134.023.0
381.766.451.940.029.559.3134.023.0
481.466.351.839.829.559.1133.922.9
), ArticleFig(id=1329435710475580021, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, label=表6, caption=

EAFCC-base的PF模块中迭代次数消融实验结果

, figureFileSmall=null, figureFileBig=null, tableContent=
迭代次
数N1
B1↑B2↑B3↑B4↑M↑R↑C↑S↑
181.566.552.040.229.659.2133.622.9
281.666.752.340.329.659.5134.023.0
381.766.451.940.029.559.3134.023.0
481.466.351.839.829.559.1133.922.9
), ArticleFig(id=1329435710593020534, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, label=Table 7, caption=

Ablation Experimental Results of Skip Intervals in the EAFCC-base Skip-Connection Encoder

, figureFileSmall=null, figureFileBig=null, tableContent=
跳跃间隔B1↑B2↑B3↑B4↑M↑R↑C↑S↑
181.466.451.840.129.358.7133.722.8
381.666.451.540.329.158.5133.522.7
681.266.251.540.029.058.1133.122.7
1,681.666.752.340.429.659.5134.023.0
3,681.666.551.740.429.459.1133.923.0
全连接81.366.251.340.029.058.0133.122.6
), ArticleFig(id=1329435710668518007, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, label=表7, caption=

EAFCC-base跳跃连接编码器中跳跃间隔消融实验结果

, figureFileSmall=null, figureFileBig=null, tableContent=
跳跃间隔B1↑B2↑B3↑B4↑M↑R↑C↑S↑
181.466.451.840.129.358.7133.722.8
381.666.451.540.329.158.5133.522.7
681.266.251.540.029.058.1133.122.7
1,681.666.752.340.429.659.5134.023.0
3,681.666.551.740.429.459.1133.923.0
全连接81.366.251.340.029.058.0133.122.6
), ArticleFig(id=1329435710777569912, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, label=Table 8, caption=

Sensitivity Analysis of Negative Sample Number K

, figureFileSmall=null, figureFileBig=null, tableContent=
评价指标K=2K=3K=5K=16K=32K=50K=64
BLEU-175.575.875.675.876.076.075.8
BLEU-435.835.935.935.736.036.336.1
METEOR27.928.027.827.927.927.927.9
ROUGE56.356.556.256.456.456.656.3
CIDEr-D114.9115.5115.3115.1115.5116.3115.4
), ArticleFig(id=1329435710848873081, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, label=表8, caption=

负样本数K的敏感性分析

, figureFileSmall=null, figureFileBig=null, tableContent=
评价指标K=2K=3K=5K=16K=32K=50K=64
BLEU-175.575.875.675.876.076.075.8
BLEU-435.835.935.935.736.036.336.1
METEOR27.928.027.827.927.927.927.9
ROUGE56.356.556.256.456.456.656.3
CIDEr-D114.9115.5115.3115.1115.5116.3115.4
), ArticleFig(id=1329435710932759162, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=EN, label=Table 9, caption=

Comparison of Image Captioning Generative Results on the Test Set

, figureFileSmall=null, figureFileBig=null, tableContent=
图像描述
GT1:A giraffe is standing in the middle of the grass.
GT2:A giraffe standing alone in a forest.
$ {\text{M}}^{\mathbf{2}} $Transformer:A giraffe standing in the forest.
DLCT:A giraffe standing alone in the green field.
DIFNet:A giraffe standing in forest.
OFATiny:A giraffe is brown and white.
EAFCC: A giraffe is standing in the tall green bushes.
GT1:A young giraffe leaning over a tall bush in a dry field.
GT2:Giraffe eating leaves off of a tree in tall bushes.
$ {\text{M}}^{\mathbf{2}} $Transformer:A giraffe is eating leaves in the field.
DLCT:A giraffe is eating green leaves in the field.
DIFNet:A giraffe eating leaves in a field.
OFATiny:A giraffe is standing in the grass.
EAFCC:A young giraffe is eating green leaves in the dry bushes.
GT1:Well made be in clean and neat bedroom setting.
GT2:A very luxurious bedroom filled with nice furniture.
$ {\text{M}}^{\mathbf{2}} $Transformer:A bedroom filled with a lot of furniture.
DLCT:A grand bedroom includes a bed and a sofa.
DIFNet:A bedroom filled with nice furniture.
OFATiny:A bedroom with a bed or beds in a hotel room.
EAFCC:A luxurious and clean bedroom filled with nice furniture such as bed and sofa.
GT1:A man in a vest and plaid pants rides a bicycle.
GT2:A man rides a bicycle down the street.
$ {\text{M}}^{\mathbf{2}} $Transformer:A man rides a bicycle on the street.
DLCT:A man with plaid and hat rides a is riding a bicycle.
DIFNet: A man riding a bicycle on the street.
OFATiny:A man with a hat riding a bicycle.
EAFCC:A man with leather hat riding a bicycle on road with greenery in the middle.
), ArticleFig(id=1329435711033422459, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, language=CN, label=表9, caption=

测试集图像描述生成结果对比

, figureFileSmall=null, figureFileBig=null, tableContent=
图像描述
GT1:A giraffe is standing in the middle of the grass.
GT2:A giraffe standing alone in a forest.
$ {\text{M}}^{\mathbf{2}} $Transformer:A giraffe standing in the forest.
DLCT:A giraffe standing alone in the green field.
DIFNet:A giraffe standing in forest.
OFATiny:A giraffe is brown and white.
EAFCC: A giraffe is standing in the tall green bushes.
GT1:A young giraffe leaning over a tall bush in a dry field.
GT2:Giraffe eating leaves off of a tree in tall bushes.
$ {\text{M}}^{\mathbf{2}} $Transformer:A giraffe is eating leaves in the field.
DLCT:A giraffe is eating green leaves in the field.
DIFNet:A giraffe eating leaves in a field.
OFATiny:A giraffe is standing in the grass.
EAFCC:A young giraffe is eating green leaves in the dry bushes.
GT1:Well made be in clean and neat bedroom setting.
GT2:A very luxurious bedroom filled with nice furniture.
$ {\text{M}}^{\mathbf{2}} $Transformer:A bedroom filled with a lot of furniture.
DLCT:A grand bedroom includes a bed and a sofa.
DIFNet:A bedroom filled with nice furniture.
OFATiny:A bedroom with a bed or beds in a hotel room.
EAFCC:A luxurious and clean bedroom filled with nice furniture such as bed and sofa.
GT1:A man in a vest and plaid pants rides a bicycle.
GT2:A man rides a bicycle down the street.
$ {\text{M}}^{\mathbf{2}} $Transformer:A man rides a bicycle on the street.
DLCT:A man with plaid and hat rides a is riding a bicycle.
DIFNet: A man riding a bicycle on the street.
OFATiny:A man with a hat riding a bicycle.
EAFCC:A man with leather hat riding a bicycle on road with greenery in the middle.
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202550936, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202550936, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202550936, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202550936, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1793462400000, fullTextJson=null, articleText=null, reference=null)
收藏切换
基于共性-特性协同建模与多专家融合的多特征图像描述模型
杨建锋 1 , 朱东坡 2 , 曹瑞 1, * , 崔选皓 1 , 王平 1 , 郑杰 1
计算机研究与发展 | 人工智能 2026,63(11): 3002-3016
收起
收藏切换
计算机研究与发展 |人工智能 2026 , 63 (11) : 3002 -3016
基于共性-特性协同建模与多专家融合的多特征图像描述模型
全屏
[Author(id=1329435706369356341, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=yjf@nwu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329435706449048119, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435706369356341, language=EN, stringName=Jianfeng Yang, firstName=Jianfeng, middleName=null, lastName=Yang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Computer Science, Northwest University, Xi’an 710127, bio={"content":"

born in 1981. Master, senior engineer, master supervisor. His main research interests include software development methodologies and data science, with a focus on applications in higher education

"}, bioImg=null, bioContent=

born in 1981. Master, senior engineer, master supervisor. His main research interests include software development methodologies and data science, with a focus on applications in higher education

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329435706511962680, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435706369356341, language=CN, stringName=杨建锋, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1西北大学计算机学院 西安 710127, bio={"img":"rl+Yix6xaTprWHjG8wlINg==","content":"

1981年生。硕士,高级工程师,硕士生导师。主要研究方向为软件开发方法与数据科学及其在高校中的应用

"}, bioImg=rl+Yix6xaTprWHjG8wlINg==, bioContent=

1981年生。硕士,高级工程师,硕士生导师。主要研究方向为软件开发方法与数据科学及其在高校中的应用

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329435706209972782, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, xref=1, ext=[AuthorCompanyExt(id=1329435706222555695, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science, Northwest University, Xi’an 710127), AuthorCompanyExt(id=1329435706230944304, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1西北大学计算机学院 西安 710127)])]), Author(id=1329435706570682938, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329435706641986108, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435706570682938, language=EN, stringName=Dongpo Zhu, firstName=Dongpo, middleName=null, lastName=Zhu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2Beijing Sankuai Technology Co., Ltd., Beijing 100190, bio={"content":"

born in 1999. Master, engineer. His main research interest includes multi-modal fusion and application

"}, bioImg=null, bioContent=

born in 1999. Master, engineer. His main research interest includes multi-modal fusion and application

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329435706717483581, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435706570682938, language=CN, stringName=朱东坡, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2北京三快科技有限公司 北京 100190, bio={"img":"x2S1WMCgYVTZqsPil0V8Dw==","content":"

1999年生。硕士,工程师。主要研究方向为多模态融合应用

"}, bioImg=x2S1WMCgYVTZqsPil0V8Dw==, bioContent=

1999年生。硕士,工程师。主要研究方向为多模态融合应用

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329435706293858865, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, xref=2, ext=[AuthorCompanyExt(id=1329435706302247474, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706293858865, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2Beijing Sankuai Technology Co., Ltd., Beijing 100190), AuthorCompanyExt(id=1329435706310636083, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706293858865, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2北京三快科技有限公司 北京 100190)])]), Author(id=1329435706776203839, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=cr@nwu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329435706851701313, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435706776203839, language=EN, stringName=Rui Cao, firstName=Rui, middleName=null, lastName=Cao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, *, address=1School of Computer Science, Northwest University, Xi’an 710127, bio={"content":"

born in 1991. PhD, lecturer, master supervisor. Member of CCF. His main research interests include text representation learning, applications and optimization of generative large language models, and art generation using diffusion models

"}, bioImg=null, bioContent=

born in 1991. PhD, lecturer, master supervisor. Member of CCF. His main research interests include text representation learning, applications and optimization of generative large language models, and art generation using diffusion models

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329435706918810178, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435706776203839, language=CN, stringName=曹瑞, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, *, address=1西北大学计算机学院 西安 710127, bio={"img":"3mjX5y/1tZNO/WqqXyMI4g==","content":"

1991年生。博士,讲师,硕士生导师。CCF会员。主要研究方向为文本表示学习、生成式大语言模型应用、优化及扩散模型艺术生成

"}, bioImg=3mjX5y/1tZNO/WqqXyMI4g==, bioContent=

1991年生。博士,讲师,硕士生导师。CCF会员。主要研究方向为文本表示学习、生成式大语言模型应用、优化及扩散模型艺术生成

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329435706209972782, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, xref=1, ext=[AuthorCompanyExt(id=1329435706222555695, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science, Northwest University, Xi’an 710127), AuthorCompanyExt(id=1329435706230944304, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1西北大学计算机学院 西安 710127)])]), Author(id=1329435706985919044, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329435707061416518, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435706985919044, language=EN, stringName=Xuanhao Cui, firstName=Xuanhao, middleName=null, lastName=Cui, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Computer Science, Northwest University, Xi’an 710127, bio={"content":"

born in 2003. Master candidate. His main research interest includes large model development and application

"}, bioImg=null, bioContent=

born in 2003. Master candidate. His main research interest includes large model development and application

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329435707162079815, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435706985919044, language=CN, stringName=崔选皓, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1西北大学计算机学院 西安 710127, bio={"img":"KCF21Vtpk9buAWULZ5uTzA==","content":"

2003年生。硕士研究生。主要研究方向为大模型开发应用

"}, bioImg=KCF21Vtpk9buAWULZ5uTzA==, bioContent=

2003年生。硕士研究生。主要研究方向为大模型开发应用

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329435706209972782, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, xref=1, ext=[AuthorCompanyExt(id=1329435706222555695, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science, Northwest University, Xi’an 710127), AuthorCompanyExt(id=1329435706230944304, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1西北大学计算机学院 西安 710127)])]), Author(id=1329435707233382985, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329435707308880459, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435707233382985, language=EN, stringName=Ping Wang, firstName=Ping, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Computer Science, Northwest University, Xi’an 710127, bio={"content":"

born in 2002. Master candidate. His main research interest includes large model visual reasoning

"}, bioImg=null, bioContent=

born in 2002. Master candidate. His main research interest includes large model visual reasoning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329435707380183628, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435707233382985, language=CN, stringName=王平, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1西北大学计算机学院 西安 710127, bio={"img":"1jrykixFy//FWewzY0cOTw==","content":"

2002年生。硕士研究生。主要研究方向为大模型视觉推理

"}, bioImg=1jrykixFy//FWewzY0cOTw==, bioContent=

2002年生。硕士研究生。主要研究方向为大模型视觉推理

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329435706209972782, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, xref=1, ext=[AuthorCompanyExt(id=1329435706222555695, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science, Northwest University, Xi’an 710127), AuthorCompanyExt(id=1329435706230944304, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1西北大学计算机学院 西安 710127)])]), Author(id=1329435707468264014, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, orderNo=5, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329435707568927312, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435707468264014, language=EN, stringName=Jie Zheng, firstName=Jie, middleName=null, lastName=Zheng, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Computer Science, Northwest University, Xi’an 710127, bio={"content":"

born in 1985. PhD, professor, master supervisor. His main research interest includes edge general intelligence

"}, bioImg=null, bioContent=

born in 1985. PhD, professor, master supervisor. His main research interest includes edge general intelligence

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329435707648619089, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, authorId=1329435707468264014, language=CN, stringName=郑杰, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1西北大学计算机学院 西安 710127, bio={"img":"9Vnp7bpLM1DolD2CFV6lXQ==","content":"

1985年生。博士,教授,硕士生导师。主要研究方向为边缘通用智能

"}, bioImg=9Vnp7bpLM1DolD2CFV6lXQ==, bioContent=

1985年生。博士,教授,硕士生导师。主要研究方向为边缘通用智能

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329435706209972782, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, xref=1, ext=[AuthorCompanyExt(id=1329435706222555695, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Computer Science, Northwest University, Xi’an 710127), AuthorCompanyExt(id=1329435706230944304, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329435702628037129, companyId=1329435706209972782, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1西北大学计算机学院 西安 710127)])])]
杨建锋1 , 朱东坡2, 曹瑞1, * , 崔选皓1, 王平1, 郑杰1
作者信息
  • 1西北大学计算机学院 西安 710127
  • 2北京三快科技有限公司 北京 100190
通讯作者:
曹瑞()
作者简介:

作者贡献声明

杨建锋提出研究构思并指导研究开展,审阅和修改论文;朱东坡负责方法实现、模型搭建、实验与数据分析,并撰写论文;曹瑞审阅和修改论文,统筹审稿意见回复并负责最终定稿;崔选皓参与论文审阅与修改;王平参与论文审阅与修改;郑杰提出改进思路,并提供方法优化建议与分析支持。

1981年生。硕士,高级工程师,硕士生导师。主要研究方向为软件开发方法与数据科学及其在高校中的应用

1999年生。硕士,工程师。主要研究方向为多模态融合应用

1991年生。博士,讲师,硕士生导师。CCF会员。主要研究方向为文本表示学习、生成式大语言模型应用、优化及扩散模型艺术生成

2003年生。硕士研究生。主要研究方向为大模型开发应用

2002年生。硕士研究生。主要研究方向为大模型视觉推理

1985年生。博士,教授,硕士生导师。主要研究方向为边缘通用智能

A Multi-Feature Image Captioning Model with Common-Specific Collaborative Modeling and Mixture-of-Experts Fusion
Jianfeng Yang1 , Dongpo Zhu2, Rui Cao1, * , Xuanhao Cui1, Ping Wang1, Jie Zheng1
Affiliations
  • 1School of Computer Science, Northwest University, Xi’an 710127
  • 2Beijing Sankuai Technology Co., Ltd., Beijing 100190
  • born in 1981. Master, senior engineer, master supervisor. His main research interests include software development methodologies and data science, with a focus on applications in higher education

    born in 1999. Master, engineer. His main research interest includes multi-modal fusion and application

    born in 1991. PhD, lecturer, master supervisor. Member of CCF. His main research interests include text representation learning, applications and optimization of generative large language models, and art generation using diffusion models

    born in 2003. Master candidate. His main research interest includes large model development and application

    born in 2002. Master candidate. His main research interest includes large model visual reasoning

    born in 1985. PhD, professor, master supervisor. His main research interest includes edge general intelligence

出版时间: 2026-11-01 doi: 10.7544/issn1000-1239.202550936
文章导航
收藏切换

针对现有图像描述方法难以同时兼顾全局语义与细粒度细节、生成结果易偏泛化或关键信息遗漏的问题,提出一种基于共性-特性协同建模与多专家融合的多特征图像描述生成模型EAFCC(expert-aware fusion with common-specific collaboration)。该模型并行接收网格特征、区域特征与分割特征,在编码阶段首先通过预融合模块对多源异构特征进行隐式对齐。在共享语义空间中提取跨特征一致的共性信息,同时为各特征流保留差异化表达基础。随后利用共享Transformer编码器建模共性表征,并通过对比学习约束进一步增强其一致性与判别性。同时,采用3条独立编码分支对不同特征流进行特性建模,以保留与特征类型相关的细粒度补充信息,并借助多专家门控路由机制实现特性信息的动态加权融合。最后,将增强后的共性特征与融合后的特性特征进行协同整合,经跳跃连接编码器后送入Transformer解码器生成图像描述。在MS-COCO数据集上的实验表明,EAFCC在CIDEr-D指标上取得了较优性能,并优于现有单特征或双特征方法,验证了共性与特性联合建模的有效性。

图像描述  /  多特征融合  /  共性-特性协同建模  /  对比学习  /  Transformer

Existing image captioning methods often struggle to simultaneously capture global semantics and fine-grained details, leading to overly generic descriptions or missing key information. To address this issue, we propose EAFCC, short for expert-aware fusion with common-specific collaboration, a multi-feature image captioning model with mixture-of-experts fusion. The model takes grid features, region features, and segmentation features as parallel inputs. In the encoding stage, a pre-fusion module first performs implicit alignment of heterogeneous visual features and models cross-feature common information within a shared semantic space, while preserving the basis for feature-specific representations. A shared Transformer encoder is then employed to learn the common representations, whose consistency and discriminability are further enhanced by a contrastive learning constraint. Meanwhile, three independent encoding branches are used to model feature-specific representations, so as to retain fine-grained complementary information associated with different feature types. A gated routing mechanism inspired by mixture-of-experts is further introduced to dynamically weight and fuse the specific information. Finally, the enhanced common features and the fused specific features are collaboratively integrated, processed by a skip-connected encoder, and fed into a Transformer decoder to generate image captions. Experimental results on MS-COCO dataset show that EAFCC achieves competitive performance on the CIDEr-D metric and outperforms existing single-feature and dual-feature methods, demonstrating the effectiveness of common-specific collaborative modeling and dynamic multi-feature fusion for image captioning.

image captioning  /  multi-feature fusion  /  common-specific collaborative modeling  /  contrastive learning  /  Transformer
杨建锋, 朱东坡, 曹瑞, 崔选皓, 王平, 郑杰. 基于共性-特性协同建模与多专家融合的多特征图像描述模型. 计算机研究与发展, 2026 , 63 (11) : 3002 -3016 . DOI: 10.7544/issn1000-1239.202550936
Jianfeng Yang, Dongpo Zhu, Rui Cao, Xuanhao Cui, Ping Wang, Jie Zheng. A Multi-Feature Image Captioning Model with Common-Specific Collaborative Modeling and Mixture-of-Experts Fusion[J]. Journal of Computer Research and Development, 2026 , 63 (11) : 3002 -3016 . DOI: 10.7544/issn1000-1239.202550936
图像描述生成是计算机视觉与自然语言处理的交叉任务,其目标为根据给定图像自动生成准确且符合人类认知的自然语言描述,涵盖图像特征提取、图像理解、特征融合等关键环节[1]。随着深度学习技术的快速发展,虽然模型结构不断更新,但生成质量的瓶颈往往来自视觉信息供给不足与融合不充分,因此输入特征的选择及其融合策略逐渐成为决定描述准确性与细节丰富度的核心因素。
在早期的主流范式中,主要依赖目标检测网络提取区域特征作为图像信息[2]。区域特征以对象级表示刻画图像中的具体事物,显著降低视觉嵌入复杂度,有效提升描述准确性。然而,区域特征的对象级特性导致其缺乏全局视觉理解与细粒度信息,限制了描述的丰富性:生成结果常仅涵盖图像主体而忽视背景,且描述长度偏短。
为弥补区域特征的全局信息缺失,部分学者引入网格特征,利用其对整幅图像的全局概括能力补充细粒度与背景信息[3-4]。但网格特征与区域特征在特征粒度与语义空间上难以对齐,且网格特征的碎片化表达可能与区域特征产生信息冲突,导致模型语义混淆。
为进一步缓解上述2类特征的互补缺陷,本文在区域特征与网格特征基础上引入语义分割特征[5]。该特征以粗粒度语义边界刻画场景布局,可同时弥补区域特征背景信息不足与网格特征信息碎片化问题,使视觉信息更为全面、准确,为后续描述生成提供坚实基础。
基于上述分析,本文借鉴因式分解思想,将多源视觉表征拆分为跨特征共享的共性与各特征特有的特性,并据此提出多专家多特征共性与特性提取融合的图像描述模型EAFCC(expert-aware fusion with common-specific collaboration)。针对多特征输入下的场景语义理解不足、特征融合冲突与冗余以及细节刻画不足等问题,本文分别引入预融合(pre-fusion,PF)模块、共性-特性协同建模与对比学习、多专家融合机制以及跳跃连接编码与解码策略,从而提升多特征建模能力与描述生成性能。为直观说明本文方法的设计动机及各模块之间的逻辑关系,图1给出了“问题-方法-效果”的整体对应框架。模型首先并行输入网格特征、区域特征与分割特征;随后通过预融合模块隐式对齐3类特征的语义空间,并在拉近特征距离的同时保留各特征独有信息;接着利用共享Transformer[6]编码器提炼跨特征的共性表征,并引入困难负样本对比学习策略进一步增强共性特征之间的一致性;针对各特征独有的特性信息,采用3条参数独立的Transformer编码器分别提取,并借鉴多专家系统的门控路由机制动态加权融合特性信息;最后,将融合后的特性特征与增强后的共性特征拼接,经跳跃连接编码器后送入Transformer解码器完成描述生成,以减少编码过程中的信息丢失。
本文的主要贡献包括4个方面。
1)提出共性-特性协同建模的多特征融合框架。针对单特征或拼接式融合难以兼顾一致性与互补性的问题,提出共性-特性协同建模的多特征融合框架:共性建模跨特征语义一致性,特性建模细粒度互补细节,并通过门控机制实现自适应融合。
2)设计基于对比学习的共性增强机制。引入困难负样本对比学习,在共享Transformer编码器输出的共性特征空间中,强化不同特征间语义一致性,抑制跨模态差异,提升共性表征的判别力与鲁棒性。
3)提出多专家门控路由的特性融合策略。借鉴混合专家网络(mixture-of-experts,MoE)思想,将各特征的特性表示视为“专家”,通过可学习的门控网络动态分配权重,实现特性信息的自适应加权融合,避免冗余,突出差异。
4)构建参数量上更具优势的高性能图像描述模型EAFCC。在无需大规模预训练数据的前提下,于MS-COCO数据集上取得135.2的CIDEr-D分数,超越现有单特征、双特征及多数预训练模型,验证“小数据+细粒度架构”在资源受限场景下的有效性。
国内外图像描述研究已由早期CNN+RNN模板或检索范式全面转向端到端Transformer架构,并在特征融合、预训练与轻量化等方向持续深入。现有工作聚焦单特征增强与大规模视觉-语言预训练[7],国内近年来则提出跨层动态聚合、双重路径视觉-语言、多级语义对比、知识提示及蒸馏-压缩框架等创新,在MS-COCO数据集上CIDEr-D指标达到了135.2,并兼顾中文场景与边缘部署需求。然而,现有方法仍普遍采用网格-区域双特征,对共性-特性协同建模与分割特征利用不足,本文所研究的EAFCC模型针对此空白展开系统研究。
图像描述研究已经由早期基于模板、检索和CNN-RNN的方法,逐步发展到以Transformer为核心的编解码框架[8-9]。在该方向上,研究人员分别从注意力分配优化[10]、层间特征交互[11]、记忆增强[12]以及视觉-语言上下文联合建模[13]等方面改进了Transformer图像描述框架,从而提升了模型对复杂视觉语义和语言依赖关系的表征能力。DLCT[14],GRIT[15]和DIFNet[16]进一步说明,Transformer框架下的信息组织方式已成为影响性能的关键因素。与此同时,视觉语言预训练模型如BLIP[17],OFA[18],OFASys[19],SimVLM[20],UNIMO[21],Unified VLP[22]推动了图像描述与更广义视觉语言生成的融合。近2年,该方向又进一步延伸到检索增强和可控描述等新范式,例如EVCap[23]利用外部视觉名称记忆(visual-name memory)增强开放世界描述能力,FineCaption[24]则支持任意粒度区域与掩码控制下的组合式描述生成。
图像描述模型的性能高度依赖输入视觉特征的质量与互补性。现有方法常使用区域特征、网格特征以及分割特征等不同表征。区域特征通常由目标检测器提取,典型如文献[25]所依赖的检测式区域表示,其优势在于突出显著目标及局部属性;网格特征更强调全局空间布局与上下文关系;分割特征则能够补充像素级语义区域与边界信息,相关提取网络如Faster R-CNN[26]和UPSNet[27]分别对应区域级和全景/分割级表征来源。此外,SGAE[28],GCN-LSTM[29]等方法还通过显式关系建模增强结构化语义表达。总体而言,不同视觉特征在感受野、语义粒度和关注重点上各有优势,单一特征往往难以同时兼顾全局场景、局部实体和区域边界,因此多视觉特征联合建模已成为提升图像描述准确性与细粒度表达能力的重要思路。
围绕多特征利用,已有研究主要从多路输入和融合机制2个方面展开。文献[1]较早探索了多视角视觉表示在Transformer图像描述中的作用。代表性工作[30-33]分别从交叉注意力建模、循环融合、层次注意力以及双层协同交互等方面改进多特征信息的组织方式,从而增强了图像描述模型对异构视觉特征的联合表征能力。GRIT则进一步表明双视觉特征能够在效率与性能之间取得较好平衡。除视觉特征间融合外,近年来还出现了检索增强图像描述方向,例如GFF[33],SmallCap[34],M-RAT[35],EVCap[23]通过引入检索文本、对象名称或外部记忆,为描述生成提供额外语言先验。与此同时,ConCap[36]和FineCaption[24]代表了可控与组合式图像描述的发展趋势。尽管这些方法在融合策略上取得了显著进展,但多数工作更关注“如何融合”,而较少显式区分不同特征间的共享语义与特异信息。
在多源信息建模中,仅通过拼接、加权或注意力交互进行融合,容易忽略不同特征之间同时存在的“共性”与“特性”。从更一般的视觉语言学习视角看,UNIMO[21]通过跨模态对比学习增强统一表征,NAAF[37]强调困难样本与判别性关系建模,EVE[38]则从模态感知与专家路由角度提升多模态表示能力。这些研究表明,显式建模共享信息与差异信息,并借助对比约束提升语义一致性,是增强多源特征表达能力的重要方向。对于图像描述任务而言,现有多特征方法虽然已经开始关注跨特征交互,但对共享语义如何增强、特异信息如何保留以及两者如何协同融合的讨论仍不充分。基于此,本文在Transformer编解码框架下引入多视觉特征的共性-特性协同建模,通过预融合、共性增强和动态融合机制,提升标准图像描述任务中的多源视觉信息组织能力与描述生成质量。
本文提出的EAFCC模型整体结构如图2所示。模型首先对输入图像进行多源视觉表征提取,利用UPSNet提取分割特征${\boldsymbol{F}}_{\mathrm{S}} $,利用Faster R-CNN提取区域特征${\boldsymbol{F}}_{\mathrm{R}} $和网格特征${\boldsymbol{F}}_{\mathrm{G}} $。3类特征分别从场景语义布局、目标级局部信息和全局视觉上下文3个层面刻画图像内容,具有显著的互补性。然而,由于三者在表示粒度、语义侧重点和分布空间上存在差异,若直接拼接或粗粒度融合,容易引入语义错位与信息干扰,进而影响描述生成的准确性与细节完整性。
为此,本文围绕共性提取—特性建模—协同融合的思路构建统一的编码与解码框架。在编码阶段,模型首先将分割特征、区域特征和网格特征送入预融合共享编码器(PF)模块。该模块一方面对多源异构特征进行初步对齐与交互,缩小不同特征之间的表示差异;另一方面提取不同特征流之间共享的高层语义信息,为后续共性增强提供统一的表示基础。与传统仅关注特征拼接或简单注意力加权的方法不同,PF模块不仅承担输入对齐作用,同时也是后续共性分支的语义起点。
在此基础上,模型进一步将经PF处理后的多路特征送入3条参数独立的特征编码分支,即Encoder-S,Encoder-R,Encoder-G,分别针对分割特征流、区域特征流和网格特征流提取各自保留的特性信息。这样可以在共享语义空间对齐的前提下,最大限度保留不同特征在目标边界、区域实体和场景全局结构上的差异化表达。随后,3路特性特征被输入多专家路由融合网络(multi-feature fusion with mixture-of-experts,MMF),通过门控路由机制自适应建模各特征分支对当前图像描述任务的贡献程度,实现多特征特性信息的动态加权融合,从而得到融合后的特性表征。
与此同时,为进一步提升不同特征流在共享语义层面的表达一致性,本文在共性分支中引入对比学习增强模块(contrastive learning,CL)。该模块以PF输出的共享表征为基础,通过对比约束拉近语义相关的共性表示、区分潜在干扰表示,从而增强共性信息的判别性与鲁棒性。经过CL模块处理后,模型获得语义更加一致、表达更加稳定的共性特征表示。
最后,增强后的共性特征与MMF输出的融合特性特征在编码器末端进行协同汇聚,并共同送入Transformer解码器。解码阶段,模型结合词嵌入与位置嵌入,利用多层Transformer解码结构逐步生成自然语言描述,最终经线性映射与Softmax输出完整句子。通过上述设计,EAFCC模型能够同时利用多源视觉特征中的共享语义与差异化细节,从而提升图像描述的语义准确性、信息完整性和表达丰富性。后文将依次介绍各模块的具体结构与实现方式。
为缓解分割特征、区域特征和网格特征在表示粒度与语义分布上的差异,并在初步对齐的同时尽可能保留各自的特异性信息,本文在编码器前端引入预融合模块,其结构如图3所示。PF模块由共享参数的多头自注意力(multi-head attention)子层、共享层归一化子层、共享位置逐元素前馈子层以及特征流独立的层归一化子层组成,整体采用残差连接形式堆叠。
具体而言,给定3类输入特征$ {\boldsymbol{F}}_{\text{G}},{\boldsymbol{F}}_{\text{R}},{\boldsymbol{F}}_{\text{S}} $,首先通过共享参数的多头自注意力机制对3路特征进行统一建模,并结合共享层归一化操作对其进行初步对齐,从而缩小不同特征流在共享语义空间中的表示偏移。由于该阶段采用共享参数,不同特征流能够在一致的映射约束下学习更稳定的高层共性表示。随后,在对齐后的表示基础上,引入共享位置逐元素前馈网络进一步增强特征表达,并通过各特征流独立的层归一化操作来保留分割特征、区域特征和网格特征各自的特异性,从而避免过度共享导致的个性信息损失。通过这一设计,PF模块能够在共享对齐与特性保持之间取得平衡,为后续共性增强与特性建模提供更合适的输入表示。
在本文实现中,PF模块的堆叠层数设为$ {N}_{1}=2 $,设共享多头自注意力$ MHS {A}_{\text{sh}}(\cdot ) $和共享位置逐元素前馈网络$ L{N}_{\text{sh}}(\cdot ) $的参数分别为$ {\boldsymbol{\theta }}_{\text{att}} $和$ {\boldsymbol{\theta }}_{\text{ff}} $,共享层归一化参数为$ \left({\boldsymbol{\alpha }}_{\text{sh}},{\boldsymbol{\beta }}_{\text{sh}}\right) $,3条特征流独立层归一化参数分别为$ \left({\boldsymbol{\alpha }}_{\text{G}},{\boldsymbol{\beta }}_{\text{G}}\right),\left({\boldsymbol{\alpha }}_{\text{R}},{\boldsymbol{\beta }}_{\text{R}}\right),\left({\boldsymbol{\alpha }}_{\text{S}},{\boldsymbol{\beta }}_{\text{S}}\right) $。则单层PF模块可表示为
$ {\boldsymbol{M}}_{\text{G}}=L{N}_{\text{sh}}\left(MHS {A}_{\text{sh}}\left({\boldsymbol{F}}_{\text{G}};{\boldsymbol{\theta }}_{\text{att}}\right)+{\boldsymbol{F}}_{\text{G}};{\boldsymbol{\alpha }}_{\text{sh}},{\boldsymbol{\beta }}_{\text{sh}}\right)\;,$
$ {\boldsymbol{M}}_{\text{R}}=L{N}_{\text{sh}}\left(MHS {A}_{\text{sh}}\left({\boldsymbol{F}}_{\text{R}};{\boldsymbol{\theta }}_{\text{att}}\right)+{\boldsymbol{F}}_{\text{R}};{\boldsymbol{\alpha }}_{\text{sh}},{\boldsymbol{\beta }}_{\text{sh}}\right)\;,$
$ {\boldsymbol{M}}_{\text{S}}=L{N}_{\text{sh}}\left(MHS {A}_{\text{sh}}\left({\boldsymbol{F}}_{\text{S}};{\boldsymbol{\theta }}_{\text{att}}\right)+{\boldsymbol{F}}_{\text{S}};{\boldsymbol{\alpha }}_{\text{sh}},{\boldsymbol{\beta }}_{\text{sh}}\right)\;,$
其中$ {\boldsymbol{M}}_{\text{G}}{,\boldsymbol{M}}_{\text{R}}{,\boldsymbol{M}}_{\text{S}} $表示经过共享对齐后的特征。在此基础上,进一步通过共享前馈网络$ PWF{F}_{\text{sh}}(\cdot ) $和特征独立的层归一化$ L{N}_{i}(\cdot ),i \in \{{ {\mathrm{G}},{\mathrm{R}},{\mathrm{S}}}\} $得到最终输出为
$ {\boldsymbol{Z}}_{\text{G}}=L{N}_{\text{G}}\left(PWF{F}_{\text{sh}}\left({\boldsymbol{M}}_{\text{G}};{\boldsymbol{\theta }}_{\text{ff}}\right)+{\boldsymbol{M}}_{\text{G}}\text{;}{\boldsymbol{\alpha }}_{\text{G}},{\boldsymbol{\beta }}_{\text{G}}\right)\;,$
$ {\boldsymbol{Z}}_{\text{R}}=L{N}_{\text{R}}\left(PWF{F}_{\text{sh}}\left({\boldsymbol{M}}_{\text{R}};{\boldsymbol{\theta }}_{\text{ff}}\right)+{\boldsymbol{M}}_{\text{R}}\text{;}{\boldsymbol{\alpha }}_{\text{R}},{\boldsymbol{\beta }}_{\text{R}}\right)\;,$
$ {\boldsymbol{Z}}_{\text{S}}=L{N}_{\text{S}}\left(PWF{F}_{\text{sh}}\left({\boldsymbol{M}}_{\text{S}};{\boldsymbol{\theta }}_{\text{ff}}\right)+{\boldsymbol{M}}_{\text{S}}\text{;}{\boldsymbol{\alpha }}_{\text{S}},{\boldsymbol{\beta }}_{\text{S}}\right)\;,$
其中$ {\boldsymbol{Z}}_{\text{G}},{\boldsymbol{Z}}_{\text{R}},{\boldsymbol{Z}}_{\text{S}} $分别表示PF模块输出的3路特征表示。需要说明的是,式(1)~(6)描述了单层PF,实际重复堆叠$ {N}_{1}=2 $,上层输出作为下层输入。
为了捕捉不同特征之间的共性信息,模型采用共享编码器机制。共享Transformer编码器采用标准结构,由多层堆叠,每层含多头自注意力机制、前馈神经网络,使用残差连接和层归一化技术,经多层堆叠对网格特征${\boldsymbol{F}}_{\mathrm{G}} $、区域特征${\boldsymbol{F}}_{\mathrm{R}} $和分割特征${\boldsymbol{F}}_{\mathrm{S}} $统一编码,将不同输入特征映射到共享语义空间,提取共有语义信息,最终提炼出共性特征表示$ {\boldsymbol{C}}_{\text{G}},{\boldsymbol{C}}_{\text{R}},{\boldsymbol{C}}_{\text{S}} $。对于每个输入的特征信息流$ {\boldsymbol{Z}}_{\text{G}},{\boldsymbol{Z}}_{\text{R}},{\boldsymbol{Z}}_{\text{S}} $,共享编码器计算过程可以表示为
$ {\boldsymbol{C}}_{i}=S hareEncoder({\boldsymbol{Z}}_{i})\;,$
其中$ S hareEncoder(\cdot ) $是共享编码器操作,$ i\in \{ {\mathrm{G}},{\mathrm{R}}, {\mathrm{S}}\} $时,$ {\boldsymbol{C}}_{i} $分别为对应的共性特征表示$ {\boldsymbol{C}}_{\text{G}},{\boldsymbol{C}}_{\text{R}},{\boldsymbol{C}}_{\text{S}} $。
为保留不同特征流的差异化信息,本文在共享语义基础上进一步建模各特征流的特性表征。对于PF模块输出的分支特征表示$ {\boldsymbol{Z}}_{i} $,采用参数独立的Transformer编码器进一步提取特性特征,计算过程可以表示为
$ {\boldsymbol{P}}_{i}={PrivateEncoder}_{i}({\boldsymbol{Z}}_{i})\;,$
其中$ {PrivateEncoder}_{i}(\cdot ) $为特征编码器编码操作,$ i\in \{{\mathrm{G}},{\mathrm{R}},{\mathrm{S}}\} $时,$ {\boldsymbol{P}}_{i} $分别为对应的特性表征$ {\boldsymbol{P}}_{\text{G}},{\boldsymbol{P}}_{\text{R}},{\boldsymbol{P}}_{\text{S}} $。
在共享Transformer编码器获得初步共性表征后,不同特征流之间虽然已具备一定的共享语义基础,但由于分割特征、区域特征和网格特征在表示粒度与语义侧重点上存在天然差异,其共性表示仍可能存在分布偏移,进而影响后续特征协同与描述生成质量。为进一步提升共享语义表示的一致性与鲁棒性,本文在共性分支中引入基于对比学习的共性特征增强模块(CL)。
该模块以共享编码器输出的3路共性特征作为输入,通过对共享语义空间中的表征关系进行约束,增强语义相关共性表示之间的聚合性,并抑制无关干扰信息带来的偏移,从而获得更加紧致、稳定且具有判别性的共性表征。经过CL模块增强后,模型输出3路增强的共性特征,并将其与后续融合得到的特性特征共同用于描述生成,其具体损失函数形式与优化过程将在2.7节中详细介绍。
为了有效地融合特性特征$ {\boldsymbol{P}}_{i} $,模块采用基于多专家的图像多特征融合机制。该机制借鉴了多专家系统(mixture of expert,MoE)的思想[29],将每个特征信息流的特性特征视为一个“专家”,利用门控路由网络学习每个特性特征的重要性权重,并根据权重对不同的特性特征进行加权融合。模块使用一个门控网络(gating network)来计算每个专家特性流的权重[10]。门控网络采用MLP结构,输入是拼接的特性特征$ {\boldsymbol{P}}_{i} $,其公式为
$ \boldsymbol{w}\coloneqq \left(\begin{array}{c}{\omega }_{\text{G}}\\{\omega }_{\text{R}}\\{\omega }_{\text{S}}\end{array}\right)=S oftmax\left(MLP\left(Concat\left({\boldsymbol{P}}_{\text{G}},{\boldsymbol{P}}_{\text{R}},{\boldsymbol{P}}_{\text{S}}\right)\right)\right)\;, $
其中$ {\omega }_{\text{G}} $,$ {\omega }_{\text{R}} $,$ {\omega }_{\text{S}} $分别表示3个分支的权重,经Softmax归一化后满足$ {\omega }_{\text{G}}+{\omega }_{\text{R}}+{\omega }_{\text{S}}=1 $。得到门控权重$ \boldsymbol{w} $之后,融合后的特性特征$ {\boldsymbol{F}}_{\text{fused}} $可以通过门控加权拼接的方式计算为
$ {\boldsymbol{F}}_{\text{fused}}=Concat\left({\omega }_{\text{G}}{\boldsymbol{P}}_{\text{G}},{\omega }_{\text{R}}{\boldsymbol{P}}_{\text{R}},{\omega }_{\text{S}}{\boldsymbol{P}}_{\text{S}}\right)\;。 $
为了减少图像信息在编码过程中丢失的问题,提出了跳跃连接编码器模块,如图4所示。
与常规编码方式相比,在对特征进行提取增强时,输入到下一层编码器的特征不仅包含上一层编码器的输出,还包含上一层的输入信息。在输入到下一个编码器之前,为降低计算资源的消耗,采用2个输入直接相加的方式,并且为最大程度降低特征在增强提取过程中的丢失,采用将初始输入的信息流直接与最高层编码器的输出流相加的方式得到最终输出。
设FSC模块采用的是N层Transformer编码器进行堆叠,$ {\boldsymbol{F}}_{0} $是原始输入的特征,那么第1层到第N$ - $1层的输出可以表示为
$ {\boldsymbol{F}}_{n}={Encoder}_{n}\left({\boldsymbol{F}}_{n-1}\right)+{\boldsymbol{F}}_{n-1}\;,$
其中$ n\in \{1{,}2,3,… ,N-1\} $,$ {\boldsymbol{F}}_{n} $表示经过第n层编码器的输出,$ {Encoder}_{n}(\cdot ) $表示对输入编码器层的输入进行一系列操作,其中主要包括多头自注意力机制、前馈神经网络以及残差连接和层归一化处理。在第N层时,由于使用跳跃连接,其公式为
$ {\boldsymbol{F}}_{N}={Encoder}_{N}\left({\boldsymbol{F}}_{N-1}\right)+{\boldsymbol{F}}_{0}\;。 $
经过多层编码以及跳跃连接的特征将不同层次的特征进行汇总,使得最终特征既包含低层语义信息,又包含高层语义信息,实现了多尺度信息的保留。需要说明的是,FSC作为共享编码器和分支专属编码器的基本实现结构,当输入为$ {\boldsymbol{Z}}_{i} $时,共享FSC编码器输出的$ {\boldsymbol{F}}_{N} $为$ {\boldsymbol{C}}_{i} $,分支专属FSC编码器输出的$ {\boldsymbol{F}}_{N} $为$ {\boldsymbol{P}}_{i} $。
共享编码器输出的共性特征$ {\boldsymbol{C}}_{\text{G}},{\boldsymbol{C}}_{\text{R}},{\boldsymbol{C}}_{\text{S}} $在训练过程中受到对比学习损失的约束,从而获得更强的一致性与判别性。在解码生成阶段,融合后的特性特征$ {\boldsymbol{F}}_{\text{fused}} $以及增强后的共性特征$ {\boldsymbol{C}}_{\text{G}},{\boldsymbol{C}}_{\text{R}},{\boldsymbol{C}}_{\text{S}} $将被拼接后作为最终Transformer解码器的图像特征输入为
$ {\boldsymbol{F}}_{\text{in}}=Concat\left({\boldsymbol{F}}_{\text{fused}},{\boldsymbol{C}}_{\text{G}},{\boldsymbol{C}}_{\text{R}},{\boldsymbol{C}}_{\text{S}}\right)\;,$
其中$ {\boldsymbol{F}}_{\text{in}} $表示解码器的输入特征。最后,通过多层堆叠的Transformer解码器的掩码多头自注意力机制和交叉注意力机制处理,生成完整的图像描述。
为同时增强多特征共性表征的判别能力,并提升图像描述生成的词级预测精度与序列级生成质量,本文采用对比学习约束与监督训练及强化学习微调的2阶段优化策略。整体而言,对比学习损失作用于编码阶段的共性增强模块,用于约束不同特征流在共享语义空间中的表示一致性;交叉熵损失用于第1阶段监督训练,以学习图像特征与目标描述之间的基本对齐关系;在此基础上,进一步引入基于自我批评序列训练(self-critical sequence training,SCST)[39]的强化学习方法,以CIDEr-D作为奖励函数对模型进行微调,从而直接优化序列级评价目标。
1)对比学习损失
为增强不同特征流共性信息的一致性并学习细粒度差异,本文在共性增强模块中引入基于InfoNCE(noise contrastive estimation)[11]的对比学习损失函数。以网格特征${\boldsymbol{F}}_{\mathrm{G}} $和区域特征${\boldsymbol{F}}_{\mathrm{R}} $为例,设同一训练批次内的每个图像样本j的正样本对为$ (\boldsymbol{C}_{j}^{\text{G}},\boldsymbol{C}_{j}^{\text{R}}) $,完整的负样本池可表示为$ N_{j}^{\text{(G,R})}=\left\{\boldsymbol{C}_{k}^{\text{R}}\mid k\neq j\right\}\cup \left\{\boldsymbol{C}_{l}^{\text{S}}\mid l\neq j\right\} $,其对比学习损失函数如式(14)所示。
$ \mathcal{L}_{\text{CL}}^{(j)}=-\text{ln} \frac{\exp \left(\mathit{si}m (\boldsymbol{C}_{j}^{\text{G}},\boldsymbol{C}_{j}^{\text{R}})/\tau \right)}{ \mathrm{exp}\left(\mathit{sim} (\boldsymbol{C}_{j}^{\text{G}},\boldsymbol{C}_{j}^{\text{R}})/\tau \right) +\displaystyle\sum_{k\neq j} \exp\left(\mathit{sim} (\boldsymbol{C}_{j}^{\text{G}},\boldsymbol{C}_{k}^{\text{R}})/\tau \right) +\displaystyle\sum_{l\neq j}\exp \left(\mathit{sim} (\boldsymbol{C}_{j}^{\text{G}},\boldsymbol{C}_{l}^{\text{S}})/\tau \right)}\;, $
其中$ j $表示当前图像样本的索引,$ k $和$ l $表示同一批次内其他图像样本的索引,$ \boldsymbol{C}_{j}^{\text{G}},\boldsymbol{C}_{j}^{\text{R}},\boldsymbol{C}_{j}^{\text{S}} $分别表示第j个图像样本的网格特征、区域特征和分割特征的共性特征,$ sim(\cdot ,\cdot ) $表示相似度函数,$ \tau $表示温度系数,$ B $表示批次大小。
${\boldsymbol{F}}_{\mathrm{G}} $与${\boldsymbol{F}}_{\mathrm{S}} $以及${\boldsymbol{F}}_{\mathrm{S}} $与${\boldsymbol{F}}_{\mathrm{R}} $之间的计算方式与式(14)类似,最终通过3组对比损失的相加,确保所有特征信息流的两两之间的共性均被增强,从而增强3种特征信息之间的共性信息。
为了提升对比学习的难度和效率,模型进一步采用了困难负样本采样(hard negative sampling)策略。困难负样本指的是那些与正样本相似度较高的负样本,与随机负采样相比,困难负样本采样策略能够更有效地挖掘和区分不同特征信息之间的细微差异,从而提升对比学习的效果。
以${\boldsymbol{F}}_{\mathrm{G}} $和${\boldsymbol{F}}_{\mathrm{R}} $为例,困难负样本的选择策略为:对于每个正样本对$ (\boldsymbol{C}_{j}^{{\mathrm{G}}},\boldsymbol{C}_{j}^{{\mathrm{R}}}) $,模型首先计算其与批次内所有其他负样本的相似度,然后选择相似度最高的$ K $个负样本作为困难负样本,构建困难负样本集合$ {H}_{j} $,$H_j $可以表示为
$ {H}_{j}=TopK\left(\left\{sim (\boldsymbol{C}_{j}^{\text{G}},\boldsymbol{z} )| \boldsymbol{z}\in N_{j}^{\text{(G,R})}\right\}\right)\;,$
其中$ \boldsymbol{z} $表示从负样本池$ N_{j}^{\text{(G,R})} $中选取的任意候选负样本共性表征。将该策略融入对比学习损失函数之后,计算如式(16)所示。
$ \mathcal{L}_{{\text{CL}}_{\text{HNS}}}^{\left(j\right)}=-\ln \frac{\exp \left(\mathit{sim} (\boldsymbol{C}_{j}^{\text{G}},\boldsymbol{C}_{j}^{\text{R}})/\tau \right)}{\mathrm{exp}\left(\mathit{sim} (\boldsymbol{C}_{j}^{\text{G}},\boldsymbol{C}_{j}^{\text{R}})/\tau \right)\\+\displaystyle\sum_{h\in {{H}_{j}}}\exp \left(h/\tau \right)}\;, $
其中$ h $表示困难负样本集合$ {H}_{j} $中的一个相似度值。根据式(15),$ {H}_{j} $由候选负样本与锚点特征$ \boldsymbol{C}_{j}^{{\mathrm{G}}} $之间相似度最高的$ K $个数值组成。
通过最小化基于困难负样本采样的对比学习损失函数$ \mathcal{L}_{{\text{CL}}_{\text{HNS}}}^{\left(j\right)} $,模型能够有效地增强不同特征共性的一致性,并学习细粒度的差异,最终得到增强后的共性特征$ {\boldsymbol{C}}_{\text{G}},{\boldsymbol{C}}_{\text{R}},{\boldsymbol{C}}_{\text{S}} $。
2)交叉熵监督训练
在第1阶段训练中,本文采用单词级交叉熵(CE)损失对模型进行监督训练,以提高解码器对目标词的预测能力。设目标描述序列为$ {Y}^{*}=\left(y_{1}^{*},y_{2}^{*},… ,y_{T}^{*}\right) $,融合后的视觉表示为$ {\boldsymbol{F}}_{\text{in}} $,则交叉熵损失定义为
$ {\mathcal{L}}_{\text{CE}}=-\displaystyle\sum_{t=1}^{T}\ln p\left(y_{t}^{*}\mid y_{1}^{*},… ,y_{t-1}^{*},{\boldsymbol{F}}_{\text{in}}\right)\;,$
其中$ T $表示目标描述序列的长度,$ t $表示词语生成的时间步,$ y_{t}^{*} $表示第$ t $个真实目标词,上标“$ * $”表示真实标注,$ p(\cdot ) $表示模型在时刻t对目标词的条件概率。该阶段主要用于建立稳定的图像-文本对齐关系,并为后续序列级优化提供良好的初始化参数。考虑到对比学习损失同样在第1阶段参与优化,因此第1阶段的总体目标函数可写为
$ {\mathcal{L}}_{\text{stage1}}={\mathcal{L}}_{\text{CE}}+\frac{1}{B}\displaystyle\sum_{j=1}^{B}\mathcal{L}_{{\text{CL}}_{\text{HNS}}}^{(j)}\;,$
其中$ B $表示批次大小,$ j $表示批次内的样本索引。
3)基于SCST的强化学习微调
在第1阶段训练收敛后,本文进一步采用基于自我批评序列训练(self-critical sequence training,SCST)的强化学习方法对模型进行微调,以直接优化图像描述任务中的序列级评价指标。具体而言,模型首先通过采样方式生成描述序列$ {Y}^{\text{D}} $,并通过贪心解码生成基线序列$ \hat{Y} $。以CIDEr-D作为奖励函数,强化学习损失定义为
$ {\mathcal{L}}_{\text{RL}}= -\left(r ({Y}^{\text{D}} )-r (\hat{Y} )\right)\displaystyle\sum_{t=1}^{T}\ln p\left(y_{t}^{\text{D}}\mid y_{1}^{\text{D}},… ,y_{t-1}^{\text{D}},{\boldsymbol{F}}_{\text{in}}\right)\;,$
其中,$ y_{t}^{\text{D}} $表示采样描述序列中的第$ t $个词,$ T $表示生成序列的长度,$ r(\cdot ) $表示基于CIDEr-D计算得到的奖励值,$ \hat{Y} $作为基线序列用于降低策略梯度估计的方差。当采样序列的奖励高于基线序列时,模型将增加该序列的生成概率;反之,则降低其概率。与仅使用交叉熵损失相比,该策略能够在一定程度上缓解暴露偏差问题,并使模型优化目标与最终评价指标更加一致。
在第2阶段中,模型以强化学习损失为主进行微调,因此总体目标可表示为
$ {\mathcal{L}}_{\text{stage2}}={\mathcal{L}}_{\text{RL}}\;。 $
4)总体训练流程
综上,本文采用2阶段训练策略:第1阶段通过交叉熵损失与对比学习损失的联合优化,学习稳定的视觉语义表示和基础语言生成能力;第2阶段在此基础上利用基于CIDEr-D奖励的SCST方法进行序列级微调,以进一步提升生成描述的整体语义相关性与评价指标表现。该训练方式使编码端的共性增强、解码端的词级监督以及最终的序列级优化形成互补,从而共同提升图像描述生成性能。
本文使用COCO 2014数据集进行实验,其中包含123287张图片,每张图片对应5句英文描述。为便于将实验结果与其他模型性能进行对比,遵循主流的数据集划分模式,将训练集图片设置为113287张,验证集和测试集图片均设置为5000张。对于文本描述的处理,采用通用处理方式,将所有图片参考描述中出现次数少于5的单词筛除,使用经过处理后的10201个词汇作为候选词库。此外,对于数据集中的每一句文本描述,删除其中所有标点符号,并将其中的大写格式转换为小写格式,在每句描述的开头插入<bos>表示句子的开始,在每句描述的结尾插入<eos>表示句子的结束,并且使用<pad>对长度不足20个单词的图片描述句子进行填充。
在训练实现上,参考M2Transformer[12]的常用训练范式,并结合本文模型进行调整。第1阶段以词级交叉熵损失为主,对模型进行监督训练,同时结合对比学习损失对共享语义表征进行约束,以提升模型的基础生成能力和共性特征表示质量;第2阶段在第1阶段参数初始化的基础上,采用基于SCST的强化学习方法进行微调,并以CIDEr-D作为奖励信号,直接优化图像描述任务中的序列级评价指标。模型训练过程中采用Adam优化器,并使用分段学习率策略进行参数更新。所有实验均在NVIDIA RTX3090上完成,训练轮数设为60,批次大小设为50。
为便于复现与对比,本文配置汇总如表1所示,所有实验均重复5次并取平均值,以确保结果的可重复性与公平性。
为验证本文方法的有效性,采用以下评价指标进行性能比较。
1)BLEU-n[5]。基于生成描述与参考描述在1~4元词组上的重叠情况进行评价,其中n越大越关注生成句在长序列层面的连贯性与流畅性。具体而言,BLEU-1主要反映词汇层面的匹配准确性,BLEU-4则更强调短语组合和整体表达的自然性。
2)ROUGE[40]。基于生成描述与参考描述之间的最长公共子序列进行评价,从召回率角度衡量生成句对参考句关键信息的覆盖程度。该指标更关注整体内容的完整性与句子结构相似性,ROUGE-L值越高,说明生成描述越能保留人工参考句中的主要语义信息。
3)METEOR[7]。引入同义词、词干还原和词序惩罚,用加权F-measure调和精度与召回,并惩罚片段乱序,缓解BLEU对同义表达过于苛刻的问题,更贴近人工判断。
4)CIDEr-D[41]。先用TF-IDF给n-gram赋权,突出稀有却关键的视觉词,再计算余弦相似并叠加长度惩罚,抑制重复刷分,使高分描述必须与多数人编写的参考句达成共识。
5)SPICE[42]。句子解析成(对象,属性,关系)三元组场景图,按F1计算语义元组匹配率,专评内容准确性:即使词面不同,只要语义关系对得上,就能获得高分。
本文方法在COCO 2014数据集上进行实验,并与公开文献中报告的代表性方法结果进行对比。
在相同测试集上,将本文方法与Up-Down,AoANet,X-Transformer等代表性方法和训练策略SCST[39]进行对比。模型的网格特征和分割特征提取的骨干网络均为ResNet-101,分割特征大小为7×7,并且在共享编码器部分添加了PF模块,所有编码器均采用提出的FSC模块。实验结果如表2所示。
表2结果显示,EAFCC在5类指标上均优于单特征输入的M2Transformer[12],CIDEr-D相对提升3.05%,其余指标平均提升2.65%。相较于双特征融合的代表模型DLCT,BLEU-4提升2.01%,CIDEr-D提升1.05%,验证了引入分割特征并在共享语义基础上建模分支依赖补充信息的有效性。与2022年CIIC及2023年ConCap相比,CIDEr-D分别提高1.58%与1.12%。上述增益可归因于3个特征输入以及共性-特性2阶段融合策略:PF模块先对齐语义空间并保留特征个性,MMF模块再动态加权特性向量,使解码器同时获得一致的高层语义与细粒度线索。然而,EAFCC在METEOR与SPICE上仍低于ConCap,差异分别为1.1个百分点与0.6个百分点。分析表明ConCap通过文本提示显式注入语义重心,而这2个指标对同义词与语义命题更敏感,因此提示机制会带来额外增益。
表3给出了各模型在第1阶段交叉熵训练下的对比结果,主要用于考察模型在不引入序列级强化学习优化时的基础生成能力,而表2中的最终结果则对应在第1阶段训练基础上进一步经过强化学习微调后的性能表现。通过分别报告2阶段结果,可以更清晰地分析模型结构设计本身的有效性,以及强化学习微调对最终描述生成质量的增益作用。可以看出,在不引入强化学习微调的情况下,EAFCC已表现出较强的基础生成能力,在多数评价指标上优于对比模型,并在SPICE指标上与X-Transformer持平。这说明,本文所提出的多特征共性-特性协同建模框架在交叉熵训练阶段即可学习到较为有效的视觉语义表示与语言生成能力。
在此基础上,进一步引入基于CIDEr-D奖励的SCST强化学习微调后,模型最终性能继续提升,说明2阶段训练策略具有必要性。其中,第1阶段交叉熵训练主要用于建立稳定的词级预测能力;第2阶段则通过直接优化序列级评价指标,进一步提升生成描述的整体语义相关性与表达质量。由此可见,本文模型的性能提升既来源于网络结构设计本身,也得益于2阶段训练策略所带来的进一步优化。
随着视觉-语言预训练模型规模的扩大,模型通过亿级图文对预训练获得强泛化能力,但在COCO描述任务上通常仅采用交叉熵微调,且推理开销大。相比于这些预训练多模态多任务模型,本文研究的优势在于能够在计算资源受限的情况下针对特定领域进行细粒度的图像描述任务。表4给出了与近年预训练多模态多任务模型在COCO数据集上的图像描述生成任务的性能对比结果。
如表4所示,BLIP系列、OFA系列、SimVLM等模型依赖14×106~1.8×109级图文预训练,在COCO上仅做轻量交叉熵微调即可取得133以上的CIDEr-D,表现出强大的跨任务泛化能力。然而,其大规模预训练与推理资源需求在边缘或域特定场景中难以承受。
表4的结果表明,尽管BLIP、OFA和SimVLM等方法依赖大规模视觉语言预训练,并在模型参数量和预训练数据规模上具有明显优势,EAFCC在不借助额外预训练图像的条件下,仍在MS-COCO测试集上取得了135.2的CIDEr-D和40.6的BLEU-4,优于表中多数预训练模型。进一步从模型规模来看,EAFCC的参数量为31.9×106,明显低于BLIPbase的223×106和OFASys的270×106。该结果表明,在资源受限条件下,针对图像描述任务进行专门的多特征建模与融合设计,同样能够取得具有竞争力的结果。
为系统验证本文所提出各组成模块的有效性,以不包含预融合共享编码器(PF)、跳跃连接编码器(FSC)、对比学习共性增强模块(CL)以及基于多专家的多特征融合模块(MMF)的基础模型构建EAFCC-base,并在此基础上逐步引入不同模块开展消融实验。同时,为进一步说明多特征共性-特性协同建模框架的整体优势,还将EAFCC-base与单特征模型M2Transformer、双特征模型DLCT[14]以及DIFNet[16]进行对比,实验结果如表5所示。
从整体结果来看,即使不引入CL和MMF,EAFCC-base仍表现出较强的竞争力,整体性能优于单特征建模的M2Transformer和双特征融合的DLCT,并在多数指标上超过DIFNet。这说明,相较于传统单一视觉表征或简单双特征融合方式,本文构建的3特征共性-特性协同建模框架能够更充分地利用分割特征、区域特征和网格特征之间的互补信息,从而为图像描述生成提供更有效的视觉语义表示。
在单模块消融方面,PF与FSC的作用呈现出不同特点。仅引入PF模块时,模型整体性能未出现明显改善,部分指标甚至略有波动。这表明,预融合操作虽然能够在一定程度上缩小多源特征之间的表示差异,但若缺乏后续特性保持与信息补偿机制,其优势尚难以充分转化为下游生成性能。相比之下,仅引入FSC模块后,模型在若干指标上表现出一定提升,说明跳跃连接机制有助于缓解深层编码过程中信息衰减的问题,并在一定程度上改善描述生成质量。但从整体上看,FSC单独使用所带来的增益仍较为有限,说明仅依赖层间信息传递尚不足以充分释放多特征建模的潜力。
进一步地,当PF与FSC联合引入后,模型性能较基线模型出现更稳定的提升,且整体表现优于两者单独作用时的结果。这说明,预融合所带来的跨特征对齐能力需要借助跳跃连接机制进行信息保留与补偿,才能更有效地服务于后续特征建模与描述生成。换言之,PF与FSC在模型中并非彼此独立地发挥作用,而是存在较明显的协同关系。
在此基础上,进一步分析高层语义增强与动态融合机制的贡献。实验结果表明,在PF+FSC结构上引入CL模块后,模型在核心生成指标上继续取得提升,说明对比学习约束能够增强多源特征在共享语义空间中的一致性,从而提升共性表征的判别能力。进一步引入MMF模块后,模型性能再次改善,表明多专家路由融合机制能够更灵活地建模不同特征分支对当前图像描述任务的贡献,并有效整合差异化特性信息。总体来看,CL更侧重于提升共享语义表示的紧致性与一致性,而MMF更侧重于增强特性信息的动态互补能力,两者在功能上具有较强的互补性。
最终,当PF,FSC,CL,MMF全部引入后,模型取得了表5中的最优结果。该结果说明,本文所设计的各模块均能对性能提升产生积极作用,且不同模块之间并非简单叠加关系,而是通过预融合对齐、特性保持、共性增强与动态融合协同发挥作用,共同提升了图像描述生成的语义准确性、细节完整性和整体表达质量。
此外,为进一步分析PF的迭代次数N1对模型性能的影响,本文在EAFCC-base上设置不同迭代次数开展消融实验,结果如表6所示。
通过表6可知,迭代次数为2时模型效果最好,当迭代次数再增加时,模型性能反而出现下降趋势。类似现象在文献[16]中也得到了验证,证明迭代次数过多会导致图像的低层语义特征占比增加,从而忽略相对重要的高层语义特征,导致模型性能下降。因此本文EAFCC模型最终采用2次迭代,达到了相对较好的性能。
在EAFCC-base模型上对FSC的跳连方式进行消融实验,由于编码器一共由6层Transformer编码器堆叠而成,所以分别采用跳跃间隔为“1”“3”“6”“1,6”“3,6”以及全连接进行实验,结果如表7所示。
表7的实验结果证实,本文对跳跃连接方式进行了系统性消融:在单独验证逐级跳连与首尾跳连的基础上,进一步考察了两者并联以及稠密全连接跳连的配置,结果一致表明逐级与首尾越级并联这一稀疏方案在BLEU-4,CIDEr-D,SPICE上均取得最优值,验证其可为梯度回传提供充足且低冗余的路径,从而强化模型对底层视觉特征的利用并生成更细粒度的描述。相比之下,稠密全连接跳跃连接在相同层数编码器下性能显著下降,提示当网络深度不足时,过度密集的连接易加剧参数耦合并诱发过拟合,导致泛化性能受损。
本文进一步分析负样本数参数K对模型性能的影响,结果如表8所示。
在其他训练设置保持不变的条件下,分别测试了$ K $=2, 3, 5, 16, 32, 50, 64的情况。实验结果表明,不同评价指标下的最优K值并不完全一致,这主要是由于BLEU,METEOR,ROUGE,CIDEr-D对生成描述质量的关注重点不同。总体来看,模型在合理范围内对K的变化较为稳定,但当K取值过小时,性能会出现一定下降;随着K增大,模型表现逐渐提升并趋于平稳。其中,K=50在核心指标CIDEr-D上取得最佳结果,并在其他指标上保持了较好的综合表现,因此本文最终将K固定为50。
为验证模型特征融合的性能,从测试集中选取了不同场景的图片进行图像描述生成,并保留生成每个词时的注意力权重,以注意力图的方式呈现,如图5所示。为验证多特征输入融合模型与双特征输入模型(DLCT)以及采用大规模数据训练的OFATiny模型在解码过程中的差异,采用这3个模型对相同图像进行描述并生成对应的注意力热图,如图6所示。
从图5和图6中可以看到,模型在生成背景信息的相关描述时,注意力关注更加准确,并且对相关动作的注意力体现得明显细致,如“Laying”一词对应的注意力,且生成的描述也更加具体准确。表9报告了EAFCC与DLCT,DIFNet及多模态预训练基线OFATiny在测试集上的自动生成结果对比。其中,样本1~2具备高视觉相似度,用以评估对比学习模块的区分能力,样本3~4包含复杂背景与遮挡,用以检验多特征融合机制对细节与语境的保持能力。相比于使用大规模数据训练的OFATiny模型,可视化结果显示,EAFCC在生成每个词语时均能准确聚焦对应的图像语义区域,并在多种场景下保持较高的语义对齐精度。例如对于包含“two elephants walking under a large tree in a forest”的图像,模型在生成“walking”时关注象腿区域,在生成“under”时聚焦于树下空间,在生成“forest”时则激活背景区域,体现出对空间关系的理解。
EAFCC在生成的描述上更加具体,且由于采用多特征融合方式,对图像背景和细节信息的描述更加完整,体现了本文研究方法的有效性。这说明EAFCC在生成描述过程中,能够实现词语与图像区域之间的准确对齐,保证描述内容在主体、动作、背景及空间关系等方面的一致性与准确性。
为验证共性分支与特性分支之间的表示差异,本文基于已训练模型在验证集上提取各分支输出特征,并计算其平均余弦相似度,结果如图7所示。可以看到,共性分支内部表示的相似度整体较高,其中$ {\boldsymbol{C}}_{\text{G}},{\boldsymbol{C}}_{\text{R}},{\boldsymbol{C}}_{\text{S}} $之间的平均余弦相似度约为0.57;相比之下,特性分支$ {\boldsymbol{P}}_{\text{G}},{\boldsymbol{P}}_{\text{R}},{\boldsymbol{P}}_{\text{S}} $之间的平均余弦相似度约为0.24,明显低于共性分支。该结果说明,共享编码与共性增强机制能够在一定程度上拉近不同特征流在共享语义空间中的表示,而分支专属编码器则进一步保留了与特征类型相关的差异化信息。
进一步地,分割特性分支$ {\boldsymbol{P}}_{\text{S}} $与其他分支之间的相似度整体较低,表明该分支更倾向于保留与语义区域和边界结构相关的特有信息。与此同时,部分共性分支与特性分支之间仍具有较高相关性,这说明EAFCC中的特性并非是与共性完全割裂的独立表示,而是在共享语义基础上进一步形成的分支依赖补充表示。
针对图像描述任务中视觉表征利用不足、跨特征对齐与互补融合不充分的问题,本文在Transformer编码器-解码器框架下提出共性表征与特性表征协同融合机制。首先对网格、区域和分割3类特征进行共享语义建模;继而以独立编码器在共性基底上强化模态专有细节,形成特性表征。其次,在共性分支中引入对比学习约束,拉近语义匹配对、推远异类对;特性分支通过门控路由实现动态加权,抑制冗余、突出差异。最后,增强后的共性与特性向量拼接输入解码器,使交叉注意力于每个时间步精准对齐对应视觉成分,生成细粒度且高保真的描述。消融实验与对比实验验证所提方法的有效性。

参考文献 引证文献
排序方式:
1
Gurari D, Li Qing, Stangl A J, et al. VizWiz grand challenge: Answering visual questions from blind people[C]//Proc of the 31st IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 3608−3617
2
Aneja J, Deshpande A, Schwing A G. Convolutional image captioning[C]//Proc of the 31st IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 5561−5570
3
Yu Jun, Li Jing, Yu Zhou, et al. Multimodal Transformer with multi-view visual representation for image captioning[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 30(12): 4467−4480
4
Lou Liangshan, Lu Ke, Xue Jian. A novel cross-fusion method of different types of features for image captioning[C]//Proc of the 33rd Int Joint Conf on Neural Networks. Piscataway, NJ: IEEE, 2023: 1−8
5
Mathur N, Baldwin T, Cohn T. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics[C]//Proc of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 4984−4997
6
Cui Yin, Yang Guandao, Veit A, et al. Learning to evaluate image captioning[C]//Proc of the 31st IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 5804−5812
7
Bernardi R, Cakici R, Elliott D, et al. Automatic description generation from images: A survey of models, datasets, and evaluation measures[J]. Journal of Artificial Intelligence Research, 2016, 55: 409−442
8
Al-Shamayleh A S, Adwan O, Alsharaiah M A, et al. A comprehensive literature review on image captioning methods and metrics based on deep learning technique[J]. Multimedia Tools and Applications, 2024, 83(12): 34219−34268
9
Ghandi T, Pourreza H, Mahyar H. Deep learning approaches on image captioning: A review[J]. ACM Computing Surveys, 2023, 56(3): 1−39
10
Huang Lun, Wang Wenmin, Chen Jie, et al. Attention on attention for image captioning[C]//Proc of the 17th IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 4634−4643
11
Li Guang, Zhu Linchao, Liu Ping, et al. Entangled Transformer for image captioning[C]//Proc of the 17th IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 8928−8937
12
Cornia M, Stefanini M, Baraldi L, et al. Meshed-memory Transformer for image captioning[C]//Proc of the 33rd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 10578−10587
13
Pan Yingwei, Yao Ting, Li Yehao, et al. X-linear attention networks for image captioning[C]//Proc of the 33rd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 10971−10980
14
Luo Yunpeng, Ji Jiayi, Sun Xiaoshuai, et al. Dual-level collaborative Transformer for image captioning[C]//Proc of the 35th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2021: 2286−2293
15
Nguyen V Q, Suganuma M, Okatani T. GRIT: Faster and better image captioning transformer using dual visual features[G]//LNCS 13696: Proc of the 17th European Conf on Computer Vision. Berlin: Springer, 2022: 167−184
16
Wu Mingrui, Zhang Xuying, Sun Xiaoshuai, et al. DIFNet: Boosting visual information flow for image captioning[C]//Proc of the 35th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 18020−18029
17
Li Junnan, Li Dongxu, Xiong Caiming, et al. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation[C]//Proc of the 39th Int Conf on Machine Learning. New York: PMLR, 2022: 12888−12900
18
Wang Peng, Yang An, Men Rui, et al. OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework[C]//Proc of the 39th Int Conf on Machine Learning. New York: PMLR, 2022: 23318−23340
19
Bai Jinze, Men Rui, Yang Hao, et al. OFASys: A multi-modal multi-task learning system for building generalist models[J]. arXiv preprint, arXiv: 2212.04408, 2022
20
Wang Zirui, Yu Jiahui, Yu Adams Wei, et al. SimVLM: Simple visual language model pretraining with weak supervision[J]. arXiv preprint, arXiv: 2108.10904, 2021
21
Li Wei, Gao Can, Niu Guocheng, et al. UNIMO: Towards unified-modal understanding and generation via cross-modal contrastive learning[J]. arXiv preprint, arXiv: 2012.15409, 2020
22
Zhou Luowei, Palangi H, Zhang Lei, et al. Unified vision-language pre-training for image captioning and VQA[C]//Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 13041−13049
23
Li Jiaxuan, Vo D M, Sugimoto A, et al. EVCap: Retrieval-augmented image captioning with external visual-name memory for open-world comprehension[C]//Proc of the 37th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2024: 13733−13742
24
Hua Hang, Liu Qing, Zhang Lingzhi, et al. FineCaption: Compositional image captioning focusing on wherever you want at any granularity[C]//Proc of the 38th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2025: 24763−24773
25
Anderson P, He Xiaodong, Buehler C, et al. Bottom-up and top-down attention for image captioning and visual question answering[C]//Proc of the 31st IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 6077−6086
26
Ren Shaoqing, He Kaiming, Girshick R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137−1149
27
Xiong Yuwen, Liao Renjie, Zhao Hengshuang, et al. UPSNet: A unified panoptic segmentation network[C]//Proc of the 32nd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 8818−8826
28
Yang Xu, Tang Kaihua, Zhang Hanwang, et al. Auto-encoding scene graphs for image captioning[C]//Proc of the 32nd IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 10685−10694
29
Yao Ting, Pan Yingwei, Li Yehao, et al. Exploring visual relationship for image captioning[G]//LNCS 11218: Proc of the 15th European Conf on Computer Vision. Berlin: Springer, 2018: 711−727
30
Zhang Jing, Xie Yingshuai, Ding Weichao, et al. Cross on cross attention: Deep fusion transformer for image captioning[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(8): 4257−4268
31
Jiang Wenhao, Ma Lin, Jiang Yugang, et al. Recurrent fusion network for image captioning[G]//LNCS 11206: Proc of the 15th European Conf on Computer Vision. Berlin: Springer, 2018: 499−515
32
Wang Weixuan, Chen Zhihong, Hu Haifeng. Hierarchical attention network for image captioning[C]//Proc of the 33rd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 8957−8964
33
Zhao Shanshan, Li Lixiang, Peng Haipeng. Incorporating retrieval-based method for feature enhanced image captioning[J]. Applied Intelligence, 2023, 53(8): 9731−9743
34
Ramos R, Martins B, Elliott D, et al. SmallCap: Lightweight image captioning prompted with retrieval augmentation[C]//Proc of the 36th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 2840−2849
35
Song Jiayan, Pan Renjie, Zhou Jun, et al. M-RAT: A multi-grained retrieval augmentation Transformer for image captioning[C]//Proc of the 17th Asian Conf on Computer Vision. Berlin: Springer, 2024: 185−203
36
Wang Ning, Xie Jiahao, Wu Jihao, et al. Controllable image captioning via prompting[C]//Proc of the 37th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023: 2617−2625
37
Zhang Kun, Mao Zhendong, Wang Quan, et al. Negative-aware attention framework for image-text matching[C]//Proc of the 35th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 15661−15670
38
Chen Junyi, Guo Longteng, Sun Jia, et al. EVE: Efficient vision-language pre-training with masked prediction and modality-aware MoE[C]//Proc of the 38th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 1110−1119
39
Rennie S J, Marcheret E, Mroueh Y, et al. Self-critical sequence training for image captioning[C]//Proc of the 30th IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2017: 7008−7024
40
Lin C Y. ROUGE: A package for automatic evaluation of summaries[C]//Proc of the Text Summarization Branches Out. Stroudsburg, PA: ACL, 2004: 74−81
41
Vedantam R, Lawrence Z C, Parikh D. CIDEr: Consensus-based image description evaluation[C]//Proc of the 28th IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2015: 4566−4575
42
Anderson P, Fernando B, Johnson M, et al. SPICE: Semantic propositional image caption evaluation[G]//LNCS 9909: Proc of the 14th European Conf on Computer Vision. Berlin: Springer, 2016: 382−398
43
Liu Bing, Wang Dong, Yang Xu, et al. Show, deconfound and tell: Image captioning with causal inference[C]//Proc of the 35th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 18041−18050
44
Zhang Xuying, Sun Xiaoshuai, Luo Yunpeng, et al. RSTNet: Captioning with adaptive attention on visual and non-visual words[C]//Proc of the 34th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2021: 15465−15474
2026年第63卷第11期
106
51
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202550936
  • 接收时间:2025-12-24
  • 首发时间:2026-11-17
  • 出版时间:2026-11-01
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2025-12-24
  • 修回日期:2026-07-23
基金
作者信息
    1西北大学计算机学院 西安 710127
    2北京三快科技有限公司 北京 100190

通讯作者:

曹瑞()
参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202550936
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏