首页 期刊 论文 学科刊群 资讯 加盟 关于
EN
image
Article(id=1241697947633643796, tenantId=1146029695717560320, journalId=1238841944844054536, issueId=1241697942122328272, articleNumber=null, orderNo=null, doi=10.12347/j.ycyk.20240326002, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=null, receivedDate=1711382400000, receivedDateStr=2024-03-26, revisedDate=1720713600000, revisedDateStr=2024-07-12, acceptedDate=null, acceptedDateStr=null, onlineDate=1773973460347, onlineDateStr=2026-03-20, pubDate=1726329600000, pubDateStr=2024-09-15, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1773973460347, onlineIssueDateStr=2026-03-20, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1773973460347, creator=13701087609, updateTime=1773973460347, updator=13701087609, issue=Issue{id=1241697942122328272, tenantId=1146029695717560320, journalId=1238841944844054536, year='2024', volume='45', issue='5', pageStart='1', pageEnd='128', issueExtLink='null', onlineDate='null', pubDate='null', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1773973459034, creator=13701087609, updateTime=1773973945698, updator=13701087609, preIssue=null, nextIssue=null, ext={EN=IssueExt(id=1241699983414580120, tenantId=1146029695717560320, journalId=1238841944844054536, issueId=1241697942122328272, language=EN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=), CN=IssueExt(id=1241699983414580121, tenantId=1146029695717560320, journalId=1238841944844054536, issueId=1241697942122328272, language=CN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=)}, issueFiles=null}, startPage=109, endPage=119, ext={EN=ArticleExt(id=1241697947902079265, articleId=1241697947633643796, tenantId=1146029695717560320, journalId=1238841944844054536, language=EN, title=Infrared and Visible Image Fusion Based on Autoencoder Composed of CNN-transformer, columnId=1239133500033528732, journalTitle=Journal of Telemetry, Tracking and Command, columnName=Radar and Countermeasures, runingTitle=null, highlight=null, articleAbstract=

Image fusion model based on autoencoder network gets more attention because it does not need to design fusion rules manually. However, most autoencoder-based fusion networks use two-stream CNNs with the same structure as the encoder,which are unable to extract global features due to the local receptive field of convolutional operations and lack the ability to extract unique features from infrared and visible images. A novel autoencoder-based image fusion network which consist of encoder module, fusion module and decoder module is constructed in this paper. In the encoder module, the CNN and Transformer are combined to capture the local and global feature of the source images simultaneously. In addition, novel contrast and gradient enhancement feature extraction blocks are designed respectively for infrared and visible images to maintain the information specific to each source images. The feature images obtained by encoder module are concatenated by the fusion module and input to the decoder module to obtain the fused image. Experimental results on three datasets show that the proposed network can better preserve both the clear target and detailed information of infrared and visible images respectively, and outperforms some state-of-the-art methods in both subjective and objective evaluation. Meanwhile, the fused image obtained by the proposed network can acquire the highest mean average precision in the target detection which proves that image fusion is beneficial for downstream tasks.

, correspAuthors=null, authorNote=null, correspAuthorsNote=null, copyrightStatement=null, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, authorCompany=null, fund=null, authors=null, authorsList=Lin LI, Yongjian SHEN, Pengyu ZHANG, Hao YUAN, Chao WANG), CN=ArticleExt(id=1241697953514058224, articleId=1241697947633643796, tenantId=1146029695717560320, journalId=1238841944844054536, language=CN, title=基于CNN-Transformer的自编码器红外和可见光图像融合方法, columnId=1239133500683645881, journalTitle=遥测遥控, columnName=雷达与对抗, runingTitle=null, highlight=null, articleAbstract=

基于自编码器的图像融合模型因无需手动设计融合规则而受到更多关注。然而，该融合网络编码器采用的卷积神经网络仅对局部感受野敏感，无法提取图像全局特征，且缺乏从红外图像和可见光图像中提取独特特征的能力。本文构建了一种基于自动编码器的新型图像融合网络，该网络由编码器模块、融合模块和解码器模块组成。在编码器模块中，结合使用CNN和Transformer模块以同时捕捉原图像的局部和全局特征。此外，为提取原图像特定信息，分别为原红外和可见光图像设计对比度增强和梯度增强模块。编码器模块获得的特征图像经融合模块串联后输入解码器模块，从而获得融合图像。在三个数据集上的实验结果表明，本文提出的融合网络能较好地保留了红外图像和可见光图像的清晰目标和细节信息，在主观和客观评价方面均优于其他先进方法。同时，本文提出的网络所获得的融合图像 在目标检测中获得了最高的平均精度，证明图像融合有利于下游任务。

, correspAuthors=null, authorNote=null, correspAuthorsNote=null, copyrightStatement=null, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=Bs1Qd+zUwodGhrHUbQ2UWg==, magXml=ca6JUivdcLTmr+aq17YEnQ==, pdfUrl=null, pdf=YlHQ8+0SjrtV27gLzlfGjg==, pdfFileSize=10871962, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=xGnVXMwmIXijfnJpAjz4og==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=704sxac4lt0Vil0R19oQIg==, mapNumber=null, authorCompany=null, fund=null, authors=

李霖 1995年生，硕士，助理工程师。

沈永健 1985年生，博士，研究员。

张鹏宇 1996年生，硕士，工程师。

原昊 1998年生，硕士，工程师。

王超 1994年生，硕士，助理工程师。

, authorsList=李霖, 沈永健, 张鹏宇, 原昊, 王超)}, authors=[Author(id=1241712918128022267, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1241712918203519745, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, authorId=1241712918128022267, language=EN, stringName=Lin LI, firstName=Lin, middleName=null, lastName=LI, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=Beijing Research Institute of Telemetry, Beijing 100076, China, bio=null, bioImg=null, bioContent=null, aboutCorrespAuthor=null), CN=AuthorExt(id=1241712918299988741, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, authorId=1241712918128022267, language=CN, stringName=李霖, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=北京遥测技术研究所　北京　100076, bio={"content":"

李霖 1995年生，硕士，助理工程师。

"}, bioImg=null, bioContent=

李霖 1995年生，硕士，助理工程师。

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1241712918031553267, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, xref=null, ext=[AuthorCompanyExt(id=1241712918039941876, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, companyId=1241712918031553267, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Beijing Research Institute of Telemetry, Beijing 100076, China), AuthorCompanyExt(id=1241712918048330486, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, companyId=1241712918031553267, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=北京遥测技术研究所　北京　100076)])]), Author(id=1241712918400652040, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1241712918497121038, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, authorId=1241712918400652040, language=EN, stringName=Yongjian SHEN, firstName=Yongjian, middleName=null, lastName=SHEN, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=Beijing Research Institute of Telemetry, Beijing 100076, China, bio=null, bioImg=null, bioContent=null, aboutCorrespAuthor=null), CN=AuthorExt(id=1241712918572618514, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, authorId=1241712918400652040, language=CN, stringName=沈永健, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=北京遥测技术研究所　北京　100076, bio={"content":"

沈永健 1985年生，博士，研究员。

"}, bioImg=null, bioContent=

沈永健 1985年生，博士，研究员。

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1241712918031553267, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, xref=null, ext=[AuthorCompanyExt(id=1241712918039941876, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, companyId=1241712918031553267, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Beijing Research Institute of Telemetry, Beijing 100076, China), AuthorCompanyExt(id=1241712918048330486, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, companyId=1241712918031553267, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=北京遥测技术研究所　北京　100076)])]), Author(id=1241712918652310297, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1241712918769750816, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, authorId=1241712918652310297, language=EN, stringName=Pengyu ZHANG, firstName=Pengyu, middleName=null, lastName=ZHANG, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=Beijing Research Institute of Telemetry, Beijing 100076, China, bio=null, bioImg=null, bioContent=null, aboutCorrespAuthor=null), CN=AuthorExt(id=1241712918857831203, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, authorId=1241712918652310297, language=CN, stringName=张鹏宇, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=北京遥测技术研究所　北京　100076, bio={"content":"

张鹏宇 1996年生，硕士，工程师。

"}, bioImg=null, bioContent=

张鹏宇 1996年生，硕士，工程师。

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1241712918031553267, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, xref=null, ext=[AuthorCompanyExt(id=1241712918039941876, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, companyId=1241712918031553267, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Beijing Research Institute of Telemetry, Beijing 100076, China), AuthorCompanyExt(id=1241712918048330486, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, companyId=1241712918031553267, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=北京遥测技术研究所　北京　100076)])]), Author(id=1241712918954300200, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1241712919038186286, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, authorId=1241712918954300200, language=EN, stringName=Hao YUAN, firstName=Hao, middleName=null, lastName=YUAN, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=Beijing Research Institute of Telemetry, Beijing 100076, China, bio=null, bioImg=null, bioContent=null, aboutCorrespAuthor=null), CN=AuthorExt(id=1241712919143043889, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, authorId=1241712918954300200, language=CN, stringName=原昊, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=北京遥测技术研究所　北京　100076, bio={"content":"

原昊 1998年生，硕士，工程师。

"}, bioImg=null, bioContent=

原昊 1998年生，硕士，工程师。

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1241712918031553267, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, xref=null, ext=[AuthorCompanyExt(id=1241712918039941876, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, companyId=1241712918031553267, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Beijing Research Institute of Telemetry, Beijing 100076, China), AuthorCompanyExt(id=1241712918048330486, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, companyId=1241712918031553267, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=北京遥测技术研究所　北京　100076)])]), Author(id=1241712919231124282, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1241712919331787580, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, authorId=1241712919231124282, language=EN, stringName=Chao WANG, firstName=Chao, middleName=null, lastName=WANG, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=Beijing Research Institute of Telemetry, Beijing 100076, China, bio=null, bioImg=null, bioContent=null, aboutCorrespAuthor=null), CN=AuthorExt(id=1241712919436645183, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, authorId=1241712919231124282, language=CN, stringName=王超, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=北京遥测技术研究所　北京　100076, bio={"content":"

王超 1994年生，硕士，助理工程师。

"}, bioImg=null, bioContent=

王超 1994年生，硕士，助理工程师。

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1241712918031553267, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, xref=null, ext=[AuthorCompanyExt(id=1241712918039941876, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, companyId=1241712918031553267, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Beijing Research Institute of Telemetry, Beijing 100076, China), AuthorCompanyExt(id=1241712918048330486, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, companyId=1241712918031553267, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=北京遥测技术研究所　北京　100076)])])], keywords=[Keyword(id=1241712919621194565, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, orderNo=1, keyword=Image fusion), Keyword(id=1241712919705080649, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, orderNo=2, keyword=Convolutional neural network), Keyword(id=1241712919788966734, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, orderNo=3, keyword=Transformer), Keyword(id=1241712919893824342, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, orderNo=4, keyword=Infrared image), Keyword(id=1241712919981904730, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, orderNo=5, keyword=Visible image), Keyword(id=1241712920074179421, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, orderNo=1, keyword=图像融合), Keyword(id=1241712920212591455, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, orderNo=2, keyword=卷积神经网络), Keyword(id=1241712920288088932, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, orderNo=3, keyword=Transformer), Keyword(id=1241712921789649768, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, orderNo=4, keyword=红外图像), Keyword(id=1241712921898701676, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, orderNo=5, keyword=可见光图像)], refs=[Reference(id=1241712926793454547, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2021, volume=50, issue=9, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=[1], rfOrder=0, authorNames=沈英, 黄春红, 黄峰, journalName=红外与激光工程, refType=null, unstructuredReference=沈英, 黄春红, 黄峰, 等. 红外与可见光图像融合技术的研究进展[J]. 红外与激光工程, 2021, 50(9): 20200467., articleTitle=红外与可见光图像融合技术的研究进展, refAbstract=null), Reference(id=1241712926877340631, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2021, volume=50, issue=9, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=[1], rfOrder=1, authorNames=SHEN Ying, HUANG Chunhong, HUANG Feng, journalName=Infrared and Laser Engineering, refType=null, unstructuredReference=SHEN Ying, HUANG Chunhong, HUANG Feng, et al. Research progress of infrared and visible image fusion technology [J]. Infrared and Laser Engineering, 2021, 50(9): 20200467., articleTitle=Research progress of infrared and visible image fusion technology, refAbstract=null), Reference(id=1241712926957032411, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2022, volume=51, issue=12, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=[2], rfOrder=2, authorNames=李霖, 王红梅, 李辰凯, journalName=红外与激光工程, refType=null, unstructuredReference=李霖, 王红梅, 李辰凯. 红外与可见光图像深度学习融合方法综述[J]. 红外与激光工程, 2022, 51(12):20220125., articleTitle=红外与可见光图像深度学习融合方法综述, refAbstract=null), Reference(id=1241712927045112798, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2022, volume=51, issue=12, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=[2], rfOrder=3, authorNames=LI Lin, WANG Hongmei, LI Chenkai, journalName=Infrared and Laser Engineering, refType=null, unstructuredReference=LI Lin, WANG Hongmei, LI Chenkai. A review of deep learning fusion methods for infrared and visible images[J]. Infrared and Laser Engineering, 2022, 51(12):20220125., articleTitle=A review of deep learning fusion methods for infrared and visible images, refAbstract=null), Reference(id=1241712927158359009, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2017, volume=null, issue=null, pageStart=5108, pageEnd=5115, url=null, language=null, rfNumber=[3], rfOrder=4, authorNames=HA Q, WATANABE K, KARASAWA T, journalName=null, refType=null, unstructuredReference=HA Q, WATANABE K, KARASAWA T, et al. MFNet:Towards realtime semantic segmentation for autonomous vehicles with multi-spectral scenes[C]//2017 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS). IEEE, 2017: 5108-5115., articleTitle=MFNet:Towards realtime semantic segmentation for autonomous vehicles with multi-spectral scenes, refAbstract=null), Reference(id=1241712927246439396, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2014, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=[4], rfOrder=5, authorNames=TOET A, journalName=null, refType=null, unstructuredReference=TOET A. TNO Image fusion dataset[DS/OL]. [2024-03-28].https://figshare.com/articles/dataset/TNO_Image-_Fusion_Dataset/1008029, 2014., articleTitle=TNO Image fusion dataset, refAbstract=null), Reference(id=1241712927334519782, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2007, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=[5], rfOrder=6, authorNames=DAVIS J W, SHARMA V, journalName=null, refType=null, unstructuredReference=DAVIS J W, SHARMA V. OTCBVS benchmark dataset collection[DS/OL]. [2024-03-28]. http://www.cse.ohio-state.edu/otcbvs-bench, 2007., articleTitle=OTCBVS benchmark dataset collection, refAbstract=null), Reference(id=1241712927430988776, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2020, volume=null, issue=1, pageStart=502, pageEnd=518, url=null, language=null, rfNumber=[6], rfOrder=7, authorNames=XU H, MA J, JIANG J, journalName=IEEE Transactions on Pattern Analysis and Machine Intelligence, refType=null, unstructuredReference=XU H, MA J, JIANG J, et al. U2Fusion: A unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020(1):502-518., articleTitle=U2Fusion: A unified unsupervised image fusion network, refAbstract=null), Reference(id=1241712927527457772, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2015, volume=24, issue=null, pageStart=147, pageEnd=164, url=null, language=null, rfNumber=[7], rfOrder=8, authorNames=LIU Y, LIU S, WANG Z, journalName=Information Fusion, refType=null, unstructuredReference=LIU Y, LIU S, WANG Z. A general framework for image fusion based on multi-scale transform and sparse representation[J]. Information Fusion, 2015, 24: 147-164., articleTitle=A general framework for image fusion based on multi-scale transform and sparse representation, refAbstract=null), Reference(id=1241712927594566637, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2016, volume=31, issue=null, pageStart=100, pageEnd=109, url=null, language=null, rfNumber=[8], rfOrder=9, authorNames=MA J, CHEN C, LI C, journalName=Information Fusion, refType=null, unstructuredReference=MA J, CHEN C, LI C. Infrared and visible image fusion via gradient transfer and total variation minimization[J]. Information Fusion, 2016, 31: 100-109., articleTitle=Infrared and visible image fusion via gradient transfer and total variation minimization, refAbstract=null), Reference(id=1241712927678452719, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2020, volume=29, issue=null, pageStart=4980, pageEnd=4995, url=null, language=null, rfNumber=[9], rfOrder=10, authorNames=MA J J, XU H, JIANG J J, journalName=IEEE Transactions on Image Processing, refType=null, unstructuredReference=MA J J, XU H, JIANG J J, et al. DDcGAN: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995., articleTitle=DDcGAN: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion, refAbstract=null), Reference(id=1241712927779116018, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2021, volume=25, issue=null, pageStart=635, pageEnd=648, url=null, language=null, rfNumber=[10], rfOrder=11, authorNames=ZHOU H B, WU W, ZHANG Y D, journalName=IEEE Transactions on Multimedia, refType=null, unstructuredReference=ZHOU H B, WU W, ZHANG Y D, et al. Semantic-supervised infrared and visible image fusion via a dual-discriminator generative adversarial network[J]. IEEE Transactions on Multimedia, 2021, 25: 635-648., articleTitle=Semantic-supervised infrared and visible image fusion via a dual-discriminator generative adversarial network, refAbstract=null), Reference(id=1241712927900750834, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2018, volume=28, issue=5, pageStart=2614, pageEnd=2623, url=null, language=null, rfNumber=[11], rfOrder=12, authorNames=LI H, WU X J, journalName=IEEE Transactions on Image Processing, refType=null, unstructuredReference=LI H, WU X J. DenseFuse: A fusion approach to infrared and visible images[J]. IEEE Transactions on Image Processing, 2018, 28(5): 2614-2623., articleTitle=DenseFuse: A fusion approach to infrared and visible images, refAbstract=null), Reference(id=1241712928043357172, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2021, volume=73, issue=null, pageStart=72, pageEnd=86, url=null, language=null, rfNumber=[12], rfOrder=13, authorNames=LI H, WU X J, KITTLER J, journalName=Information Fusion, refType=null, unstructuredReference=LI H, WU X J, KITTLER J. RFN-Nest: An end-to-end residual fusion network for infrared and visible images[J]. Information Fusion, 2021, 73: 72-86., articleTitle=RFN-Nest: An end-to-end residual fusion network for infrared and visible images, refAbstract=null), Reference(id=1241712928135631863, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2021, volume=70, issue=null, pageStart=1, pageEnd=13, url=null, language=null, rfNumber=[13], rfOrder=14, authorNames=MA J, TANG L, XU M, journalName=IEEE Transactions on Instrumentation and Measurement, refType=null, unstructuredReference=MA J, TANG L, XU M, et al. STDFusionNet: An Infrared and visible image fusion network based on salient target detection[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-13., articleTitle=STDFusionNet: An Infrared and visible image fusion network based on salient target detection, refAbstract=null), Reference(id=1241712928215323641, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2022, volume=9, issue=7, pageStart=1200, pageEnd=1217, url=null, language=null, rfNumber=[14], rfOrder=15, authorNames=MA J Y, TANG L F, FAN F, journalName=IEEE/CAA Journal of Automatica Sinica, refType=null, unstructuredReference=MA J Y, TANG L F, FAN F, et al. SwinFusion: Cross-domain long-range learning for general image fusion via swin transformer[J]. IEEE/CAA Journal of Automatica Sinica, 2022, 9(7): 1200-1217., articleTitle=SwinFusion: Cross-domain long-range learning for general image fusion via swin transformer, refAbstract=null), Reference(id=1241712928328569851, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2002, volume=38, issue=7, pageStart=313, pageEnd=315, url=null, language=null, rfNumber=[15], rfOrder=16, authorNames=QU G, ZHANG D, YAN P, journalName=Electronics Letters, refType=null, unstructuredReference=QU G, ZHANG D, YAN P. Information measure for performance of image fusion[J]. Electronics Letters, 2002, 38(7): 313-315., articleTitle=Information measure for performance of image fusion, refAbstract=null), Reference(id=1241712928399873020, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=1995, volume=43, issue=12, pageStart=2959, pageEnd=2965, url=null, language=null, rfNumber=[16], rfOrder=17, authorNames=ESKICIOGLU A M, FISHER P S, journalName=IEEE Transactions on Communications, refType=null, unstructuredReference=ESKICIOGLU A M, FISHER P S. Image quality measures and their performance[J]. IEEE Transactions on Communications, 1995, 43(12): 2959-2965., articleTitle=Image quality measures and their performance, refAbstract=null), Reference(id=1241712928492147710, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2011, volume=11, issue=7, pageStart=6533, pageEnd=6554, url=null, language=null, rfNumber=[17], rfOrder=18, authorNames=GUO W, XIONG N, CHAO H C, journalName=Sensors, refType=null, unstructuredReference=GUO W, XIONG N, CHAO H C, et al. Design and analysis of self-adapted task scheduling strategies in wireless sensor networks[J]. Sensors, 2011, 11(7): 6533-6554., articleTitle=Design and analysis of self-adapted task scheduling strategies in wireless sensor networks, refAbstract=null), Reference(id=1241712928580228094, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2015, volume=341, issue=null, pageStart=199, pageEnd=209, url=null, language=null, rfNumber=[18], rfOrder=19, authorNames=CUI G, FENG H, XU Z, journalName=Optics Communications, refType=null, unstructuredReference=CUI G, FENG H, XU Z, et al. Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition[J]. Optics Communications, 2015, 341: 199-209., articleTitle=Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition, refAbstract=null), Reference(id=1241712928672502784, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2013, volume=14, issue=2, pageStart=127, pageEnd=135, url=null, language=null, rfNumber=[19], rfOrder=20, authorNames=HAN Y, CAI Y, CAO Y, journalName=Information Fusion, refType=null, unstructuredReference=HAN Y, CAI Y, CAO Y, et al. A new image fusion performance metric based on visual information fidelity[J]. Information Fusion, 2013, 14(2): 127-135., articleTitle=A new image fusion performance metric based on visual information fidelity, refAbstract=null), Reference(id=1241712928806719490, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2000, volume=36, issue=4, pageStart=308, pageEnd=309, url=null, language=null, rfNumber=[20], rfOrder=21, authorNames=XYDEAS C S, PETROVIC V, journalName=Electronics Letters, refType=null, unstructuredReference=XYDEAS C S, PETROVIC V. Objective image fusion performance measure[J]. Electronics Letters, 2000, 36(4): 308-309., articleTitle=Objective image fusion performance measure, refAbstract=null), Reference(id=1241712928936742915, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, doi=null, pmid=null, pmcid=null, year=2015, volume=null, issue=null, pageStart=91, pageEnd=99, url=null, language=null, rfNumber=[21], rfOrder=22, authorNames=REN S, HE K, GIRSHICK R, journalName=null, refType=null, unstructuredReference=REN S, HE K, GIRSHICK R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[C]//29th Annual Conference on Neural Information Processing Systems, NIPS 2015, Canad.2015:91-99., articleTitle=Faster R-CNN: Towards real-time object detection with region proposal networks, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1241712918031553267, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, xref=null, ext=[AuthorCompanyExt(id=1241712918039941876, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, companyId=1241712918031553267, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Beijing Research Institute of Telemetry, Beijing 100076, China), AuthorCompanyExt(id=1241712918048330486, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, companyId=1241712918031553267, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=北京遥测技术研究所　北京　100076)])], figs=[ArticleFig(id=1241712922146165621, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, label=Fig.1, caption=Diagram of infrared and visible image fusion, figureFileSmall=r2w1ZCixC+4clTG62DVjWA==, figureFileBig=j4f2/d3DmQd9CCnvvEo0qQ==, tableContent=null), ArticleFig(id=1241712922234246007, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, label=图1, caption=红外与可见光图像融合示意图, figureFileSmall=r2w1ZCixC+4clTG62DVjWA==, figureFileBig=j4f2/d3DmQd9CCnvvEo0qQ==, tableContent=null), ArticleFig(id=1241712922381046654, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, label=Fig.2, caption=Network model of infrared and visible image fusion, figureFileSmall=pzoyQQS3Efk+AVcBMVHPNg==, figureFileBig=vQLSGZP0tWcmtl+QrBKlBg==, tableContent=null), ArticleFig(id=1241712922469127041, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, label=图2, caption=红外与可见光图像融合网络模型, figureFileSmall=pzoyQQS3Efk+AVcBMVHPNg==, figureFileBig=vQLSGZP0tWcmtl+QrBKlBg==, tableContent=null), ArticleFig(id=1241712922532041602, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, label=Fig.3, caption=Contrast enhanced block, figureFileSmall=6JpX88qwwzVI8vgY8QCmuw==, figureFileBig=YO3xy5Ljgx6d7qQGs7mcQQ==, tableContent=null), ArticleFig(id=1241712922615927685, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, label=图3, caption=对比度增强模块, figureFileSmall=6JpX88qwwzVI8vgY8QCmuw==, figureFileBig=YO3xy5Ljgx6d7qQGs7mcQQ==, tableContent=null), ArticleFig(id=1241712922704008072, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, label=Fig.4, caption=Gradient residual block, figureFileSmall=n4vNRqh5nBONIdS/QxvNyA==, figureFileBig=ulzlSj181AIGQwBUr9PPuQ==, tableContent=null), ArticleFig(id=1241712922787894154, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, label=图4, caption=斜率残差模块, figureFileSmall=n4vNRqh5nBONIdS/QxvNyA==, figureFileBig=ulzlSj181AIGQwBUr9PPuQ==, tableContent=null), ArticleFig(id=1241712922888557455, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, label=Fig.5, caption=Transformer Block, figureFileSmall=gcH5FRsMi0kAXZ8v8oTdcQ==, figureFileBig=uVRlvkWi/5Q/ZlBgOilNVA==, tableContent=null), ArticleFig(id=1241712922980832146, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, label=图5, caption=Transformer模块, figureFileSmall=gcH5FRsMi0kAXZ8v8oTdcQ==, figureFileBig=uVRlvkWi/5Q/ZlBgOilNVA==, tableContent=null), ArticleFig(id=1241712923060523926, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, label=Fig.6, caption=Decoder block, figureFileSmall=D7r22806QnAFY0GFCiao3g==, figureFileBig=w31g+6brJr5/s2SIHlsYDg==, tableContent=null), ArticleFig(id=1241712923140215705, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, label=图6, caption=解码模块, figureFileSmall=D7r22806QnAFY0GFCiao3g==, figureFileBig=w31g+6brJr5/s2SIHlsYDg==, tableContent=null), ArticleFig(id=1241712923215713178, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, label=Fig.7, caption=Examples of infrared and visible images[3], figureFileSmall=+M4xlkE9jF+dnE3CMBzzlQ==, figureFileBig=QkZIPMKtYFEe+vZkny0nIA==, tableContent=null), ArticleFig(id=1241712923312182172, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, label=图7, caption=红外与可见光图像示例[3], figureFileSmall=+M4xlkE9jF+dnE3CMBzzlQ==, figureFileBig=QkZIPMKtYFEe+vZkny0nIA==, tableContent=null), ArticleFig(id=1241712923383485343, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, label=Fig.8, caption=Fusion results for the TNO dataset, figureFileSmall=ioPShTxw+nPBHP1SxM5TJQ==, figureFileBig=j5LWa3tH9ZwhKwn63X0/Zg==, tableContent=null), ArticleFig(id=1241712923475760033, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, label=图8, caption=TNO数据集的融合结果, figureFileSmall=ioPShTxw+nPBHP1SxM5TJQ==, figureFileBig=j5LWa3tH9ZwhKwn63X0/Zg==, tableContent=null), ArticleFig(id=1241712923555451811, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, label=Fig.9, caption=Fusion results for the OBTCVS dataset, figureFileSmall=NWRXEsD0iQ3XZPTMH0esZg==, figureFileBig=fYtJbAVnC2cYb0p4F20/2w==, tableContent=null), ArticleFig(id=1241712923630949284, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, label=图9, caption=OBTCVS数据集的融合结果, figureFileSmall=NWRXEsD0iQ3XZPTMH0esZg==, figureFileBig=fYtJbAVnC2cYb0p4F20/2w==, tableContent=null), ArticleFig(id=1241712923727418279, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, label=Fig.10, caption=Fusion results for the RoadScene dataset, figureFileSmall=SbBD6oQKMgglLKmJVjcSLA==, figureFileBig=7nh7JPqTh96r8VdzLVAzbg==, tableContent=null), ArticleFig(id=1241712923807110058, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, label=图10, caption=RoadScene数据集的融合结果, figureFileSmall=SbBD6oQKMgglLKmJVjcSLA==, figureFileBig=7nh7JPqTh96r8VdzLVAzbg==, tableContent=null), ArticleFig(id=1241712923903579053, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, label=Fig.11, caption=Map of objective evaluation metrics for fused images, figureFileSmall=1rtQYMMSNYhaafvr8So+tw==, figureFileBig=nR/+C+c6f0GmtHtdMczgvg==, tableContent=null), ArticleFig(id=1241712924000048048, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, label=图11, caption=融合图像客观评价指标图, figureFileSmall=1rtQYMMSNYhaafvr8So+tw==, figureFileBig=nR/+C+c6f0GmtHtdMczgvg==, tableContent=null), ArticleFig(id=1241712924083934131, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, label=Fig.12, caption=Fusion image target detection results, figureFileSmall=3GB+J1RiXNVE2y/Bc/5SMg==, figureFileBig=bgSBibuYKQi84+cw5CjfNQ==, tableContent=null), ArticleFig(id=1241712924209763253, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, label=图12, caption=融合图像目标检测结果, figureFileSmall=3GB+J1RiXNVE2y/Bc/5SMg==, figureFileBig=bgSBibuYKQi84+cw5CjfNQ==, tableContent=null), ArticleFig(id=1241712924293649338, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, label=Table 1, caption=

Objective indicator of fusion results for TNO datasets

, figureFileSmall=null, figureFileBig=null, tableContent=
MISDAGPSNRVIFFQAB/F
MST_SR13.605 69.229 64.820 017.296 30.240 00.568 0
GTF13.559 98.467 75.265 620.612 70.154 60.487 2
DenseFuse13.633 29.350 43.543 417.711 30.263 90.348 9
U2Fusion13.170 98.943 74.275 017.576 40.235 40.373 5
RFN-Nest14.050 19.634 33.190 416.786 30.284 20.331 3
DDcGAN14.656 09.451 56.869 515.072 60.333 30.428 0
SDDGAN13.329 97.900 42.065 010.941 40.137 10.119 6
STDFusion14.664 29.632 16.154 812.796 80.303 10.633 0
SwinFusion14.310 49.629 45.470 621.393 60.328 80.497 1
本文方法14.659 710.065 15.685 919.129 00.384 10.615 5
), ArticleFig(id=1241712924385924026, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, label=表1, caption=

TNO数据集融合结果的客观指标

, figureFileSmall=null, figureFileBig=null, tableContent=
MISDAGPSNRVIFFQAB/F
MST_SR13.605 69.229 64.820 017.296 30.240 00.568 0
GTF13.559 98.467 75.265 620.612 70.154 60.487 2
DenseFuse13.633 29.350 43.543 417.711 30.263 90.348 9
U2Fusion13.170 98.943 74.275 017.576 40.235 40.373 5
RFN-Nest14.050 19.634 33.190 416.786 30.284 20.331 3
DDcGAN14.656 09.451 56.869 515.072 60.333 30.428 0
SDDGAN13.329 97.900 42.065 010.941 40.137 10.119 6
STDFusion14.664 29.632 16.154 812.796 80.303 10.633 0
SwinFusion14.310 49.629 45.470 621.393 60.328 80.497 1
本文方法14.659 710.065 15.685 919.129 00.384 10.615 5
), ArticleFig(id=1241712924486587324, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, label=Table 2, caption=

Objective indicator of fusion results for OTCBVS datasets

, figureFileSmall=null, figureFileBig=null, tableContent=
MISDAGPSNRVIFFQAB/F
MST_SR15.514 710.696 07.772 721.285 50.253 00.517 4
GTF13.955 59.760 07.512 321.087 00.163 20.426 8
DenseFuse14.247 310.708 65.481 821.363 90.286 20.358 5
U2Fusion14.310 610.811 17.132 621.463 30.292 20.400 7
RFN-Nest14.389 810.664 94.397 521.161 80.265 70.252 5
DDcGAN14.770 39.728 48.249 619.037 80.230 20.403 7
SDDGAN12.977 48.663 13.053 421.261 40.165 30.130 9
STDFusion14.748 810.187 67.925 020.285 50.166 40.532 1
SwinFusion15.063 610.763 88.120 021.780 00.232 40.477 0
本文方法15.173 310.959 28.832 721.952 20.279 90.555 6
), ArticleFig(id=1241712924595639231, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, label=表2, caption=

OTCBVS数据集融合结果的客观指标

, figureFileSmall=null, figureFileBig=null, tableContent=
MISDAGPSNRVIFFQAB/F
MST_SR15.514 710.696 07.772 721.285 50.253 00.517 4
GTF13.955 59.760 07.512 321.087 00.163 20.426 8
DenseFuse14.247 310.708 65.481 821.363 90.286 20.358 5
U2Fusion14.310 610.811 17.132 621.463 30.292 20.400 7
RFN-Nest14.389 810.664 94.397 521.161 80.265 70.252 5
DDcGAN14.770 39.728 48.249 619.037 80.230 20.403 7
SDDGAN12.977 48.663 13.053 421.261 40.165 30.130 9
STDFusion14.748 810.187 67.925 020.285 50.166 40.532 1
SwinFusion15.063 610.763 88.120 021.780 00.232 40.477 0
本文方法15.173 310.959 28.832 721.952 20.279 90.555 6
), ArticleFig(id=1241712924704691137, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, label=Table 3, caption=

Objective indicator of fusion results for RoadScene datasets

, figureFileSmall=null, figureFileBig=null, tableContent=
MISDAGPSNRVIFFQAB/F
MST_SR14.340 410.622 45.347 918.447 60.501 70.639 7
GTF15.116 310.580 33.186 918.242 60.315 10.337 7
DenseFuse14.636 510.824 74.506 618.242 60.561 90.540 4
U2Fusion14.320 210.750 35.409 917.927 90.504 70.569 0
RFN-Nest14.752 710.946 02.912 018.347 80.460 10.280 9
DDcGAN14.492 49.459 54.557 215.426 20.333 20.324 4
SDDGAN14.776 19.642 13.322 115.579 90.437 20.224 1
STDFusion14.646 88.709 45.388 310.259 80.494 20.378 7
SwinFusion14.445 010.664 84.604 817.096 80.454 10.388 7
本文方法14.790 710.644 56.388 415.176 90.677 20.605 4
), ArticleFig(id=1241712924809548744, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, label=表3, caption=

RoadScene数据集融合结果的客观指标

, figureFileSmall=null, figureFileBig=null, tableContent=
MISDAGPSNRVIFFQAB/F
MST_SR14.340 410.622 45.347 918.447 60.501 70.639 7
GTF15.116 310.580 33.186 918.242 60.315 10.337 7
DenseFuse14.636 510.824 74.506 618.242 60.561 90.540 4
U2Fusion14.320 210.750 35.409 917.927 90.504 70.569 0
RFN-Nest14.752 710.946 02.912 018.347 80.460 10.280 9
DDcGAN14.492 49.459 54.557 215.426 20.333 20.324 4
SDDGAN14.776 19.642 13.322 115.579 90.437 20.224 1
STDFusion14.646 88.709 45.388 310.259 80.494 20.378 7
SwinFusion14.445 010.664 84.604 817.096 80.454 10.388 7
本文方法14.790 710.644 56.388 415.176 90.677 20.605 4
), ArticleFig(id=1241712926298526664, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, label=Table 4, caption=

Mean values of objective evaluation indexes for fused images

, figureFileSmall=null, figureFileBig=null, tableContent=
实验方法MISDAGPSNRVIFFQAB/F
MST_SR13.182 48.911 74.235 417.062 70.366 80.553 9
GTF13.352 88.615 93.836 118.040 10.231 20.410 8
DenseFuse13.590 19.260 33.754 416.205 10.483 50.431 5
U2Fusion12.766 68.747 94.085 217.069 70.355 40.424 7
RFN-Nest13.787 79.343 42.940 315.868 40.448 60.325 2
DDcGAN14.303 29.233 75.189 714.168 30.431 10.378 6
SDDGAN12.614 27.856 92.058 013.842 40.246 00.139 2
STDFusion13.503 39.004 94.772 115.624 60.368 10.502 5
SwinFusion13.809 29.428 84.618 316.338 40.428 20.489 1
Our14.001 89.523 75.069 816.256 20.510 00.585 6
), ArticleFig(id=1241712926378218443, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, label=表4, caption=

融合图像客观评价指标均值

, figureFileSmall=null, figureFileBig=null, tableContent=
实验方法MISDAGPSNRVIFFQAB/F
MST_SR13.182 48.911 74.235 417.062 70.366 80.553 9
GTF13.352 88.615 93.836 118.040 10.231 20.410 8
DenseFuse13.590 19.260 33.754 416.205 10.483 50.431 5
U2Fusion12.766 68.747 94.085 217.069 70.355 40.424 7
RFN-Nest13.787 79.343 42.940 315.868 40.448 60.325 2
DDcGAN14.303 29.233 75.189 714.168 30.431 10.378 6
SDDGAN12.614 27.856 92.058 013.842 40.246 00.139 2
STDFusion13.503 39.004 94.772 115.624 60.368 10.502 5
SwinFusion13.809 29.428 84.618 316.338 40.428 20.489 1
Our14.001 89.523 75.069 816.256 20.510 00.585 6
), ArticleFig(id=1241712926474687435, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=EN, label=Table 5, caption=

Target detection accuracy

, figureFileSmall=null, figureFileBig=null, tableContent=
mAP
红外图像0.494
可见光图像0.561
MST_SR0.773
GTF0.725
DenseFuse0.699
U2Fusion0.788
RFN-Nest0.667
DDcGAN0.654
SDDGAN0.606
STDFusion0.712
SwinFusion0.730
本文方法0.800
), ArticleFig(id=1241712926596322254, tenantId=1146029695717560320, journalId=1238841944844054536, articleId=1241697947633643796, language=CN, label=表5, caption=

目标检测精度

, figureFileSmall=null, figureFileBig=null, tableContent=
mAP
红外图像0.494
可见光图像0.561
MST_SR0.773
GTF0.725
DenseFuse0.699
U2Fusion0.788
RFN-Nest0.667
DDcGAN0.654
SDDGAN0.606
STDFusion0.712
SwinFusion0.730
本文方法0.800
)], attaches=null, journal=Journal(id=1238822691516493831, delFlag=0, nameCn=遥测遥控, nameEn=Journal of Telemetry, Tracking and Command, nameHistory1=null, nameHistory2=null, issn=2095-1000, eissn=null, cn=11-1780/TP, coden=null, periodic=1, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=uUqWBk5pV4VWPULNoAHkuQ==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Telemetry, Tracking and Command, journalRemark=null, publicationField=null, createdTime=1773287945871, updatedTime=1773299995823, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Engineering, subjectName=null, subjectCodeEn=Engineering, subjectNameEn=null, picCn=uUqWBk5pV4VWPULNoAHkuQ==, picEn=RDDslHHvDBDNnJMjk6BxdA==, jcr=null, cjcr=null, exts=[JournalExt(id=1238873232732909900, language=CN, name=遥测遥控, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1773299995835, updatedTime=1773299995835, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://manuscript.spacejournal.cn/ycyk, submissionEditorUrl=https://manuscript.spacejournal.cn/ycyk, submissionReviewUrl=https://manuscript.spacejournal.cn/ycyk, submissionCeEditorUrl=, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1238873232779047245, language=EN, name=Journal of Telemetry, Tracking and Command, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1773299995846, updatedTime=1773299995846, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://manuscript.spacejournal.cn/ycyk, submissionEditorUrl=https://manuscript.spacejournal.cn/ycyk, submissionReviewUrl=https://manuscript.spacejournal.cn/ycyk, submissionCeEditorUrl=, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1238841944844054536, websiteList=[Website(id=1238843830011744952, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1238841944844054536, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/ycyk/CN, language=CN, createTime=1773292985680, createBy=18614031015, updateTime=1773293577303, updateBy=18614031015, name=遥测遥控-中文, tplId=1146099689490845704, title=遥测遥控, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1238846390617568150, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830011744952, code=articleTextType, value=kx, createTime=1773293596176, updateTime=1773293596176, creator=18614031015, updator=18614031015), WebsiteProps(id=1238846390596596627, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830011744952, code=banner, value=null, createTime=1773293596171, updateTime=1773293596171, creator=18614031015, updator=18614031015), WebsiteProps(id=1238846390638539673, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830011744952, code=grayFlag, value=0, createTime=1773293596181, updateTime=1773293596181, creator=18614031015, updator=18614031015), WebsiteProps(id=1238846390588208018, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830011744952, code=logo, value=https://castjournals.cast.org.cn/joweb/ycyk/CN/file/pic?fileId=g9S2GGWx7iZjf51o9vpmmA==, createTime=1773293596169, updateTime=1773293596169, creator=18614031015, updator=18614031015), WebsiteProps(id=1238846390651122587, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830011744952, code=minRunFlag, value=0, createTime=1773293596184, updateTime=1773293596184, creator=18614031015, updator=18614031015), WebsiteProps(id=1238846390613373845, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830011744952, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/ycyk/CN/file/pic, createTime=1773293596175, updateTime=1773293596175, creator=18614031015, updator=18614031015), WebsiteProps(id=1238846390646928282, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830011744952, code=silenceFlag, value=0, createTime=1773293596183, updateTime=1773293596183, creator=18614031015, updator=18614031015), WebsiteProps(id=1238846390604985236, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830011744952, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1773293596173, updateTime=1773293596173, creator=18614031015, updator=18614031015), WebsiteProps(id=1238846390625956759, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830011744952, code=themeColor, value=null, createTime=1773293596178, updateTime=1773293596178, creator=18614031015, updator=18614031015), WebsiteProps(id=1238846390634345368, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830011744952, code=themeStyle, value=null, createTime=1773293596180, updateTime=1773293596180, creator=18614031015, updator=18614031015)]), Website(id=1238843830078853819, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1238841944844054536, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/ycyk/EN, language=EN, createTime=1773292985696, createBy=18614031015, updateTime=1773293566698, updateBy=18614031015, name=遥测遥控-英文, tplId=1146101810881728533, title=Journal of Telemetry, Tracking and Command, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1238846419235304352, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830078853819, code=articleTextType, value=kx, createTime=1773293602999, updateTime=1773293602999, creator=18614031015, updator=18614031015), WebsiteProps(id=1238846419214332829, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830078853819, code=banner, value=null, createTime=1773293602994, updateTime=1773293602994, creator=18614031015, updator=18614031015), WebsiteProps(id=1238846419256275875, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830078853819, code=grayFlag, value=0, createTime=1773293603004, updateTime=1773293603004, creator=18614031015, updator=18614031015), WebsiteProps(id=1238846419205944220, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830078853819, code=logo, value=https://castjournals.cast.org.cn/joweb/ycyk/EN/file/pic?fileId=g9S2GGWx7iZjf51o9vpmmA==, createTime=1773293602992, updateTime=1773293602992, creator=18614031015, updator=18614031015), WebsiteProps(id=1238846419268858789, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830078853819, code=minRunFlag, value=0, createTime=1773293603007, updateTime=1773293603007, creator=18614031015, updator=18614031015), WebsiteProps(id=1238846419231110047, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830078853819, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/ycyk/EN/file/pic, createTime=1773293602998, updateTime=1773293602998, creator=18614031015, updator=18614031015), WebsiteProps(id=1238846419264664484, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830078853819, code=silenceFlag, value=0, createTime=1773293603006, updateTime=1773293603006, creator=18614031015, updator=18614031015), WebsiteProps(id=1238846419222721438, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830078853819, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1773293602996, updateTime=1773293602996, creator=18614031015, updator=18614031015), WebsiteProps(id=1238846419243692961, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830078853819, code=themeColor, value=null, createTime=1773293603001, updateTime=1773293603001, creator=18614031015, updator=18614031015), WebsiteProps(id=1238846419252081570, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1238843830078853819, code=themeStyle, value=null, createTime=1773293603003, updateTime=1773293603003, creator=18614031015, updator=18614031015)])], journalTitle=遥测遥控, weixinUrl=null, journalUrl=https://ycyk.spacejournal.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Telemetry, Tracking and Command, journalPhotoCn=uUqWBk5pV4VWPULNoAHkuQ==, journalPhotoEn=RDDslHHvDBDNnJMjk6BxdA==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false), detailUrlCn=https://castjournals.cast.org.cn/joweb/ycyk/CN/10.12347/j.ycyk.20240326002, detailUrlEn=https://castjournals.cast.org.cn/joweb/ycyk/EN/10.12347/j.ycyk.20240326002, pdfUrlCn=https://castjournals.cast.org.cn/joweb/ycyk/CN/PDF/10.12347/j.ycyk.20240326002, pdfUrlEn=https://castjournals.cast.org.cn/joweb/ycyk/EN/PDF/10.12347/j.ycyk.20240326002, aliStartDate=null, aliEndDate=null, collectionFlag=false, citedCount=null, citedUrl=null, reference=null)
收藏切换
基于CNN-Transformer的自编码器红外和可见光图像融合方法
收藏切换
PDF下载
李霖 , 沈永健 , 张鹏宇 , 原昊 , 王超
遥测遥控 | 雷达与对抗 2024,45(5): 109-119
收起
收藏切换
遥测遥控 | 雷达与对抗 2024, 45(5): 109-119
基于CNN-Transformer的自编码器红外和可见光图像融合方法
全屏
李霖, 沈永健, 张鹏宇, 原昊, 王超
作者信息
  • 北京遥测技术研究所　北京　100076

    • 李霖 1995年生，硕士，助理工程师。

    沈永健 1985年生，博士，研究员。

    张鹏宇 1996年生，硕士，工程师。

    原昊 1998年生，硕士，工程师。

    王超 1994年生，硕士，助理工程师。

Infrared and Visible Image Fusion Based on Autoencoder Composed of CNN-transformer
Lin LI, Yongjian SHEN, Pengyu ZHANG, Hao YUAN, Chao WANG
Affiliations
  • Beijing Research Institute of Telemetry, Beijing 100076, China
出版时间: 2024-09-15 doi: 10.12347/j.ycyk.20240326002
文章导航
收藏切换
摘要
收起

基于自编码器的图像融合模型因无需手动设计融合规则而受到更多关注。然而，该融合网络编码器采用的卷积神经网络仅对局部感受野敏感，无法提取图像全局特征，且缺乏从红外图像和可见光图像中提取独特特征的能力。本文构建了一种基于自动编码器的新型图像融合网络，该网络由编码器模块、融合模块和解码器模块组成。在编码器模块中，结合使用CNN和Transformer模块以同时捕捉原图像的局部和全局特征。此外，为提取原图像特定信息，分别为原红外和可见光图像设计对比度增强和梯度增强模块。编码器模块获得的特征图像经融合模块串联后输入解码器模块，从而获得融合图像。在三个数据集上的实验结果表明，本文提出的融合网络能较好地保留了红外图像和可见光图像的清晰目标和细节信息，在主观和客观评价方面均优于其他先进方法。同时，本文提出的网络所获得的融合图像 在目标检测中获得了最高的平均精度，证明图像融合有利于下游任务。

图像融合  /  卷积神经网络  /  Transformer  /  红外图像  /  可见光图像

Image fusion model based on autoencoder network gets more attention because it does not need to design fusion rules manually. However, most autoencoder-based fusion networks use two-stream CNNs with the same structure as the encoder,which are unable to extract global features due to the local receptive field of convolutional operations and lack the ability to extract unique features from infrared and visible images. A novel autoencoder-based image fusion network which consist of encoder module, fusion module and decoder module is constructed in this paper. In the encoder module, the CNN and Transformer are combined to capture the local and global feature of the source images simultaneously. In addition, novel contrast and gradient enhancement feature extraction blocks are designed respectively for infrared and visible images to maintain the information specific to each source images. The feature images obtained by encoder module are concatenated by the fusion module and input to the decoder module to obtain the fused image. Experimental results on three datasets show that the proposed network can better preserve both the clear target and detailed information of infrared and visible images respectively, and outperforms some state-of-the-art methods in both subjective and objective evaluation. Meanwhile, the fused image obtained by the proposed network can acquire the highest mean average precision in the target detection which proves that image fusion is beneficial for downstream tasks.

Image fusion  /  Convolutional neural network  /  Transformer  /  Infrared image  /  Visible image
李霖, 沈永健, 张鹏宇, 原昊, 王超. 基于CNN-Transformer的自编码器红外和可见光图像融合方法. 遥测遥控, 2024 , 45 (5) : 109 -119 . DOI: 10.12347/j.ycyk.20240326002
Lin LI, Yongjian SHEN, Pengyu ZHANG, Hao YUAN, Chao WANG. Infrared and Visible Image Fusion Based on Autoencoder Composed of CNN-transformer[J]. Journal of Telemetry, Tracking and Command, 2024 , 45 (5) : 109 -119 . DOI: 10.12347/j.ycyk.20240326002
正文
收起
0　引言
收起
图像融合技术可将来自不同场景的图像合并成具有多个原图像特征的融合图像。结合利用多模态数据可进一步增强特征表征，是提高图像分割和目标检测等任务性能的有效手段之一。具有互补性的图像有很多，如红外图像和可见光图像、红外图像和合成孔径雷达图像、可见光图像和合成孔径雷达图像、医学中的CT和MRI图像等。其中，红外图像有利于提高目标探测和识别能力，可以避免烟雾、光线、雨水等环境的影响，同时也存在一些不足，如像素分辨率低、对比度差、背景纹理不足等[1]。可见光图像分辨率高，能反映出丰富纹理、细节等场景信息，但是它容易受到天气、烟雾、遮挡等环境因素的影响[1]，导致目标不能被突出。因此，融合技术成为结合二者互补特征的必要选择，从而获得目标明亮、背景细致的融合图像。实验结果表明：与单个模态相比，该融合图像的人脸识别性能有了显著提高，如图1所示。目前，红外与可见光图像融合技术被广泛应用于图像增强、农业自动化、遥感探测、目标识别、检测和跟踪等领域[2]。
过去有许多不同的图像融合方法，包括多尺度变换、稀疏表示、神经网络、子空间、显著性、混合模型以及其他融合方法。尽管传统的融合方法已经取得了一些进展，但这些方法依赖于人工设计活动水平测量和融合规则策略，从而使得融合过程越来越复杂，且经过这些方法提取的特征缺乏多样性，往往导致融合后的图像对比度低、纹理模糊、目标出现伪影等[2]。
基于深度学习的图像融合模型通过自适应训练，借助网络的学习能力更新模型参数，从而形成端到端的融合模型。与传统方法相比，深度学习融合方法避免了活动水平测量和融合规则设计，大大降低了人为因素对融合结果的影响。此外，深度学习方法借助深度学习网络强大的特征提取能力，在融合图像中充分保留了原图像丰富内容信息，提高了融合图像的质量。
目前，基于深度神经网络的图像融合方法可分为基于CNN(卷积神经网络)的方法、基于Autoencoder(自动编码器)的方法、基于GAN(生成对抗网络)的方法和基于Transformer的方法。基于自动编码器的图像融合网络无需人工设计融合规则，已成为当今广泛研究的一种方法。然而，现有的自动编码器大多使用卷积运算，由于局部感受野的特性，无法完全提取全局特征。此外，目前融合模型的特征提取子网络没有对不同的原图像进行区分，因此融合图像中的互补信息体现不足。为解决上述问题，本文提出了一种基于CNN-Transformer的红外与可见光图像的融合网络。本文方法的特点如下：
① 建立了一种由CNN和Transformer组合的新型编码器，同时提取红外图像和可见光图像的局部和全局信息；
② 对比度增强模块和梯度残差模块分别提取红外图像和可见光图像的独特信息，以在融合图像中保持原图像的互补信息；
③ 在三个数据集(TNO、OTCBVS和Road-Scene)上进行的广泛实验表明，本文提出的方法能够获得既包含清晰目标又包含丰富纹理的融合图像，且优于近几年的一些先进方法，包括U2Fusion、DDcGAN、DenseFuse、RFN-Nest、STDFusion 和SwinFusion。
1　本文方法
收起
本文提出的融合模型包括特征提取、特征融合和特征重组。特征提取过程分为两个通道，每个通道结合使用CNN特征增强模块和Transformer模块提取原图像特征。
特征融合模块对提取的特征进行堆叠，并输入由四个卷积层组成的解码器，最终还原生成包含红外和可见光图像特征的融合图像。本文提出的融合模型的整体框架如图2所示。
1.1　编码器
① 对比度增强模块
对比度增强模块设计目的是增强红外图像的对比度，如图3所示，使用步长分别为2、4和8的池化层对特征图像进行最大池化(最大池化操作可以保留红外图像中较大的像素信息，并过滤掉不重要的信息)，并进行卷积操作，将这些特征图像线性插值到输入特征图像大小，并与输入特征图像相加，使用3×3卷积核避免叠加像素中出现伪影，最终得到对比度增强的特征图像。
② 斜率残差模块
梯度残差模块旨在增强可见图像的细节，该块采用残差连接模式，其结构如图4所示。Sobel滤波器可以提取可见图像或特征的梯度信息，主通道特征提取器采用大小为3×3的卷积核，激活函数为Leaky ReLU。使用大小为1×1的卷积核作为辅助通道特征提取器，激活函数为Leaky ReLU。
③ Transformer模块
本文设计的Transformer模块与ViT模型相似，但在输入和注意力计算方法上有所不同。首先，二维源图像被拉伸为一维矩阵，并将批次数和每批图像数等信息一起输入Transformer模型。此外，受SwinFusion的启发，特征图像被划分为多个小窗口，并针对这些窗口计算全局注意力(基于窗口的多头自注意力机制W-MSA)，以克服传统模型的计算复杂性。滑动窗口的大小设置为8，与CNN相比，它具有更大的感知域。图5展示了Transformer模块，以下是多头注意力的计算过程。
对于特征图像，将三个可学习的权重矩阵WQЄℝC×C,WKЄℝC×C和WVЄℝC×C映射至查询矩阵Q、键矩阵K和值矩阵V，其表达式如式(1)：
{Q,K,V}={XWQ,XWK,XWV}
注意力机制定义如式(2)：
其中，dk是一个常量，便于在Softmax操作后存在梯度。将特征图拉伸为固定长度的向量后，进行计算，如式(3)、(4)所示：
其中，和fl分别表示W-MSA和MLP模块的输出特征图像。
1.2　解码器
原图像经过编码器特征提取后生成32张特征图，采用堆叠的方式对两个通道的特征图进行融合，解码器对堆叠融合后的特征图序列进行重组得到融合图像。如图6所示，解码层通过四次卷积操作把64张的特征图整合成一张结果图，每一层的输出通道数分别为32、16、8、1。
1.3　损失函数
在本文损失函数设置中，使融合图像的灰度与原图像灰度值较大者相似，同时使融合图像的梯度与原图像梯度较大者相似。损失函数定义如式(5)：
L=αLint+βLtexture
其中α和β是超参数，用于平衡像素损失函数Lint和梯度损失函数Ltexture，Lint和Ltexture的定义分别如式(6)、(7)。
其中，H和W分别表示特征图像的宽度和高度；Iir、Ivis和If分别表示红外图像、可见光图像和融合图像；▽Iir、▽Ivis和▽If分别表示红外图像、可见光图像和融合图像的梯度。
2　实验结果
收起
2.1　训练数据
训练数据集选择公开且配准度较高的MFNet数据集[3]，其中包括1 083对不同场景的图像，场景目标包括人、车等。数据集中的图像大小为640×480，位深度为24。为了适合网络训练，这些图像被处理成灰度图像并将图像裁剪为128×128大小。图7展示了一些MFNet数据集样本，前四列是光线不足场景下的图像对，后四列是光线充足场景下的图像对。
2.2　模型参数
在本文提出的融合模型中，学习率设置为2×105，衰减率为0.99，权值更新规则为Adam，批量图像大小设置为8，α、β分别设置为1和9。实验操作系统为Windows 10，硬件平台为AMD Ryzen Threadripper PRO 3945WX，主频为4.0 GHz，GPU为RTX 3080，显存为10 G，软件平台为Python 3.7，模型通过Pytorch进行搭建，从三个公开数据集(TNO[4]、OTCBVS[5]和RoadScene[6])中随机抽取样本作为测试集。
2.3　算法对比
为了验证所提模型相对于其他先进方法的优势，选择两种传统方法和七种深度学习方法进行主客观比较，包括两种传统方法：MST_SR[7]、GTF[8]；一种基于CNN的方法：U2Fusion[9]；两种基于GAN的方法：DDcGAN[10]、SDDGAN[11]；三种基于自动编码器的方法：DenseFuse[12]、RFN-Nest[13]和STDFusion[14]；一种基于Transformer的方法：SwinFusion[15]。客观评价指标包括互信息(MI)[16]、标准偏差(SD)[17]、平均梯度(AG)[18]、峰值信噪比(PSNR)[19]、融合的视觉信息保真度(VIFF)[20]和基于梯度的融合性能(QAB/F)[21]。经验证表明：本文融合后的图像不仅具有高质量的视觉效果，而且在客观指标上的评价也优于其他方法。此外，实验部分对比了原图像、本文融合图像以及其他方法生成的融合图像在目标检测任务中的表现，验证了本文融合方法的优势以及在下游任务中的应用价值。
2.3.1　融合图像主观对比
图8分别展示了上述算法在TNO数据集上的一对红外和可见光图像融合结果。从红外目标角度，图(c)、(e)、(f)和(g)的目标亮度较低，与背景灰度值相似；另外，融合图像的目标局部信息被放大，如融合图像左下角绿色框所示，图(c)、(e)、(f)和(g)的目标信息弱，图(d)目标信息几乎丢失，图(h)目标边缘出现虚影，图(i)、(j)和(k)目标边缘不够完整。对比上述融合图像目标，本文算法目标明亮，结构清晰。融合图像右下角红色框对图像背景局部区域进行了放大，相对其他算法，图(c)、(e)、(f)、(g)和(i)背景层次性不够突出；图(h)、(j)、(k)和本文方法背景纹理清晰。整体而言，本文方法在目标亮度信息及图像纹理信息上优于其他算法。
如图9所示，第一行展示的是从OTCBVS视频截取的单帧红外和可见光图像。图(c)、(d)、(e)、(f)和(g)目标亮度较低，图(d)和(g)几乎没有体现红外目标亮度信息；即便图(h)、(i)和(j)的目标明亮，但是图(h)目标边缘受光晕影响，存在一定失真；图(i)目标信息不完整，目标模糊；图(j)目标仅保留红外图像信息，纹理丢失。图(k)和本文方法目标明亮，信息丰富。在红色框位置，图(e)、(g)、(i)树枝纹理模糊。相对原可见光图像，图(d)、(f)、(g)、(h)和(i)图像灰度值较低，视觉效果不够理想。本文方法的融合结果在保留目标特征亮度、纹理信息的同时，背景清晰，细节明显。因此，本文方法的融合效果优于其他对比方法。
图10第一行展示了一对马路场景下的红外和可见光图像，该图像来自RoadScene数据集。绿色、红色和蓝色框分别标定了融合图像的目标、路灯和树木背景。图(c)、(e)、(f)、(g)和(h)目标灰度值偏低，图(g)、(h)、(i)目标模糊，图(d)、(h)、(j)和(k)路灯信息不完整。图(j)树木背景信息丢失，图(d)、(g)、(i)和(k)树木背景模糊，图(h)背景出现错误，与原图像结构不一致。综上三个方面，本文方法的融合图像目标明亮、路灯信息完整且树木背景纹理清晰。
2.3.2　融合图像客观评价
基于融合主观评价，可对图像质量做出一定判断。为了进一步验证本文方法的优势，本节对各算法在上述融合图像上的客观评价指标进行对比，做出综合性的评价。指标值越大，说明图像在该指标上的性能更好。
表1记录了各算法在上述TNO数据集上融合图像的客观评价指标。在SD、VIFF以及QAB/F三个指标上，本文方法表现最优，表明其融合图像目标明亮、局部信息丰富，视觉效果良好。另外，本文算法的其他指标在所有算法中处于前列，比如：MI数值排在第二位置，仅比STDFusion算法小了0.001；AG、PSNR均排在第三位置，因此，本文算法融合图像的信息量、梯度等指标均已得到验证。
表2记录了各算法在上述OTCBVS视频数据集上融合图像的客观评价指标。从客观指标来看，本文方法除MI、VIFF值排在第二位置，其他指标都能达到最优。与主观评价一致，本文方法融合图像的信息量、纹理、对比度以及视觉效果均优于最新算法。
表3记录了各算法在上述RoadScene数据集上融合图像的客观评价指标。本文方法在AG和VIFF指标上表现最优，MI和QAB/F指标值均排在第二，因此，本文方法融合图像质量高，客观评价良好。
除上述单幅图像指标对比之外，本章从测试集随机挑选20对红外和可见光图像，以对比各算法得到的融合图像在不同指标上的表现，如图11所示。可以注意到，本文方法在SD、AG、VIFF以及QAB/F指标上整体优于其他方法，并且有13幅融合图像的MI指标值排在对比算法前列。除个别融合图像外，本文方法融合图像的PSNR值与最高值差距较小，甚至与第4、5、6幅图像相比，本文方法PSNR值最高。最后，对这些图像指标取均值，结果如表4所示。显然，本文方法的SD、VIFF和QAB/F值均高于其他方法，并且MI和AG值排在第二位。综上，本文方法的融合图像客观评价整体优于其他先进算法。
2.3.3　目标检测对比
为证明本文方法得到的融合图像在应用领域的价值，以目标检测任务为例，对比红外图像、可见光图像和其他融合方法得到的融合图像的目标检测结果。从MFNet数据集中选择50张用于目标检测的图像对，这些图像反映的是一些城市场景，笔者手动标注了一些关键的目标，比如：人、车等等。把红外图像、可见光图像和其他算法的融合图像分别输入到Faster-RCNN[21]检测器，得到对应的检测图。利用所有目标的平均精度(mAP)衡量检测性能，检测的评估结果如表5所示。显然，两幅原图像的检测精度是最低的，因此图像融合的必要性已被验证。其次，在所有方法中，本文方法的精度最高，因此本文的融合方法相较其他对比方法更利于应用。
此外，图12提供了两个可视化的例子来说明本文的融合方法在帮助物体目标检测方面的优势。在第一个场景中，红外和可见光原图像各检测到两个人，且红外图像检测到车的类型错误，而本文方法的融合图像检测到三个人，且检测到车的类别正确。对于其他方法，RFN-Nest、DDcGAN、SDDGAN、STDFusion和SwinFusion方法的融合图像仅检测到一个人，MST_SR、GTF和DenseFuse方法的融合图像仅检测出两个人。在第二个场景中，红外图像和可见光图像各检测到一个不同位置的人，而本文方法融合结果可以同时检测到两个人。在对比方法中，除MST_SR和SDDGAN方法外，其他方法均不能检测到红外图像中的目标。综上所述，本文方法的融合结果在目标检测领域的应用价值已被验证。
3　结束语
收起
本文提出了一种基于CNN-Transformer架构的新型自动编码器，用于红外图像和可见光图像的融合。一方面，针对自动编码器网络中CNN对全局信息不敏感的问题，结合Transformer模块以构建既能保留局部信息又能保留全局信息的编码器，提高融合图像的质量。另一方面，针对目前编码器使用孪生结构而无法从红外图像和可见光图像中充分提取独特特征问题，分别为红外和可见光图像设计对比度增强模块和梯度残差模块，使特征提取过程更具针对性。与其他先进的方法相比，本文提出的融合方法可以获得主观和客观评价都较优的融合图像，同时有利于包括物体检测和识别等其他下游任务。
文献
收起
参考文献 引证文献
排序方式：
[1]
沈英, 黄春红, 黄峰, 等. 红外与可见光图像融合技术的研究进展[J]. 红外与激光工程, 2021, 50(9): 20200467.
SHEN Ying, HUANG Chunhong, HUANG Feng, et al. Research progress of infrared and visible image fusion technology [J]. Infrared and Laser Engineering, 2021, 50(9): 20200467.
[2]
李霖, 王红梅, 李辰凯. 红外与可见光图像深度学习融合方法综述[J]. 红外与激光工程, 2022, 51(12):20220125.
LI Lin, WANG Hongmei, LI Chenkai. A review of deep learning fusion methods for infrared and visible images[J]. Infrared and Laser Engineering, 2022, 51(12):20220125.
[3]
HA Q, WATANABE K, KARASAWA T, et al. MFNet:Towards realtime semantic segmentation for autonomous vehicles with multi-spectral scenes[C]//2017 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS). IEEE, 2017: 5108-5115.
[4]
TOET A. TNO Image fusion dataset[DS/OL]. [2024-03-28].https://figshare.com/articles/dataset/TNO_Image-_Fusion_Dataset/1008029, 2014.
[5]
DAVIS J W, SHARMA V. OTCBVS benchmark dataset collection[DS/OL]. [2024-03-28]. http://www.cse.ohio-state.edu/otcbvs-bench, 2007.
[6]
XU H, MA J, JIANG J, et al. U2Fusion: A unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020(1):502-518.
[7]
LIU Y, LIU S, WANG Z. A general framework for image fusion based on multi-scale transform and sparse representation[J]. Information Fusion, 2015, 24: 147-164.
[8]
MA J, CHEN C, LI C. Infrared and visible image fusion via gradient transfer and total variation minimization[J]. Information Fusion, 2016, 31: 100-109.
[9]
MA J J, XU H, JIANG J J, et al. DDcGAN: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995.
[10]
ZHOU H B, WU W, ZHANG Y D, et al. Semantic-supervised infrared and visible image fusion via a dual-discriminator generative adversarial network[J]. IEEE Transactions on Multimedia, 2021, 25: 635-648.
[11]
LI H, WU X J. DenseFuse: A fusion approach to infrared and visible images[J]. IEEE Transactions on Image Processing, 2018, 28(5): 2614-2623.
[12]
LI H, WU X J, KITTLER J. RFN-Nest: An end-to-end residual fusion network for infrared and visible images[J]. Information Fusion, 2021, 73: 72-86.
[13]
MA J, TANG L, XU M, et al. STDFusionNet: An Infrared and visible image fusion network based on salient target detection[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-13.
[14]
MA J Y, TANG L F, FAN F, et al. SwinFusion: Cross-domain long-range learning for general image fusion via swin transformer[J]. IEEE/CAA Journal of Automatica Sinica, 2022, 9(7): 1200-1217.
[15]
QU G, ZHANG D, YAN P. Information measure for performance of image fusion[J]. Electronics Letters, 2002, 38(7): 313-315.
[16]
ESKICIOGLU A M, FISHER P S. Image quality measures and their performance[J]. IEEE Transactions on Communications, 1995, 43(12): 2959-2965.
[17]
GUO W, XIONG N, CHAO H C, et al. Design and analysis of self-adapted task scheduling strategies in wireless sensor networks[J]. Sensors, 2011, 11(7): 6533-6554.
[18]
CUI G, FENG H, XU Z, et al. Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition[J]. Optics Communications, 2015, 341: 199-209.
[19]
HAN Y, CAI Y, CAO Y, et al. A new image fusion performance metric based on visual information fidelity[J]. Information Fusion, 2013, 14(2): 127-135.
[20]
XYDEAS C S, PETROVIC V. Objective image fusion performance measure[J]. Electronics Letters, 2000, 36(4): 308-309.
[21]
REN S, HE K, GIRSHICK R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[C]//29th Annual Conference on Neural Information Processing Systems, NIPS 2015, Canad.2015:91-99.
2024年第45卷第5期
PDF下载
89
41
引用本文
BibTeX
文章信息
doi: 10.12347/j.ycyk.20240326002
  • 接收时间：2024-03-26
  • 首发时间：2026-03-20
  • 出版时间：2024-09-15
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期：2024-03-26
  • 修回日期：2024-07-12
基金
作者信息
    北京遥测技术研究所　北京　100076
参考文献
分享链接
https://castjournals.cast.org.cn/joweb/ycyk/CN/10.12347/j.ycyk.20240326002
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family		 属数
Number of
genus		 种数
Number of
species		 占总种数比例
Percentage of
total species (%)		 属
Genus		 种数
Number of
species		 占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
相关链接
论文
最新文章
热读文章
热点专题
内参
蓝皮书
产业发展报告
传播力报告
期刊分级目录
资讯
学术新闻
行业新闻
学术会议
行业会议
关于
关于我们
联系我们
北京市海淀区学院南路86号
100081
010-62199257
qkjq@cast.org.cn

中国科学技术协会版权所有 京ICP 备10216604号-4 海淀分局备案 1101084647
科技导报社主办 中国科协信息中心技术支持