计算机辅助设计与图形学学报

基于混合多模注意力融合矫正的深度图超分辨率重建

PDF下载

侯兵 ¹⁾ , 董秀成 ¹^,²⁾^,^* , 杨陈成 ¹⁾ , 雍萧 ¹⁾ , 雎雅玲 ¹⁾

计算机辅助设计与图形学学报 | 图像视频处理 2025,37(12): 2213-2224

收起

计算机辅助设计与图形学学报 |图像视频处理 2025 , 37 (12) : 2213 -2224

基于混合多模注意力融合矫正的深度图超分辨率重建

全屏

侯兵¹⁾, 董秀成¹^,²⁾^,^*, 杨陈成¹⁾, 雍萧¹⁾, 雎雅玲¹⁾

作者信息

¹⁾西华大学电气与电子信息学院　成都　610039

²⁾四川大学锦江学院　眉山　620860

通讯作者:

（dxc@mail.xhu.edu.cn）

作者简介:

侯兵（2000—），男，硕士研究生，主要研究方向为计算机视觉、深度学习

董秀成（1963—），男，硕士，教授，硕士生导师，论文通信作者，主要研究方向为智能信息处理、计算机视觉、智能控制；

杨陈成（1998—），女，硕士研究生，主要研究方向为图像处理、深度学习

雍萧（2001—），女，硕士研究生，主要研究方向为图像处理、深度学习

雎雅玲（2000—），女，硕士研究生，主要研究方向为图像处理、深度学习.

Depth Map Super-Resolution Reconstruction Based on Hybrid Multimodal Attention Fusion Correction

Bing Hou¹⁾, Xiucheng Dong¹^,²⁾^,^*, Chencheng Yang¹⁾, Xiao Yong¹⁾, Yaling Ju¹⁾

Affiliations

¹⁾School of Electrical Engineering and Electronic Information, Xihua University, Chengdu　610039

²⁾College of Jinjiang, Sichuan University, Meishan　620860

出版时间: 2025-12-20 doi: 10.3724/SP.J.1089.2024-00105

文章导航

摘要

收起

针对低成本深度相机捕获的深度图中通常存在模糊和分辨率低等多种退化问题，联合同场景高分辨率彩色图像，提出一种混合多模注意力融合矫正网络. 为了有效地利用彩色图像的结构信息，使用注意力机制迭代对齐及融合深度特征和彩色结构特征，并通过彩色结构特征和感知边缘图对融合特征进行矫正. 首先在深度边缘感知模块，联合低分辨率深度图和彩色边缘图提取高分辨率深度边缘图；然后使用侧窗边缘保留模块提取引导彩色图像中有用的结构信息，减少多余纹理信息的影响；最后在多模注意力融合矫正模块中，利用注意力机制提取深度特征和彩色结构特征的结构注意力分布，使2种模态特征实现对齐及融合，并利用彩色结构特征和高分辨率深度边缘图的空间注意力权重增强对深度边缘的监督，减少模糊和伪影问题. 在NYU v2数据集和Middlebury数据集（2005）上的实验结果表明，所提方法均方根误差平均测试结果分别为2.96和1.45，相对于次优方法分别降低了0.10和0.08；该网络能重建出边缘更加清晰和伪影更少的高分辨率深度图.

关键词

深度图超分辨率 / 注意力机制 / 侧窗卷积 / 可变卷积

Abstract

收起

To solve the problems of blurring and low resolution in depth maps captured by low-cost depth cameras, a hybrid multimode attention fusion correction network is proposed by combining high-resolution color images in the same scene. To effectively leverage the structure information of color images, the attention mechanism is employed to iteratively align and integrate depth features with color structural features, and the fused features are corrected through the incorporation of color structure features and edge aware maps. Initially, the deep edge aware block combines low-resolution depth maps with color edge maps to extract high-resolution depth edge maps. Then, the side-window edge preserve block is utilized to extract useful structural information from the color image, reducing the impact of extraneous texture information. Finally, multimodal attention fusion correction block utilizes attention mechanisms to extract the structural attention distribution of both depth features and color structural features, aligning and fusing these modalities. The spatial attention weights of the color structural features and high-resolution depth edge maps are used to enhance the supervision of the depth edges, reducing blurring and artifacts. Experimental results on the NYU v2 dataset and Middlebury dataset (2005) show that the average root mean square error test results are 2.96 and 1.45, respectively, which are reduced by 0.10 and 0.08 compared to suboptimal methods. The network can reconstruct high-resolution depth maps with clearer edges and fewer artifacts.

Key words

depth map super-resolution / attention mechanism / side-window convolution / deformable convolution

引用本文

侯兵, 董秀成, 杨陈成, 雍萧, 雎雅玲. 基于混合多模注意力融合矫正的深度图超分辨率重建. 计算机辅助设计与图形学学报, 2025 , 37 (12) : 2213 -2224 . DOI: 10.3724/SP.J.1089.2024-00105

Bing Hou, Xiucheng Dong, Chencheng Yang, Xiao Yong, Yaling Ju. Depth Map Super-Resolution Reconstruction Based on Hybrid Multimodal Attention Fusion Correction[J]. Journal of Computer-Aided Design & Computer Graphics, 2025 , 37 (12) : 2213 -2224 . DOI: 10.3724/SP.J.1089.2024-00105

基金

收起

国家自然科学基金(11872069)
四川省中央引导地方科技发展专项(2021ZYD0034)
四威高科-西华大学产学研联合实验室项目(2016-YF04-00044-JH)

参考文献

收起

文献

收起

参考文献引证文献

排序方式：

[1]

He K M, Sun J, Tang X O. Guided image filtering[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(6): 1397-1409

[2]

Lei J J, Li L L, Yue H J, et al. Depth map super-resolution considering view synthesis quality[J]. IEEE Transactions on Image Processing, 2017, 26(4): 1732-1745

[3]

Ferstl D, Reinbacher C, Ranftl R, et al. Image guided depth upsampling using anisotropic total generalized variation[C]//Proceedings of the IEEE International Conference on Computer Vision. Los Alamitos: IEEE Computer Society Press, 2013: 993-1000

[4]

Liu B H, Ling Q. Edge-guided depth image super-resolution based on KSVD[J]. IEEE Access, 2020, 8: 41108-41115

[5]

Hui T W, Loy C C, Tang X O. Depth map super-resolution by deep multi-scale guidance[C]//Proceedings of the 14th European Conference on Computer Vision. Heidelberg: Springer, 2016: 353-369

[6]

Li T, Lin H W, Dong X C, et al. Depth image super-resolution using correlation-controlled color guidance and multi-scale symmetric network[J]. Pattern Recognition, 2020, 107: Article No.107513

[7]

Zhong Z W, Liu X M, Jiang J J, et al. High-resolution depth maps imaging via attention-based hierarchical multi-modal fusion[J]. IEEE Transactions on Image Processing, 2022, 31: 648-663

[8]

Fan Peipei, Dong Xiucheng, Li Tao, et al. Super-resolution reconstruction of depth map based on non-local means constraint[J]. Journal of Computer-Aided Design & Computer Graphics, 2020, 32(10): 1671-1678 (in Chinese) (范佩佩, 董秀成, 李滔, 等. 基于非局部均值约束的深度图像超分辨率重建[J]. 计算机辅助设计与图形学学报, 2020, 32(10): 1671-1678)

[9]

Chen B L, Jung C. Single depth image super-resolution using convolutional neural networks[C]//Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. Los Alamitos: IEEE Computer Society Press, 2018: 1473-1477

[10]

Song X B, Dai Y C, Qin X Y. Deeply supervised depth map super-resolution as novel view synthesis[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 29(8): 2323-2336

[11]

Li Tao, Dong Xiucheng, Lin Hongwei. Depth map super-resolution reconstruction based on deeply supervised cross-scale attention network[J]. Acta Electronica Sinica, 2023, 51(1): 128-138 (in Chinese) (李滔, 董秀成, 林宏伟. 基于深监督跨尺度注意力网络的深度图像超分辨率重建[J]. 电子学报, 2023, 51(1): 128-138)

[12]

Zuo Y F, Fang Y M, Yang Y, et al. Depth map enhancement by revisiting multi-scale intensity guidance within coarse-to-fine stages[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(12): 4676-4687

[13]

Zuo Y F, Fang Y M, An P, et al. Frequency-dependent depth map enhancement via iterative depth-guided affine transformation and intensity-guided refinement[J]. IEEE Transactions on Multimedia, 2021, 23: 772-783

[14]

Zuo Y F, Wu Q, Fang Y M, et al. Multi-scale frequency reconstruction for guided depth map super-resolution via deep residual network[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(2): 297-306

[15]

Zhong Z W, Liu X M, Jiang J J, et al. Deep attentional guided image filtering[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(9): 12236-12250

[16]

Ye X C, Duan X Y, Li H J. Depth super-resolution with deep edge-inference network and edge-guided depth filling[C]//Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. Los Alamitos: IEEE Computer Society Press, 2018: 1398-1402

[17]

Wang Z H, Ye X C, Sun B L, et al. Depth upsampling based on deep edge-aware learning[J]. Pattern Recognition, 2020, 103: Article No.107274

[18]

Zhao Z X, Zhang J S, Xu S, et al. Discrete cosine transform network for guided depth map super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2022: 5687-5697

[19]

Yuan J Y, Jiang H B, Li X, et al. Recurrent structure attention guidance for depth super-resolution[C]//Proceedings of the 37th AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2023: 3331-3339

[20]

Li T, Dong X C, Lin H W. Guided depth map super-resolution using recumbent Y network[J]. IEEE Access, 2020, 8: 122695-122708

[21]

Ye X C, Sun B L, Wang Z H, et al. PMBANet: progressive multi-branch aggregation network for scene depth super-resolution[J]. IEEE Transactions on Image Processing, 2020, 29: 7427-7442

[22]

Zhang F, Liu N, Duan F Q. Coarse-to-fine depth super-resolution with adaptive RGB-D feature attention[J]. IEEE Transactions on Multimedia, 2024, 26: 2621-2633

[23]

Ariav I, Cohen I. Fully cross-attention Transformer for guided depth super-resolution[J]. Sensors, 2023, 23(5): Article No.2723

[24]

Zhang Y L, Li K P, Li K, et al. Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the 15th European Conference on Computer Vision. Heidelberg: Springer, 2018: 294-310

[25]

Yin H, Gong Y H, Qiu G P. Side window filtering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2019: 8750-8758

[26]

Zhu X Z, Hu H, Lin S, et al. Deformable ConvNets v2: more deformable, better results[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2019: 9300-9308

[27]

Silberman N, Hoiem D, Kohli P, et al. Indoor segmentation and support inference from RGBD images[C]//Proceedings of the 12th European Conference on Computer Vision. Heidelberg: Springer, 2012: 746-760

[28]

Hirschmuller H, Scharstein D. Evaluation of cost functions for stereo matching[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2007: 1-8

[29]

Butler D J, Wulff J, Stanley G B, et al. A naturalistic open source movie for optical flow evaluation[C]//Proceedings of the 12th European Conference on Computer Vision. Heidelberg: Springer, 2012: 611-625

[30]

Scharstein D, Pal C. Learning conditional random fields for stereo[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2007: 1-8

[31]

Kim B, Ponce J, Ham B. Deformable kernel networks for joint image filtering[J]. International Journal of Computer Vision, 2021, 129(2): 579-600

[32]

Wang X H, Chen X H, Ni B B, et al. Learning continuous depth representation via geometric spatial aggregator[C]//Proceedings of the 37th AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2023: 2698-2706

[33]

Zuo Y F, Xu Y P, Zeng Y F, et al. A²GSTran: depth map super-resolution via asymmetric attention with guidance selection[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2024, 34(6): 4668-4681

[34]

Miao Yongwei, Zhang Xinjie, Ren Hanshi, et al. A channel multi-scale fusion network for scene depth map super-resolution[J]. Journal of Computer-Aided Design & Computer Graphics, 2023, 35(1): 37-47 (in Chinese) (缪永伟, 张新杰, 任瀚实, 等. 基于通道多尺度融合的场景深度图超分辨率网络[J]. 计算机辅助设计与图形学学报, 2023, 35(1): 37-47)

2025年第37卷第12期

PDF下载

引用本文

BibTeX

文章信息

doi: 10.3724/SP.J.1089.2024-00105

接收时间：2024-02-28
首发时间：2026-10-10
出版时间：2025-12-20

补充材料

相关文章

文章信息

作者

出版历史

收稿日期：2024-02-28
修回日期：2024-06-20

基金

国家自然科学基金(11872069)

四川省中央引导地方科技发展专项(2021ZYD0034)

四威高科-西华大学产学研联合实验室项目(2016-YF04-00044-JH)

作者信息

¹⁾西华大学电气与电子信息学院　成都　610039

²⁾四川大学锦江学院　眉山　620860

通讯作者:

（dxc@mail.xhu.edu.cn）

参考文献

分享链接

https://castjournals.cast.org.cn/joweb/jsjfzsj/CN/10.3724/SP.J.1089.2024-00105

分享至

全文二维码

扫描看全文

引用本文

BibTeX

本文的引用情况

2种不同金属材料的力学参数

科 Family	属数 Number of genus	种数 Number of species	占总种数比例 Percentage of total species (%)	属 Genus	种数 Number of species	占总种数比例 Percentage of total species (%)
鹅膏菌科Amanitaceae	2	11	5.26	鹅膏菌属 Amanita	10	4.78
小菇科 Mycenaceae	2	12	5.74	丝盖伞属 Inocybe	5	2.39
多孔菌科 Polyporaceae	8	14	6.70	蜡蘑属 Laccaria	5	2.39
红菇科 Russulaceae	3	23	11.00	小皮伞属 Marasmius	6	2.87
				小菇属 Mycena	11	5.26
				光柄菇属 Pluteus	5	2.39
				红菇属 Russula	17	8.13
				栓菌属 Trametes	5	2.39

关闭全屏

BibTeX
EndNote
RefWorks
TxT