计算机辅助设计与图形学学报

基于局部空间自注意力的3D感知图像合成

PDF下载

宋成刚 ¹⁾ , 袁霞 ¹⁾ , 吴涛 ¹⁾ , 杨智鹏 ²⁾ , 刘伟 ³⁾ , 符颖 ¹⁾^,^*

计算机辅助设计与图形学学报 | 三维图形计算 2025,37(12): 2061-2076

收起

计算机辅助设计与图形学学报 |三维图形计算 2025 , 37 (12) : 2061 -2076

基于局部空间自注意力的3D感知图像合成

全屏

宋成刚¹⁾, 袁霞¹⁾, 吴涛¹⁾, 杨智鹏²⁾, 刘伟³⁾, 符颖¹⁾^,^*

作者信息

¹⁾成都信息工程大学计算机学院　成都　610225

²⁾成都信息工程大学电子工程学院　成都　610225

³⁾上海交通大学自动化系　上海　200240

通讯作者:

（fuying@cuit.edu.cn）

作者简介:

宋成刚（1998—），男，硕士研究生，主要研究方向为计算机视觉、3D重建

袁霞（1987—），女，博士，副研究员，硕士生导师，主要研究方向为计算机图形学、3D重建

吴涛（1984—），女，博士，教授，硕士生导师，主要研究方向为智能计算、高性能并行计算

杨智鹏（1985—），男，博士，副教授，硕士生导师，主要研究方向为图像处理及应用

刘伟（1989—），男，博士，副教授，博士生导师，主要研究方向为计算机视觉、计算机图形学

符颖（1988—），女，博士，副教授，硕士生导师，论文通信作者，主要研究方向为信号与图像处理、生成模型在机器视觉中的应用.

3D-Aware Image Synthesis via Local Spatial Self-Attention

Chenggang Song¹⁾, Xia Yuan¹⁾, Tao Wu¹⁾, Zhipeng Yang²⁾, Wei Liu³⁾, Ying Fu¹⁾^,^*

Affiliations

¹⁾School of Computer Science, Chengdu University of Information Technology, Chengdu　610225

²⁾College of Electronic Engineering, Chengdu University of Information Technology, Chengdu　610225

³⁾Department of Automation, Shanghai Jiao Tong University, Shanghai　200240

出版时间: 2025-12-20 doi: 10.3724/SP.J.1089.2024-00073

文章导航

摘要

收起

神经辐射场（neural radiance fields，NeRF）拥有将3D场景建模为连续函数的强大隐式表征能力，促进了生成模型在非结构化的2D图像集合上完成3D感知图像合成任务. 然而，由于忽略了真实世界的3D本质或依赖表达能力不足的网络架构，当前方法在图像合成方面仍然存在保真度低、一致性差和不可控制等挑战. 为此，提出一种基于自注意力机制的局部空间神经表示方法对NeRF生成器进行改良，实现可控的高质量3D感知图像合成. 首先通过结合各对象的位姿信息和预定义的3D边界框，将待合成场景进行解耦表示；然后利用自注意力局部表示模块为各对象单独建模，定义组合算子得到完整场景表示；最后经过渲染模块处理得到最终的RGB图像. 实验结果表明，所提方法能够在对空间点低频采样的情况下完成视角一致的、可控的高质量3D感知图像合成任务；且在高复杂度的CompCars公开数据集上进行64×64和256×256像素分辨率的合成实验，该方法获得的FID相较于基线方法分别提升4.69和2.51；同样在Churches公开数据集上FID分别提升1.93和3.56，进一步突出了该方法的有效性.

关键词

神经辐射场 / 隐式表征 / 注意力机制 / 3D感知图像合成

Abstract

收起

The neural radiance field (NeRF), endowed with the implicit representation capability to model 3D scenes as continuous functions, facilitates the generation model in accomplishing 3D-aware image synthesis tasks on unstructured 2D image sets. Nevertheless, current approaches face challenges in image synthesis, including low fidelity, poor consistency and uncontrollability, stemming from their inability to fully capture the three-dimensional essence of the real world or their reliance on insufficiently expressive network architectures. To address these limitations, we introduce a locally spatial neural representation method grounded in a self-attention mechanism. This approach augments the NeRF generator, empowering it to achieve controllable high-quality 3D-aware image synthesis. Initially, the scene to be synthesized is decomposed by incorporating the pose information of each object and the predefined 3D boundary box. Subsequently, the self-attention local representation module models each object distinctly. A defined combination operator is then utilized to obtain the complete scene representation. Finally, the rendering module processes the resulting RGB image. Experimental results show that the proposed method accomplishes the task of high-quality 3D-aware image synthesis with multi-view consistency and controllability, even under conditions of low-frequency spatial point sampling. Furthermore, when conducting synthesis testing with resolutions of 64×64 and 256×256 pixels on the highly complex CompCars public dataset, our proposed method achieved a 4.69 and 2.51 improvement in terms of FID scores compared to the baseline method, respectively. Similarly, on the Churches public dataset, the FID scores were improved by 1.93 and 3.56, respectively, further highlighting the effectiveness of the method.

Key words

neural radiance field / implicit representation / attention mechanism / 3D-aware image synthesis

引用本文

宋成刚, 袁霞, 吴涛, 杨智鹏, 刘伟, 符颖. 基于局部空间自注意力的3D感知图像合成. 计算机辅助设计与图形学学报, 2025 , 37 (12) : 2061 -2076 . DOI: 10.3724/SP.J.1089.2024-00073

Chenggang Song, Xia Yuan, Tao Wu, Zhipeng Yang, Wei Liu, Ying Fu. 3D-Aware Image Synthesis via Local Spatial Self-Attention[J]. Journal of Computer-Aided Design & Computer Graphics, 2025 , 37 (12) : 2061 -2076 . DOI: 10.3724/SP.J.1089.2024-00073

基金

收起

国家自然科学基金(62402318)
四川自然科学基金(2023NSFSC0469)
四川省重点研发项目(2024YFG0012; 2023YFS0428)
四川省科技厅项目(2023JDZH0034)
成都信息工程大学科技创新项目(KYQN202219)
成都信息工程大学人才引进项目(KYTZ202267)
四川省科技创新苗子工程项目(MZGC20240061)

参考文献

收起

文献

收起

参考文献引证文献

排序方式：

[1]

Chan E R, Monteiro M, Kellnhofer P, et al. pi-GAN: periodic implicit generative adversarial networks for 3D-aware image synthesis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2021: 5795-5805

[2]

Niemeyer M, Geiger A. GIRAFFE: representing scenes as compositional generative neural feature fields[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2021: 11448-11459

[3]

Schwarz K, Liao Y Y, Niemeyer M, et al. GRAF: generative radiance fields for 3D-aware image synthesis[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc, 2020: Article No.1692

[4]

Gupta S, Keshari A, Das S. Controllable image synthesis via feature mask coupling using implicit neural representation[C]//Proceedings of the 13th Indian Conference on Computer Vision, Graphics and Image Processing. New York: ACM Press, 2023: Article No.45

[5]

Goodfellow I J, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2014: 2672-2680

[6]

Henzler P, Mitra N, Ritschel T. Escaping plato's cave: 3D shape from adversarial rendering[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Los Alamitos: IEEE Computer Society Press, 2019: 9983-9992

[7]

Nguyen-Phuoc T, Li C, Theis L, et al. HoloGAN: unsupervised learning of 3D representations from natural images[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Los Alamitos: IEEE Computer Society Press, 2019: 7587-7596

[8]

Nguyen-Phuoc T, Richardt C, Mai L, et al. BlockGAN: learning 3D object-aware scene representations from unlabelled images[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc, 2020: Article No.568

[9]

Karras T, Aila T, Laine S, et al. Progressive growing of GANs for improved quality, stability, and variation[C]//Proceedings of the 6th International Conference on Learning Representations. Red Hook: Curran Associates Inc, 2018: 1-26

[10]

Karras T, Laine S, Aila T. A style-based generator architecture for generative adversarial networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(12): 4217-4228

[11]

Karras T, Laine S, Aittala M, et al. Analyzing and improving the image quality of StyleGAN[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2020: 8107-8116

[12]

Brock A, Donahue J, Simonyan K. Large scale GAN training for high fidelity natural image synthesis[C]//Proceedings of the 7th International Conference on Learning Representations. Red Hook: Curran Associates Inc, 2019: 1-35

[13]

Choi Y, Uh Y, Yoo J, et al. StarGAN v2: diverse image synthesis for multiple domains[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2020: 8185-8194

[14]

Zhu Yan, Wang Kai, Wang Linbo, et al. Pose and shape estimation of occluded humans with attention and conditional GAN[J]. Journal of Computer-Aided Design & Computer Graphics, 2024, 36(1): 142-151 (in Chinese) (朱妍, 汪楷, 汪粼波, 等. 联合注意力和条件GAN的被遮挡人体姿态和体形估计方法[J]. 计算机辅助设计与图形学学报, 2024, 36(1): 142-151)

[15]

Mildenhall B, Srinivasan P P, Tancik M, et al. NeRF: representing scenes as neural radiance fields for view synthesis[C]//Proceedings of the 16th European Conference on Computer Vision. Heidelberg: Springer, 2020: 405-421

[16]

Xie Zhifeng, Zheng Jiaheng, Wang Ji, et al. Speech-driven facial reenactment based on implicit neural representations with structured latent codes[J]. Journal of Computer-Aided Design & Computer Graphics, 2024, 36(10): 1616-1624 (in Chinese) (谢志峰, 郑迦恒, 王吉, 等. 基于结构化潜码引导NeRF的语音驱动人脸重演[J]. 计算机辅助设计与图形学学报, 2024, 36(10): 1616-1624)

[17]

Drebin R A, Carpenter L, Hanrahan P. Volume rendering[J]. ACM SIGGRAPH Computer Graphics, 1988, 22(4): 65-74

[18]

Kajiya J T, Von Herzen B P. Ray tracing volume densities[J]. ACM SIGGRAPH Computer Graphics, 1984, 18(3): 165-174

[19]

Locatello F, Bauer S, Lucic M, et al. Challenging common assumptions in the unsupervised learning of disentangled representations[C]//Proceedings of the 36th International Conference on Machine Learning. Red Hook: Curran Associates Inc, 2019: 4114-4124

[20]

Bengio Y, Courville A, Vincent P. Representation learning: a review and new perspectives[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(8): 1798-1828

[21]

Liao Y Y, Schwarz K, Mescheder L, et al. Towards unsupervised learning of generative models for 3D controllable image synthesis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2020: 5870-5879

[22]

Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc, 2017: 6000-6010

[23]

Zhang Y L, Li K P, Li K, et al. Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the 15th European Conference on Computer Vision. Heidelberg: Springer, 2018: 294-310

[24]

Zhang H, Goodfellow I J, Metaxas D N, et al. Self-attention generative adversarial networks[C]//Proceedings of the 36th International Conference on Machine Learning. Red Hook: Curran Associates Inc, 2019: 7354-7363

[25]

Woo S, Park J, Lee J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of the 15th European Conference on Computer Vision. Heidelberg: Springer, 2018: 3-19

[26]

Zheng S X, Lu J C, Zhao H S, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2021: 6877-6886

[27]

Beal J, Kim E, Tzeng E, et al. Toward transformer-based object detection[OL]. [2024-02-02]. https://arxiv.org/abs/2012.09958

[28]

Han K, Xiao A, Wu E, et al. Transformer in transformer[C]//Proceedings of the Neural Information Processing Systems. Curran Associates Inc, 2021: 15908-15919

[29]

Srinivas A, Lin T Y, Parmar N, et al. Bottleneck transformers for visual recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2021: 16514-16524

[30]

Jiang Y F, Chang S Y, Wang Z Y. TransGAN: two pure transformers can make one strong GAN, and that can scale up[C]//Proceedings of the 35th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc, 2021: Article No.1130

[31]

Parmar N, Vaswani A, Uszkoreit J, et al. Image transformer[C]//Proceedings of the 35th International Conference on Machine Learning. Red Hook: Curran Associates Inc, 2018: 4055-4064

[32]

Chen Y L, Wang Z C, Peng Y X, et al. Cascaded pyramid network for multi-person pose estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2018: 7103-7112

[33]

Zhang L, Curless, Hertzmann, et al. Shape and motion under varying illumination: unifying structure from motion, photometric stereo, and multiview stereo[C]//Proceedings of the 9th IEEE International Conference on Computer Vision. Los Alamitos: IEEE Computer Society Press, 2003: 618-625

[34]

Oechsle M, Niemeyer M, Reiser C, et al. Learning implicit surface light fields[C]//Proceedings of the International Conference on 3D Vision. Los Alamitos: IEEE Computer Society Press, 2020: 452-462

[35]

Liu Z W, Luo P, Wang X G, et al. Deep learning face attributes in the wild[C]//Proceedings of the IEEE International Conference on Computer Vision. Los Alamitos: IEEE Computer Society Press, 2015: 3730-3738

[36]

Yang J L, Li H D. Dense, accurate optical flow estimation with piecewise parametric model[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2015: 1019-1027

[37]

Yu F, Seff A, Zhang Y D, et al. LSUN: construction of a large-scale image dataset using deep learning with humans in the loop[OL]. [2024-02-02]. https://arxiv.org/abs/1506.03365

[38]

Mescheder L, Oechsle M, Niemeyer M, et al. Occupancy networks: learning 3D reconstruction in function space[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2019: 4455-4465

[39]

Park J J, Florence P, Straub J, et al. DeepSDF: learning continuous signed distance functions for shape representation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2019: 165-174

[40]

Bagautdinov T, Wu C L, Saragih J, et al. Modeling facial geometry using compositional VAEs[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2018: 3877-3886

[41]

Fan H Q, Su H, Guibas L. A point set generation network for 3D object reconstruction from a single image[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2017: 2463-2471

[42]

Yang Y Q, Feng C, Shen Y R, et al. FoldingNet: point cloud auto-encoder via deep grid deformation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2018: 206-215

[43]

Girdhar R, Fouhey D F, Rodriguez M, et al. Learning a predictable and generative vector representation for objects[C]//Proceedings of the 14th European Conference on Computer Vision. Heidelberg: Springer, 2016: 484-499

[44]

Tatarchenko M, Dosovitskiy A, Brox T. Octree generating networks: efficient convolutional architectures for high-resolution 3D outputs[C]//Proceedings of the IEEE International Conference on Computer Vision. Los Alamitos: IEEE Computer Society Press, 2017: 2107-2115

[45]

Wang Q Q, Wang Z C, Genova K, et al. IBRNet: learning multi-view image-based rendering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2021: 4688-4697

[46]

Reizenstein J, Shapovalov R, Henzler P, et al. Common objects in 3D: large-scale learning and evaluation of real-life 3D category reconstruction[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Los Alamitos: IEEE Computer Society Press, 2021: 10881-10891

[47]

T M V, Wang P H, Chen X X, et al. Is attention all that NeRF needs?[C]//Proceedings of the 11th International Conference on Learning Representations. Red Hook: Curran Associates Inc, 2023: 1-22

[48]

Sohl-Dickstein J, Weiss E A, Maheswaranathan N, et al. Deep unsupervised learning using nonequilibrium thermodynamics[C]//Proceedings of the 32nd International Conference on International Conference on Machine Learning. Red Hook: Curran Associates Inc, 2015: 2256-2265

[49]

Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc, 2020: Article No.574

[50]

Song Y, Sohl-Dickstein J, Kingma D P, et al. Score-based generative modeling through stochastic differential equations[C]//Proceedings of the 37th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc, 2021: 37799-37812

[51]

Nichol A Q, Dhariwal P, Ramesh A, et al. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models[C]//Proceedings of the 39th International Conference on Machine Learning. New York: PMLR, 2022: 16784-16804

[52]

Berrada T, Verbeek J, Couprie C, et al. Unlocking pre-trained image backbones for semantic image synthesis[OL]. [2024-02-02]. https://arxiv.org/abs/2312.13314

[53]

Pan X G, Tewari A, Leimkuhler T, et al. Drag your GAN: interactive point-based manipulation on the generative image manifold[C]//Proceedings of the ACM SIGGRAPH 2023 Conference Proceedings. New York: ACM Press, 2023: Article No.78

[54]

Xiao Z S, Kreis K, Vahdat A. Tackling the generative learning trilemma with denoising diffusion GANs[C]//Proceedings of the 10th International Conference on Learning Representations. Red Hook: Curran Associates Inc, 2022: 1-28

[55]

Chen Z Q, Kim V G, Fisher M, et al. DECOR-GAN: 3D shape detailization by conditional refinement[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2021: 15735-15744

[56]

Wu J J, Zhang C K, Xue T F, et al. Learning a probabilistic latent space of object shapes via 3D generative-adversarial modeling[C]//Proceedings of the 30th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc, 2016: 82-90

[57]

Zhu J Y, Zhang Z T, Zhang C K, et al. Visual object networks: image generation with disentangled 3D representations[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc, 2018: 118-129

[58]

Gadelha M, Maji S, Wang R. 3D shape induction from 2D views of multiple objects[C]//Proceedings of the International Conference on 3D Vision. Los Alamitos: IEEE Computer Society Press, 2017: 402-411

[59]

Szabó A, Meishvili G, Favaro P. Unsupervised generative 3D shape learning from natural images[OL]. [2024-02-02]. https://arxiv.org/abs/1910.00287

[60]

Tancik M, Srinivasan P P, Mildenhall B, et al. Fourier features let networks learn high frequency functions in low dimensional domains[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc, 2020: Article No.632

[61]

Shi W Z, Caballero J, Huszár F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2016: 1874-1883

[62]

Radford A, Metz L, Chintala S. Unsupervised representation learning with deep convolutional generative adversarial networks[OL]. [2024-02-02]. https://arxiv.org/abs/1511.06434

[63]

Mescheder L M, Geiger A, Nowozin S. Which training methods for GANs do actually converge?[C]//Proceedings of the 35th International Conference on Machine Learning. Red Hook: Curran Associates Inc, 2018: 3481-3490

[64]

Heusel M, Ramsauer H, Unterthiner T, et al. GANs trained by a two time-scale update rule converge to a local nash equilibrium[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc, 2017: 6629-6640

[65]

He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2016: 770-778

2025年第37卷第12期

PDF下载

引用本文

BibTeX

文章信息

doi: 10.3724/SP.J.1089.2024-00073

接收时间：2024-02-02
首发时间：2026-10-10
出版时间：2025-12-20

补充材料

相关文章

文章信息

作者

出版历史

收稿日期：2024-02-02
修回日期：2024-09-29

基金

国家自然科学基金(62402318)

四川自然科学基金(2023NSFSC0469)

四川省重点研发项目(2024YFG0012; 2023YFS0428)

四川省科技厅项目(2023JDZH0034)

成都信息工程大学科技创新项目(KYQN202219)

成都信息工程大学人才引进项目(KYTZ202267)

四川省科技创新苗子工程项目(MZGC20240061)

作者信息

¹⁾成都信息工程大学计算机学院　成都　610225

²⁾成都信息工程大学电子工程学院　成都　610225

³⁾上海交通大学自动化系　上海　200240

通讯作者:

（fuying@cuit.edu.cn）

参考文献

分享链接

https://castjournals.cast.org.cn/joweb/jsjfzsj/CN/10.3724/SP.J.1089.2024-00073

分享至

全文二维码

扫描看全文

引用本文

BibTeX

本文的引用情况

2种不同金属材料的力学参数

科 Family	属数 Number of genus	种数 Number of species	占总种数比例 Percentage of total species (%)	属 Genus	种数 Number of species	占总种数比例 Percentage of total species (%)
鹅膏菌科Amanitaceae	2	11	5.26	鹅膏菌属 Amanita	10	4.78
小菇科 Mycenaceae	2	12	5.74	丝盖伞属 Inocybe	5	2.39
多孔菌科 Polyporaceae	8	14	6.70	蜡蘑属 Laccaria	5	2.39
红菇科 Russulaceae	3	23	11.00	小皮伞属 Marasmius	6	2.87
				小菇属 Mycena	11	5.26
				光柄菇属 Pluteus	5	2.39
				红菇属 Russula	17	8.13
				栓菌属 Trametes	5	2.39

关闭全屏

BibTeX
EndNote
RefWorks
TxT