计算机辅助设计与图形学学报

卷积与自注意力结合和时空融合的无参考视频质量评价

PDF下载

牛玉贞 ¹^,²^,³⁾ , 王镇隆 ¹⁾ , 张博林 ¹⁾ , 陈羽中 ¹^,²^,³⁾

计算机辅助设计与图形学学报 | 图像视频处理 2025,37(12): 2170-2180

收起

计算机辅助设计与图形学学报 |图像视频处理 2025 , 37 (12) : 2170 -2180

卷积与自注意力结合和时空融合的无参考视频质量评价

全屏

牛玉贞¹^,²^,³⁾, 王镇隆¹⁾, 张博林¹⁾, 陈羽中¹^,²^,³⁾

作者信息

¹⁾福州大学计算机与大数据学院　福州　350108

²⁾福建省网络计算与智能信息处理重点实验室（福州大学）　福州　350108

³⁾大数据智能教育部工程研究中心　福州　350108

通讯作者:

（nyzhen1982@fzu.edu.cn）

作者简介:

牛玉贞（1982—），女，博士，教授，博士生导师，CCF会员，主要研究方向为计算机视觉、人工智能

王镇隆（1998—），男，硕士研究生，主要研究方向为深度学习、视频质量评估

张博林（2000—），男，硕士研究生，主要研究方向为深度学习、图像恢复

陈羽中（1979—），男，博士，教授，博士生导师，主要研究方向为计算智能、数据挖掘.

No-Reference Video Quality Assessment Based on Convolution and Self-Attention Combined Network and Spatial-Temporal Fusion

Yuzhen Niu¹^,²^,³⁾, Zhenlong Wang¹⁾, Bolin Zhang¹⁾, Yuzhong Cheng¹^,²^,³⁾

Affiliations

¹⁾College of Computer and Data Science, Fuzhou University, Fuzhou　350108

²⁾Fujian Key Laboratory of Network Computing and Intelligent Information Processing (Fuzhou University), Fuzhou　350108

³⁾Engineering Research Center of Big Data Intelligence, Ministry of Education, Fuzhou　350108

出版时间: 2025-12-20 doi: 10.3724/SP.J.1089.2023-00808

文章导航

摘要

收起

随着互联网上的视频分享日益普遍，对视频进行质量评估越来越重要. 现有的视频质量评价研究中，卷积被广泛应用于视频的空域和时域特征提取，但由于其固有的局部感知特性，缺乏对于视频全局信息的感知；同时，已有方法未对空域和时域特征进行充分融合. 针对上述问题，提出一种结合卷积与自注意力机制和时空融合的无参考视频质量评价方法. 为了提取更有效的空域和时域特征，首先分别设计空域特征提取模块和时域特征提取模块，其中，空域特征提取模块通过结合卷积和自注意力的特征感知增强模块，对卷积提取的局部特征通过长距离依赖进行全局特征增强，时域特征提取模块联合视频帧序列和视频帧差序列，通过Video Swin Transformer和3D卷积学习视频的时域特征；然后使用时空特征融合增强模块，通过自注意力机制对时域和空域特征进行融合并增强；最后对视频质量进行预测. 在KoNVid-1k和YouTube-UGC数据集上的实验结果表明，与文中对比方法中最优的视频质量评价方法相比，所提方法具有更出色的表现，并通过消融实验验证了每个模块的有效性.

关键词

无参考视频质量评价 / 3D卷积神经网络 / 自注意力机制

Abstract

收起

Sharing videos on the Internet is becoming increasingly common, therefore, video quality assessment is becoming increasingly important. In existing video quality assessment research, convolution is widely used for feature extraction in the spatial and temporal domains of video, but due to its inherent local perceptual characteristics, the global information perception for video is neglected. At the same time, existing methods do not fully integrate spatial and temporal features. To solve the above problems, this paper proposes a no-reference video quality assessment method that combines convolution with a self-attention mechanism and spatiotemporal fusion. To extract more effective spatial and temporal features, the method first designs a spatial feature extraction module and a temporal feature extraction module, where the spatial feature extraction module enhances the global information of the local features extracted by convolution through long-range dependencies by the proposed feature perception and enhancement module that combines convolution and self-attention. The temporal feature extraction module jointly learns the temporal features of video using Video Swin Transformer and 3D convolution based on video frame sequences and video frame difference sequences. Then, the spatiotemporal feature fusion and enhancement module is used to fuse and enhance the temporal and spatial features through the self-attention mechanism, and finally predict the video quality. Experimental results on the KoNVid-1k and YouTube-UGC datasets show that the proposed method has better performance compared to existing video quality assessment methods, and ablation experiments also verify the effectiveness of each proposed module.

Key words

no-reference video quality assessment / 3D convolutional neural network / self-attention mechanism

引用本文

牛玉贞, 王镇隆, 张博林, 陈羽中. 卷积与自注意力结合和时空融合的无参考视频质量评价. 计算机辅助设计与图形学学报, 2025 , 37 (12) : 2170 -2180 . DOI: 10.3724/SP.J.1089.2023-00808

Yuzhen Niu, Zhenlong Wang, Bolin Zhang, Yuzhong Cheng. No-Reference Video Quality Assessment Based on Convolution and Self-Attention Combined Network and Spatial-Temporal Fusion[J]. Journal of Computer-Aided Design & Computer Graphics, 2025 , 37 (12) : 2170 -2180 . DOI: 10.3724/SP.J.1089.2023-00808

基金

收起

国家自然科学基金面上项目(61972097)
福建省自然科学基金(2023J01067)

参考文献

收起

文献

收起

参考文献引证文献

排序方式：

[1]

Duanmu Z F, Zeng K, Ma K D, et al. A quality-of-experience index for streaming video[J]. IEEE Journal of Selected Topics in Signal Processing, 2017, 11(1): 154-166

[2]

Yan Jiebin, Fang Yuming, Liu Xuelin, et al. A survey on recent advances in video quality assessment[J]. ChineseJournal of Computers, 2023, 46(10): 2196-2224 (in Chinese) (鄢杰斌, 方玉明, 刘学林, 等. 视频质量评价研究综述[J]. 计算机学报, 2023, 46(10): 2196-2224)

[3]

Liao L, Xu K M, Wu H N, et al. Exploring the effectiveness of video perceptual representation in blind video quality assessment[C]//Proceedings of the 30th ACM International Conference on Multimedia. New York: ACM Press, 2022: 837-846

[4]

Mittal A, Saad M A, Bovik A C. A completely blind video integrity oracle[J]. IEEE Transactions on Image Processing, 2016, 25(1): 289-300

[5]

Korhonen J. Two-level approach for no-reference consumer video quality assessment[J]. IEEE Transactions on Image Processing, 2019, 28(12): 5923-5938

[6]

Golestaneh S A, Dadsetan S, Kitani K M. No-reference image quality assessment via transformers, relative ranking, and self-consistency[C]//Proceedings of the IEEE Winter Conference on Applications of Computer Vision. Los Alamitos: IEEE Computer Society Press, 2022: 3989-3999

[7]

Deng J, Dong W, Socher R, et al. ImageNet: a large-scale hierarchical image database[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2009: 248-255

[8]

He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2016: 770-778

[9]

Liu Z, Ning J, Cao Y, et al. Video swin transformer[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2022: 3192-3201

[10]

Zou Liangtao, Jiang Gangyi, Yu Mei, et al. No-reference image quality assessment of high dynamic range image based on tensor domain perceptual features[J]. Journal of Computer-Aided Design & Computer Graphics, 2018, 30(10): 1850-1858 (in Chinese) (邹良涛, 蒋刚毅, 郁梅, 等. 基于张量域感知特征的无参考高动态范围图像质量评价[J]. 计算机辅助设计与图形学学报, 2018, 30(10): 1850-1858)

[11]

Mittal A, Moorthy A K, Bovik A C. No-reference image quality assessment in the spatial domain[J]. IEEE Transactions on Image Processing, 2012, 21(12): 4695-4708

[12]

Saad M A, Bovik A C, Charrier C. Blind prediction of natural video quality[J]. IEEE Transactions on Image Processing, 2014, 23(3): 1352-1365

[13]

Tu Z Z, Wang Y L, Birkbeck N, et al. UGC-VQA: benchmarking blind video quality assessment for user generated content[J]. IEEE Transactions on Image Processing, 2021, 30: 4449-4464

[14]

Korhonen J, Su Y C, You J Y. Blind natural video quality prediction via statistical temporal features and deep spatial features[C]//Proceedings of the 28th ACM International Conference on Multimedia. New York: ACM Press, 2020: 3311-3319

[15]

Tu Z Z, Yu X X, Wang Y L, et al. RAPIQUE: rapid and accurate video quality prediction of user generated content[J]. IEEE Open Journal of Signal Processing, 2021, 2: 425-440

[16]

Hosu V, Hahn F, Jenadeleh M, et al. The konstanz natural video database (KoNViD-1k)[C]//Proceedings of the 9th International Conference on Quality of Multimedia Experience. Los Alamitos: IEEE Computer Society Press, 2017: 1-6

[17]

Wang Y L, Inguva S, Adsumilli B. YouTube UGC dataset for video compression research[C]//Proceedings of the 21st IEEE International Workshop on Multimedia Signal Processing. Los Alamitos: IEEE Computer Society Press, 2019: 1-5

[18]

Li D Q, Jiang T T, Jiang M. Quality assessment of in-the-wild videos[C]//Proceedings of the 27th ACM International Conference on Multimedia. New York: ACM Press, 2019: 2351-2359

[19]

Varga D, Szirányi T. No-reference video quality assessment via pretrained CNN and LSTM networks[J]. Signal, Image and Video Processing, 2019, 13(8): 1569-1576

[20]

Shahreza H O, Amini A, Behroozi H. No-reference video quality assessment using recurrent neural networks[C]//Proceedings of the 5th Iranian Conference on Signal Processing and Intelligent Systems. Los Alamitos: IEEE Computer Society Press, 2019: 1-5

[21]

Chen P F, Li L D, Ma L, et al. RIRNet: recurrent-in-recurrent network for video quality assessment[C]//Proceedings of the 28th ACM International Conference on Multimedia. New York: ACM Press, 2020: 834-842

[22]

Hou R, Zhao Y H, Hu Y, et al. No-reference video quality evaluation by a deep transfer CNN architecture[J]. Signal Processing: Image Communication, 2020, 83: 115782

[23]

Ying Z Q, Mandal M, Ghadiyaram D, et al. Patch-VQ: ‘patching up’ the video quality problem[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2021: 14014-14024

[24]

Sun W, Min X K, Lu W, et al. A deep learning based no-reference quality assessment model for UGC videos[C]//Proceedings of the 30th ACM International Conference on Multimedia. New York: ACM Press, 2022: 856-865

[25]

Xu J H, Li J, Zhou X G, et al. Perceptual quality assessment of internet videos[C]//Proceedings of the 29th ACM International Conference on Multimedia. New York: ACM Press, 2021: 1248-1257

[26]

Li B W, Zhang W X, Tian M, et al. Blindly assess quality of in-the-wild videos via quality-aware pre-training and motion perception[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(9): 5944-5958

[27]

Zhang Z C, Wu W, Sun W, et al. MD-VQA: multi-dimensional quality assessment for UGC live videos[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2023: 1746-1755

[28]

Zhang A X, Wang Y G, Tang W X, et al. HVS revisited: acomprehensive video quality assessment framework[OL]. [2023-12-18]. https://arxiv.org/abs/2210.04158

[29]

Kay W, Carreira J, Simonyan K, et al. The kinetics human action video dataset[OL]. [2023-12-18]. https://arxiv.org/abs/1705.06950

[30]

Wu Jun, Cui Yue, Zhao Xuemei, et al. SSA-PointNet++: as-paceself-attention CNN for the semantic segmentation of 3D point cloud[J]. Journal of Computer-Aided Design & Computer Graphics, 2022, 34(3): 437-448 (in Chinese) (吴军, 崔玥, 赵雪梅, 等. SSA-PointNet++: 空间自注意力机制下的3D点云语义分割网络[J]. 计算机辅助设计与图形学学报, 2022, 34(3): 437-448)

[31]

Guo Danqing, Fu Ying, Zhu Ye, et al. Semantic segmentation of remote sensing image via self-attention-based multi-scale feature fusion[J]. Journal of Computer-Aided Design & Computer Graphics, 2023, 35(8): 1259-1268 (in Chinese) (郭丹青, 符颖, 朱烨, 等. 自注意力多尺度特征融合的遥感图像语义分割算法[J]. 计算机辅助设计与图形学学报, 2023, 35(8): 1259-1268)

[32]

Wen Yuanbo, Gao Tao, Chen Ting, et al. Frequency-guided dual sparse self-attention algorithm for single image deraining[J]. Acta Electronica Sinica, 2023, 51(10): 2812-2820 (in Chinese) (文渊博, 高涛, 陈婷, 等. 频率引导的双稀疏自注意力单图像去雨算法[J]. 电子学报, 2023, 51(10): 2812-2820)

[33]

Huan Ying, He Zifen, Yang Hongkuan, et al. Multi-scale segmentation of episodic video instance through polarized self-attention manipulation[J]. Chinese Journal of Computers, 2022, 45(12): 2605-2618 (in Chinese) (黄滢, 何自芬, 杨宏宽, 等. 极化自注意力调控的情景式视频实例多尺度分割[J]. 计算机学报, 2022, 45(12): 2605-2618)

[34]

Hara K, Kataoka H, Satoh Y. Learning spatio-temporal features with 3D residual networks for action recognition[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops. Los Alamitos: IEEE Computer Society Press, 2017: 3154-3160

[35]

Huynh-Thu Q, Ghanbari M. Modelling of spatio-temporal interaction for video quality assessment[J]. Signal Processing: Image Communication, 2010, 25(7): 535-546

[36]

Tu Z Z, Chen C J, Chen L H, et al. A comparative evaluation of temporal pooling methods for blind video quality assessment[C]//Proceedings of the IEEE International Conference on Image Processing. Los Alamitos: IEEE Computer Society Press, 2020: 141-145

[37]

Thomee B, Shamma D A, Friedland G, et al. YFCC100M: the new data in multimedia research[J]. Communications of the ACM, 2016, 59(2): 64-73

[38]

Chen X N, Liang C, Huang D, et al. Symbolic discovery of optimization algorithms[C]//Proceedings of the 37th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2023: Article No.2140

[39]

Wu H N, Liao L, Wang A N, et al. Towards robust text-prompted semantic criterion for in-the-wild video quality assessment[OL]. [2023-12-18]. https://arxiv.org/abs/2304.14672

[40]

Wang Y L, Ke J J, Talebi H, et al. Rich features for perceptual quality assessment ofUGC videos[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2021: 13430-13439

[41]

Madhusudana P C, Birkbeck N, Wang Y L, et al. CONVIQT: contrastive video quality estimator[J]. IEEE Transactions on Image Processing, 2023, 32: 5138-5152

[42]

Chen J R, Kao S H, He H, et al. Run, Don't walk: chasing higher flops for faster neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Alamitos: IEEE Computer Society Press, 2023: 12021-12031

[43]

Woo S, Park J, Lee J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of the 15th European Conference on Computer Vision. Heidelberg: Springer, 2018: 3-19

2025年第37卷第12期

PDF下载

引用本文

BibTeX

文章信息

doi: 10.3724/SP.J.1089.2023-00808

接收时间：2023-12-18
首发时间：2026-10-10
出版时间：2025-12-20

补充材料

相关文章

文章信息

作者

出版历史

收稿日期：2023-12-18
修回日期：2024-07-15

基金

国家自然科学基金面上项目(61972097)

福建省自然科学基金(2023J01067)

作者信息

¹⁾福州大学计算机与大数据学院　福州　350108

²⁾福建省网络计算与智能信息处理重点实验室（福州大学）　福州　350108

³⁾大数据智能教育部工程研究中心　福州　350108

通讯作者:

（nyzhen1982@fzu.edu.cn）

参考文献

分享链接

https://castjournals.cast.org.cn/joweb/jsjfzsj/CN/10.3724/SP.J.1089.2023-00808

分享至

全文二维码

扫描看全文

引用本文

BibTeX

本文的引用情况

2种不同金属材料的力学参数

科 Family	属数 Number of genus	种数 Number of species	占总种数比例 Percentage of total species (%)	属 Genus	种数 Number of species	占总种数比例 Percentage of total species (%)
鹅膏菌科Amanitaceae	2	11	5.26	鹅膏菌属 Amanita	10	4.78
小菇科 Mycenaceae	2	12	5.74	丝盖伞属 Inocybe	5	2.39
多孔菌科 Polyporaceae	8	14	6.70	蜡蘑属 Laccaria	5	2.39
红菇科 Russulaceae	3	23	11.00	小皮伞属 Marasmius	6	2.87
				小菇属 Mycena	11	5.26
				光柄菇属 Pluteus	5	2.39
				红菇属 Russula	17	8.13
				栓菌属 Trametes	5	2.39

关闭全屏

BibTeX
EndNote
RefWorks
TxT