收藏切换
Surprise Adequacy-Guided Deep Neural Network Test Inputs Generation
收藏切换
Hongjing Guo1, Chuanqi Tao1, 2, 3, *, Zhiqiu Huang1, 2
Journal of Computer Research and Development | 2024, 61(4) : 1003 - 1017
Less
收藏切换
Journal of Computer Research and Development | 2024, 61(4): 1003-1017
• Software Technology •
Surprise Adequacy-Guided Deep Neural Network Test Inputs Generation
Full
Hongjing Guo1, Chuanqi Tao1, 2, 3, *, Zhiqiu Huang1, 2
Affiliations
  • 1College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing 210016
  • 2Ministry Key Laboratory for Safety-Critical Software Development and Verification (Nanjing University of Aeronautics and Astronautics), Nanjing 210016
  • 3National Key Laboratory for Novel Software Technology (Nanjing University), Nanjing 210023
  • born in 1996. PhD candidate. Student member of CCF. Her main research interest includes intelligent software testing

    born in 1984. PhD, associate professor. Senior member of CCF. His main research interests include intelligent software development and quality assurance of intelligent software

    born in 1965. PhD, professor. Distinguished member of CCF. His main research interests include software quality assurance, system safety, and formal methods

About Author:

born in 1996. PhD candidate. Student member of CCF. Her main research interest includes intelligent software testing

born in 1984. PhD, associate professor. Senior member of CCF. His main research interests include intelligent software development and quality assurance of intelligent software

born in 1965. PhD, professor. Distinguished member of CCF. His main research interests include software quality assurance, system safety, and formal methods

Published: 2024-04-01 doi: 10.7544/issn1000-1239.202220745
Outline
收藏切换

Due to the complexity and uncertainty of deep neural network (DNN) models, generating test inputs to comprehensively test general and corner case behaviors of DNN models is of great significance for ensuring model quality. Current research primarily focuses on designing coverage criteria and utilizing fuzzing testing technique to generate test inputs, thereby improving test adequacy. However, few studies have taken into consideration the diversity and individual fault-revealing ability of test inputs. Surprise adequacy quantifies the neuron activation differences between a test input and the training set. It is an important metric to measure test adequacy, which has not been leveraged for test input generation. Therefore, we propose a surprise adequacy-guided test input generation approach. Firstly, the approach selects important neurons that contribute more to decision-making. Activation values of these neurons are used as features to improve the surprise adequacy metric. Then, seed test inputs are selected with error-revealing capability based on the improved surprise adequacy measurements. Finally, the approach utilizes the idea of coverage-guided fuzzing testing to jointly optimize the surprise adequacy value of test inputs and the prediction probability differences among classes. The gradient ascent algorithm is adopted to calculate the perturbation and iteratively generate test inputs. Empirical studies on 5 DNN models covering 4 different image datasets demonstrate that the improved surprise adequacy metric effectively captures surprising test inputs and reduces the time cost of the calculation. Concerning test input generation, compared with DeepGini and RobOT, the follow-up test set generated by using the proposed seed input selection strategy exhibits the highest surprise coverage improvement of 5.9% and 15.9%, respectively. Compared with DLFuzz and DeepXplore, the proposed approach achieves the highest surprise coverage improvement of 26.5% and 33.7%, respectively.

software testing  /  test input generation  /  test coverage  /  deep neural network  /  surprise adequacy
郭虹静, 陶传奇, 黄志球. 意外充分性引导的深度神经网络测试样本生成. 计算机研究与发展, 2024 , 61 (4) : 1003 -1017 . DOI: 10.7544/issn1000-1239.202220745
Hongjing Guo, Chuanqi Tao, Zhiqiu Huang. Surprise Adequacy-Guided Deep Neural Network Test Inputs Generation[J]. Journal of Computer Research and Development, 2024 , 61 (4) : 1003 -1017 . DOI: 10.7544/issn1000-1239.202220745
Year 2024 volume 61 Issue 4
97
50
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202220745
  • Receive Date:2022-08-24
  • Online Date:2026-11-17
  • Published:2024-04-01
Article Data
Affiliations
History
  • Received:2022-08-24
  • Revised:2023-08-15
Affiliations
    1College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing 210016
    2Ministry Key Laboratory for Safety-Critical Software Development and Verification (Nanjing University of Aeronautics and Astronautics), Nanjing 210016
    3National Key Laboratory for Novel Software Technology (Nanjing University), Nanjing 210023
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202220745
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT