收藏切换
Towards Trustworthy Evaluation and Analysis of Intelligent Vulnerability Detection Models under Pairwise Prediction
收藏切换
Jiahao Zhu1, 2, 3, 5, 6, Zichao Wei1, 2, 3, 5, 6, Jingyi Guo3, Ming Wen1, 2, 3, 5, 6, *, Hai Jin1, 2, 4
Journal of Computer Research and Development | 2026, 63(5) : 1158 - 1176
Less
收藏切换
Journal of Computer Research and Development | 2026, 63(5): 1158-1176
• Special Issue on Security and Trustworthiness for AI-Enabled Human-Cyber-Physical Systems •
Towards Trustworthy Evaluation and Analysis of Intelligent Vulnerability Detection Models under Pairwise Prediction
Full
Jiahao Zhu1, 2, 3, 5, 6, Zichao Wei1, 2, 3, 5, 6, Jingyi Guo3, Ming Wen1, 2, 3, 5, 6, *, Hai Jin1, 2, 4
Affiliations
  • 1Services Computing Technology and System Lab (Huazhong University of Science and Technology), Ministry of Education, Wuhan 430074
  • 2National Engineering Research Center for Big Data Technology and System (Huazhong University of Science and Technology), Wuhan 430074
  • 3School of Cyber Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074
  • 4School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074
  • 5Hubei Key Laboratory of Distributed System Security (Huazhong University of Science and Technology), Wuhan 430074
  • 6Hubei Engineering Research Center on Big Data Security (Huazhong University of Science and Technology), Wuhan 430074
  • born in 2000. Master. His main research interests include code security and vulnerability detection

    born in 2000. PhD candidate. His main research interests include code security and automated vulnerability repair

    born in 2005. Undergraduate. Her main research interest includes code security

    born in 1991. PhD, professor. Senior member of CCF. His main research interests include software security, software supply chain security, and big code analysis

    born in 1966. PhD, professor. Fellow of CCF. His main research interests include computer architecture, parallel and distributed processing, cloud computing, and big data

About Author:

born in 2000. Master. His main research interests include code security and vulnerability detection

born in 2000. PhD candidate. His main research interests include code security and automated vulnerability repair

born in 2005. Undergraduate. Her main research interest includes code security

born in 1991. PhD, professor. Senior member of CCF. His main research interests include software security, software supply chain security, and big code analysis

born in 1966. PhD, professor. Fellow of CCF. His main research interests include computer architecture, parallel and distributed processing, cloud computing, and big data

Published: 2026-05-01 doi: 10.7544/issn1000-1239.202550902
Outline
收藏切换

Graph neural networks (GNNs) and code pre-trained models (CodePTMs) have been widely applied in vulnerability detection tasks, achieving the state-of-the-art (SOTA) performance on vulnerability benchmark datasets. However, such datasets (denoted as regular datasets) commonly collect vulnerable and non-vulnerable samples from different sources, resulting in significantly divergent code characteristics. Consequently, existing learning models tend to capture spurious features specific to certain datasets, rather than understanding the actual root causes of vulnerabilities. A robust and reliable vulnerability detection model should be able to make pairwise predictions, that is, recognizing the vulnerable code as vulnerable while the corresponding fixed version as non-vulnerable. However, the capability of existing learning models to make such pairwise predictions is significant and has not been explored extensively. Specifically, we first construct a large-scale real-world dataset containing vulnerable code and the corresponding fixed version. This ensures that positive and negative samples are derived from the same source, thus forming similar distributions in terms of code characteristics. We then evaluate eight mainstream SOTA vulnerability detection models in the rigorous setting to see if they can distinguish vulnerable code from its fixed versions (i.e., paired vulnerability instances). Our results reveal that all models perform poorly in making pairwise predictions when trained on regular datasets. We also find that retraining with these paired instances can improve the models’ performance by 17% on average. Furthermore, we highlight the limitations of existing CodePTMs in aggregating local code information (i.e., a key statement concerning vulnerable semantics). Driven by our findings, we devise a new approach and the evaluation results demonstrate that it outperforms the SOTA model UniXcoder by 58.47% towards making pairwise predictions.

software engineering  /  code security  /  intelligent vulnerability detection  /  trustworthy model evaluation  /  code representation learning
朱嘉豪, 魏子超, 郭靖宜, 文明, 金海. 基于成对预测的智能漏洞检测模型可信评估与分析. 计算机研究与发展, 2026 , 63 (5) : 1158 -1176 . DOI: 10.7544/issn1000-1239.202550902
Jiahao Zhu, Zichao Wei, Jingyi Guo, Ming Wen, Hai Jin. Towards Trustworthy Evaluation and Analysis of Intelligent Vulnerability Detection Models under Pairwise Prediction[J]. Journal of Computer Research and Development, 2026 , 63 (5) : 1158 -1176 . DOI: 10.7544/issn1000-1239.202550902
Year 2026 volume 63 Issue 5
79
38
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202550902
  • Receive Date:2025-12-15
  • Online Date:2026-11-17
  • Published:2026-05-01
Article Data
Affiliations
History
  • Received:2025-12-15
  • Revised:2026-03-12
Affiliations
    1Services Computing Technology and System Lab (Huazhong University of Science and Technology), Ministry of Education, Wuhan 430074
    2National Engineering Research Center for Big Data Technology and System (Huazhong University of Science and Technology), Wuhan 430074
    3School of Cyber Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074
    4School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074
    5Hubei Key Laboratory of Distributed System Security (Huazhong University of Science and Technology), Wuhan 430074
    6Hubei Engineering Research Center on Big Data Security (Huazhong University of Science and Technology), Wuhan 430074
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202550902
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT