收藏切换
CodeScore-R: An Automated Robustness Metric for Assessing the Functional Correctness of Code Synthesis
收藏切换
Guang Yang1, Yu Zhou1, *, Xiang Chen2, Xiangyu Zhang1
Journal of Computer Research and Development | 2024, 61(2) : 291 - 306
Less
收藏切换
Journal of Computer Research and Development | 2024, 61(2): 291-306
• Special Issue on Generative AI and Software Automation •
CodeScore-R: An Automated Robustness Metric for Assessing the Functional Correctness of Code Synthesis
Full
Guang Yang1, Yu Zhou1, *, Xiang Chen2, Xiangyu Zhang1
Affiliations
  • 1College of Computer Science and Technology/College of Artificial Intelligence/College of Software, Nanjing University of Aeronautics and Astronautics, Nanjing 211106
  • 2School of Information Science and Technology, Nantong University, Nantong 226019
  • born in 1997. PhD candidate. Student Member of CCF. His main research interests include code generation, code translation, and robustness research in code synthesis

    born in 1981. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include software evolution analysis, mining software repositories, software architecture, and reliability analysis

    born in 1980. PhD, associate professor. Member of CCF. His main research interests include software defect prediction, software defect location, regression testing, and combination testing

    born in 1999. Master candidate. His main research interests include code generation and code pre-trained model

About Author:

born in 1997. PhD candidate. Student Member of CCF. His main research interests include code generation, code translation, and robustness research in code synthesis

born in 1981. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include software evolution analysis, mining software repositories, software architecture, and reliability analysis

born in 1980. PhD, associate professor. Member of CCF. His main research interests include software defect prediction, software defect location, regression testing, and combination testing

born in 1999. Master candidate. His main research interests include code generation and code pre-trained model

Published: 2024-02-02 doi: 10.7544/issn1000-1239.202330715
Outline
收藏切换

Evaluation metrics are crucial in the field of code synthesis. Commonly used code evaluation metrics can be classified into three types: match-based, semantic-based, and execution-based. Among them, the execution-based Pass@k metric accurately assesses the functionality of predicted code by executing test cases. However, calculating this metric requires a significant amount of overhead, necessitating the design of an automated evaluation metric that can assess the functionality of predicted code without the need for test cases. Additionally, a good evaluation metric should be robust, that is the metric can maintain its accuracy even when the predicted code undergoes minor changes. To address these challenges, we propose an automated robust metric, called CodeScore-R, based on UniXcoder and contrastive learning, for evaluating the functionality of code synthesis. CodeScore-R employs techniques such as sketch-based processing, syntactic-equivalent transformations, and mutation testing to effectively mitigate the interference caused by identifiers, syntax structures, and operators on evaluation results. Experimental results demonstrate that in the tasks of code generation and migration in Java and Python, CodeScore-R outperforms other evaluation metrics and is more closely aligned with the Pass@k metric, while exhibiting stronger robustness.

code synthesis evaluation metric  /  functional correctness  /  robustness  /  code synthesis  /  neural network
杨光, 周宇, 陈翔, 张翔宇. CodeScore-R:用于评估代码合成功能准确性的自动化鲁棒指标. 计算机研究与发展, 2024 , 61 (2) : 291 -306 . DOI: 10.7544/issn1000-1239.202330715
Guang Yang, Yu Zhou, Xiang Chen, Xiangyu Zhang. CodeScore-R: An Automated Robustness Metric for Assessing the Functional Correctness of Code Synthesis[J]. Journal of Computer Research and Development, 2024 , 61 (2) : 291 -306 . DOI: 10.7544/issn1000-1239.202330715
Year 2024 volume 61 Issue 2
64
30
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202330715
  • Receive Date:2023-09-11
  • Online Date:2026-11-17
  • Published:2024-02-02
Article Data
Affiliations
History
  • Received:2023-09-11
  • Revised:2023-12-02
Affiliations
    1College of Computer Science and Technology/College of Artificial Intelligence/College of Software, Nanjing University of Aeronautics and Astronautics, Nanjing 211106
    2School of Information Science and Technology, Nantong University, Nantong 226019
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202330715
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT