Article(id=1329455905067872752, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1329455653363495339, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202330715, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=null, receivedDate=1694361600000, receivedDateStr=2023-09-11, revisedDate=1701446400000, revisedDateStr=2023-12-02, acceptedDate=null, acceptedDateStr=null, onlineDate=1794896587863, onlineDateStr=2026-11-17, pubDate=1706803200000, pubDateStr=2024-02-02, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1794896587863, onlineIssueDateStr=2026-11-17, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1794896587863, creator=13701087609, updateTime=1794896587863, updator=13701087609, issue=Issue{id=1329455653363495339, tenantId=1146029695717560320, journalId=1324759914146529478, year='2024', volume='61', issue='2', pageStart='273', pageEnd='538', issueExtLink='null', onlineDate='null', pubDate='1706803200000', pubDateStr='2024-02-02', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1794896527853, creator='13701087609', updateTime=1794896527853, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext=null, issueFiles=null, downloadFileDto=null}, startPage=291, endPage=306, ext={EN=ArticleExt(id=1329455905315336689, articleId=1329455905067872752, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=CodeScore-R: An Automated Robustness Metric for Assessing the Functional Correctness of Code Synthesis, columnId=1329455654147830188, journalTitle=Journal of Computer Research and Development, columnName=Special Issue on Generative AI and Software Automation, runingTitle=null, highlight=null, articleAbstract=

Evaluation metrics are crucial in the field of code synthesis. Commonly used code evaluation metrics can be classified into three types: match-based, semantic-based, and execution-based. Among them, the execution-based Pass@k metric accurately assesses the functionality of predicted code by executing test cases. However, calculating this metric requires a significant amount of overhead, necessitating the design of an automated evaluation metric that can assess the functionality of predicted code without the need for test cases. Additionally, a good evaluation metric should be robust, that is the metric can maintain its accuracy even when the predicted code undergoes minor changes. To address these challenges, we propose an automated robust metric, called CodeScore-R, based on UniXcoder and contrastive learning, for evaluating the functionality of code synthesis. CodeScore-R employs techniques such as sketch-based processing, syntactic-equivalent transformations, and mutation testing to effectively mitigate the interference caused by identifiers, syntax structures, and operators on evaluation results. Experimental results demonstrate that in the tasks of code generation and migration in Java and Python, CodeScore-R outperforms other evaluation metrics and is more closely aligned with the Pass@k metric, while exhibiting stronger robustness.

, authors=Guang Yang1, Yu Zhou1, *, Xiang Chen2, Xiangyu Zhang1, authorsList=Guang Yang, Yu Zhou, Xiang Chen, Xiangyu Zhang, authorCompany=null, correspAuthors=Yu Zhou, authorNote=

born in 1997. PhD candidate. Student Member of CCF. His main research interests include code generation, code translation, and robustness research in code synthesis

born in 1981. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include software evolution analysis, mining software repositories, software architecture, and reliability analysis

born in 1980. PhD, associate professor. Member of CCF. His main research interests include software defect prediction, software defect location, regression testing, and combination testing

born in 1999. Master candidate. His main research interests include code generation and code pre-trained model

, correspAuthorsNote=null, copyrightStatement=Copyright © 2024 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1329455909304119814, articleId=1329455905067872752, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=CodeScore-R:用于评估代码合成功能准确性的自动化鲁棒指标, columnId=1329455654328185262, journalTitle=计算机研究与发展, columnName=生成式AI与软件自动化专题, runingTitle=null, highlight=null, articleAbstract=

评估指标在代码合成领域中至关重要. 常用的代码评估指标可以分为3种类型:基于匹配、基于语义和基于执行. 其中,基于执行的Pass@k指标通过执行测试用例,能够准确判断预测代码的功能准确性. 然而,该指标的计算需要大量开销,因此亟需设计一种自动化评估指标,在无需测试用例时仍可评估预测代码的功能准确性. 此外,好的评估指标应当具有鲁棒性,即预测代码发生微小改变时,评估指标仍能保持其准确性. 为此,提出了一种基于UniXcoder和对比学习的自动化鲁棒指标CodeScore-R,用于评估代码合成的功能准确性. CodeScore-R采用草图化处理、语法等价转换和变异测试等技术手段,有效减轻了标识符、语法结构和运算符对评估结果的干扰. 实验结果表明,在Java和Python语言上的代码生成和迁移任务中,CodeScore-R的表现优于其他无需测试用例的评估指标,且更接近Pass@k指标,并具有更强的鲁棒性.

, authors=杨光1, 周宇1, *, 陈翔2, 张翔宇1, authorsList=杨光, 周宇, 陈翔, 张翔宇, authorCompany=null, correspAuthors=周宇, authorNote=

1997年生,博士研究生.CCF学生会员. 主要研究方向为代码生成、代码迁移、代码合成中的鲁棒性研究

1981年生. 博士,教授,博士生导师. CCF高级会员. 主要研究方向为软件演化分析、挖掘软件仓库、软件架构、可靠性分析

1980年生. 博士,副教授. CCF会员. 主要研究方向为软件缺陷预测、软件缺陷定位、回归测试、组合测试

1999年生. 硕士研究生. 主要研究方向为代码生成、代码预训练模型

, correspAuthorsNote=
周宇()
, copyrightStatement=版权所有 © 2024 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=qB2UkE6AxCKY0zWOazKpxQ==, magXml=7HLM99xNGvrwR+qdYcUHxQ==, pdfUrl=null, pdf=HOe49R0qLKJzl4anv7Tq5A==, pdfFileSize=2494981, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=GxECN8L9k95Enwcj6f9pBw==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=77oHNYVoWjObYAmrBW2uxA==, mapNumber=null, fund=null)}, authors=[Author(id=1329455909694190094, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=yang.guang@nuaa.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329455909752910352, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, authorId=1329455909694190094, language=EN, stringName=Guang Yang, firstName=Guang, middleName=null, lastName=Yang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1College of Computer Science and Technology/College of Artificial Intelligence/College of Software, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, bio={"content":"

born in 1997. PhD candidate. Student Member of CCF. His main research interests include code generation, code translation, and robustness research in code synthesis

"}, bioImg=null, bioContent=

born in 1997. PhD candidate. Student Member of CCF. His main research interests include code generation, code translation, and robustness research in code synthesis

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329455909815824913, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, authorId=1329455909694190094, language=CN, stringName=杨光, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1南京航空航天大学计算机科学与技术学院/人工智能学院/软件学院 南京 211106, bio={"img":"hTaOpXwIaigQWB2WlwsylA==","content":"

1997年生,博士研究生.CCF学生会员. 主要研究方向为代码生成、代码迁移、代码合成中的鲁棒性研究

"}, bioImg=hTaOpXwIaigQWB2WlwsylA==, bioContent=

1997年生,博士研究生.CCF学生会员. 主要研究方向为代码生成、代码迁移、代码合成中的鲁棒性研究

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329455909505446407, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, xref=1, ext=[AuthorCompanyExt(id=1329455909518029320, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909505446407, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1College of Computer Science and Technology/College of Artificial Intelligence/College of Software, Nanjing University of Aeronautics and Astronautics, Nanjing 211106), AuthorCompanyExt(id=1329455909526417929, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909505446407, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1南京航空航天大学计算机科学与技术学院/人工智能学院/软件学院 南京 211106)])]), Author(id=1329455909870350867, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=zhouyu@nuaa.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329455909937459733, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, authorId=1329455909870350867, language=EN, stringName=Yu Zhou, firstName=Yu, middleName=null, lastName=Zhou, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, *, address=1College of Computer Science and Technology/College of Artificial Intelligence/College of Software, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, bio={"content":"

born in 1981. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include software evolution analysis, mining software repositories, software architecture, and reliability analysis

"}, bioImg=null, bioContent=

born in 1981. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include software evolution analysis, mining software repositories, software architecture, and reliability analysis

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329455910000374294, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, authorId=1329455909870350867, language=CN, stringName=周宇, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, *, address=1南京航空航天大学计算机科学与技术学院/人工智能学院/软件学院 南京 211106, bio={"img":"ZTQ2scsjnI5M7Gx4hWh1YA==","content":"

1981年生. 博士,教授,博士生导师. CCF高级会员. 主要研究方向为软件演化分析、挖掘软件仓库、软件架构、可靠性分析

"}, bioImg=ZTQ2scsjnI5M7Gx4hWh1YA==, bioContent=

1981年生. 博士,教授,博士生导师. CCF高级会员. 主要研究方向为软件演化分析、挖掘软件仓库、软件架构、可靠性分析

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329455909505446407, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, xref=1, ext=[AuthorCompanyExt(id=1329455909518029320, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909505446407, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1College of Computer Science and Technology/College of Artificial Intelligence/College of Software, Nanjing University of Aeronautics and Astronautics, Nanjing 211106), AuthorCompanyExt(id=1329455909526417929, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909505446407, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1南京航空航天大学计算机科学与技术学院/人工智能学院/软件学院 南京 211106)])]), Author(id=1329455910067483160, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329455910147174938, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, authorId=1329455910067483160, language=EN, stringName=Xiang Chen, firstName=Xiang, middleName=null, lastName=Chen, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2School of Information Science and Technology, Nantong University, Nantong 226019, bio={"content":"

born in 1980. PhD, associate professor. Member of CCF. His main research interests include software defect prediction, software defect location, regression testing, and combination testing

"}, bioImg=null, bioContent=

born in 1980. PhD, associate professor. Member of CCF. His main research interests include software defect prediction, software defect location, regression testing, and combination testing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329455910222672411, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, authorId=1329455910067483160, language=CN, stringName=陈翔, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2南通大学信息科学技术学院 江苏南通 226019, bio={"img":"h0uP8aCYKrwOP2ZxjzqIEg==","content":"

1980年生. 博士,副教授. CCF会员. 主要研究方向为软件缺陷预测、软件缺陷定位、回归测试、组合测试

"}, bioImg=h0uP8aCYKrwOP2ZxjzqIEg==, bioContent=

1980年生. 博士,副教授. CCF会员. 主要研究方向为软件缺陷预测、软件缺陷定位、回归测试、组合测试

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329455909601915402, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, xref=2, ext=[AuthorCompanyExt(id=1329455909610304011, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909601915402, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Information Science and Technology, Nantong University, Nantong 226019), AuthorCompanyExt(id=1329455909614498316, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909601915402, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2南通大学信息科学技术学院 江苏南通 226019)])]), Author(id=1329455910293975581, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329455910369473055, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, authorId=1329455910293975581, language=EN, stringName=Xiangyu Zhang, firstName=Xiangyu, middleName=null, lastName=Zhang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1College of Computer Science and Technology/College of Artificial Intelligence/College of Software, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, bio={"content":"

born in 1999. Master candidate. His main research interests include code generation and code pre-trained model

"}, bioImg=null, bioContent=

born in 1999. Master candidate. His main research interests include code generation and code pre-trained model

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329455910428193312, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, authorId=1329455910293975581, language=CN, stringName=张翔宇, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1南京航空航天大学计算机科学与技术学院/人工智能学院/软件学院 南京 211106, bio={"img":"Y10r9fosbnliLKdhZslTmw==","content":"

1999年生. 硕士研究生. 主要研究方向为代码生成、代码预训练模型

"}, bioImg=Y10r9fosbnliLKdhZslTmw==, bioContent=

1999年生. 硕士研究生. 主要研究方向为代码生成、代码预训练模型

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329455909505446407, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, xref=1, ext=[AuthorCompanyExt(id=1329455909518029320, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909505446407, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1College of Computer Science and Technology/College of Artificial Intelligence/College of Software, Nanjing University of Aeronautics and Astronautics, Nanjing 211106), AuthorCompanyExt(id=1329455909526417929, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909505446407, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1南京航空航天大学计算机科学与技术学院/人工智能学院/软件学院 南京 211106)])])], keywords=[Keyword(id=1329455910533050913, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, orderNo=1, keyword=code synthesis evaluation metric), Keyword(id=1329455910591771170, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, orderNo=2, keyword=functional correctness), Keyword(id=1329455910663074339, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, orderNo=3, keyword=robustness), Keyword(id=1329455910730183204, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, orderNo=4, keyword=code synthesis), Keyword(id=1329455910797292069, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, orderNo=5, keyword=neural network), Keyword(id=1329455912361767462, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, orderNo=1, keyword=代码合成评估指标), Keyword(id=1329455912445653543, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, orderNo=2, keyword=功能准确性), Keyword(id=1329455912550511144, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, orderNo=3, keyword=鲁棒性), Keyword(id=1329455912609231401, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, orderNo=4, keyword=代码合成), Keyword(id=1329455912688923178, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, orderNo=5, keyword=神经网络)], refs=[Reference(id=1329455917634007637, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=Gulwani S, Polozov O, Singh R. Program synthesis[J]. Foundations and Trends in Programming Languages, 2017, 4(1-2): 1−119, articleTitle=null, refAbstract=null), Reference(id=1329455917692727894, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Ren Shuo, Guo Daya, Lu Shuai, et al. CodeBLEU: A method for automatic evaluation of code synthesis[J]. arXiv preprint, arXiv: 2009. 10297, 2020, articleTitle=null, refAbstract=null), Reference(id=1329455917755642455, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Lu Shuai, Guo Daya, Ren Shuo, et al. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation[J]. arXiv preprint, arXiv: 2102. 04664, 2021, articleTitle=null, refAbstract=null), Reference(id=1329455917822751320, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Yujia, Choi D, Chung J, et al. Competition-level code generation with alphacode[J]. Science, 2022, 378(6624): 1092−1097, articleTitle=null, refAbstract=null), Reference(id=1329455917881471577, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Weisz J D, Muller M, Houde S, et al. Perfection not required human-AI partnerships in code translation[C]//Proc of the 26th Int Conf on Intelligent User Interfaces. New YorK: ACM, 2021: 402−412, articleTitle=null, refAbstract=null), Reference(id=1329455917944386138, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Jain N, Vaidyanath S, Iyer A, et al. Jigsaw: Large language models meet program synthesis[C]//Proc of the 44th Int Conf on Software Engineering. New York: ACM, 2022: 1219−1231, articleTitle=null, refAbstract=null), Reference(id=1329455917998912091, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Chaoning, Zhang Chenshuang, Zheng Sheng, et al. A complete survey on generative AI: Is ChatGPT from GPT-4 to GPT-5 all you need?[J]. arXiv preprint, arXiv: 2303. 11717, 2023, articleTitle=null, refAbstract=null), Reference(id=1329455918049243740, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Yue, Wang Weishi, Joty S, et al. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation[C]//Proc of the 2021 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2021: 8696−8708, articleTitle=null, refAbstract=null), Reference(id=1329455918107963997, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Nijkamp E, Pang B, Hayashi H, et al. CodeGen: An open large language model for code with multi-turn program synthesis[J]. arXiv preprint, arXiv: 2203. 13474, 2022, articleTitle=null, refAbstract=null), Reference(id=1329455918166684254, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Zheng Qinkai, Xia Xiao, Zou Xu, et al. CodeGeeX: A pre-trained model for code generation with multilingual evaluations on humaneval-x[J]. arXiv preprint, arXiv: 2303. 17568, 2023, articleTitle=null, refAbstract=null), Reference(id=1329455918229598815, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Liang Qingyuan, Sun Zeyu, Zhu Qihao, et al. Lyra: A benchmark for Turducken-style code generation[C]// Proc of the Thirty-First Int Joint Conf on Artificial Intelligence, San Francisco, CA: Morgan Kaufmann. 2022: 4238−4244, articleTitle=null, refAbstract=null), Reference(id=1329455918296707680, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Austin J, Odena A, Nye M, et al. Program synthesis with large language models[J]. arXiv preprint, arXiv: 2108. 07732, 2021, articleTitle=null, refAbstract=null), Reference(id=1329455918359622241, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Iyer S, Konstas I, Cheung A, et al. Mapping language to code in programmatic context[C]//Proc of the 2018 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2018: 1643−1652, articleTitle=null, refAbstract=null), Reference(id=1329455918426731106, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Papineni K, Roukos S, Ward T, et al. BLEU: A method for automatic evaluation of machine translation[C]//Proc of the 40th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2002: 311−318, articleTitle=null, refAbstract=null), Reference(id=1329455918481257059, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Popović M. chrF: Character n-gram F-Score for automatic MT evaluation[C]//Proc of the 10th workshop on Statistical Machine Translation. Stroudsburg, PA: ACL, 2015: 392−395, articleTitle=null, refAbstract=null), Reference(id=1329455918544171620, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Kulal S, Pasupat P, Chandra K, et al. Spoc: Search-based pseudocode to code[J]. Advances in Neural Information Processing Systems, 2019, 32: 11906−11917, articleTitle=null, refAbstract=null), Reference(id=1329455918607086181, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Niu Changan, Li Chuanyi, Luo Bin, et al. Deep learning meets software engineering: A survey on pre-trained models of source code[C]// Proc of the 31st Int Joint Conf on Artificial Intelligence. San Francisco, CA: Morgan Kaufmann, 2022: 5546−5555, articleTitle=null, refAbstract=null), Reference(id=1329455918695166566, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhuo T Y. Large language models are state-of-the-art evaluators of code generation[J]. arXiv preprint, arXiv: 2304. 14317, 2023, articleTitle=null, refAbstract=null), Reference(id=1329455918774858343, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Guo Daya, Lu Shuai, Duan Nan, et al. UniXcoder: Unified cross-modal pre-training for code representation[C]//Proc of the 60th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL. 2022: 7212−7225, articleTitle=null, refAbstract=null), Reference(id=1329455918858744424, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen M, Tworek J, Jun H, et al. Evaluating large language models trained on code[J]. arXiv preprint, arXiv: 2107. 03374, 2021, articleTitle=null, refAbstract=null), Reference(id=1329455918959407721, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Ahmad W U, Tushar M G R, Chakraborty S, et al. AVATAR: A parallel corpus for Java-Python program translation[J]. arXiv preprint, arXiv: 2108. 11590, 2021, articleTitle=null, refAbstract=null), Reference(id=1329455919039099498, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Liguori P, Improta C, Natella R, et al. Who evaluates the evaluators? On automatic metrics for assessing AI-based offensive code generators[J]. Expert Systems with Applications, 2023, 225: 120073, articleTitle=null, refAbstract=null), Reference(id=1329455919118791275, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Lin C Y. Rouge: A package for automatic evaluation of summaries[C]//Proc of the Text Summarization Branches Out. Stroudsburg, PA: ACL, 2004: 74−81, articleTitle=null, refAbstract=null), Reference(id=1329455919181705836, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=Evtikhiev M, Bogomolov E, Sokolov Y, et al. Out of the BLEU: How should we assess quality of the code generation models?[J]. Journal of Systems and Software, 2023, 203: 111741, articleTitle=null, refAbstract=null), Reference(id=1329455919248814701, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=Eghbali A, Pradel M. CrystalBLEU: Precisely and efficiently measuring the similarity of code[C]//Proc of the 37th IEEE/ACM Int Conf on Automated Software Engineering. Piscataway, NJ: IEEE, 2022: 1−12, articleTitle=null, refAbstract=null), Reference(id=1329455919320117870, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=Dong Yihong, Ding Jiazheng, Jiang Xue, et al. CodeScore: Evaluating code generation by learning code execution[J]. arXiv preprint, arXiv: 2301. 09043, 2023, articleTitle=null, refAbstract=null), Reference(id=1329455919408198255, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=26, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhou Shuyan, Alon U, Agarwal S, et al. CodeBERTScore: Evaluating code generation with pretrained models of code[J]. arXiv preprint, arXiv: 2302. 05527, 2023, articleTitle=null, refAbstract=null), Reference(id=1329455919483695728, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=28, rfOrder=27, authorNames=null, journalName=null, refType=null, unstructuredReference=Feng Zhangyin, Guo Daya, Tang Duyu, et al. CodeBERT: A pre-trained model for programming and natural languages[C]//Proc of the Findings of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 1536−1547, articleTitle=null, refAbstract=null), Reference(id=1329455919550804593, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=29, rfOrder=28, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Guang, Zhou Yu, Chen Xiang, et al. ExploitGen: Template-augmented exploit code generation based on CodeBERT[J]. Journal of Systems and Software, 2023, 197: 111577, articleTitle=null, refAbstract=null), Reference(id=1329455919622107762, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=30, rfOrder=29, authorNames=null, journalName=null, refType=null, unstructuredReference=Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 2017: 6000−6010, articleTitle=null, refAbstract=null), Reference(id=1329455919680828019, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=31, rfOrder=30, authorNames=null, journalName=null, refType=null, unstructuredReference=Husain H, Wu H H, Gazit T, et al. CodeSearchNet challenge: Evaluating the state of semantic code search[J]. arXiv preprint, arXiv: 1909. 09436, 2019, articleTitle=null, refAbstract=null), Reference(id=1329455919752131188, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=32, rfOrder=31, authorNames=null, journalName=null, refType=null, unstructuredReference=Gao Tianyu, Yao Xingcheng, Chen Danqi. SimCSE: Simple contrastive learning of sentence embeddings[C]//Proc of the 2021 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2021: 6894−6910, articleTitle=null, refAbstract=null), Reference(id=1329455920964285045, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=33, rfOrder=32, authorNames=null, journalName=null, refType=null, unstructuredReference=Chakraborty S, Ahmed T, Ding Yangruibo, et al. NatGen: Generative pre-training by “naturalizing” source code[C]//Proc of the 30th ACM Joint European Software Engineering Conf and Symp. on the Foundations of Software Engineering. New York: ACM, 2022: 18−30, articleTitle=null, refAbstract=null), Reference(id=1329455921031393910, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=34, rfOrder=33, authorNames=null, journalName=null, refType=null, unstructuredReference=Jia Yue, Harman M. An analysis and survey of the development of mutation testing[J]. IEEE Transactions on Software Engineering, 2010, 37(5): 649−678, articleTitle=null, refAbstract=null), Reference(id=1329455921090114167, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=35, rfOrder=34, authorNames=null, journalName=null, refType=null, unstructuredReference=van den Heuvel E, Zhan Z. Myths about linear and monotonic associations: Pearson’s r, Spearman’s ρ, and Kendall’s τ[J]. The American Statistician, 2022, 76(1): 44−52, articleTitle=null, refAbstract=null), Reference(id=1329455921157223032, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=36, rfOrder=35, authorNames=null, journalName=null, refType=null, unstructuredReference=Willmott C J, Matsuura K. Advantages of the mean absolute error over the root mean square error in assessing average model performance[J]. Climate Research, 2005, 30(1): 79−82, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1329455909505446407, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, xref=1, ext=[AuthorCompanyExt(id=1329455909518029320, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909505446407, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1College of Computer Science and Technology/College of Artificial Intelligence/College of Software, Nanjing University of Aeronautics and Astronautics, Nanjing 211106), AuthorCompanyExt(id=1329455909526417929, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909505446407, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1南京航空航天大学计算机科学与技术学院/人工智能学院/软件学院 南京 211106)]), AuthorCompany(id=1329455909601915402, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, xref=2, ext=[AuthorCompanyExt(id=1329455909610304011, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909601915402, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Information Science and Technology, Nantong University, Nantong 226019), AuthorCompanyExt(id=1329455909614498316, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909601915402, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2南通大学信息科学技术学院 江苏南通 226019)])], figs=[ArticleFig(id=1329455912797975083, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Figure 1, caption=Example of a robustness problem for code synthesis evaluation metrics, figureFileSmall=mSca0ykRYIwA/uyiVCy4Mw==, figureFileBig=GxECN8L9k95Enwcj6f9pBw==, tableContent=null), ArticleFig(id=1329455912865083948, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=图1, caption=代码合成评估指标的一个鲁棒性问题示例, figureFileSmall=mSca0ykRYIwA/uyiVCy4Mw==, figureFileBig=GxECN8L9k95Enwcj6f9pBw==, tableContent=null), ArticleFig(id=1329455913074799149, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Figure 2, caption=CodeScore-R framework, figureFileSmall=4uAExSiaqzpoRQiYPJF1VQ==, figureFileBig=Ak8KfcUmKCV+Llw8ATGg+Q==, tableContent=null), ArticleFig(id=1329455913137713710, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=图2, caption=CodeScore-R框架, figureFileSmall=4uAExSiaqzpoRQiYPJF1VQ==, figureFileBig=Ak8KfcUmKCV+Llw8ATGg+Q==, tableContent=null), ArticleFig(id=1329455913200628271, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Figure 3, caption=An example of code pre-processing, figureFileSmall=4DQaLr6H41Nq7sNbStnsKQ==, figureFileBig=hwyK4BydRD69KLTkqU9vEw==, tableContent=null), ArticleFig(id=1329455913263542832, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=图3, caption=代码预处理示例, figureFileSmall=4DQaLr6H41Nq7sNbStnsKQ==, figureFileBig=hwyK4BydRD69KLTkqU9vEw==, tableContent=null), ArticleFig(id=1329455913330651697, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Figure 4, caption=Illustration of code representation learning tasks, figureFileSmall=doL5R2tzkiF5UE5WGiUvXw==, figureFileBig=dwqn1X3PypMKGS6W6frr7Q==, tableContent=null), ArticleFig(id=1329455913389371954, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=图4, caption=掩码表征学习任务示例图, figureFileSmall=doL5R2tzkiF5UE5WGiUvXw==, figureFileBig=dwqn1X3PypMKGS6W6frr7Q==, tableContent=null), ArticleFig(id=1329455913477452339, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Figure 5, caption=Classification results of CodeScore-R and functional correctness, figureFileSmall=afniHUv3IHM/DGJTtRcCLw==, figureFileBig=wF+ATE6z/cFD4qAwrdH5xA==, tableContent=null), ArticleFig(id=1329455913565532724, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=图5, caption=CodeScore-R与功能准确性的分类结果, figureFileSmall=afniHUv3IHM/DGJTtRcCLw==, figureFileBig=wF+ATE6z/cFD4qAwrdH5xA==, tableContent=null), ArticleFig(id=1329455913636835893, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Figure 6, caption=Process diagram of CodeBERTScore calculation, figureFileSmall=OoSOqtuEOz4kd0CdfyHMJw==, figureFileBig=p9rqUkK7f15mdhYbKrBLXA==, tableContent=null), ArticleFig(id=1329455913695556150, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=图6, caption=CodeBERTScore计算过程图, figureFileSmall=OoSOqtuEOz4kd0CdfyHMJw==, figureFileBig=p9rqUkK7f15mdhYbKrBLXA==, tableContent=null), ArticleFig(id=1329455913783636535, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Table 1, caption=

Table of Code Syntax Equivalence Transformation Rules

, figureFileSmall=null, figureFileBig=null, tableContent=
规则代码示例
循环交换(loop exchange)for ⇔ while
表达式交换(expression exchange)a+=b ⇔ a=a+b
判断体交换(permute exchange)if(a){A}else{B} ⇔ if(!a){B}else{A}
判断条件交换(condition exchange)a>b ⇔ b>a or True ⇔ !False
), ArticleFig(id=1329455913850745400, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=表1, caption=

代码语法等价转换规则表

, figureFileSmall=null, figureFileBig=null, tableContent=
规则代码示例
循环交换(loop exchange)for ⇔ while
表达式交换(expression exchange)a+=b ⇔ a=a+b
判断体交换(permute exchange)if(a){A}else{B} ⇔ if(!a){B}else{A}
判断条件交换(condition exchange)a>b ⇔ b>a or True ⇔ !False
), ArticleFig(id=1329455913951408697, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Table 2, caption=

Table of Code Mutation Testing Rules

, figureFileSmall=null, figureFileBig=null, tableContent=
规则运算符
算术运算符+, −, ×, /, ××, %
关系运算符>, <, >=, <=, ==, ≠
条件运算符&&, $|| $, &, |, ^
移位运算符<<, >>, >>>
逻辑运算符&, |, ^
赋值运算符=, +=, …−=, ×=, /=, %=, ××=, <<=, >>=, >>>=
), ArticleFig(id=1329455914022711866, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=表2, caption=

代码变异测试规则表

, figureFileSmall=null, figureFileBig=null, tableContent=
规则运算符
算术运算符+, −, ×, /, ××, %
关系运算符>, <, >=, <=, ==, ≠
条件运算符&&, $|| $, &, |, ^
移位运算符<<, >>, >>>
逻辑运算符&, |, ^
赋值运算符=, +=, …−=, ×=, /=, %=, ××=, <<=, >>=, >>>=
), ArticleFig(id=1329455914089820731, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Table 3, caption=

Comparison Results Between CodeScore-R and Baseline Metrics for MAE

, figureFileSmall=null, figureFileBig=null, tableContent=
指标代码生成代码迁移
JavaPythonJavaPython
BLEU0.4530.4980.2680.347
ROUGE-L0.3910.4090.2250.192
ChrF0.3950.4240.2290.262
ED0.3920.4010.2080.198
WeightBLEU0.4150.4670.2520.329
SyntaxMatch0.4040.4360.2830.420
DataflowMatch0.3980.4150.2170.313
CodeBLEU0.4180.4540.2550.352
CrystalBLEU0.4010.4320.2350.220
CodeBERTScore0.3420.3240.2020.151
CodeScore-R0.3170.2870.1750.125
), ArticleFig(id=1329455914152735292, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=表3, caption=

CodeScore-R与基准指标的MAE对比结果

, figureFileSmall=null, figureFileBig=null, tableContent=
指标代码生成代码迁移
JavaPythonJavaPython
BLEU0.4530.4980.2680.347
ROUGE-L0.3910.4090.2250.192
ChrF0.3950.4240.2290.262
ED0.3920.4010.2080.198
WeightBLEU0.4150.4670.2520.329
SyntaxMatch0.4040.4360.2830.420
DataflowMatch0.3980.4150.2170.313
CodeBLEU0.4180.4540.2550.352
CrystalBLEU0.4010.4320.2350.220
CodeBERTScore0.3420.3240.2020.151
CodeScore-R0.3170.2870.1750.125
), ArticleFig(id=1329455914219844157, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Table 4, caption=

Comparison Results for Token Perturbation on Code Generation Tasks

, figureFileSmall=null, figureFileBig=null, tableContent=
指标Java代码生成Python代码生成
O2SS2SO2SS2S
BLEU0.5440.4350.5930.465
ROUGE-L0.5010.3900.4440.398
ChrF0.4830.3760.5370.398
ED0.4290.3810.4540.385
WeightBLEU0.5220.3940.5740.432
SyntaxMatch0.4040.4010.4500.439
DataflowMatch0.4020.3950.4220.409
CodeBLEU0.4680.4060.5100.436
CrystalBLEU0.4860.3890.5080.405
CodeBERTScore0.3760.3400.3710.316
CodeScore-R0.3170.287
), ArticleFig(id=1329455914282758718, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=表4, caption=

在代码生成任务上针对词素扰动的对比结果

, figureFileSmall=null, figureFileBig=null, tableContent=
指标Java代码生成Python代码生成
O2SS2SO2SS2S
BLEU0.5440.4350.5930.465
ROUGE-L0.5010.3900.4440.398
ChrF0.4830.3760.5370.398
ED0.4290.3810.4540.385
WeightBLEU0.5220.3940.5740.432
SyntaxMatch0.4040.4010.4500.439
DataflowMatch0.4020.3950.4220.409
CodeBLEU0.4680.4060.5100.436
CrystalBLEU0.4860.3890.5080.405
CodeBERTScore0.3760.3400.3710.316
CodeScore-R0.3170.287
), ArticleFig(id=1329455914362450495, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Table 5, caption=

Comparison Results for Token Perturbation on Code Migration Tasks

, figureFileSmall=null, figureFileBig=null, tableContent=
指标Python-to-JavaJava-to-Python
O2SS2SO2SS2S
BLEU0.5900.2650.7180.340
ROUGE-L0.4850.2270.3100.191
ChrF0.4930.2230.5980.242
ED0.3350.2080.3800.198
WeightBLEU0.5790.2500.7090.323
SyntaxMatch0.2830.2830.4200.411
DataflowMatch0.2410.2180.3500.304
CodeBLEU0.4230.2540.5490.345
CrystalBLEU0.4740.2130.4490.211
CodeBERTScore0.2820.2010.2730.149
CodeScore-R0.1750.125
), ArticleFig(id=1329455914425365056, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=表5, caption=

在代码迁移任务上针对词素扰动的对比结果

, figureFileSmall=null, figureFileBig=null, tableContent=
指标Python-to-JavaJava-to-Python
O2SS2SO2SS2S
BLEU0.5900.2650.7180.340
ROUGE-L0.4850.2270.3100.191
ChrF0.4930.2230.5980.242
ED0.3350.2080.3800.198
WeightBLEU0.5790.2500.7090.323
SyntaxMatch0.2830.2830.4200.411
DataflowMatch0.2410.2180.3500.304
CodeBLEU0.4230.2540.5490.345
CrystalBLEU0.4740.2130.4490.211
CodeBERTScore0.2820.2010.2730.149
CodeScore-R0.1750.125
), ArticleFig(id=1329455914488279617, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Table 6, caption=

Comparison Results of CodeScore-R and Baseline Metrics for Syntax Perturbation

, figureFileSmall=null, figureFileBig=null, tableContent=
指标代码生成代码迁移
JavaPythonJavaPython
BLEU0.4510.4790.3180.388
ROUGE-L0.4530.4930.3770.502
ChrF0.3850.4210.2510.318
ED0.3910.3930.2380.230
WeightBLEU0.4120.4440.3000.362
SyntaxMatch0.4140.4480.3040.444
DataflowMatch0.3960.4140.2210.324
CodeBLEU0.4180.4460.2860.379
CrystalBLEU0.4300.4470.3160.369
CodeBERTScore0.3470.3200.2190.167
CodeScore-R0.3110.2780.1820.129
), ArticleFig(id=1329455914567971394, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=表6, caption=

CodeScore-R与基准指标针对语法扰动的对比结果

, figureFileSmall=null, figureFileBig=null, tableContent=
指标代码生成代码迁移
JavaPythonJavaPython
BLEU0.4510.4790.3180.388
ROUGE-L0.4530.4930.3770.502
ChrF0.3850.4210.2510.318
ED0.3910.3930.2380.230
WeightBLEU0.4120.4440.3000.362
SyntaxMatch0.4140.4480.3040.444
DataflowMatch0.3960.4140.2210.324
CodeBLEU0.4180.4460.2860.379
CrystalBLEU0.4300.4470.3160.369
CodeBERTScore0.3470.3200.2190.167
CodeScore-R0.3110.2780.1820.129
), ArticleFig(id=1329455914643468867, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Table 7, caption=

Comparison Results for Semantic Perturbation on Java Code Generation Task

, figureFileSmall=null, figureFileBig=null, tableContent=
指标Java代码生成
Top25%Top50%Top75%Top100%
BLEU0.4600.4710.4730.450
ROUGE-L0.4410.4760.4990.498
ChrF0.4890.5750.6470.673
ED0.4910.5720.6440.675
WeightBLEU0.4730.5260.5650.559
SyntaxMatch0.4810.5290.5650.570
DataflowMatch0.4760.5460.5870.590
CodeBLEU0.4730.5180.5470.542
CrystalBLEU0.4610.5170.5600.582
CodeBERTScore0.5150.6600.7880.872
CodeScore-R0.3260.3040.3200.335
), ArticleFig(id=1329455914714772036, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=表7, caption=

在Java代码生成任务上针对语义扰动的对比结果

, figureFileSmall=null, figureFileBig=null, tableContent=
指标Java代码生成
Top25%Top50%Top75%Top100%
BLEU0.4600.4710.4730.450
ROUGE-L0.4410.4760.4990.498
ChrF0.4890.5750.6470.673
ED0.4910.5720.6440.675
WeightBLEU0.4730.5260.5650.559
SyntaxMatch0.4810.5290.5650.570
DataflowMatch0.4760.5460.5870.590
CodeBLEU0.4730.5180.5470.542
CrystalBLEU0.4610.5170.5600.582
CodeBERTScore0.5150.6600.7880.872
CodeScore-R0.3260.3040.3200.335
), ArticleFig(id=1329455914777686597, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Table 8, caption=

Comparison Results for Semantic Perturbation on Python Code Generation Task

, figureFileSmall=null, figureFileBig=null, tableContent=
指标Python代码生成
Top25%Top50%Top75%Top100%
BLEU0.4810.4680.4280.391
ROUGE-L0.4790.5580.5830.598
ChrF0.4840.5500.5750.592
ED0.4720.5550.5910.619
WeightBLEU0.4890.5080.4890.471
SyntaxMatch0.5010.5100.5040.492
DataflowMatch0.4820.5760.6110.637
CodeBLEU0.4880.5150.5080.498
CrystalBLEU0.4840.5520.5770.596
CodeBERTScore0.4700.6450.7590.862
CodeScore-R0.3010.3170.3200.312
), ArticleFig(id=1329455914840601158, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=表8, caption=

在Python代码生成任务上针对语义扰动的对比结果

, figureFileSmall=null, figureFileBig=null, tableContent=
指标Python代码生成
Top25%Top50%Top75%Top100%
BLEU0.4810.4680.4280.391
ROUGE-L0.4790.5580.5830.598
ChrF0.4840.5500.5750.592
ED0.4720.5550.5910.619
WeightBLEU0.4890.5080.4890.471
SyntaxMatch0.5010.5100.5040.492
DataflowMatch0.4820.5760.6110.637
CodeBLEU0.4880.5150.5080.498
CrystalBLEU0.4840.5520.5770.596
CodeBERTScore0.4700.6450.7590.862
CodeScore-R0.3010.3170.3200.312
), ArticleFig(id=1329455914911904327, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Table 9, caption=

Comparison Results for Semantic Perturbation on Python-to-Java Code Migration Task

, figureFileSmall=null, figureFileBig=null, tableContent=
指标Python-to-Java代码迁移
Top25%Top50%Top75%Top100%
BLEU0.3770.4870.5860.722
ROUGE-L0.3420.4510.5550.681
ChrF0.3780.5310.6780.851
ED0.3780.5450.7110.901
WeightBLEU0.3770.5010.6140.759
SyntaxMatch0.3950.4990.6060.738
DataflowMatch0.3590.4800.6150.769
CodeBLEU0.3770.4920.6050.747
CrystalBLEU0.3520.4840.6150.769
CodeBERTScore0.3870.5670.7470.947
CodeScore-R0.2430.3070.3810.456
), ArticleFig(id=1329455914995790408, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=表9, caption=

在Python-to-Java代码迁移任务上针对语义扰动的对比结果

, figureFileSmall=null, figureFileBig=null, tableContent=
指标Python-to-Java代码迁移
Top25%Top50%Top75%Top100%
BLEU0.3770.4870.5860.722
ROUGE-L0.3420.4510.5550.681
ChrF0.3780.5310.6780.851
ED0.3780.5450.7110.901
WeightBLEU0.3770.5010.6140.759
SyntaxMatch0.3950.4990.6060.738
DataflowMatch0.3590.4800.6150.769
CodeBLEU0.3770.4920.6050.747
CrystalBLEU0.3520.4840.6150.769
CodeBERTScore0.3870.5670.7470.947
CodeScore-R0.2430.3070.3810.456
), ArticleFig(id=1329455915079676489, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Table 10, caption=

Comparison Results for Semantic Perturbation on Java-to-Python Code Migration Task

, figureFileSmall=null, figureFileBig=null, tableContent=
指标Java-to-Python代码迁移
Top25%Top50%Top75%Top100%
BLEU0.4030.4610.5240.607
ROUGE-L0.3600.5230.6960.873
ChrF0.3780.5040.6410.787
ED0.3650.5250.6950.867
WeightBLEU0.4000.4640.5380.631
SyntaxMatch0.4490.4750.5180.581
DataflowMatch0.3950.4710.5860.715
CodeBLEU0.4120.4680.5410.634
CrystalBLEU0.3680.5230.6830.851
CodeBERTScore0.3550.5520.7570.961
CodeScore-R0.1970.2570.3030.371
), ArticleFig(id=1329455915150979658, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=表10, caption=

在Java-to-Python代码迁移任务上针对语义扰动的对比结果

, figureFileSmall=null, figureFileBig=null, tableContent=
指标Java-to-Python代码迁移
Top25%Top50%Top75%Top100%
BLEU0.4030.4610.5240.607
ROUGE-L0.3600.5230.6960.873
ChrF0.3780.5040.6410.787
ED0.3650.5250.6950.867
WeightBLEU0.4000.4640.5380.631
SyntaxMatch0.4490.4750.5180.581
DataflowMatch0.3950.4710.5860.715
CodeBLEU0.4120.4680.5410.634
CrystalBLEU0.3680.5230.6830.851
CodeBERTScore0.3550.5520.7570.961
CodeScore-R0.1970.2570.3030.371
), ArticleFig(id=1329455915213894219, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Table 11, caption=

Comparison Results of Different Semantic Extraction Methods for CodeScore-R on Code Generation Tasks

, figureFileSmall=null, figureFileBig=null, tableContent=
方法Java代码生成Python代码生成
MAE准确率精度召回率F1-ScoreMAE准确率精度召回率F1-Score
last-avg0.3230.6770.740.8130.7740.3410.6590.7610.7610.761
first-last-avg0.3110.6890.7440.830.7850.3230.6770.7620.7950.778
CLS0.3230.6770.7480.7950.7710.3540.6460.7610.7350.748
CLS+ReLU0.3170.6830.7170.8840.7920.2870.7130.7650.8630.811
), ArticleFig(id=1329455915276808780, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=表11, caption=

不同的语义提取方法在代码生成任务上对CodeScore-R的对比结果

, figureFileSmall=null, figureFileBig=null, tableContent=
方法Java代码生成Python代码生成
MAE准确率精度召回率F1-ScoreMAE准确率精度召回率F1-Score
last-avg0.3230.6770.740.8130.7740.3410.6590.7610.7610.761
first-last-avg0.3110.6890.7440.830.7850.3230.6770.7620.7950.778
CLS0.3230.6770.7480.7950.7710.3540.6460.7610.7350.748
CLS+ReLU0.3170.6830.7170.8840.7920.2870.7130.7650.8630.811
), ArticleFig(id=1329455915343917645, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Table 12, caption=

Comparison Results of Different Semantic Extraction Methods for CodeScore-R on Code Migration Tasks

, figureFileSmall=null, figureFileBig=null, tableContent=
方法Python-to-Java代码迁移Java-to-Python代码迁移
MAE准确率精度召回率F1-ScoreMAE准确率精度召回率F1-Score
last-avg0.1750.8250.8280.9880.9010.1350.8650.8760.9830.926
first-last-avg0.1700.8300.8290.9940.9040.1300.8700.8770.9880.929
CLS0.1800.8200.8240.9880.8980.1400.8600.8760.9770.923
CLS+ReLU0.1750.8250.8211.0000.9020.1250.8750.8780.9940.932
), ArticleFig(id=1329455916946141774, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=表12, caption=

不同的语义提取方法在代码迁移任务上对CodeScore-R的对比结果

, figureFileSmall=null, figureFileBig=null, tableContent=
方法Python-to-Java代码迁移Java-to-Python代码迁移
MAE准确率精度召回率F1-ScoreMAE准确率精度召回率F1-Score
last-avg0.1750.8250.8280.9880.9010.1350.8650.8760.9830.926
first-last-avg0.1700.8300.8290.9940.9040.1300.8700.8770.9880.929
CLS0.1800.8200.8240.9880.8980.1400.8600.8760.9770.923
CLS+ReLU0.1750.8250.8211.0000.9020.1250.8750.8780.9940.932
), ArticleFig(id=1329455917067776591, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Table 13, caption=

Comparison Results of Different Metrics Score on Code Generation Tasks

, figureFileSmall=null, figureFileBig=null, tableContent=
指标Java代码生成Python代码生成
原始词素扰动语法扰动语义扰动原始词素扰动语法扰动语义扰动
BLEU0.4590.2710.4570.4560.3950.2000.4110.417
ROUGE-L0.6550.3850.4780.5040.6180.5500.4160.626
ChrF0.6450.4600.6780.6800.5890.3530.5820.621
ED0.6630.5810.6730.6810.6240.5210.6410.649
WeightBLEU0.5690.3380.5670.5660.4760.2400.4980.498
SyntaxMatch0.6390.6390.6200.5800.5130.4970.5020.507
DataflowMatch0.6270.6300.6220.6120.6220.6210.6180.627
CodeBLEU0.5730.4700.5670.5530.5010.3890.5070.512
CrystalBLEU0.6260.4360.5710.5890.5830.4370.5360.622
CodeBERTScore0.8790.7780.8750.8780.8810.7810.8920.897
CodeScore-R0.8410.8410.8130.3410.8050.8050.8060.349
), ArticleFig(id=1329455917193605712, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=表13, caption=

代码生成任务上不同指标分数对比结果

, figureFileSmall=null, figureFileBig=null, tableContent=
指标Java代码生成Python代码生成
原始词素扰动语法扰动语义扰动原始词素扰动语法扰动语义扰动
BLEU0.4590.2710.4570.4560.3950.2000.4110.417
ROUGE-L0.6550.3850.4780.5040.6180.5500.4160.626
ChrF0.6450.4600.6780.6800.5890.3530.5820.621
ED0.6630.5810.6730.6810.6240.5210.6410.649
WeightBLEU0.5690.3380.5670.5660.4760.2400.4980.498
SyntaxMatch0.6390.6390.6200.5800.5130.4970.5020.507
DataflowMatch0.6270.6300.6220.6120.6220.6210.6180.627
CodeBLEU0.5730.4700.5670.5530.5010.3890.5070.512
CrystalBLEU0.6260.4360.5710.5890.5830.4370.5360.622
CodeBERTScore0.8790.7780.8750.8780.8810.7810.8920.897
CodeScore-R0.8410.8410.8130.3410.8050.8050.8060.349
), ArticleFig(id=1329455917294269009, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Table 14, caption=

Comparison Results of Different Metrics Score on Code Migration Tasks

, figureFileSmall=null, figureFileBig=null, tableContent=
指标Python-to-Java代码迁移Java-to-Python代码迁移
原始词素扰动语法扰动语义扰动原始词素扰动语法扰动语义扰动
BLEU0.7790.3200.7120.7170.6700.1940.6170.607
ROUGE-L0.8900.4890.6570.6790.8910.7400.4810.873
ChrF0.8620.4790.8410.8500.7950.3560.7250.786
ED0.9140.7240.8790.9000.8820.6460.8470.867
WeightBLEU0.8150.3430.7500.7540.6950.2070.6530.631
SyntaxMatch0.7910.7910.7670.7340.5650.5690.5340.555
DataflowMatch0.8830.8500.8750.8640.6940.6700.6820.693
CodeBLEU0.8170.5760.7760.7670.6560.4100.6220.621
CrystalBLEU0.8670.5070.7530.7680.8530.5550.6580.851
CodeBERTScore0.9660.8440.9430.9460.9660.8030.9470.960
CodeScore-R0.9800.9800.9450.4500.9800.9800.9680.371
), ArticleFig(id=1329455917378155090, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=表14, caption=

代码迁移任务上不同指标分数对比结果

, figureFileSmall=null, figureFileBig=null, tableContent=
指标Python-to-Java代码迁移Java-to-Python代码迁移
原始词素扰动语法扰动语义扰动原始词素扰动语法扰动语义扰动
BLEU0.7790.3200.7120.7170.6700.1940.6170.607
ROUGE-L0.8900.4890.6570.6790.8910.7400.4810.873
ChrF0.8620.4790.8410.8500.7950.3560.7250.786
ED0.9140.7240.8790.9000.8820.6460.8470.867
WeightBLEU0.8150.3430.7500.7540.6950.2070.6530.631
SyntaxMatch0.7910.7910.7670.7340.5650.5690.5340.555
DataflowMatch0.8830.8500.8750.8640.6940.6700.6820.693
CodeBLEU0.8170.5760.7760.7670.6560.4100.6220.621
CrystalBLEU0.8670.5070.7530.7680.8530.5550.6580.851
CodeBERTScore0.9660.8440.9430.9460.9660.8030.9470.960
CodeScore-R0.9800.9800.9450.4500.9800.9800.9680.371
), ArticleFig(id=1329455917449458259, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=EN, label=Table 15, caption=

Comparison Results of Different Similarity Thresholds

, figureFileSmall=null, figureFileBig=null, tableContent=
阈值代码生成代码迁移
JavaPythonJavaPython
0.10.3170.2380.1750.125
0.20.3170.2380.1750.125
0.30.3290.2560.1750.125
0.40.3230.2680.1750.125
0.50.3170.2870.1750.125
0.60.3230.3480.1800.135
0.70.3530.4330.1750.140
0.80.3960.4330.1650.165
0.90.4210.5000.2000.260
), ArticleFig(id=1329455917529150036, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, language=CN, label=表15, caption=

不同相似度阈值的对比结果

, figureFileSmall=null, figureFileBig=null, tableContent=
阈值代码生成代码迁移
JavaPythonJavaPython
0.10.3170.2380.1750.125
0.20.3170.2380.1750.125
0.30.3290.2560.1750.125
0.40.3230.2680.1750.125
0.50.3170.2870.1750.125
0.60.3230.3480.1800.135
0.70.3530.4330.1750.140
0.80.3960.4330.1650.165
0.90.4210.5000.2000.260
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202330715, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202330715, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202330715, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202330715, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1706803200000, fullTextJson=null, articleText=null, reference=null)
收藏切换
CodeScore-R:用于评估代码合成功能准确性的自动化鲁棒指标
杨光 1 , 周宇 1, * , 陈翔 2 , 张翔宇 1
计算机研究与发展 | 生成式AI与软件自动化专题 2024,61(2): 291-306
收起
收藏切换
计算机研究与发展 |生成式AI与软件自动化专题 2024 , 61 (2) : 291 -306
CodeScore-R:用于评估代码合成功能准确性的自动化鲁棒指标
全屏
[Author(id=1329455909694190094, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=yang.guang@nuaa.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329455909752910352, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, authorId=1329455909694190094, language=EN, stringName=Guang Yang, firstName=Guang, middleName=null, lastName=Yang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1College of Computer Science and Technology/College of Artificial Intelligence/College of Software, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, bio={"content":"

born in 1997. PhD candidate. Student Member of CCF. His main research interests include code generation, code translation, and robustness research in code synthesis

"}, bioImg=null, bioContent=

born in 1997. PhD candidate. Student Member of CCF. His main research interests include code generation, code translation, and robustness research in code synthesis

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329455909815824913, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, authorId=1329455909694190094, language=CN, stringName=杨光, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1南京航空航天大学计算机科学与技术学院/人工智能学院/软件学院 南京 211106, bio={"img":"hTaOpXwIaigQWB2WlwsylA==","content":"

1997年生,博士研究生.CCF学生会员. 主要研究方向为代码生成、代码迁移、代码合成中的鲁棒性研究

"}, bioImg=hTaOpXwIaigQWB2WlwsylA==, bioContent=

1997年生,博士研究生.CCF学生会员. 主要研究方向为代码生成、代码迁移、代码合成中的鲁棒性研究

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329455909505446407, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, xref=1, ext=[AuthorCompanyExt(id=1329455909518029320, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909505446407, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1College of Computer Science and Technology/College of Artificial Intelligence/College of Software, Nanjing University of Aeronautics and Astronautics, Nanjing 211106), AuthorCompanyExt(id=1329455909526417929, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909505446407, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1南京航空航天大学计算机科学与技术学院/人工智能学院/软件学院 南京 211106)])]), Author(id=1329455909870350867, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=zhouyu@nuaa.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329455909937459733, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, authorId=1329455909870350867, language=EN, stringName=Yu Zhou, firstName=Yu, middleName=null, lastName=Zhou, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, *, address=1College of Computer Science and Technology/College of Artificial Intelligence/College of Software, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, bio={"content":"

born in 1981. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include software evolution analysis, mining software repositories, software architecture, and reliability analysis

"}, bioImg=null, bioContent=

born in 1981. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include software evolution analysis, mining software repositories, software architecture, and reliability analysis

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329455910000374294, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, authorId=1329455909870350867, language=CN, stringName=周宇, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, *, address=1南京航空航天大学计算机科学与技术学院/人工智能学院/软件学院 南京 211106, bio={"img":"ZTQ2scsjnI5M7Gx4hWh1YA==","content":"

1981年生. 博士,教授,博士生导师. CCF高级会员. 主要研究方向为软件演化分析、挖掘软件仓库、软件架构、可靠性分析

"}, bioImg=ZTQ2scsjnI5M7Gx4hWh1YA==, bioContent=

1981年生. 博士,教授,博士生导师. CCF高级会员. 主要研究方向为软件演化分析、挖掘软件仓库、软件架构、可靠性分析

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329455909505446407, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, xref=1, ext=[AuthorCompanyExt(id=1329455909518029320, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909505446407, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1College of Computer Science and Technology/College of Artificial Intelligence/College of Software, Nanjing University of Aeronautics and Astronautics, Nanjing 211106), AuthorCompanyExt(id=1329455909526417929, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909505446407, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1南京航空航天大学计算机科学与技术学院/人工智能学院/软件学院 南京 211106)])]), Author(id=1329455910067483160, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329455910147174938, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, authorId=1329455910067483160, language=EN, stringName=Xiang Chen, firstName=Xiang, middleName=null, lastName=Chen, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2School of Information Science and Technology, Nantong University, Nantong 226019, bio={"content":"

born in 1980. PhD, associate professor. Member of CCF. His main research interests include software defect prediction, software defect location, regression testing, and combination testing

"}, bioImg=null, bioContent=

born in 1980. PhD, associate professor. Member of CCF. His main research interests include software defect prediction, software defect location, regression testing, and combination testing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329455910222672411, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, authorId=1329455910067483160, language=CN, stringName=陈翔, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2南通大学信息科学技术学院 江苏南通 226019, bio={"img":"h0uP8aCYKrwOP2ZxjzqIEg==","content":"

1980年生. 博士,副教授. CCF会员. 主要研究方向为软件缺陷预测、软件缺陷定位、回归测试、组合测试

"}, bioImg=h0uP8aCYKrwOP2ZxjzqIEg==, bioContent=

1980年生. 博士,副教授. CCF会员. 主要研究方向为软件缺陷预测、软件缺陷定位、回归测试、组合测试

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329455909601915402, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, xref=2, ext=[AuthorCompanyExt(id=1329455909610304011, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909601915402, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Information Science and Technology, Nantong University, Nantong 226019), AuthorCompanyExt(id=1329455909614498316, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909601915402, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2南通大学信息科学技术学院 江苏南通 226019)])]), Author(id=1329455910293975581, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329455910369473055, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, authorId=1329455910293975581, language=EN, stringName=Xiangyu Zhang, firstName=Xiangyu, middleName=null, lastName=Zhang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1College of Computer Science and Technology/College of Artificial Intelligence/College of Software, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, bio={"content":"

born in 1999. Master candidate. His main research interests include code generation and code pre-trained model

"}, bioImg=null, bioContent=

born in 1999. Master candidate. His main research interests include code generation and code pre-trained model

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329455910428193312, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, authorId=1329455910293975581, language=CN, stringName=张翔宇, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1南京航空航天大学计算机科学与技术学院/人工智能学院/软件学院 南京 211106, bio={"img":"Y10r9fosbnliLKdhZslTmw==","content":"

1999年生. 硕士研究生. 主要研究方向为代码生成、代码预训练模型

"}, bioImg=Y10r9fosbnliLKdhZslTmw==, bioContent=

1999年生. 硕士研究生. 主要研究方向为代码生成、代码预训练模型

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329455909505446407, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, xref=1, ext=[AuthorCompanyExt(id=1329455909518029320, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909505446407, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1College of Computer Science and Technology/College of Artificial Intelligence/College of Software, Nanjing University of Aeronautics and Astronautics, Nanjing 211106), AuthorCompanyExt(id=1329455909526417929, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329455905067872752, companyId=1329455909505446407, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1南京航空航天大学计算机科学与技术学院/人工智能学院/软件学院 南京 211106)])])]
杨光1 , 周宇1, * , 陈翔2, 张翔宇1
作者信息
  • 1南京航空航天大学计算机科学与技术学院/人工智能学院/软件学院 南京 211106
  • 2南通大学信息科学技术学院 江苏南通 226019
通讯作者:
作者简介:

1997年生,博士研究生.CCF学生会员. 主要研究方向为代码生成、代码迁移、代码合成中的鲁棒性研究

1981年生. 博士,教授,博士生导师. CCF高级会员. 主要研究方向为软件演化分析、挖掘软件仓库、软件架构、可靠性分析

1980年生. 博士,副教授. CCF会员. 主要研究方向为软件缺陷预测、软件缺陷定位、回归测试、组合测试

1999年生. 硕士研究生. 主要研究方向为代码生成、代码预训练模型

CodeScore-R: An Automated Robustness Metric for Assessing the Functional Correctness of Code Synthesis
Guang Yang1 , Yu Zhou1, * , Xiang Chen2, Xiangyu Zhang1
Affiliations
  • 1College of Computer Science and Technology/College of Artificial Intelligence/College of Software, Nanjing University of Aeronautics and Astronautics, Nanjing 211106
  • 2School of Information Science and Technology, Nantong University, Nantong 226019
  • born in 1997. PhD candidate. Student Member of CCF. His main research interests include code generation, code translation, and robustness research in code synthesis

    born in 1981. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include software evolution analysis, mining software repositories, software architecture, and reliability analysis

    born in 1980. PhD, associate professor. Member of CCF. His main research interests include software defect prediction, software defect location, regression testing, and combination testing

    born in 1999. Master candidate. His main research interests include code generation and code pre-trained model

出版时间: 2024-02-02 doi: 10.7544/issn1000-1239.202330715
文章导航
收藏切换

评估指标在代码合成领域中至关重要. 常用的代码评估指标可以分为3种类型:基于匹配、基于语义和基于执行. 其中,基于执行的Pass@k指标通过执行测试用例,能够准确判断预测代码的功能准确性. 然而,该指标的计算需要大量开销,因此亟需设计一种自动化评估指标,在无需测试用例时仍可评估预测代码的功能准确性. 此外,好的评估指标应当具有鲁棒性,即预测代码发生微小改变时,评估指标仍能保持其准确性. 为此,提出了一种基于UniXcoder和对比学习的自动化鲁棒指标CodeScore-R,用于评估代码合成的功能准确性. CodeScore-R采用草图化处理、语法等价转换和变异测试等技术手段,有效减轻了标识符、语法结构和运算符对评估结果的干扰. 实验结果表明,在Java和Python语言上的代码生成和迁移任务中,CodeScore-R的表现优于其他无需测试用例的评估指标,且更接近Pass@k指标,并具有更强的鲁棒性.

代码合成评估指标  /  功能准确性  /  鲁棒性  /  代码合成  /  神经网络

Evaluation metrics are crucial in the field of code synthesis. Commonly used code evaluation metrics can be classified into three types: match-based, semantic-based, and execution-based. Among them, the execution-based Pass@k metric accurately assesses the functionality of predicted code by executing test cases. However, calculating this metric requires a significant amount of overhead, necessitating the design of an automated evaluation metric that can assess the functionality of predicted code without the need for test cases. Additionally, a good evaluation metric should be robust, that is the metric can maintain its accuracy even when the predicted code undergoes minor changes. To address these challenges, we propose an automated robust metric, called CodeScore-R, based on UniXcoder and contrastive learning, for evaluating the functionality of code synthesis. CodeScore-R employs techniques such as sketch-based processing, syntactic-equivalent transformations, and mutation testing to effectively mitigate the interference caused by identifiers, syntax structures, and operators on evaluation results. Experimental results demonstrate that in the tasks of code generation and migration in Java and Python, CodeScore-R outperforms other evaluation metrics and is more closely aligned with the Pass@k metric, while exhibiting stronger robustness.

code synthesis evaluation metric  /  functional correctness  /  robustness  /  code synthesis  /  neural network
杨光, 周宇, 陈翔, 张翔宇. CodeScore-R:用于评估代码合成功能准确性的自动化鲁棒指标. 计算机研究与发展, 2024 , 61 (2) : 291 -306 . DOI: 10.7544/issn1000-1239.202330715
Guang Yang, Yu Zhou, Xiang Chen, Xiangyu Zhang. CodeScore-R: An Automated Robustness Metric for Assessing the Functional Correctness of Code Synthesis[J]. Journal of Computer Research and Development, 2024 , 61 (2) : 291 -306 . DOI: 10.7544/issn1000-1239.202330715
代码合成是一项自动化程序开发技术,旨在根据给定的规范和约束条件生成代码[1]. 根据输入的模态,Ren等人[2]将代码合成分为文本到代码 (text-to-code)和代码到代码 (code-to-code)这2种类型[3]. 其中,文本到代码是将自然语言描述转换为可执行代码的过程,例如代码生成[4];而代码到代码则是将现有代码转换为满足新需求的代码,例如代码迁移[5]. 这2种技术都可以提高软件开发效率和质量,在实际应用中都具有重要的价值. 此外,代码合成还可以为程序员提供一种更加友好的编程方式[6],并吸引那些在特定编程语言上经验有限的程序员新手参与软件开发. 因此,代码合成对软件开发行业具有重要意义,并在其发展过程中变得越来越重要.
近年来,随着生成式AI模型[7]的快速迭代,代码合成领域取得显著的发展,包括各种强大模型 (例如CodeT5[8],CodeGen[9]和CodeGeeX[10])的出现和多种数据集(例如Lyra[11],MBPP[12] 和CONCODE[13])的构建. 尽管有这些令人振奋的进展,但代码合成的评价指标在现有的研究中受到的关注有限. 评估指标在代码合成领域起着至关重要的作用,它们可以为研究人员和开发人员提供重要的指导和支持,帮助改进模型性能. 目前,常用的代码评估指标可以分为3类:基于匹配、基于语义和基于执行的指标. 基于匹配的指标,例如BLEU[14]和ChrF[15],将代码视为文本,并仅关注词素 (token)级别的匹配分数. 基于语义的指标,例如CodeBLEU[2],考虑到代码的句法结构和数据流信息可作为代码的语义信息,并将代码与词素信息混合计算. 基于执行的指标,例如Pass@k[16],通过人工编写测试用例,并根据代码能否通过测试用例来评估模型的性能. 其中,基于执行的Pass@k指标能更精准地判断预测代码的功能准确性,因此,本文以Pass@k指标作为标准的功能准确性. 但该指标在实际评估时,需要大量的人力、时间和成本开销,这在实际应用场景中并不总是可行的. 因此,亟需设计一种自动化评估指标,在无需测试用例时仍可评估预测代码的功能准确性.
此外,一个好的评估指标应该具备鲁棒性,即当预测代码发生微小改变时,评估指标仍能保持准确性. 然而现有的评估指标并不鲁棒,如图1所示,只有当预测代码 b与参考代码 a完全一致时,现有的指标才能给出正确的分数. 然而,当代码中的用户标识符被修改或对代码进行了等价的语法转换时,即预测代码 c和代码 d,此时代码 c、代码 d与参考代码 a的功能是一致的,但现有的指标无法识别它们,特别是BLEU指标,认为代码 c与参考代码 a的相似度仅为0.040. 另外,即使对代码进行微小的语义改变,即预测代码 e,此时代码 e与参考代码 a的功能并不一致,但现有的评估指标仍无法正确识别它们,这种不鲁棒问题限制了现有评估指标的有效性. 因此,本文对代码合成评估指标的鲁棒性检验提出了3种假设:
假设1. 当生成的代码中发生用户自定义标识符替换时,指标分数应基本不变;
假设2. 当生成的代码中发生语法等价转换时,指标分数应基本不变;
假设3. 当对语义正确的代码进行语义变异时,原本得分高的指标应该大幅度降低.
假设1~3旨在评估现有评估指标在应对代码改变时的鲁棒性,并为改进和开发更可靠的评估指标提供指导.
近年来,评估代码合成的一个流行趋势是设计基于代码的预训练语言模型[17](pre-trained language models of code,CodePTMs). 相较于依赖词素层面的评估指标,通过提取代码内在的语义向量进行评估被认为是一个具有潜力的方向[18]. 本文提出了一种鲁棒的评估指标CodeScore-R,它利用代码预训练语言模型UniXcoder[19]来评估代码合成的功能准确性. 为了减轻用户标识符命名风格对指标的干扰,本文提出了一种基于抽象语法树 (abstract syntax tree,AST)的代码草图化处理方法,该方法从代码中提取用户自定义标识符,并用统一的占位符进行替换. 为了提升指标评估代码合成的功能准确性和鲁棒性,本文提出了针对代码的对比学习框架ConCE (contrastive learning of code embedding). 该框架通过最小化正样本之间的语义距离和最大化负样本之间的语义距离来学习代码的表征. 通过语义向量之间的余弦相似度,可以评估代码合成的功能准确性. 在构造正样本方面,CodeScore-R通过进行语法等价转换来生成代码变体;而对于负样本,则通过利用变异测试中的运算符变异来对代码进行微小的改变.
为评估CodeScore-R的有效性,本文在代码生成和代码迁移任务上进行实验. 在代码生成任务中,使用带有测试用例的HumanEval[20]和HumanEval-X[10]数据集,而在代码翻译任务中,本文使用AVATAR[21]数据集,并通过众包的方式手工构造了相应的测试用例. 实验结果表明,对于Java与Python语言上的代码生成和迁移任务中,CodeScore-R的表现优于现有评估指标,可以更接近Pass@k指标,并具有更强的鲁棒性.
本文的主要贡献有3点:
1) 提出一种基于UniXcoder和对比学习的鲁棒指标CodeScore-R,用于自动化评估代码合成功能的准确性.
2) 引入草图化处理、语法等价转换和变异测试等技术,以减轻标识符、语法结构和运算符对评估结果的扰动.
3) 在Java和Python语言上的代码生成和迁移任务中,CodeScore-R的表现优于现有评估指标,并具有更强的鲁棒性.
当前已经有部分研究专注于不同的代码合成评估指标,这些指标对于衡量代码合成系统生成的代码质量至关重要. Liguori等人[22]对现有代码合成评估指标和人工评估在生成攻击性代码 (offensive code)的相似性进行了全面分析,他们的实验结果指出现有的指标通常不能提供准确的评估结果. 目前,自动化代码评估指标主要可以分为3类:基于匹配、基于语义和基于执行的指标. 此外,一些研究还通过人工评估的方法对预测代码进行分析.
基于匹配的评估指标主要局限于对词素粒度的相似性计算,而忽略了代码的潜在语义信息. 这类指标源自机器翻译、文本摘要等领域,包括BLEU[14],Rouge[23],和ChrF[15]等指标. 它们通过比较参考文本和预测文本之间的n-gram匹配程度来计算分数. 此外,精准匹配度 (exact match,EM)指标也被广泛应用于代码合成任务[24]中.
然而,Eghbali等人[25]指出,由于编程语言冗长的语法和编码约定,即使是完全不相关的代码片段也可能存在许多共同的n-grams信息. 例如,在Java代码中,会出现大量的括号与分号的组合,这些冗余的信息可能导致BLEU指标的结果虚高;为了提高评估指标的准确性和可区分性,提出了CrystalBLEU指标,该指标在BLEU指标的基础上移除出现频率最高的n-grams信息. 此外,Liguori等人[22]认为,与其他基于匹配的指标相比,编辑距离(edit distance,ED)能够更好地衡量代码之间的相似性.
基于语义的评估指标考虑了代码的语法结构、数据流信息和潜在语义信息,但是仍不够鲁棒. Ren等人[2]认为基于匹配的指标忽略了代码中的句法和语义特征,而EM指标过于严苛,无法识别出语义相同但语法不同的样本. 为此,他们提出了CodeBLEU指标,该指标根据代码中的关键词进一步改进BLEU指标,并进一步通过AST注入代码语法和通过数据流注入代码语义.
Dong等人[26]则基于CodePTM提出CodeScore指标,在带有测试用例的数据集上进行有监督学习,以此进行代码合成的功能性评估. 然而,这种方法在训练时依赖于带有测试用例的数据集,成本较高. 另一方面,Zhou等人[27]利用CodePTM提取代码的潜在语义. 具体而言,他们提出了CodeBERTScore,利用CodeBERT模型[28]在大规模数据集上进行自监督学习,再对参考代码和预测代码进行上下文编码,计算每个词素之间的相似性分数.
基于执行的评估指标能准确判断预测代码的功能准确性,但需要投入大量的成本,包括测试用例的构造、解决运行环境的依赖问题,以及耗费大量时间进行测试用例的运行.
Kulal等人[16]首次提出了基于测试用例的评估指标Pass@k,为每个问题生成了K个代码样本,并评估了K个样本中任意一个样本能够通过单元测试集的比率. 另一方面,Liang等人[11]则提出了Code Executable指标,用于判断预测代码是否能通过编译器的编译,该指标旨在没有测试用例的情况下观察预测代码的质量.
除了自动化评估指标外,人工评估也是一种可靠的方法,可以全面评估预测代码的准确性、语法、语义、复杂性、可读性、安全性等方面. 然而,对每个编程问题进行手动评估是不切实际的,因为这需要大量的时间和成本.
因此,Yang等人[29]提出了基于采样的方法,并将该方法应用于小规模实证研究中. 这种方法可以在可承受的时间和资源范围内获取对预测代码的高质量评估.
在代码合成任务中,通常面临以下问题:给定一个上下文x(例如,自然语言功能描述或者源代码),将x输入到代码合成模型M中,生成一个预测代码片段$\hat y $,然后使用自动化评估指标来评估预测代码的质量. 评估通常使用度量函数$f(\hat y,\;y^*)$来比较预测代码$\hat y $与参考代码$ y^* $.
$f(\hat y,\;y^*) $的值越大,表示预测代码与参考代码在功能准确性上越相似. 在理想条件下,该度量函数应具备良好的鲁棒性,即满足引言提出的3个假设. 具体来说,当预测代码$\hat y_1 $与参考代码$ y^* $之间语义一致,但词素不同(预测代码记为$\hat y_2 $)或者语法上的实现不同(预测代码记为$\hat y_3 $)时,评估指标计算的分数应该是一致的;当预测代码$\hat y_4 $与参考代码$ y^* $之间词素上相似但语义不一致时,评估指标应能正确识别出差异. 因此,本文希望提出一个满足$f (\hat y_1,\;y^*) \approx f(\hat y_2,\;y^* ) \approx f(\hat y_3,\;y^*) \gg f (\hat y_4,\;y^* )$的度量函数$f(\;)$.
UniXcoder[19]是一种基于Transformer[30]的代码预训练语言模型,它在多项编程任务中表现出色. 该模型使用CodeSearchNet数据集[31]进行了预训练,该数据集包含了来自6种编程语言(Ruby,Java,Python,PHP,Go和Javascript)的约2.3×107个代码样本. UniXcoder通过使用3种不同类型的自注意力掩码策略来控制模型的行为,同时兼容编码、解码和编解码3种模式. 除了基本的预训练任务,如掩码语言模型(masked language modeling,MLM),还提出了一些新的预训练任务,包括统一语言建模、去噪任务和代码片段表征学习等.
与其他CodePTMs相比,UniXcoder能更全面地利用AST提供的代码结构信息,从而弥补了AST和词素表示的差异. 此外,UniXcoder还通过对比学习来学习代码片段的表征,使其在计算代码之间的相似度任务中更加适用.
本节主要介绍CodeScore-R指标的设计方法及具体实现,该指标的整体框架如图2所示. CodeScore-R以UniXcoder为基座模型,并采用本文提出的ConCE框架进行自监督的代码表征学习. 通过利用经过表征学习的UniXcoder模型,CodeScore-R能够对生成的代码进行评估计算. 接下来将介绍CodeScore-R的具体实现细节.
ConCE框架由数据预处理、正负样本构造以及表征学习任务3部分组成.
为了降低不同标识符命名风格对评估指标的干扰,即为了满足假设1,CodeScore-R在数据预处理阶段对代码片段进行处理. 首先,CodeScore-R将代码片段解析为抽象语法树(abstract syntax tree,AST),并根据预定义的规则识别出代码中用户自定义的标识符,例如函数名、参数名和变量名. 然后,采用草图化的方法统一变量命名风格. 图3展示了预处理前后的对比示例图. 由于参数、变量及函数名的命名与程序员的编码习惯相关,这可能会对评估指标的计算产生干扰. 但是,对参数、变量及函数名进行草图化处理并不会影响代码的语法和语义信息.
具体来说,CodeScore-R利用tree-sitter工具1对代码片段进行解析,并根据每个类型为identifier的叶子节点的父节点进行判断,以识别出函数名、参数名和变量名的词素信息. 草图化是将这些词素以统一的编码方式进行逐一替换. 例如,函数名替换为“f”,参数名和变量名分别按照先后顺序替换为{arg_0, arg_1, …, arg_m}和{var_0, var_1, …, var_n}. 在后续的正负样本构造及表征学习中,CodeScore-R都是使用草图代码(sketch code)而不是原始代码(origin code)进行实验,以尽可能地避免标识符词素对评估指标的影响.
为了进行后续的表征学习,CodeScore-R需要构造正负样本. 对于正样本的构造,CodeScore-R借鉴了SimCSE[32]中的思想,一方面采用2次不同的dropout方法生成正样本,以增强其对代码语义的理解和鲁棒性. 另一方面,本文根据代码的语法特性,对代码进行语法上等价的转换,生成与原始代码功能相同但形式不同的代码片段. 这种转换可以包括改变代码中的控制流结构、使用不同的语法特性等. 通过这样的等价转换,可以扩展训练数据集,提供更多样化的正样本供CodeScore-R进行学习. 为了确保生成语法等价的代码质量,选择了代码重构[33]中常见的4种规则进行操作,具体规则见表1.
循环交换是将for循环替换为等效的while循环或将while循环替换为等效的for循环. 这样的交换可以改变循环结构的表达方式,但保持了代码功能的等效性. 表达式交换是基于运算符特性进行等价替换,例如将“a+=b”替换为“a=a+b”. 这种交换在语义上保持了表达式的等效性,但改变了表达式的形式. 判断体交换和判断条件交换是常见的代码重构规则,用于条件语句的等价转换. 判断体交换将条件语句中的if分支和else分支进行交换,而判断条件交换交换条件表达式中的2个子表达式. 这样的交换保持了条件语句的等效性,但改变了条件语句的结构或条件的顺序.
对于负样本的构造,CodeScore-R采用了变异测试的思想. 变异测试[34]是一种软件测试方法,通过对原始代码进行变异 (如改变运算符、改变变量引用等),生成具有细微差异的代码,从而检验测试用例的充分性和代码的健壮性. 在构造负样本时,CodeScore-R也使用了类似的方法. 它通过对原始代码进行运算符的变异操作,例如将加法操作变异为减法操作或将逻辑运算符变异为移位运算符等,可以生成与原始代码形式上相似但在语义上存在不同的负样本,具体变异规则如表2所示. 通过引入这样的负样本,CodeScore-R能够区分细微差异的代码,并提升对代码变异的鲁棒性.
为了进行表征学习,CodeScore-R采用了2个自监督预训练任务,分别是掩码语言模型和对比学习,如图4所示. 这些任务旨在让模型学习代码的语义信息,并能够区分代码之间的微小差异.
1)掩码语言模型. 掩码语言模型的任务是最大化正确预测被屏蔽词素的概率,以使CodeScore-R能够学习到代码的语义表示. 在数据预处理阶段,由于对代码进行了草图化处理,导致草图代码的词素失去了原始代码中部分语义信息. 为了确保模型能够学习到完整的语义信息,本文引入了代码对应的注释NL,并随机选择草图代码中15%的词素进行掩码操作. 具体而言,该任务以一定的概率对选定的词素进行替换,其中,以80%的概率将词素直接替换为特殊标记“<mask>”,以10%的概率将词素随机替换为其他词素,而剩下的以10%的概率保持词素不变. 通过这种方式,模型需要预测被掩码的词素,从而学习到代码的语义信息.
给定一个NL-code对的数据 (d ={w, c})作为输入,其中w是NL的序列,c是代码词素的序列,可以将掩码操作定义为:
$ m_i^{{c}}\sim{{\mathrm{unif}}} \{ 1,|c|\}\;{\mathrm{ for }}\;i = 1 \;{\mathrm{to}}\; |c|. $
掩码语言模型的目标是预测被屏蔽掉的原始标记,可以定义为:
$ {\mathcal{L}_{{\text{MLM}}}}(\theta ) = \sum\limits_{i \in m_i^c} - \log p\left( {{d_i}\mid {c^{{\text{masked }}}}} \right) .$
2)对比学习. 对比学习任务旨在使模型能够区分正样本和负样本,以增强对微小差异的敏感性. 对于正样本的构造,本文采用3.1.2节提出的2种方法:50%的概率通过2次dropout得到正样本和50%的概率通过代码语法等价转换得到正样本. 这样的设计既避免了模型对输入长度的敏感性问题,又增加了模型对不同语法的识别能力,提高了模型的语法鲁棒性. 对于负样本的构造,同样采用3.1.2节提出的基于变异测试的方法. 考虑到正、负样本的构造都是基于多种规则的,为了提升样本的多样性,本文使用组合的方式将这些规则进行融合,再从中随机抽选出正样本和负样本.
给定一个代码片段ci,我们将ci+记为正样本,将ci−记为负样本,一起组合为 (ci,ci+,ci−),利用UniXcoder对它们进行语义向量提取,得到向量 hi,hi+,hi−. 具体的提取方法是先对输入的代码进行上下文编码得到其隐藏向量$ {\boldsymbol{H}} \in {\mathbb{R}^{batch \times N \times d}} $,其中N表示为输入代码的长度,d表示为向量的维度. 接着提取第一个词素,即[CLS]的向量值,并经过一层ReLU层进行激活,最终作为代码的语义表征h. 对比学习任务的损失函数定义为:
$ {\mathcal{L}_{{\mathrm{CL}}}}(\theta ) = - \log \frac{{{e^{{{\mathrm{sim}}} \left( {{{\boldsymbol{h}}_i},{\boldsymbol{h}}_i^ + } \right)/\tau }}}}{{\displaystyle\sum\limits_{j = 1}^{batch} {\left( {{e^{{{\mathrm{sim}}} \left( {{{\boldsymbol{h}}_i},{\boldsymbol{h}}_j^ + } \right)/\tau }} + {e^{{{\mathrm{sim}}} \left( {{{\boldsymbol{h}}_i},{\boldsymbol{h}}_j^ - } \right)/\tau }}} \right)} }}\text{,} $
其中batch代表批次的大小,sim代表余弦相似度. 最终,ConCE框架的优化目标可设置为:
$ \mathcal{L}(\theta ) = {\mathcal{L}_{{\text{MLM}}}}(\theta ) + {\mathcal{L}_{{\mathrm{CL}}}}(\theta ) .$
CodeScore-R的评估流程为:首先,以预测代码和参考代码作为输入. 为了避免预测代码包含语法错误,CodeScore-R借助编译器判断预测代码是否能够编译通过. 如果预测代码无法通过编译,则直接输出的功能准确性分数为0. 如果预测代码通过了编译,则对预测代码和参考代码进行草图化处理,以减轻标识符对评估结果的影响. 接着,CodeScore-R借助经过表征学习后的UniXcoder,分别提取预测代码和参考代码的语义表征向量,并进行余弦相似度计算. 通常来说,余弦相似度计算结果的值域为[−1, 1]. 然而,由于在语义表征向量提取过程中使用了ReLU激活函数,语义表征向量保持非负值不变. 因此,经过ReLU激活后的语义表征向量中的每个元素都是非负的.
余弦相似度的计算公式为:
$ {{{cosine}}}\_{{{similarity}}}(\boldsymbol{x},\boldsymbol{y})=\frac{\boldsymbol{x}\cdot \boldsymbol{y}}{\Vert {{\boldsymbol{x}}}\Vert \cdot \Vert {{\boldsymbol{y}}}\Vert } \text{,} $
其中,$\boldsymbol{x}\cdot \boldsymbol{y} $表示向量x和y的内积,$\left\|{\boldsymbol{x}}\right\| $和$\left\|{\boldsymbol{y}}\right\| $分别表示向量x和y的范数. 由于语义表征向量中的元素都是非负的,内积和范数的结果也是非负的. 因此,余弦相似度的分子是非负的内积,分母是非负的范数乘积,所以余弦相似度的结果也是非负的,即值域为[0, 1].
为了更好地拟合Pass@k指标,CodeScore-R加入了后处理操作,即对相似度分数进行二值化处理,最终输出预测代码的功能准确性分数:若相似度分数大于0.5,输出为1;否则输出为0.
为了验证本文提出的CodeScore-R的有效性,本文设计了5个实验问题.
问题1:与现有指标相比,CodeScore-R对功能准确性的拟合程度如何?
问题2:与现有指标相比,CodeScore-R针对代码标识符扰动下的鲁棒性如何?
问题3:与现有指标相比,CodeScore-R针对代码语法扰动下的鲁棒性如何?
问题4:与现有指标相比,CodeScore-R针对代码语义扰动下的鲁棒性如何?
问题5:不同的语义提取方法对CodeScore-R有什么影响?
本文以CodeSearchNet数据集[31]作为表征学习的数据集,具体来说,我们提取其中的Java代码和Python代码以及它们相应的功能注释,并对它们进行预处理和正、负样本的构造. 由于不是每个代码都能根据规则转换为相应的变体代码,因此过滤出那些无法构造正、负样本的代码以作为最终的预训练数据集.
对于下游任务,本文在2种编程语言Java和Python任务上评估了CodeScore-R,包括代码生成任务和代码迁移任务.
代码生成任务是根据用户的功能描述加上代码签名(signature)来生成相应的代码. 本文使用由Python语言构造的HumanEval数据集[20]和包含了Java语言的HumanEval-X数据集[10]作为实验对象,这些数据集由包含了164条带有测试用例的样本构成.
代码迁移任务是根据一种编程语言的源代码生成另一种编程语言的目标代码. 本文使用AVATAR数据集[21],其中包含了挖掘自在线编程网站上的Java和Python的代码对. 我们从中随机挑选了200对数据并手工构造测试用例,为了保证测试用例的质量,我们通过众包的方式聘请具有2~3年开发经验的工程师为每个代码段构造5条测试用例以确保测试用例的多样性.
本文选择10种代码合成评估的基准指标,包括BLEU[14],ROUGE-L[23],ChrF[15],ED[22],WeightBLEU[2],SyntaxMatch[2],DataflowMatch[2],CodeBLEU[2],CrystalBLEU[25],和CodeBERTScore[27]. 这些指标被广泛使用在代码合成的相关研究中. 其中,BLEU,ROUGE-L,ChrF,ED和CrystalBLEU是基于匹配的评估指标;CodeBLEU和CodeBERTScore是基于语义的评估指标;WeightBLEU,SyntaxMatch,DataflowMatch是CodeBLEU计算过程中设计出的指标,分别对应改进的BLEU、语法匹配与数据流匹配的计算.
为了评估现有评估指标和代码功能准确性之间的拟合程度,现有的研究通常选择相关系数指标[35] (如Pearson相关系数、Spearman相关系数和Kendall相关系数)和拟合指标[36] (如平均绝对误差). 其中,相关系数是用来衡量2个变量之间线性相关程度的指标,而拟合指标则是用来衡量模型的拟合程度的指标.
本文的目标是检验CodeScore-R能否自动评估预测代码的功能准确性,即是否能拟合Pass@1指标的结果,因此本文选择平均绝对误差(mean absolute error,MAE)来评估预测的结果$M_i^1 $与实际Pass@1结果$M_i^2 $之间的绝对误差. MAE值越大,说明拟合程度越差;反之则说明拟合程度越好. MAE可以定义为:
$ {{MAE}} = \frac{1}{N}{{\displaystyle\sum\limits_{i = 1}^N {\left| {M_i^1 - M_i^2} \right|} }}. $
在后续的实证研究中,所有实验均运行在Windows 10操作系统上,其内存为32 GB,CPU处理器为Intel Core i7-9750H. 对于编译器,本文使用Python 3.9.1版本2作为Python代码的编译器,并使用JDK 18.0.2.1版本3编译Java代码.
随着语言模型和训练数据的规模不断增长,基于大型语言模型的生成式AI表现出各种涌现行为. 其中一种能力是零样本学习,它允许模型在特定指令或提示中回答. 这些生成式AI模型取得了出色的性能,并在代码合成任务上也展示了广阔的潜力. 因此,本文通过调用ChatGPT的接口4 (gpt-3.5-turbo)完成代码生成与代码迁移任务. 为了保证结果的稳定性,本文将接口中的temperature参数设置为0,并只返回1个代码结果,用于计算Pass@1指标.
经测试,ChatGPT在Java代码生成任务上的Pass@1指标为68.29%,在Python代码生成任务上的Pass@1指标为71.34%;在Python-to-Java代码合成任务上的Pass@1指标为80.5%,在Java-to-Python代码合成任务上的Pass@1指标为86.5%.
为了评估CodeScore-R的拟合程度,本文选用了机器翻译领域以及代码合成领域中常用的指标作为基准评估指标. 表3给出了CodeScore-R和选择的基准方法与功能准确性之间的MAE拟合结果.
表3显示,与所有基准指标相比,CodeScore-R在所选的4个下游任务数据集上对功能准确性的拟合更好. 在Java代码生成、Python代码生成、Python-to-Java代码迁移和Java-to-Python代码迁移任务中,与最优的基准指标相比,CodeScore-R的拟合误差分别降低了7.31%、11.42%、13.37%和17.22%. 其中,在代码迁移任务上的拟合程度普遍优于在代码生成任务上的拟合程度;在Python编程语言上的拟合程度普遍优于在Java编程语言上的拟合程度.
此外,表3的结果指出,基于大语言模型(large language model,LLM)的评估指标普遍优于其他的基准指标,这也体现出该类指标的优势和可行性. 在基于匹配的基准指标中,ED的拟合结果最优,这与文献[22]的研究结论一致,而在基于语义的基准指标中,CodeBERTScore则表现出最好的拟合结果.
考虑到CodeScore-R输出的是0~1的值,因此相较于其他指标,我们可以将CodeScore-R的输出与功能准确性视作分类任务,通过比较CodeScore-R的输出与功能准确性之间的准确率、精度、召回率和F1-Score以进一步评估CodeScore-R的性能. 同时可以分析表3结果中的MAE指标是否存在假阳性或者假阴性数据过多而导致的误差虚低的情况. 结果如图5所示,在代码生成任务上,CodeScore-R的F1-Score值在0.8左右,而在代码迁移任务上,CodeScore-R的F1-Score值在0.9以上,该结果可以进一步表明CodeScore-R的准确性.
基于问题1中的分析结果,我们发现CodeScore-R在所有任务中对功能准确性的拟合程度超过了本文考虑的10种基准指标. 然而,一个好的评估指标应具备鲁棒性. 为了研究CodeScore-R与现有基准指标在代码标识符扰动下的鲁棒性(即是否满足假设1),以及我们提出的代码草图化处理的有效性,我们进行了2组对比实验:针对Origin-to-Sketch (O2S)的对比实验和针对Sketch-to-Sketch (S2S)的对比实验.
O2S仅对预测代码进行草图化处理,而参考代码保持不变. 由于草图化处理会识别代码中的所有用户自定义标识符并进行统一替换,我们将其视为代码标识符扰动的一种极端情况.
S2S则对参考代码和预测代码都进行草图化处理,以探究草图化处理是否能缓解对指标的影响.
表4和表5分别展示了CodeScore-R和所选基准方法在代码生成和代码迁移任务上的评估结果. 在问题2的分析结果中,除了CodeScore-R之外,现有基准指标对于代码词素扰动普遍不具备鲁棒性. 以Java-to-Python代码迁移任务为例,在原始设置下,其BLEU的MAE值为0.347. 然而,在O2S设置下,即仅对预测代码进行草图化处理的情况下,MAE值增加至0.718,即拟合误差增加了106.92%. 而在S2S设置下,即对参考代码和预测代码都进行草图化处理的情况下,MAE值降至0.340,与原始设置相比,误差减少了2.02%. 这表明草图化处理能够提升现有基准指标的评估准确性和鲁棒性. 在后续的问题3和问题4中,为了公平比较CodeScore-R和现有基准方法的鲁棒性,我们提前对参考代码和预测代码都进行了草图化处理.
为了探究CodeScore-R与现有基准指标对代码语法扰动下的鲁棒性,即是否满足假设2,我们根据表1中提出的4种规则,并通过相关组合构造出多个语法转换方法. 在问题3中,为了避免实验的偶然性,我们选择了5个不同的随机种子,并为每个预测代码生成了5种变体代码. 然后,我们计算每个变体代码与参考代码之间的评估指标,并在表6中给出了这些指标的5次MAE值的均值.
从表6可以观察到,相对于代码中的词素扰动,现有指标在面对语法扰动时受到的影响较小. 然而,ROUGE-L受语法扰动的影响最为显著,在Java-to-Python代码迁移任务中,其MAE值从0.192增加到0.502,即误差增加了161.46%. 与此相比,CodeScore-R对语法扰动的影响较小,并且在语法扰动下,其MAE值的拟合结果仍然是最佳的. 表6的结果进一步证明了ConCE中的表征学习方法的有效性.
为了探究CodeScore-R与现有基准指标在代码语义扰动下的鲁棒性,即是否满足假设3,我们根据表2中提出的6种运算符变异规则,并通过相关组合构造出多个语义扰动方法. 在问题4中,我们控制了变异的比例,分别对代码数据的25%,50%,75%和100%进行变异,以观察不同语义扰动比例下CodeScore-R和现有基准指标的鲁棒性. 在实验中,我们仅对能通过测试用例的代码进行语义扰动. 如果扰动成功,将对应的Pass@1值变为0. 为了避免实验结果的偶然性,同样选择了5个不同的随机种子,并相应地为每个预测代码生成5种变异代码. 然后,计算每个变异代码与参考代码的评估指标,并在表7~10中分别给出在Java代码生成任务、Python代码生成任务、Python-to-Java代码迁移任务和Java-to-Python代码迁移任务上的MAE拟合均值.
从表7~10可知,现有指标在面对语义扰动时受到的影响最严重,尤其是CodeBERTScore,在代码迁移任务上的MAE值超过0.94. 一方面,其模型不具备识别代码中微小语义误差的能力,导致计算的相似度过高;另一方面,CodeBERTScore计算的是每个词素之间的相似度,导致改变语义的关键词素信息被忽视. 以图6为例,我们根据CodeBERTScore计算代码段“a = a+b”和“a = a−b”之间的相似度,其计算公式为 (0.995 + 0.986 + 0.990 + 0.730 + 0.998) / 5 = 0.9398. 其中,关键的语义词素为“−”,但是在CodeBERTScore的计算过程中没有考虑为该词素赋予更大的权重,导致了计算结果的虚高.
相比于现有指标存在一定的不鲁棒问题,CodeScore-R在下游任务上都能表现出对语义扰动具有良好的鲁棒能力. 一方面,这得益于ConCE框架让模型具有区分语义差异的能力;另一方面也得益于使用的编译器进行预测代码语法正确的判断,能够避免变异后有语法问题的代码对指标计算的干扰.
本文通过提取[CLS]的语义向量并经过ReLU函数进行激活来提取代码的语义信息. 为证明该提取方法的优势,将本文方法其与3种常见的语义提取方法进行比较. 这3种语义提取方法为:
1)last-avg. 该方法对隐藏向量$ {\boldsymbol{H}} $的最后一层进行语义提取,并通过均值化处理得到语义向量.
2)first-last-avg. 该方法对隐藏向量$ {\boldsymbol{H}} $的第一层和最后一层进行语义提取,并通过均值化处理得到语义向量.
3)CLS. 该方法对隐藏向量$ {\boldsymbol{H}} $最后一层的[CLS]进行语义提取,不额外加入激活函数层.
将这3种语义提取方法和本文提出的语义提取方法在S2S的设定下进行比较. 从表11和表12中可以分析出,在针对Python编程语言的下游任务中,本文提出的[CLS]+ReLU能够取得最优结果,而在Java编程语言的下游任务中,first-last-avg方法可能取得更好的性能结果,因此,可以将语义提取方法当作CodeScore-R的超参数,以更灵活地为下游任务进行评估.
先前的实验结果表明CodeScore-R的指标误差均小于现有基准指标,但是并没有直观地分析该指标满足对代码合成评估指标的鲁棒性提出的3个假设. 为进一步探究CodeScore-R是否满足对代码合成评估指标的鲁棒性提出的3个假设,本节旨在通过直观的分数值对比来进行验证.
具体而言,分别给出了CodeScore-R和选择的基准指标在Java代码生成、Python代码生成、Python-to-Java代码迁移和Java-to-Python代码迁移的计算分数. 如表13~14所示,其中,“原始”意为不对预测代码进行扰动,“词素扰动”意为对所有的预测代码进行草图化处理,而参考代码保持不变,“语法扰动”意为对所有的预测代码进行等价语法转换,而参考代码保持不变,“语义扰动”意味对所有的预测代码进行变异,而参考代码保持不变. 这些扰动的方法的设置与实验部分的设置保持一致.
从表13~14中可以分析出,无论是在代码生成任务还是代码迁移任务上,现有的指标均不能同时满足提出的3种假设. 其中,大部分指标满足假设2,即当生成的代码中发生语法等价转换时,指标分数应基本不变. 然而,指标分数对于词素的扰动和语义的扰动并不敏感,以CodeBERTScore在Java-to-Python代码迁移任务上的指标分数为例,其在“原始”条件下的指标分数为0.966,但是在“词素扰动”条件下的分数为0.803,并不符合假设1;此外,在“语义扰动”条件下,其指标分数为0.960,表明该指标不能真实分辨出代码的功能语义,即不符合假设3.
相比之下,CodeScore-R能同时满足提出的3种假设. 特别是假设3,以Java-to-Python代码迁移任务上的指标分数为例,CodeScore-R在“语义扰动”条件下计算出的指标分数为0.371,远低于其在“原始”条件下计算出的指标分数0.980,符合假设3. 因此,CodeScore-R相较于其他基准指标,具有更强的鲁棒性.
CodeScore-R计算相似度并进行二值化操作后,通过比较阈值返回最终的预测结果. 为了进一步分析阈值的取值,我们对阈值0~1,以步长为0.1进行了详细的分析. 表15展示了不同相似度阈值下Java与Python语言的代码生成和代码迁移任务的对比结果. 这些结果展示了CodeScore-R在不同阈值下的拟合性能.
观察结果可以发现,阈值的选择对代码生成和代码迁移任务的评估指标有类似的影响. 具体而言,当阈值过大时,拟合效果会变差. 例如,在Python代码生成任务中,当阈值设定为0.9时,MAE值会从0.2上升到0.5. 此外,阈值的最优选择也会受到不同数据集和任务的影响. 综合考虑整体效果,我们建议将阈值默认取值设定为0.5,这样可以获得比较综合的优势.
本文提出了一种自动化鲁棒指标CodeScore-R,用于评估代码合成功能的准确性. 然而,该指标仍然存在一些局限性. 一方面,CodeScore-R需要依赖编译器来检测预测代码的语法正确性,而现有的基准指标则不存在上述限制,这可能导致CodeScore-R与基准指标的比较存在一定的不公平性. 另一方面,由于深度神经网络本身存在“各向异性” (anisotropic)的特性,在一些情况下,即使2个代码片段存在一定的语义差异,但由于它们之间仍然具有一些共享特征,导致其在余弦相似度计算时一般会得到较高的分值.
第1个内部有效性威胁因素是CodeScore-R在实现过程中可能存在潜在缺陷,包括代码的草图化处理方法、正负样本的构造和模型的表征学习等方面. 为了减轻这些缺陷的影响,我们对处理后的代码进行了检验,以验证其是否能通过测试用例. 在模型的表征学习方面,我们采用了PyTorch框架5和成熟的第三方库Transformers6进行实现,以确保实现的正确性和可靠性.
第2个内部有效性威胁是选择的基准指标是否合理. 考虑到CodeScore工作需要在相关数据集上进行有监督学习,而其余指标则是通过自监督或者无监督的方法进行计算. 为了确保结果的公平性和可比性,我们选择了在先前的代码合成研究中广泛采用的基准指标,并使用公开的第三方库进行了统一实验.
外部有效性是指研究结果在多大程度上可以推广到实验所使用的数据集之外. 我们在2个下游任务上进行了实验,并同时关注Java和Python编程语言. 对于Java和Python编程语言的其他数据集,CodeScore-R可以直接使用. 然而,对于其他编程语言的代码合成任务,需要先在相应编程语言的数据集上进行ConCE表征学习,然后才能应用于该编程语言的代码合成任务中.
此外,在研究中,我们仅调研了ChatGPT模型而未考虑传统的方法,这是因为传统方法在代码生成和迁移任务中存在的一些限制和挑战,例如代码的复杂性和语义的多样性,传统的方法一般难以解决这些问题,导致这些代码的Pass@1指标不高. 因此,我们选择了ChatGPT这样的大规模语言模型作为验证工具,因为它具有更强大的语言理解和生成能力,可以应对复杂的代码生成和代码迁移任务.
构造有效性威胁是指实证研究中使用的评价指标是否真实反映了评估方法的性能. 在本文的研究中,考虑了MAE作为评判指标对功能准确性的拟合程度. 此外,为了避免假阳性或者假阴性数据过多而导致的误差虚低的情况,本文额外将CodeScore-R与功能准确性的拟合程度视为分类任务,根据准确率、精度、召回率和F1-Score来进一步评估CodeScore-R指标的性能.
本文旨在提出一种新的代码质量评估指标CodeScore-R,该指标基于UniXcoder和对比学习. 实验结果证明,CodeScore-R相对于现有基准指标在功能准确性上拟合程度更好. 此外,CodeScore-R与现有基准指标相比,在代码词素扰动、代码语法扰动和代码变异扰动下的鲁棒性表现更好.
在后续研究工作中,我们将在更大规模、更多样性的数据集上验证CodeScore-R的性能,同时考虑能力更强大的代码模型作为底座模型,以进一步探索和改进其在特定领域或特定编程语言上的适用性.
同时,在构造正负样本的方法上也将进一步地探索,丰富正负样本的多样性,以适应其在真实数据上的评估.
此外,本文仅在ChatGPT模型上进行实证研究,在未来工作中,我们将对传统的方法和基于大规模语言模型的方法进行统一验证,以进一步探索和比较CodeScore-R在不同方法中的性能和效果.
作者贡献声明:杨光提出了算法思路,负责完成实验并撰写论文;周宇对实验思路提出改进和指导,并给出修改意见;陈翔提出指导意见并修改论文;张翔宇帮助推进实验并检查实验结果.

参考文献 引证文献
排序方式:
1
Gulwani S, Polozov O, Singh R. Program synthesis[J]. Foundations and Trends in Programming Languages, 2017, 4(1-2): 1−119
2
Ren Shuo, Guo Daya, Lu Shuai, et al. CodeBLEU: A method for automatic evaluation of code synthesis[J]. arXiv preprint, arXiv: 2009. 10297, 2020
3
Lu Shuai, Guo Daya, Ren Shuo, et al. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation[J]. arXiv preprint, arXiv: 2102. 04664, 2021
4
Li Yujia, Choi D, Chung J, et al. Competition-level code generation with alphacode[J]. Science, 2022, 378(6624): 1092−1097
5
Weisz J D, Muller M, Houde S, et al. Perfection not required human-AI partnerships in code translation[C]//Proc of the 26th Int Conf on Intelligent User Interfaces. New YorK: ACM, 2021: 402−412
6
Jain N, Vaidyanath S, Iyer A, et al. Jigsaw: Large language models meet program synthesis[C]//Proc of the 44th Int Conf on Software Engineering. New York: ACM, 2022: 1219−1231
7
Zhang Chaoning, Zhang Chenshuang, Zheng Sheng, et al. A complete survey on generative AI: Is ChatGPT from GPT-4 to GPT-5 all you need?[J]. arXiv preprint, arXiv: 2303. 11717, 2023
8
Wang Yue, Wang Weishi, Joty S, et al. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation[C]//Proc of the 2021 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2021: 8696−8708
9
Nijkamp E, Pang B, Hayashi H, et al. CodeGen: An open large language model for code with multi-turn program synthesis[J]. arXiv preprint, arXiv: 2203. 13474, 2022
10
Zheng Qinkai, Xia Xiao, Zou Xu, et al. CodeGeeX: A pre-trained model for code generation with multilingual evaluations on humaneval-x[J]. arXiv preprint, arXiv: 2303. 17568, 2023
11
Liang Qingyuan, Sun Zeyu, Zhu Qihao, et al. Lyra: A benchmark for Turducken-style code generation[C]// Proc of the Thirty-First Int Joint Conf on Artificial Intelligence, San Francisco, CA: Morgan Kaufmann. 2022: 4238−4244
12
Austin J, Odena A, Nye M, et al. Program synthesis with large language models[J]. arXiv preprint, arXiv: 2108. 07732, 2021
13
Iyer S, Konstas I, Cheung A, et al. Mapping language to code in programmatic context[C]//Proc of the 2018 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2018: 1643−1652
14
Papineni K, Roukos S, Ward T, et al. BLEU: A method for automatic evaluation of machine translation[C]//Proc of the 40th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2002: 311−318
15
Popović M. chrF: Character n-gram F-Score for automatic MT evaluation[C]//Proc of the 10th workshop on Statistical Machine Translation. Stroudsburg, PA: ACL, 2015: 392−395
16
Kulal S, Pasupat P, Chandra K, et al. Spoc: Search-based pseudocode to code[J]. Advances in Neural Information Processing Systems, 2019, 32: 11906−11917
17
Niu Changan, Li Chuanyi, Luo Bin, et al. Deep learning meets software engineering: A survey on pre-trained models of source code[C]// Proc of the 31st Int Joint Conf on Artificial Intelligence. San Francisco, CA: Morgan Kaufmann, 2022: 5546−5555
18
Zhuo T Y. Large language models are state-of-the-art evaluators of code generation[J]. arXiv preprint, arXiv: 2304. 14317, 2023
19
Guo Daya, Lu Shuai, Duan Nan, et al. UniXcoder: Unified cross-modal pre-training for code representation[C]//Proc of the 60th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL. 2022: 7212−7225
20
Chen M, Tworek J, Jun H, et al. Evaluating large language models trained on code[J]. arXiv preprint, arXiv: 2107. 03374, 2021
21
Ahmad W U, Tushar M G R, Chakraborty S, et al. AVATAR: A parallel corpus for Java-Python program translation[J]. arXiv preprint, arXiv: 2108. 11590, 2021
22
Liguori P, Improta C, Natella R, et al. Who evaluates the evaluators? On automatic metrics for assessing AI-based offensive code generators[J]. Expert Systems with Applications, 2023, 225: 120073
23
Lin C Y. Rouge: A package for automatic evaluation of summaries[C]//Proc of the Text Summarization Branches Out. Stroudsburg, PA: ACL, 2004: 74−81
24
Evtikhiev M, Bogomolov E, Sokolov Y, et al. Out of the BLEU: How should we assess quality of the code generation models?[J]. Journal of Systems and Software, 2023, 203: 111741
25
Eghbali A, Pradel M. CrystalBLEU: Precisely and efficiently measuring the similarity of code[C]//Proc of the 37th IEEE/ACM Int Conf on Automated Software Engineering. Piscataway, NJ: IEEE, 2022: 1−12
26
Dong Yihong, Ding Jiazheng, Jiang Xue, et al. CodeScore: Evaluating code generation by learning code execution[J]. arXiv preprint, arXiv: 2301. 09043, 2023
27
Zhou Shuyan, Alon U, Agarwal S, et al. CodeBERTScore: Evaluating code generation with pretrained models of code[J]. arXiv preprint, arXiv: 2302. 05527, 2023
28
Feng Zhangyin, Guo Daya, Tang Duyu, et al. CodeBERT: A pre-trained model for programming and natural languages[C]//Proc of the Findings of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 1536−1547
29
Yang Guang, Zhou Yu, Chen Xiang, et al. ExploitGen: Template-augmented exploit code generation based on CodeBERT[J]. Journal of Systems and Software, 2023, 197: 111577
30
Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 2017: 6000−6010
31
Husain H, Wu H H, Gazit T, et al. CodeSearchNet challenge: Evaluating the state of semantic code search[J]. arXiv preprint, arXiv: 1909. 09436, 2019
32
Gao Tianyu, Yao Xingcheng, Chen Danqi. SimCSE: Simple contrastive learning of sentence embeddings[C]//Proc of the 2021 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2021: 6894−6910
33
Chakraborty S, Ahmed T, Ding Yangruibo, et al. NatGen: Generative pre-training by “naturalizing” source code[C]//Proc of the 30th ACM Joint European Software Engineering Conf and Symp. on the Foundations of Software Engineering. New York: ACM, 2022: 18−30
34
Jia Yue, Harman M. An analysis and survey of the development of mutation testing[J]. IEEE Transactions on Software Engineering, 2010, 37(5): 649−678
35
van den Heuvel E, Zhan Z. Myths about linear and monotonic associations: Pearson’s r, Spearman’s ρ, and Kendall’s τ[J]. The American Statistician, 2022, 76(1): 44−52
36
Willmott C J, Matsuura K. Advantages of the mean absolute error over the root mean square error in assessing average model performance[J]. Climate Research, 2005, 30(1): 79−82
2024年第61卷第2期
69
33
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202330715
  • 接收时间:2023-09-11
  • 首发时间:2026-11-17
  • 出版时间:2024-02-02
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2023-09-11
  • 修回日期:2023-12-02
基金
作者信息
    1南京航空航天大学计算机科学与技术学院/人工智能学院/软件学院 南京 211106
    2南通大学信息科学技术学院 江苏南通 226019

通讯作者:

参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202330715
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏