Article(id=1327018836873818180, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1327018536993665336, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202440551, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=research-article, receivedDate=1718812800000, receivedDateStr=2024-06-20, revisedDate=1729526400000, revisedDateStr=2024-10-22, acceptedDate=1731340800000, acceptedDateStr=2024-11-12, onlineDate=1794315545527, onlineDateStr=2026-11-10, pubDate=1753977600000, pubDateStr=2025-08-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1794315545527, onlineIssueDateStr=2026-11-10, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1794315545527, creator=13701087609, updateTime=1794315545527, updator=13701087609, issue=Issue{id=1327018536993665336, tenantId=1146029695717560320, journalId=1324759914146529478, year='2025', volume='62', issue='8', pageStart='1833', pageEnd='2124', issueExtLink='null', onlineDate='null', pubDate='1753977600000', pubDateStr='2025-08-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1794315474031, creator='13701087609', updateTime=1794376881014, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext={EN=IssueExt(id=1327276097470837643, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1327018536993665336, language=EN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=), CN=IssueExt(id=1327276097470837644, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1327018536993665336, language=CN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=)}, issueFiles=null, downloadFileDto=null}, startPage=1902, endPage=1917, ext={EN=ArticleExt(id=1327018838362796101, articleId=1327018836873818180, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=Conversation Generation Based on Variational Attention Knowledge Selection and Pre-trained Language Model, columnId=1325489891789910943, journalTitle=Journal of Computer Research and Development, columnName=Artificial Intelligence, runingTitle=null, highlight=null, articleAbstract=

Research on knowledge-grounded dialogue often suffers from the problem of external knowledge containing redundant or even noisy information irrelevant to the conversation topic, which leads to a degradation in the performance of the dialogue system. Knowledge selection becomes an important approach to solve this issue. However, existing work has not yet investigated in depth some issues involving it such as how to design a knowledge selector, how to exploit the selected knowledge, what are the suitable scenarios for the knowledge selection conversation methods, etc. In this paper, we propose a new neural conversation method based on conditional variational attention knowledge selection and a pre-trained language model. This method employs a knowledge selection algorithm based on conditional variational autoencoder (CVAE) and a multi-layer attention mechanism to pick up the most relevant textual knowledge collection to the current conversation, which effectively exploits the dialogue response in training data to improve the efficiency of knowledge selection. Our novel model adopts the pre-trained language model Bart as encoder-decoder architecture and incorporates selected textual knowledge into the Bart model to fine-tune it during the training process. The experimental results show that the model proposed, in contrast to the current representative dialog models, can generate more diverse and coherent dialogue responses with higher accuracy.

, authors=Naizhou Zhang1, Wei Cao1, Xiaojian Zhang1, Shijun Li2, authorsList=Naizhou Zhang, Wei Cao, Xiaojian Zhang, Shijun Li, authorCompany=null, correspAuthors=null, authorNote=

born in 1970. PhD,associate professor. Member of CCF. His main research interests include Web search, data mining, and machine learning

born in 1977. Master, associate professor. Her main research interests include machine learning and Web data mining

born in 1980. PhD, professor, master supervisor. His main research interests include differential privacy, privacy analysis in language model, and graph data management

born in 1964. PhD, professor, PhD supervisor. His main research interests include database theory, Web search engine, and Web mining

, correspAuthorsNote=null, copyrightStatement=Copyright © 2025 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1327018840250232918, articleId=1327018836873818180, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=基于变分注意力知识选择和预训练语言模型的对话生成, columnId=1325489892104483745, journalTitle=计算机研究与发展, columnName=人工智能, runingTitle=null, highlight=null, articleAbstract=

基于知识的神经对话研究常常面临外部知识包含冗余甚至与对话主题不相关信息的问题,从而导致对话系统性能下降. 知识选择成为解决该问题的重要途径,但现有研究对诸如知识选择器的设计、选择出的知识的利用以及知识选择对话方法适用的场景等问题,还缺乏深入研究. 针对这些问题,提出了一个新的基于变分注意力知识选择和预训练模型的神经对话方法,使用一个基于条件变分自编码(conditional variational autoencoder,CVAE)和多层注意力机制的知识选择算法,自动选择出与当前对话最相关文本知识集合. 该算法有效利用了训练数据中的对话回复来提高知识选择的效率. 使用预训练语言模型Bart作为编码器-解码器架构,将选择的文本知识合并到Bart模型中,并在训练过程中对其进行微调. 实验结果表明,与现有的一些代表性研究方法相比,提出的模型能生成多样性和连贯性更好、准确率更高的对话回复.

, authors=张乃洲1, 曹薇1, 张啸剑1, 李石君2, authorsList=张乃洲, 曹薇, 张啸剑, 李石君, authorCompany=null, correspAuthors=null, authorNote=

1970年生. 博士,副教授. CCF会员. 主要研究方向为Web检索、数据挖掘、机器学习

1977年生. 硕士,副教授. 主要研究方向为机器学习、Web数据挖掘

1980年生. 博士,教授,硕士生导师. 主要研究方向为差分隐私、语言模型隐私分析、图形数据管理

1964年生. 博士,教授,博士生导师. 主要研究方向为数据库理论、Web搜索引擎、Web挖掘

, correspAuthorsNote=null, copyrightStatement=版权所有 © 2025 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=WAptGI/uWT+ls3epcASqow==, magXml=UgMsTczUJCYtQn0XNFdLUg==, pdfUrl=null, pdf=yZbDdeUVNnMmvAxmoT/zqA==, pdfFileSize=1133126, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=Oh6fgEopHAT6TcyBrVHuPw==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=YQct/DqyhnHmutYlBOY9vA==, mapNumber=null, fund=null)}, authors=[Author(id=1327018840678051934, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=zhangnz@126.com, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327018840766132320, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, authorId=1327018840678051934, language=EN, stringName=Naizhou Zhang, firstName=Naizhou, middleName=null, lastName=Zhang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1College of Computer and Information Engineering, Henan University of Economics and Law, Zhengzhou 450046, bio={"content":"

born in 1970. PhD,associate professor. Member of CCF. His main research interests include Web search, data mining, and machine learning

"}, bioImg=null, bioContent=

born in 1970. PhD,associate professor. Member of CCF. His main research interests include Web search, data mining, and machine learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327018840845824098, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, authorId=1327018840678051934, language=CN, stringName=张乃洲, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1河南财经政法大学计算机与信息工程学院 郑州 450046, bio={"img":"Vmy7tbtLP6tH74yc5PW25A==","content":"

1970年生. 博士,副教授. CCF会员. 主要研究方向为Web检索、数据挖掘、机器学习

"}, bioImg=Vmy7tbtLP6tH74yc5PW25A==, bioContent=

1970年生. 博士,副教授. CCF会员. 主要研究方向为Web检索、数据挖掘、机器学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327018840480919639, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, xref=1, ext=[AuthorCompanyExt(id=1327018840489308248, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840480919639, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1College of Computer and Information Engineering, Henan University of Economics and Law, Zhengzhou 450046), AuthorCompanyExt(id=1327018840497696857, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840480919639, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1河南财经政法大学计算机与信息工程学院 郑州 450046)])]), Author(id=1327018840908738660, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327018840975847526, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, authorId=1327018840908738660, language=EN, stringName=Wei Cao, firstName=Wei, middleName=null, lastName=Cao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1College of Computer and Information Engineering, Henan University of Economics and Law, Zhengzhou 450046, bio={"content":"

born in 1977. Master, associate professor. Her main research interests include machine learning and Web data mining

"}, bioImg=null, bioContent=

born in 1977. Master, associate professor. Her main research interests include machine learning and Web data mining

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327018841034567783, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, authorId=1327018840908738660, language=CN, stringName=曹薇, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1河南财经政法大学计算机与信息工程学院 郑州 450046, bio={"img":"TLWpAnQkRgFW20hotF8+lg==","content":"

1977年生. 硕士,副教授. 主要研究方向为机器学习、Web数据挖掘

"}, bioImg=TLWpAnQkRgFW20hotF8+lg==, bioContent=

1977年生. 硕士,副教授. 主要研究方向为机器学习、Web数据挖掘

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327018840480919639, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, xref=1, ext=[AuthorCompanyExt(id=1327018840489308248, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840480919639, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1College of Computer and Information Engineering, Henan University of Economics and Law, Zhengzhou 450046), AuthorCompanyExt(id=1327018840497696857, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840480919639, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1河南财经政法大学计算机与信息工程学院 郑州 450046)])]), Author(id=1327018841093288041, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327018842766815339, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, authorId=1327018841093288041, language=EN, stringName=Xiaojian Zhang, firstName=Xiaojian, middleName=null, lastName=Zhang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1College of Computer and Information Engineering, Henan University of Economics and Law, Zhengzhou 450046, bio={"content":"

born in 1980. PhD, professor, master supervisor. His main research interests include differential privacy, privacy analysis in language model, and graph data management

"}, bioImg=null, bioContent=

born in 1980. PhD, professor, master supervisor. His main research interests include differential privacy, privacy analysis in language model, and graph data management

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327018842842312812, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, authorId=1327018841093288041, language=CN, stringName=张啸剑, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1河南财经政法大学计算机与信息工程学院 郑州 450046, bio={"img":"/XuCL2abPZbvlbrwR0l3Yg==","content":"

1980年生. 博士,教授,硕士生导师. 主要研究方向为差分隐私、语言模型隐私分析、图形数据管理

"}, bioImg=/XuCL2abPZbvlbrwR0l3Yg==, bioContent=

1980年生. 博士,教授,硕士生导师. 主要研究方向为差分隐私、语言模型隐私分析、图形数据管理

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327018840480919639, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, xref=1, ext=[AuthorCompanyExt(id=1327018840489308248, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840480919639, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1College of Computer and Information Engineering, Henan University of Economics and Law, Zhengzhou 450046), AuthorCompanyExt(id=1327018840497696857, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840480919639, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1河南财经政法大学计算机与信息工程学院 郑州 450046)])]), Author(id=1327018842909421678, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327018842989113456, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, authorId=1327018842909421678, language=EN, stringName=Shijun Li, firstName=Shijun, middleName=null, lastName=Li, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2School of Computer Science, Wuhan University, Wuhan 430072, bio={"content":"

born in 1964. PhD, professor, PhD supervisor. His main research interests include database theory, Web search engine, and Web mining

"}, bioImg=null, bioContent=

born in 1964. PhD, professor, PhD supervisor. His main research interests include database theory, Web search engine, and Web mining

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327018843056222321, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, authorId=1327018842909421678, language=CN, stringName=李石君, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2武汉大学计算机学院 武汉 430072, bio={"img":"z1WRwFY0PigBq2Siy92GBA==","content":"

1964年生. 博士,教授,博士生导师. 主要研究方向为数据库理论、Web搜索引擎、Web挖掘

"}, bioImg=z1WRwFY0PigBq2Siy92GBA==, bioContent=

1964年生. 博士,教授,博士生导师. 主要研究方向为数据库理论、Web搜索引擎、Web挖掘

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327018840564805722, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, xref=2, ext=[AuthorCompanyExt(id=1327018840573194331, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840564805722, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Computer Science, Wuhan University, Wuhan 430072), AuthorCompanyExt(id=1327018840581582940, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840564805722, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2武汉大学计算机学院 武汉 430072)])])], keywords=[Keyword(id=1327018843177857138, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=EN, orderNo=1, keyword=knowledge-grounded dialogue), Keyword(id=1327018843240771699, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=EN, orderNo=2, keyword=knowledge selection), Keyword(id=1327018843307880564, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=EN, orderNo=3, keyword=pre-trained language model), Keyword(id=1327018843429515381, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=EN, orderNo=4, keyword=CVAE), Keyword(id=1327018843492429942, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=EN, orderNo=5, keyword=attention mechanism), Keyword(id=1327018843555344503, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=EN, orderNo=6, keyword=memory network), Keyword(id=1327018843609870456, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=CN, orderNo=1, keyword=基于知识的对话), Keyword(id=1327018843664396409, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=CN, orderNo=2, keyword=知识选择), Keyword(id=1327018843718922362, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=CN, orderNo=3, keyword=预训练语言模型), Keyword(id=1327018843794419835, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=CN, orderNo=4, keyword=条件变分自编码), Keyword(id=1327018843882500220, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=CN, orderNo=5, keyword=注意力机制), Keyword(id=1327018843953803389, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=CN, orderNo=6, keyword=记忆网络)], refs=[Reference(id=1327018847577682072, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=Ni Jinjie, Young T, Pandelea V, et al. Recent advances in deep learning based dialogue systems: A systematic survey[J]. Artificial Intelligence Review, 2023, 56(4): 3055−3155, articleTitle=null, refAbstract=null), Reference(id=1327018847657373849, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Serban I V, Sordoni A, Bengio Y, et al. Building end-to-end dialogue systems using generative hierarchical neural network models[C]//Proc of the 30th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2016: 3776−3784, articleTitle=null, refAbstract=null), Reference(id=1327018847737065626, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Serban I V, Sordoni A, Lowe R, et al. A hierarchical latent variable encoder-decoder model for generating dialogues[C]//Proc of the 31st AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2017: 3295–3301, articleTitle=null, refAbstract=null), Reference(id=1327018847791591579, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Jiwei, Galley M, Brocket C, et al. A diversity-promoting objective function for neural conversation models[C]//Proc of the 2016 Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, PA: ACL, 2016: 110−119, articleTitle=null, refAbstract=null), Reference(id=1327018847854506140, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Tiancheng, Zhao Ran, Eskénazi M. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders[C]//Proc of the 55th Conf of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2017: 654−664, articleTitle=null, refAbstract=null), Reference(id=1327018847909032093, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Rongjunchen, Wu Tingmin, Chen Xiao, et al. A transformer-based dialogue system with dynamic attention[C]//Proc of the Web Conf 2023. New York: ACM, 2023: 1604−1615, articleTitle=null, refAbstract=null), Reference(id=1327018847984529566, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Golovanov S, Kurbanov R, Sergey I N, et al. Large-scale transfer learning for natural language generation[C]//Proc of the 57th Conf of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2019: 6053−6058, articleTitle=null, refAbstract=null), Reference(id=1327018848043249823, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Ghazvininejad M, Brockett C, Chang Mingwei, et al. A knowledge-grounded neural conversation model[C]//Proc of the 32nd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2018: 5110−5117, articleTitle=null, refAbstract=null), Reference(id=1327018848110358688, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Zekang, Niu Cheng, Meng Fandong, et al. Incremental transformer with deliberation decoder for document grounded conversations[C]//Proc of the 57th Conf of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2019: 12−21, articleTitle=null, refAbstract=null), Reference(id=1327018848173273249, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Shilei, Zhao Xiaofeng, Li Bochao, et al. A three-stage learning framework for low-resource knowledge-grounded dialogue generation[C]//Proc of the 2021 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2021: 2262−2272, articleTitle=null, refAbstract=null), Reference(id=1327018848240382114, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Dinan E, Roller S, Shuster K, et al. Wizard of Wikipedia: Knowledge-powered conversational agents[C/OL]//Proc of the 7th Int Conf on Learning Representations. Ithaca, NY: Cornell University, 2019[2024-09-25]. https://openreview.net/forum?id=HtLWEQ69sf9, articleTitle=null, refAbstract=null), Reference(id=1327018848303296675, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Lian Rongzhong, Xie Min, Wang Fan, et al. Learning to select knowledge for response generation in dialog systems[C]//Proc of the 28th Int Joint Conf on Artificial Intelligence. San Francisco, CA: Morgan Kaufmann, 2019: 5081−5087, articleTitle=null, refAbstract=null), Reference(id=1327018848374599844, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Kim B, Ahn J, Kim G. Sequential latent knowledge selection for knowledge-grounded dialogue[C/OL]//Proc of the 8th Int Conf on Learning Representations. Ithaca, NY: Cornell University, 2020[2024-09-25]. https://openreview.net/forum?id=pm6GlJQp4Oo, articleTitle=null, refAbstract=null), Reference(id=1327018848437514405, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Xiuyi, Meng Fandong, Li Peng, et al. Bridging the gap between prior and posterior knowledge selection for knowledge-grounded dialogue generation[C]//Proc of the 2020 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2020: 3426−3437, articleTitle=null, refAbstract=null), Reference(id=1327018848496234662, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Xiuyi, Chen Feilong, Meng Fandong, et al. Unsupervised knowledge selection for dialogue generation[C]//Proc of the Joint Conf of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int Joint Conf on Natural Language Processing: Findings. Stroudsburg, PA: ACL, 2021: 1230−1244, articleTitle=null, refAbstract=null), Reference(id=1327018848571732135, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Zhibin, Niu Zhengyu, Wu Hua, et al. Knowledge aware conversation generation with explainable reasoning over augmented graphs[C]//Proc of the 2019 Conf on Empirical Methods in Natural Language Processing and the 9th Int Joint Conf on Natural Language Processing. Stroudsburg, PA: ACL, 2019: 1782−1792, articleTitle=null, refAbstract=null), Reference(id=1327018848634646696, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Meng Chuan, Ren Pengjie, Chen Zhumin, et al. Initiative-aware self-supervised learning for knowledge-grounded conversations[C]//Proc of the 44th Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2021: 522−532, articleTitle=null, refAbstract=null), Reference(id=1327018848718532777, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Xueliang, Wu Wei, Xu Can, et al. Knowledge-grounded dialogue generation with pre-trained language models[C]//Proc of the 2020 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2020: 115−125, articleTitle=null, refAbstract=null), Reference(id=1327018848777253034, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Zheng Chujie, Cao Yunbo, Jiang Daxin, et al. Difference-aware knowledge selection for knowledge-grounded conversation generation[C]//Proc of the 2020 Conf on Empirical Methods in Natural Language Processing: Findings. Stroudsburg, PA: ACL, 2020: 115−125, articleTitle=null, refAbstract=null), Reference(id=1327018848835973291, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Sun Weiwei, Ren Pengjie, Ren Zhaochun. Generative knowledge selection for knowledge-grounded dialogues[C]//Proc of the 17th Conf of the European Chapter of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2023: 2032−2043, articleTitle=null, refAbstract=null), Reference(id=1327018848907276460, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=孙润鑫,马龙轩,张伟男,等. 基于文档的对话研究[J]. 计算机研究与发展,2021,58(9):1915−1924, articleTitle=null, refAbstract=null), Reference(id=1327018849003745453, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Sun Runxin, Ma Longxuan, Zhang Weinan, et al. Research on document grounded conversations[J]. Journal of Computer Research and Development, 2021, 58(9): 1915−1924 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1327018849070854318, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Yizhe, Huang Heyan, Liu Yuhang, et al. Graph vs. sequence: An empirical study on knowledge forms for knowledge-grounded dialogue[C]//Proc of the 2023 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2023: 15846−15858, articleTitle=null, refAbstract=null), Reference(id=1327018849133768879, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=Sukhbaatar S, Szlam A, Weston J, et al. End-to-end memory networks[C]//Proc of the 28th Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2015: 2440−2448, articleTitle=null, refAbstract=null), Reference(id=1327018849200877744, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=Lewis M, Liu Yinhan, Goyal N, et al. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[C]//Proc of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 7871−7880, articleTitle=null, refAbstract=null), Reference(id=1327018849263792305, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhou Kangyan, Prabhumoye S, Black A W. A dataset for document grounded conversations[C]//Proc of the 2018 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2018: 708−713, articleTitle=null, refAbstract=null), Reference(id=1327018849326706866, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=26, authorNames=null, journalName=null, refType=null, unstructuredReference=Papineni K, Roukos S, Ward T, et al. Bleu: A method for automatic evaluation of machine translation[C]//Proc of the 40th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2002: 311−318, articleTitle=null, refAbstract=null), Reference(id=1327018849398010035, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=27, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Pengfei, Yuan Weizhe, Fu Jinlan, et al. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing[J]. ACM Computing Surveys, 2023, 55(9): 195: 1−195: 35, articleTitle=null, refAbstract=null), Reference(id=1327018849452535988, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=28, rfOrder=28, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Tianyi, Kishore V, Wu F, et al. BERTScore: Evaluating text generation with BERT[C/OL]//Proc of the 8th Int Conf on Learning Representations. Ithaca, NY: Cornell University, 2020[2024-09-25]. https://openreview.net/forum?id=SkeHuCVFDr, articleTitle=null, refAbstract=null), Reference(id=1327018849523839157, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=29, rfOrder=29, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Chiawei, Lowe R, Serban I, et al. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation[C]//Proc of the 2016 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2016: 2122−2132, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1327018840480919639, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, xref=1, ext=[AuthorCompanyExt(id=1327018840489308248, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840480919639, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1College of Computer and Information Engineering, Henan University of Economics and Law, Zhengzhou 450046), AuthorCompanyExt(id=1327018840497696857, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840480919639, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1河南财经政法大学计算机与信息工程学院 郑州 450046)]), AuthorCompany(id=1327018840564805722, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, xref=2, ext=[AuthorCompanyExt(id=1327018840573194331, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840564805722, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Computer Science, Wuhan University, Wuhan 430072), AuthorCompanyExt(id=1327018840581582940, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840564805722, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2武汉大学计算机学院 武汉 430072)])], figs=[ArticleFig(id=1327018844083826814, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=EN, label=Figure 1, caption=Comparison of the conversation methods for implicit knowledge selection and explicit knowledge selection, figureFileSmall=TudyEnESBeirfDwPUPlZDQ==, figureFileBig=Oh6fgEopHAT6TcyBrVHuPw==, tableContent=null), ArticleFig(id=1327018844163518591, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=CN, label=图1, caption=隐式知识选择与显式知识选择对话方法比较, figureFileSmall=TudyEnESBeirfDwPUPlZDQ==, figureFileBig=Oh6fgEopHAT6TcyBrVHuPw==, tableContent=null), ArticleFig(id=1327018844348067968, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=EN, label=Figure 2, caption=Overall architecture of the CVAKSPLM-based knowledge dialogue model, figureFileSmall=7yBKOxMPiVaKzW/a2ZMCmg==, figureFileBig=cU/9j4Ky8GqEs34QAA9oeA==, tableContent=null), ArticleFig(id=1327018844415176833, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=CN, label=图2, caption=基于CVAKSPLM知识对话模型的总体架构, figureFileSmall=7yBKOxMPiVaKzW/a2ZMCmg==, figureFileBig=cU/9j4Ky8GqEs34QAA9oeA==, tableContent=null), ArticleFig(id=1327018844478091394, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=EN, label=Figure 3, caption=Comparison of CVAKSPLM and other models in terms of computation, parameter quantity, and inference speed, figureFileSmall=MbWUq0zQIdISZ+ur0z4+PQ==, figureFileBig=YIL6IQkigk/07WltB4nV3w==, tableContent=null), ArticleFig(id=1327018844549394563, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=CN, label=图3, caption=CVAKSPLM与其他模型在计算量、参数量和推理速度方面的比较, figureFileSmall=MbWUq0zQIdISZ+ur0z4+PQ==, figureFileBig=YIL6IQkigk/07WltB4nV3w==, tableContent=null), ArticleFig(id=1327018844603920516, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=EN, label=Figure 4, caption=Examples of responses generated by various models, figureFileSmall=/lqp9tgv8UD4yuvKsVjOfQ==, figureFileBig=PlSRGCc/OgzoNIQSZUpyIA==, tableContent=null), ArticleFig(id=1327018844666835077, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=CN, label=图4, caption=不同模型生成的回复实例, figureFileSmall=/lqp9tgv8UD4yuvKsVjOfQ==, figureFileBig=PlSRGCc/OgzoNIQSZUpyIA==, tableContent=null), ArticleFig(id=1327018844733943942, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=EN, label=Table 1, caption=Comparison of Three Explicit Knowledge Selection Based Conversation Methods, figureFileSmall=null, figureFileBig=null, tableContent=
模型编码器解码器知识选择方法
PostKS[12]带有GRU单元的双向RNN带有GRU单元的RNN条件先验和条件后验概率使用Dot Product注意力机制来计算;并采用Gumbel-Softmax重参数化技巧,解决在选择最大概率知识样本时,离散分布的不可微性.
SKT[13]BERT+
GRU
带有Copy Mechanism的Transformer模型考虑了知识选择的历史,采用序列潜变量模型来追踪知识上的先验和后验分布;也采用了Gumbel-Softmax重参数化技巧,并引入了知识损失.
CVAKSPLM(本文)Bart+
BERT
Bart使用CVAE建模知识查询向量的条件先验和后验分布;抽样潜变量空间得到知识查询向量,利用一个多层注意力模块Attention计算知识查询向量与文本知识向量之间的相关度来进行知识选择.
), ArticleFig(id=1327018844805247111, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=CN, label=表1, caption=

3种显式知识选择对话方法的比较

, figureFileSmall=null, figureFileBig=null, tableContent=
模型编码器解码器知识选择方法
PostKS[12]带有GRU单元的双向RNN带有GRU单元的RNN条件先验和条件后验概率使用Dot Product注意力机制来计算;并采用Gumbel-Softmax重参数化技巧,解决在选择最大概率知识样本时,离散分布的不可微性.
SKT[13]BERT+
GRU
带有Copy Mechanism的Transformer模型考虑了知识选择的历史,采用序列潜变量模型来追踪知识上的先验和后验分布;也采用了Gumbel-Softmax重参数化技巧,并引入了知识损失.
CVAKSPLM(本文)Bart+
BERT
Bart使用CVAE建模知识查询向量的条件先验和后验分布;抽样潜变量空间得到知识查询向量,利用一个多层注意力模块Attention计算知识查询向量与文本知识向量之间的相关度来进行知识选择.
), ArticleFig(id=1327018844876550280, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=EN, label=Table 2, caption=Performance of Various Models on Wizard Test Seen Dataset, figureFileSmall=null, figureFileBig=null, tableContent=
模型准确率多样性召回率
BLEU-1BLEU-2BLEU-3BLEU-4Dist1Dist2ROUGE-1ROUGE-2
注:黑体数值表示实验结果为最优值,SKT去除了知识损失.
ITDD[9]0.14450.05130.02180.01010.00740.02230.18380.0224
PostKS[12]0.24640.14010.08930.05660.04220.20430.27160.0695
SKT[13]0.23170.11880.07230.04770.07180.32350.17330.0512
KAT_TSLF[10]0.23810.12250.07700.05460.08880.35380.18190.0520
GenKS[20]0.26860.13950.08490.05860.06590.23890.26490.0707
CVAKSPLM(本文)0.28830.19400.14690.11800.11030.42300.26540.1226
), ArticleFig(id=1327018844956242057, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=CN, label=表2, caption=

各种模型在Wizard Test Seen数据集上的表现

, figureFileSmall=null, figureFileBig=null, tableContent=
模型准确率多样性召回率
BLEU-1BLEU-2BLEU-3BLEU-4Dist1Dist2ROUGE-1ROUGE-2
注:黑体数值表示实验结果为最优值,SKT去除了知识损失.
ITDD[9]0.14450.05130.02180.01010.00740.02230.18380.0224
PostKS[12]0.24640.14010.08930.05660.04220.20430.27160.0695
SKT[13]0.23170.11880.07230.04770.07180.32350.17330.0512
KAT_TSLF[10]0.23810.12250.07700.05460.08880.35380.18190.0520
GenKS[20]0.26860.13950.08490.05860.06590.23890.26490.0707
CVAKSPLM(本文)0.28830.19400.14690.11800.11030.42300.26540.1226
), ArticleFig(id=1327018845031739530, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=EN, label=Table 3, caption=Performance of Various Models on Wizard Test Unseen Dataset, figureFileSmall=null, figureFileBig=null, tableContent=
模型准确率多样性召回率
BLEU-1BLEU-2BLEU-3BLEU-4Dist1Dist2ROUGE-1ROUGE-2
黑体数值表示实验结果为最优值,SKT去除了知识损失.
ITDD[9]0.13910.04810.01950.00840.00570.01680.17860.0203
PostKS[12]0.23860.13750.09030.05530.01530.09090.23350.0406
SKT[13]0.21640.10120.06330.04040.05250.21740.16290.0427
KAT_TSLF[10]0.22770.11210.06770.04630.05970.23270.17460.0442
GenKS[20]0.26280.13650.08250.05610.06140.22930.25990.0688
CVAKSPLM(本文)0.28220.19330.14510.11590.06460.23330.26340.1174
), ArticleFig(id=1327018845107237003, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=CN, label=表3, caption=

各种模型在Wizard Test Unseen数据集上的表现

, figureFileSmall=null, figureFileBig=null, tableContent=
模型准确率多样性召回率
BLEU-1BLEU-2BLEU-3BLEU-4Dist1Dist2ROUGE-1ROUGE-2
黑体数值表示实验结果为最优值,SKT去除了知识损失.
ITDD[9]0.13910.04810.01950.00840.00570.01680.17860.0203
PostKS[12]0.23860.13750.09030.05530.01530.09090.23350.0406
SKT[13]0.21640.10120.06330.04040.05250.21740.16290.0427
KAT_TSLF[10]0.22770.11210.06770.04630.05970.23270.17460.0442
GenKS[20]0.26280.13650.08250.05610.06140.22930.25990.0688
CVAKSPLM(本文)0.28220.19330.14510.11590.06460.23330.26340.1174
), ArticleFig(id=1327018845182734476, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=EN, label=Table 4, caption=Performance of Various Models on CMU_DoG Dataset, figureFileSmall=null, figureFileBig=null, tableContent=
模型准确率多样性召回率
BLEU-1BLEU-2BLEU-3BLEU-4Dist1Dist2ROUGE-1ROUGE-2
黑体数值表示实验结果为最优值,SKT去除了知识损失.
ITDD[9]0.09970.04790.02920.01880.00100.00260.14240.0183
PostKS[12]0.13940.04990.02070.01020.01880.10460.14190.0174
SKT[13]0.17830.08680.04180.02990.02560.12380.12760.0238
KAT_TSLF[10]0.19740.09830.05990.03840.03990.18380.15450.0292
GenKS[20]0.20220.07190.03970.02790.03530.17420.14520.0255
CVAKSPLM(本文)0.20560.10500.06370.03920.02800.12690.14200.0242
), ArticleFig(id=1327018845262426253, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=CN, label=表4, caption=

各种模型在CMU_DoG数据集上的表现

, figureFileSmall=null, figureFileBig=null, tableContent=
模型准确率多样性召回率
BLEU-1BLEU-2BLEU-3BLEU-4Dist1Dist2ROUGE-1ROUGE-2
黑体数值表示实验结果为最优值,SKT去除了知识损失.
ITDD[9]0.09970.04790.02920.01880.00100.00260.14240.0183
PostKS[12]0.13940.04990.02070.01020.01880.10460.14190.0174
SKT[13]0.17830.08680.04180.02990.02560.12380.12760.0238
KAT_TSLF[10]0.19740.09830.05990.03840.03990.18380.15450.0292
GenKS[20]0.20220.07190.03970.02790.03530.17420.14520.0255
CVAKSPLM(本文)0.20560.10500.06370.03920.02800.12690.14200.0242
), ArticleFig(id=1327018845325340814, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=EN, label=Table 5, caption=Artificial Evaluation Results on Wizard Dataset, figureFileSmall=null, figureFileBig=null, tableContent=
模型Wizard Test Seen数据集Wizard Test Unseen数据集
流利性相关性连贯性评价一致性流利性相关性连贯性评价一致性
PostKS[12]1.711.681.250.651.701.631.240.62
SKT[13]1.671.521.170.621.631.481.140.64
KAT_TSLF[10]1.741.721.490.681.761.691.460.66
GenKS[20]1.801.761.560.671.791.741.520.68
CVAKSPLM(本文)1.871.811.680.691.861.791.670.71
), ArticleFig(id=1327018845400838287, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=CN, label=表5, caption=

在Wizard数据集上的人工评价结果

, figureFileSmall=null, figureFileBig=null, tableContent=
模型Wizard Test Seen数据集Wizard Test Unseen数据集
流利性相关性连贯性评价一致性流利性相关性连贯性评价一致性
PostKS[12]1.711.681.250.651.701.631.240.62
SKT[13]1.671.521.170.621.631.481.140.64
KAT_TSLF[10]1.741.721.490.681.761.691.460.66
GenKS[20]1.801.761.560.671.791.741.520.68
CVAKSPLM(本文)1.871.811.680.691.861.791.670.71
), ArticleFig(id=1327018845480530065, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=EN, label=Table 6, caption=Ablation Study Results on Wizard Test Unseen Dataset, figureFileSmall=null, figureFileBig=null, tableContent=
模型准确率多样性召回率
BLEU-3/BLEU-4Dist1/Dist2ROUGE-1/ROUGE-2
CVKASPLM (本文)0.1451/0.11590.0646/0.23330.2634/0.1174
CVKASPLM with KAT0.1152/0.08640.0603/0.22480.2391/0.0892
CVKASPLM with random selection0.1026/0.07870.0582/0.21350.2255/0.0826
), ArticleFig(id=1327018845539250322, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=CN, label=表6, caption=

在Wizard Test Unseen数据集上的消融实验结果

, figureFileSmall=null, figureFileBig=null, tableContent=
模型准确率多样性召回率
BLEU-3/BLEU-4Dist1/Dist2ROUGE-1/ROUGE-2
CVKASPLM (本文)0.1451/0.11590.0646/0.23330.2634/0.1174
CVKASPLM with KAT0.1152/0.08640.0603/0.22480.2391/0.0892
CVKASPLM with random selection0.1026/0.07870.0582/0.21350.2255/0.0826
), ArticleFig(id=1327018845614747795, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=EN, label=Table 7, caption=Accuracy of Knowledge Selection, figureFileSmall=null, figureFileBig=null, tableContent=
模型数据集
Wizard Test SeenWizard Test Unseen
PostKS[12]0.390.37
SKT[13]0.360.33
CVAKSPLM (本文)0.470.46
), ArticleFig(id=1327018847317635221, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=CN, label=表7, caption=

知识选择的准确率

, figureFileSmall=null, figureFileBig=null, tableContent=
模型数据集
Wizard Test SeenWizard Test Unseen
PostKS[12]0.390.37
SKT[13]0.360.33
CVAKSPLM (本文)0.470.46
), ArticleFig(id=1327018847384744086, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=EN, label=Table 8, caption=Experimental Results for CVAKS on Large Language Models, figureFileSmall=null, figureFileBig=null, tableContent=
模型Wizard Test Seen数据集Wizard Test Unseen数据集
精度召回率嵌入平均向量极值贪心匹配精度召回率嵌入平均向量极值贪心匹配
ChatGPT-2-Medium+CVAKS(本文)0.3380.3860.7510.2930.5240.3360.3840.7480.2910.523
ChatGPT-2-Medium+Random0.3310.3740.7280.2880.5130.3300.3720.7260.2870.511
Llamma-2-7b+CVAKS(本文)0.4070.4650.8210.3750.6050.4020.4590.8190.3740.601
Llamma-2-7b+Random0.3960.4520.8040.3670.5930.3940.4510.8010.3660.590
Llamma-2-13b+CVAKS(本文)0.4110.4670.8350.3920.6150.4080.4640.8310.3890.613
Llamma-2-13b+Random0.4030.4560.8210.3850.6040.4020.4530.8190.3840.602
), ArticleFig(id=1327018847468630167, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, language=CN, label=表8, caption=

CVAKS在大语言模型上的实验结果

, figureFileSmall=null, figureFileBig=null, tableContent=
模型Wizard Test Seen数据集Wizard Test Unseen数据集
精度召回率嵌入平均向量极值贪心匹配精度召回率嵌入平均向量极值贪心匹配
ChatGPT-2-Medium+CVAKS(本文)0.3380.3860.7510.2930.5240.3360.3840.7480.2910.523
ChatGPT-2-Medium+Random0.3310.3740.7280.2880.5130.3300.3720.7260.2870.511
Llamma-2-7b+CVAKS(本文)0.4070.4650.8210.3750.6050.4020.4590.8190.3740.601
Llamma-2-7b+Random0.3960.4520.8040.3670.5930.3940.4510.8010.3660.590
Llamma-2-13b+CVAKS(本文)0.4110.4670.8350.3920.6150.4080.4640.8310.3890.613
Llamma-2-13b+Random0.4030.4560.8210.3850.6040.4020.4530.8190.3840.602
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202440551, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202440551, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202440551, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202440551, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1753977600000, fullTextJson=null, articleText=null, reference=null)
收藏切换
基于变分注意力知识选择和预训练语言模型的对话生成
张乃洲 1 , 曹薇 1 , 张啸剑 1 , 李石君 2
计算机研究与发展 | 人工智能 2025,62(8): 1902-1917
收起
收藏切换
计算机研究与发展 |人工智能 2025 , 62 (8) : 1902 -1917
基于变分注意力知识选择和预训练语言模型的对话生成
全屏
[Author(id=1327018840678051934, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=zhangnz@126.com, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327018840766132320, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, authorId=1327018840678051934, language=EN, stringName=Naizhou Zhang, firstName=Naizhou, middleName=null, lastName=Zhang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1College of Computer and Information Engineering, Henan University of Economics and Law, Zhengzhou 450046, bio={"content":"

born in 1970. PhD,associate professor. Member of CCF. His main research interests include Web search, data mining, and machine learning

"}, bioImg=null, bioContent=

born in 1970. PhD,associate professor. Member of CCF. His main research interests include Web search, data mining, and machine learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327018840845824098, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, authorId=1327018840678051934, language=CN, stringName=张乃洲, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1河南财经政法大学计算机与信息工程学院 郑州 450046, bio={"img":"Vmy7tbtLP6tH74yc5PW25A==","content":"

1970年生. 博士,副教授. CCF会员. 主要研究方向为Web检索、数据挖掘、机器学习

"}, bioImg=Vmy7tbtLP6tH74yc5PW25A==, bioContent=

1970年生. 博士,副教授. CCF会员. 主要研究方向为Web检索、数据挖掘、机器学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327018840480919639, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, xref=1, ext=[AuthorCompanyExt(id=1327018840489308248, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840480919639, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1College of Computer and Information Engineering, Henan University of Economics and Law, Zhengzhou 450046), AuthorCompanyExt(id=1327018840497696857, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840480919639, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1河南财经政法大学计算机与信息工程学院 郑州 450046)])]), Author(id=1327018840908738660, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327018840975847526, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, authorId=1327018840908738660, language=EN, stringName=Wei Cao, firstName=Wei, middleName=null, lastName=Cao, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1College of Computer and Information Engineering, Henan University of Economics and Law, Zhengzhou 450046, bio={"content":"

born in 1977. Master, associate professor. Her main research interests include machine learning and Web data mining

"}, bioImg=null, bioContent=

born in 1977. Master, associate professor. Her main research interests include machine learning and Web data mining

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327018841034567783, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, authorId=1327018840908738660, language=CN, stringName=曹薇, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1河南财经政法大学计算机与信息工程学院 郑州 450046, bio={"img":"TLWpAnQkRgFW20hotF8+lg==","content":"

1977年生. 硕士,副教授. 主要研究方向为机器学习、Web数据挖掘

"}, bioImg=TLWpAnQkRgFW20hotF8+lg==, bioContent=

1977年生. 硕士,副教授. 主要研究方向为机器学习、Web数据挖掘

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327018840480919639, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, xref=1, ext=[AuthorCompanyExt(id=1327018840489308248, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840480919639, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1College of Computer and Information Engineering, Henan University of Economics and Law, Zhengzhou 450046), AuthorCompanyExt(id=1327018840497696857, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840480919639, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1河南财经政法大学计算机与信息工程学院 郑州 450046)])]), Author(id=1327018841093288041, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327018842766815339, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, authorId=1327018841093288041, language=EN, stringName=Xiaojian Zhang, firstName=Xiaojian, middleName=null, lastName=Zhang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1College of Computer and Information Engineering, Henan University of Economics and Law, Zhengzhou 450046, bio={"content":"

born in 1980. PhD, professor, master supervisor. His main research interests include differential privacy, privacy analysis in language model, and graph data management

"}, bioImg=null, bioContent=

born in 1980. PhD, professor, master supervisor. His main research interests include differential privacy, privacy analysis in language model, and graph data management

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327018842842312812, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, authorId=1327018841093288041, language=CN, stringName=张啸剑, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1河南财经政法大学计算机与信息工程学院 郑州 450046, bio={"img":"/XuCL2abPZbvlbrwR0l3Yg==","content":"

1980年生. 博士,教授,硕士生导师. 主要研究方向为差分隐私、语言模型隐私分析、图形数据管理

"}, bioImg=/XuCL2abPZbvlbrwR0l3Yg==, bioContent=

1980年生. 博士,教授,硕士生导师. 主要研究方向为差分隐私、语言模型隐私分析、图形数据管理

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327018840480919639, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, xref=1, ext=[AuthorCompanyExt(id=1327018840489308248, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840480919639, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1College of Computer and Information Engineering, Henan University of Economics and Law, Zhengzhou 450046), AuthorCompanyExt(id=1327018840497696857, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840480919639, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1河南财经政法大学计算机与信息工程学院 郑州 450046)])]), Author(id=1327018842909421678, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1327018842989113456, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, authorId=1327018842909421678, language=EN, stringName=Shijun Li, firstName=Shijun, middleName=null, lastName=Li, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2School of Computer Science, Wuhan University, Wuhan 430072, bio={"content":"

born in 1964. PhD, professor, PhD supervisor. His main research interests include database theory, Web search engine, and Web mining

"}, bioImg=null, bioContent=

born in 1964. PhD, professor, PhD supervisor. His main research interests include database theory, Web search engine, and Web mining

, aboutCorrespAuthor=null), CN=AuthorExt(id=1327018843056222321, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, authorId=1327018842909421678, language=CN, stringName=李石君, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2武汉大学计算机学院 武汉 430072, bio={"img":"z1WRwFY0PigBq2Siy92GBA==","content":"

1964年生. 博士,教授,博士生导师. 主要研究方向为数据库理论、Web搜索引擎、Web挖掘

"}, bioImg=z1WRwFY0PigBq2Siy92GBA==, bioContent=

1964年生. 博士,教授,博士生导师. 主要研究方向为数据库理论、Web搜索引擎、Web挖掘

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1327018840564805722, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, xref=2, ext=[AuthorCompanyExt(id=1327018840573194331, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840564805722, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2School of Computer Science, Wuhan University, Wuhan 430072), AuthorCompanyExt(id=1327018840581582940, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1327018836873818180, companyId=1327018840564805722, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2武汉大学计算机学院 武汉 430072)])])]
张乃洲1 , 曹薇1, 张啸剑1, 李石君2
作者信息
  • 1河南财经政法大学计算机与信息工程学院 郑州 450046
  • 2武汉大学计算机学院 武汉 430072
作者简介:

1970年生. 博士,副教授. CCF会员. 主要研究方向为Web检索、数据挖掘、机器学习

1977年生. 硕士,副教授. 主要研究方向为机器学习、Web数据挖掘

1980年生. 博士,教授,硕士生导师. 主要研究方向为差分隐私、语言模型隐私分析、图形数据管理

1964年生. 博士,教授,博士生导师. 主要研究方向为数据库理论、Web搜索引擎、Web挖掘

Conversation Generation Based on Variational Attention Knowledge Selection and Pre-trained Language Model
Naizhou Zhang1 , Wei Cao1, Xiaojian Zhang1, Shijun Li2
Affiliations
  • 1College of Computer and Information Engineering, Henan University of Economics and Law, Zhengzhou 450046
  • 2School of Computer Science, Wuhan University, Wuhan 430072
  • born in 1970. PhD,associate professor. Member of CCF. His main research interests include Web search, data mining, and machine learning

    born in 1977. Master, associate professor. Her main research interests include machine learning and Web data mining

    born in 1980. PhD, professor, master supervisor. His main research interests include differential privacy, privacy analysis in language model, and graph data management

    born in 1964. PhD, professor, PhD supervisor. His main research interests include database theory, Web search engine, and Web mining

出版时间: 2025-08-01 doi: 10.7544/issn1000-1239.202440551
文章导航
收藏切换

基于知识的神经对话研究常常面临外部知识包含冗余甚至与对话主题不相关信息的问题,从而导致对话系统性能下降. 知识选择成为解决该问题的重要途径,但现有研究对诸如知识选择器的设计、选择出的知识的利用以及知识选择对话方法适用的场景等问题,还缺乏深入研究. 针对这些问题,提出了一个新的基于变分注意力知识选择和预训练模型的神经对话方法,使用一个基于条件变分自编码(conditional variational autoencoder,CVAE)和多层注意力机制的知识选择算法,自动选择出与当前对话最相关文本知识集合. 该算法有效利用了训练数据中的对话回复来提高知识选择的效率. 使用预训练语言模型Bart作为编码器-解码器架构,将选择的文本知识合并到Bart模型中,并在训练过程中对其进行微调. 实验结果表明,与现有的一些代表性研究方法相比,提出的模型能生成多样性和连贯性更好、准确率更高的对话回复.

基于知识的对话  /  知识选择  /  预训练语言模型  /  条件变分自编码  /  注意力机制  /  记忆网络

Research on knowledge-grounded dialogue often suffers from the problem of external knowledge containing redundant or even noisy information irrelevant to the conversation topic, which leads to a degradation in the performance of the dialogue system. Knowledge selection becomes an important approach to solve this issue. However, existing work has not yet investigated in depth some issues involving it such as how to design a knowledge selector, how to exploit the selected knowledge, what are the suitable scenarios for the knowledge selection conversation methods, etc. In this paper, we propose a new neural conversation method based on conditional variational attention knowledge selection and a pre-trained language model. This method employs a knowledge selection algorithm based on conditional variational autoencoder (CVAE) and a multi-layer attention mechanism to pick up the most relevant textual knowledge collection to the current conversation, which effectively exploits the dialogue response in training data to improve the efficiency of knowledge selection. Our novel model adopts the pre-trained language model Bart as encoder-decoder architecture and incorporates selected textual knowledge into the Bart model to fine-tune it during the training process. The experimental results show that the model proposed, in contrast to the current representative dialog models, can generate more diverse and coherent dialogue responses with higher accuracy.

knowledge-grounded dialogue  /  knowledge selection  /  pre-trained language model  /  CVAE  /  attention mechanism  /  memory network
张乃洲, 曹薇, 张啸剑, 李石君. 基于变分注意力知识选择和预训练语言模型的对话生成. 计算机研究与发展, 2025 , 62 (8) : 1902 -1917 . DOI: 10.7544/issn1000-1239.202440551
Naizhou Zhang, Wei Cao, Xiaojian Zhang, Shijun Li. Conversation Generation Based on Variational Attention Knowledge Selection and Pre-trained Language Model[J]. Journal of Computer Research and Development, 2025 , 62 (8) : 1902 -1917 . DOI: 10.7544/issn1000-1239.202440551
随着移动互联网技术的飞速发展,当前使用人机对话系统(如聊天机器人)取代人工服务,在诸多应用领域,如电子商务、银行、通信等,已成为一种趋势,对话系统已显示出其潜在的巨大商业价值和应用前景. 因此近年来,对话系统的相关研究已成为自然语言处理领域的研究热点之一[1-7]. 当前采用基于深度神经网络技术的编码器-解码器(encoder-decoder)架构作为端到端生成式模型,已成为开放域对话系统研究的主流方法[2-3,5-7]. 但与人类的真实对话相比,目前的对话系统生成的回复还存在很大差距. 如现有研究已表明:采用编码器-解码器模型在生成文本时,容易产生一些常见的无聊与泛泛的回复,诸如“I don’t know”等[4-5],从而造成模型生成的回复信息量小、多样性差. 而基于知识的对话(knowledge-grounded dialogue)主要研究如何有效利用外部知识改善对话生成的多样性和信息性(informative),目前已经成为对话系统研究的一个重要方向[8-22].
外部知识具有多种形式:可以是非结构化的文档[21],也可以是结构化的知识图谱[22]. 非结构化文档的优点是数据极其丰富且获取方便,省去了知识图谱构建的繁琐过程;但缺点是获取的知识信息冗余大、准确率低,且不能直接应用于知识推理. 而本文的研究主要集中在非结构化文档. 当前采用非结构化文档作为外部知识的对话系统相关研究,其基本思路是:将对话数据和外部文本知识共同作为模型的输入,主要研究如何采用各种不同的神经网络架构来有效利用输入的训练数据,最终提升对话系统的性能. 但当前该类研究面临的一个主要困难是:外部的文本知识主要通过信息检索技术自动获取[11]. 如前所述,这些非结构化文本知识往往包含一些冗余甚至与对话主题不相关(噪声)的信息,从而降低了对话系统的性能. 而要去除这些噪音信息,并对相关知识进行人工标注,以明确针对每轮次对话真实的知识(ground truth)是什么,则需要耗费大量的人工费用,这使得该类方法的研究面临不小的挑战. 因此,在对话过程中采用知识选择(knowledge selection)方法 [8-20]成为解决上述问题的有效途径.
从如何利用知识的角度,本文将当前基于知识文档的对话系统研究分为2类:基于隐式知识选择的对话方法和基于显式知识选择的对话方法. 图1显示了这2种方法的区别. 如图1(a)所示,隐式知识选择方法[8-10,20]使用对话历史编码器和知识编码器分别将对话历史和文本知识编码为向量,然后在解码器端采用特定的注意力机制与对话回复进行对齐. 模型对知识选择采用了一种隐式方式,即由编码器来自动选择. 这种方式的优点是模型结构简单,但缺点是文本知识集所包含的噪声知识,可能会使模型在解码对齐过程中产生较大偏差,从而导致对话系统性能下降. 此外,由于这种方法输入解码器的文本知识数量较多,会导致模型在解码过程中占用较多的GPU资源. 图1(b)则给出了另一种知识选择范式:显式知识选择方法[11-19]. 该方法与隐式知识选择方法的主要区别在于:将知识向量送入解码器之前,会通过一个知识选择器预先选择出与当前对话最相关的知识,然后将选中的知识向量送入解码器进行解码训练. 该类方法的知识选择采用了一种显式方式,即专门训练一个知识选择器来负责知识的选择. 由于预先选择出了与当前对话最相关的文本知识,因此有效减少了模型在解码对齐过程中产生偏差的概率. 此外,由于输入解码器的文本知识数量较少,显式知识选择方法在解码过程中占用的GPU资源也少于隐式知识选择方法. 本文的实验结果也表明从总体上看,基于显式知识选择的对话方法在性能指标上要优于基于隐式知识选择的对话方法.
显式知识选择对话方法发挥作用的一个关键因素是能够在模型解码前有效利用训练数据中的对话回复来进行知识选择,从而提高了知识选择的效率. 有研究表明,利用训练数据中的对话回复能有效提高知识选择的精度. 其基本假设为:如果模型能够提前知道(或看到)对话回复,那么知识选择就变得比较容易[13]. 但在当前的基于编码器-解码器结构的知识对话方法框架中,对话回复只有在训练阶段才是可见的,而在预测阶段如何利用对话回复则是一个具有挑战性的任务. 作为一种生成式模型,条件变分自编码模型(conditional variational autoencoder,CVAE)[5]能够在训练阶段建模数据的条件先验和条件后验分布,并最小化条件先验分布与条件后验分布之间的KL散度(Kullback-Leibler divergence). 这样在预测阶段便能使用条件先验分布近似条件后验分布. 这种方法解决了前面提到的在预测阶段如何利用对话回复的问题. 这也是文本提出的方法中使用CVAE的主要动机之一.
尽管研究者们对显式知识选择对话方法已经进行了一些研究,但当前仍存在一些重要问题需要解决:1)如何对文本知识、对话历史和对话回复进行有效编码;2)如何设计高效的知识选择器以提高知识选择的效率;3)如何设计有效的对话回复解码器,以高效利用知识选择器选择出的知识生成高质量的对话回复;4)显式知识选择对话方法适用的场景有哪些. 目前对这些问题的研究还不够深入. 为此,本文提出了一个新的基于条件变分注意力知识选择和预训练语言模型(conditional variational attention knowledge selection and pre-trained language model, CVAKSPLM)的对话生成方法,试图对以上4个问题进行研究. 本文研究目标为:在训练数据提供了大量包含一定噪声信息的非结构化文本知识但数据集没有提供真实文本知识标签(无监督知识选择[15])的场景下,模型如何有效学习并自动选择出最相关的若干文本知识,从而显著提升对话系统的性能. 本文的主要贡献有3点.
1)提出了一个基于CVAKSPLM的神经对话模型. 该模型的核心为知识选择算法,其原理为:首先使用BERT模型将训练数据中的对话历史、文本知识和对话回复统一映射到相同维度的特征空间中. 然后使用条件变分自编码模型将编码后的对话历史和对话回复映射到潜变量空间中,这使得在训练过程中,对话历史和对话回复能够纠缠(entangle)在同一个特征空间. 通过最小化潜变量的条件先验分布和条件后验分布之间的KL散度,便能在预测阶段使用对话数据的条件先验分布近似条件后验分布. 通过抽样潜变量空间得到知识查询向量q,再利用一个多层注意力模块Attention计算知识查询向量q与BERT模型编码后的文本知识向量之间的相关度,最后按相关度从大到小排序,取top-m文本知识集$\{ {k_i}\} _{i = 1}^m$作为模型选择出的最相关文本知识集合.
2)CVAKSPLM模型采用预训练语言模型(pre-trained language model,PLM)Bart作为编码器-解码器架构,将选择的文本知识合并到Bart模型中,并在训练过程中对其进行微调. 其动机是希望利用Bart模型较强的文本生成能力. Bart编码器分别编码了对话历史和知识选择模块选择出的文本知识集$\{ {k_i}\} _{i = 1}^m$,并对其进行连接,然后将连接向量以及训练数据中的对话回复共同输入到Bart解码器中,进行自回归解码训练.
3)将本文提出的CVAKSPLM模型在不同数据集上与多个基线模型进行了比较,实验结果表明:在训练数据集提供的非结构化文本知识数量较多且不包含真实文本知识标签的情况下,在衡量对话回复生成性能(如多样性、连贯性和准确率等)的多个指标上,本文提出的方法明显优于现有的一些代表性研究方法.
早期基于编码器-解码器模型的神经对话生成研究主要集中在神经网络结构的设计上[1]:如采用基于循环神经网络(recurrent neural network,RNN)和注意力机制的方法[2]、基于潜变量RNN模型[3]和基于条件变分自编码模型[5]的方法、基于Transformer模型[6]和预训练语言模型[7]的方法等. 而近年来,许多研究集中在如何有效利用外部知识来提升对话系统的性能[8-22]. 如本文前面所提到的,可将当前基于知识的对话系统研究分为2类.
第1类研究为基于隐式知识选择的方法[8-10,20]. 如Ghazvininejad等人[8] 提出了一种基于RNN的Seq2Seq模型,用于编码对话历史和事实性知识,并用于解码生成对话回复. Li等人[9] 提出了一种基于增量Transformer推敲解码器模型,用于基于文档知识的多轮次对话任务,能有效提升回复生成的上下文一致性和知识的正确性. 而针对基于知识的对话系统常常面临低资源(low resources)应用场景中的训练数据不易获得以及很难迁移到新领域等问题,Liu等人[10] 提出了一个新的3阶段学习框架(TSLF),并使用了知识感知的Transformer(KAT)模型构建端到端对话系统,用于低资源环境下的对话生成.
第2类研究为基于显式知识选择的对话方法[11-19]. Dinan等人[11]首先提出了将基于知识的对话任务分解为知识选择和回复生成2个子任务的思想,并针对开放域对话系统研究缺乏大型基于知识的对话语料,造成知识使用比较困难的问题,从Wikipedia搜集并推出了一个大型基于知识的著名知识对话数据集Wizard-of-Wikipedia,同时提出了一个基于2阶段生成式Transformer记忆网络(TMN)模型用于对话回复的生成. 目前使用2阶段基于知识的对话生成方法已经成为知识选择方法的标准范式. 针对文本知识中常常包含一些冗余甚至噪声信息而导致对话系统的性能下降的问题, Lian等人[12] 提出了一个端到端的学习知识选择对话模型PostKS,其中编码器采用了带有GUR单元的双向RNN结构,而解码器采用了带有GUR单元的RNN结构. 编码器分别用于编码话语、对话回复与知识,设计了一个知识管理器,采用先验知识分布近似后验证知识分布的方法进行知识的选择;而解码器则通过合并选择的知识和输入数据生成对话回复. Kim等人[13] 提出了一个序列潜变量模型-序列知识转移(SKT)模型,可以追踪到知识上的先验和后验分布. 该模型不但能利用对话回复进行对话选择,同时也减少了知识选择的多样性造成的模糊性. 针对在近似过程中先验知识和后验知识信息存在差距的问题,Chen等人[14] 提出了一种桥接先验知识选择和后验知识选择之间差距的方法:使用了一个PIPM模块从后验证信息中获取必要的信息来增强先验知识的选择;同时使用了一个基于知识蒸馏的模块来训练解码器从先验知识分布中选择知识.
针对当前不少显式知识选择对话方法研究[13-14,16- 17]采用了有监督知识学习方法的问题,即模型在训练知识选择过程中依赖人工标记的真实知识,Chen等人[15]又提出了一种无监督知识选择方法,用于真实的知识标签未知的知识选择场景. 其主要思想是先利用远监督方法获得Oracle知识标签,然后再使用知识蒸馏的方法来减轻噪声标签问题.
近年来也出现了一些使用预训练语言模型的知识对话方法[18-20]. 如Zhao等人[18]提出了一种使用预训练模型的无监督对话方法:先使用BERT模型编码知识,然后采用了一个序列知识选择器生成伪知识标记,将生成的伪知识、对话历史和回复输入到GPT-2解码器进行自回归解码训练;在训练过程中还采用了强化学习方法来提升知识选择的精度.
本文的研究受CVAE模型 [5]和记忆网络(memory network)[23]相关研究的启发. 一方面,使用CVAE模型能够有效利用训练数据中的对话回复,使得知识选择受到对话回复语义的约束. 另一方面,本文借鉴了记忆网络多层注意力机制来计算文本知识与CVAE模型输出的知识查询向量之间的相关度,以提高知识选择的精度. 总体上看,本文的研究与文献[9-10,12-13]比较相关,但与之存在诸多不同. 表1显示了本文提出的方法在编码器、解码器和知识选择方法等方面与其他2种代表性的显式知识选择对话方法的区别.
从表1可以看到,本文提出的CVAKSPLM模型在解码器、编码器和知识选择方法等方面均与其他2种方法存在差别. 尤其在知识选择方法上,主要存在4方面不同:1)本文使用CVAE建模的是对话潜变量的条件先验分布(对话历史)和条件后验分布(对话历史+对话回复),通过抽样后得到查询向量,然后计算查询向量与知识向量之间的注意力权重. 而其他2种方法直接建模了知识上的条件先验分布和条件后验分布,这种计算是通过采用Dot Product注意力机制来计算知识与对话编码向量之间的注意力权重得到的[12-13]. 2)本文使用了一个多层注意力模块来计算知识查询向量与文本知识向量之间的相关度,而PostKS方法仅使用了单层Dot Product注意力机制,SKT方法在每个时间步上与PostKS方法类似. 3)其他2种方法在选择出最相关文本后,直接将原先的文本编码向量送入解码器. 而本文采用了Bart编码器对选择出的相关文本进行再编码的策略(参见图2). 这种方式使得模型对文本知识的选择和解码这2个过程,可以在不同的语义向量空间中进行. 4)SKT方法属于有监督知识选择方法,模型在训练过程中依赖人工标记的真实知识,这显然增加了训练的成本. 而与PostKS方法类似,本文提出CVAKSPLM模型以无监督方式进行知识选择,不需要任何真实的文本知识标签.
首先给出本文提出的CVAKSPLM神经对话模型的问题描述:
定义1. 对话历史(dialogue history). 对话历史Ui可表示为一个话语序列的集合$ \{ {u_1},{u_2},…,{u_k}\} $,而$ {u_t} = \{ w_{1}^{\,t},w_2^{\,t},…,w_m^{\,t}\} $表示Ui中的一条话语,其中$ w_j^{\,t} $表示话语中的一个词.
定义2. 对话回复(dialogue response). 对话回复ri可表示为一个句子$ \{ {w_1},{w_2},…,{w_l}\} $,wi表示其中一个词.
定义3. 文本知识(knowledge). 文本知识Ki可表示为一个知识的集合$ \{ {k_1},{k_2},…,{k_n}\} $,而$ {k_t} = \{ w_{1}^{\,t},w_2^{\,t},…,w_s^{\,t}\} $表示Ki中的一条知识语句.
由以上给出的3个基本定义可将本文提出的CVAKSPLM模型对话生成问题定义为:给定训练数据集$ T = \{ ({U_i},{K_i},{r_i})\} _{i = 1}^n,\;R = \{ {r_i}\} _{i = 1}^n $,学习一个映射$ \hat f:T \to \hat R $. 其中,$ \hat R = \{ {\hat r_i}\} _{i = 1}^n $为模型$ \hat f $输出的对话回复预测值集合. 设模型的参数为$ \theta $,参数空间为$ \varTheta $,则模型训练过程的目标是获得$ {\theta ^*} = \mathop {\arg \min }\limits_{\theta \in \varTheta } (L(R,\hat f(T))) $,其中损失函数$ L(R,\hat f(T)) = - \displaystyle\sum_{i = 1}^n {\log P({r_i}|{U_i},{K_i})} $.
图2描述了本文提出的CVAKSPLM神经对话模型的总体架构,主要包含3个部分.
1)条件变分注意力知识选择模块. 该模块对应图2上部虚线方框,又分为2个子模块: CVAE和层次注意力知识选择模块. 其原理为:首先将对话历史和对话回复进行连接,然后使用BERT编码器进行编码,得到变分自编码模块的后验输入(虚线表示);同理将对话历史使用BERT编码器进行编码,得到先验输入(实线表示). 然后使用CVAE模块将先验输入和后验输入变换为知识查询向量. 使用BERT编码器对文本知识集Ki进行编码,生成层次注意力知识选择模块中的知识记忆向量;将知识查询作为层次注意力知识选择模块的输入,采用基于多层注意力机制的知识选择算法,计算知识查询与知识向量之间的注意力权重向量,最后得到选择的知识索引.
2)Bart编码器. 如图2左下部所示,使用Bart编码器将对话历史编码为向量;使用由条件变分注意力知识选择模块得到的选择知识索引,生成最相关文本知识集,然后使用Bart编码器对其进行重新编码,得到相关文本知识编码向量. 将对话历史编码向量与相关文本知识编码向量进行连接并作为Bart编码器的输出.
3)Bart解码器. 如图2右下部所示,将对话回复和得到的Bart编码器输出一起作为Bart解码器的输入,进行自回归解码训练.
条件变分注意力知识选择模块的处理过程可以分为如下2个步骤.
1)生成知识查询
CVAE的基本原理[5]可表示为
$\begin{split} \mathcal{L}(\theta ;{\boldsymbol{x}},{\boldsymbol{y}}) = & - {D_{\rm{KL}}}(Q({\boldsymbol{z}}|{\boldsymbol{x}},{\boldsymbol{y}})||P({\boldsymbol{z}}|{\boldsymbol{y}}))+ \\& {E_{{\boldsymbol{z}}\sim Q}}[\log P({\boldsymbol{x}}|{\boldsymbol{z}},{\boldsymbol{y}})] \leqslant \log P({\boldsymbol{x}}|{\boldsymbol{y}}), \end{split}$
其中x表示样本数据,y为条件,θ表示模型参数. $ \mathcal{L}(\theta ;{\boldsymbol{x}},{\boldsymbol{y}}) $为样本观察数据条件分布$ P({\boldsymbol{x}}|{\boldsymbol{y}}) $对数的一个下界ELBO(evidence lower bound),$ {D}_{\rm KL}(\cdot ) $表示KL散度. 当公式中潜变量${\boldsymbol{z}} $的条件先验分布$ P({\boldsymbol{z}}|{\boldsymbol{y}}) $与近似条件后验分布$ Q({\boldsymbol{z}}|{\boldsymbol{x}},{\boldsymbol{y}}) $的KL散度最小时,取得该下界,即模型生成样本数据的条件概率达到最大.
本文中对话回复代表样本数据为x,对话历史代表条件为y. 如图2所示,CVAE模块的输入有${\boldsymbol{ pos}}{{\boldsymbol{t}}_{\boldsymbol{x},{\boldsymbol y}}} $和${\boldsymbol{pri}}{{\boldsymbol{o}}_{\boldsymbol{y}}} $.
$ \begin{aligned}&{\boldsymbol{ pos}}{{\boldsymbol{t}}_{\boldsymbol{x},{\boldsymbol y}}} = Bert([{U_i},{r_i}]), \\&{\boldsymbol{pri}}{{\boldsymbol{o}}_{\boldsymbol{y}}} = Bert({U_i}), \end{aligned} $
其中$ Bert( \cdot ) $表示BERT编码器映射函数,$ [ \cdot , \cdot ] $表示连接操作;$ {U_i} = [{\rm{CLS}}]{u_1}{u_2}…{u_k}[{\rm{SEP}}] $,$ [{U_i},{r_i}] = [{\rm{CLS}}]{u_1}{u_2}… {u_k}[{\rm{SEP}}]{w_1}{w_2}…{w_l}[{\rm{SEP}}] $分别表示对定义1和定义2描述的对话历史、对话回复添加类别标记$ [{\rm{CLS}}] $和分隔标记$ [{\rm{SEP}}] $.
本文最终取BERT编码器输出的$ [{\rm{CLS}}] $标记所对应的向量,因此有$ {\boldsymbol{ pos}}{{\boldsymbol{t}}_{\boldsymbol{x},{\boldsymbol y}}}\in {\mathbb{R}^d} $,d=768.
CVAE模块生成知识查询的过程分为训练和预测2个阶段. 在训练阶段,模型将最小化条件先验分布与条件后验分布的KL散度:${D_{\rm{KL}}}(q||p)$,同时抽样条件后验潜变量$ {\boldsymbol{z}} $生成知识查询. 而在预测阶段,模型抽样条件先验潜变量$ {\boldsymbol{z}}' $生成知识查询.
以训练阶段生成知识查询为例,后验分布潜变量${\boldsymbol{ z}} $的均值与方差生成过程可表示为
$ {\boldsymbol{ \mu}} ,{{\boldsymbol{\sigma}} ^2} = PosteriorNW({\boldsymbol{pos}}{{\boldsymbol{t}}_{{\boldsymbol{x}},{\boldsymbol{y}}}}), \;\;{\boldsymbol{z}}\sim \mathcal{N}({\boldsymbol{\mu}} ,{{\boldsymbol{\sigma}} ^2}).$
则后验分布潜变量$ {\boldsymbol{z}} $的抽样过程为
$ {\boldsymbol{z}} = {\boldsymbol{\mu}} + {\boldsymbol{\sigma}} \odot{\boldsymbol{ \varepsilon}} , \;\;{\boldsymbol{\varepsilon}} \sim \mathcal{N}({\boldsymbol{0}},{\boldsymbol{I}}){\text ,}$
其中符号$ \odot $表示哈达玛积运算.
使用多层感知机对${\boldsymbol{z}} $进行变换,生成知识查询
$ {{\boldsymbol{q}}_{\rm{kno}}} = MLP({\boldsymbol{z}}). $
式(3)~(5)描述了训练阶段知识查询的生成过程. 在图2上部的CVAE框图中,使用虚线表示其生成过程. 而在预测阶段,知识查询的生成过程类似,对应图2上部的CVAE框图中的实线部分.
2)层次注意力知识选择
受记忆网络 [23]的启发,本文采用了一种层次注意力知识选择方法,用于计算知识查询$ {{\boldsymbol{q}}_{\rm{kno}}} $与文本知识集Ki之间的注意力权重向量,以生成最相关文本知识. 为此先对文本知识集Ki做如下变换:
$ {\boldsymbol{M}} = Bert({K_i}), $
其中$ {K_i} = \{ [{\rm{CLS}}]{k_j}[{\rm{SEP}}]\} _j^n $,表示对定义3描述的文本知识集中的每条文本知识,添加类别标记$ [{\rm{CLS}}] $和分隔标记$ [{\rm{SEP}}] $后得到的集合. M为编码后的知识张量. 与式(2)相同,这里有$ {\boldsymbol{M}} \in {\mathbb{R}^{n \times d}} $,n为文本知识Ki中知识的数量,d=768.
层次注意力知识选择过程采用如下公式计算:
$ {{\boldsymbol{Q}}^0} = {\boldsymbol{B}}{{\boldsymbol{q}}_{{\mathrm{kno}}}}, $
$ \begin{aligned}& {\boldsymbol{Pro}}{{\boldsymbol{b}}^{\,t}} = softmax({{\boldsymbol{Q}}^{t - 1}}{({\boldsymbol{M}}{{\boldsymbol{A}}^{\,t}})^{\mathrm{T}}}), \\& {{\boldsymbol{Q}}^{\,t}} = {\boldsymbol{Pro}}{{\boldsymbol{b}}^{\,t}}({\boldsymbol{M}}{{\boldsymbol{C}}^{\,t}}) + {{\boldsymbol{Q}}^{t - 1}} .\end{aligned} $
本文采用的注意力模块的层数为3. 因此式(8)中,t=1,2,3.
式(7)中$ {{\boldsymbol{q}}_{\rm{kno}}} $表示由式(5)得到的知识查询向量,$ {\boldsymbol{B}} $为知识查询向量的嵌入表示矩阵,$ {{\boldsymbol{Q}}^0} $为注意力-1层$ {{\boldsymbol{q}}_{\rm{kno}}} $的嵌入表示输入.
式(8)为迭代计算过程. M代表图2中的知识记忆,由式(6)计算得到;$ {{\boldsymbol{A}}^{\,t}} $表示当前第t层M的前嵌入表示矩阵,而$ {{\boldsymbol{Q}}^{t - 1}} $表示当前第t层的知识查询向量的输入. $ softmax(\cdot ) $表示归一化处理算子,$ {\boldsymbol{Pro}}{{\boldsymbol{b}}^{\,t}} $则表示$ {{\boldsymbol{Q}}^{t - 1}} $与当前M的前嵌入表示向量之间的注意力权重分布. $ {{\boldsymbol{C}}^{\,t}} $表示当前第t层M的后嵌入表示矩阵,M与$ {{\boldsymbol{C}}^{\,t}} $相乘进行嵌入化表示之后,再与$ {\boldsymbol{Pro}}{{\boldsymbol{b}}^{\,t}} $相乘,得到按注意力权重分配的知识表示向量. 最后再将知识表示向量与$ {{\boldsymbol{Q}}^{t - 1}} $相加(类似残差连接),得到第t层知识查询向量输出$ {{\boldsymbol{Q}}^{\,t}} $.
最终选择的知识索引为
$ \begin{aligned}& {\boldsymbol{Id}}{{\boldsymbol{x}}^{\rm{sel}}} = {\mathrm{arg sort}}({\boldsymbol{Pro}}{{\boldsymbol{b}}^{\rm{final}}}), \\&K_i^{\rm{sel}} = \# ({K_i},{\boldsymbol{Id}}{{\boldsymbol{x}}^{\rm{sel}}}), \end{aligned} $
其中$ {\boldsymbol{Pro}}{{\boldsymbol{b}}^{\rm{final}}} \in {\mathbb{R}^n} $为经过t次迭代后$ {\boldsymbol{Pro}}{{\boldsymbol{b}}^{\,t}} $的终值,n为文本知识集Ki中知识的数量;$ {\rm{argsort}}(\cdot ) $表示索引排序函数,其作用是对输入的一组概率向量按分量值从大到小排序,最后返回top-m向量索引值;$ {\boldsymbol{Id}}{{\boldsymbol{x}}^{\rm{sel}}} $表示最终模型自动选择的最相关文本知识索引向量;$ \# ( \cdot , \cdot ) $表示按索引对知识取值算子,$ K_i^{\rm{sel}} $为最终模型选择的最相关文本知识集,满足$ |K_i^{\rm{sel}}| = m $,$m \ll n$.
本文使用预训练模型Bart [24]作为基本架构,该模型采用了编码器-解码器结构,其中编码器和解码器均使用了Transformer作为基本单元. Bart编码器结构与BERT模型类似,属于一种双向降噪自编码器. 如图2所示,本文中Bart编码器的输入有2个:一个是对话历史,由多个话语组成;另一个是选择的文本知识,由一组句子组成.
Bart编码器处理对话历史的过程为
$ {\boldsymbol{h}}{{\boldsymbol{u}}_i} = Bar{t_{\rm{enc}}}({U_i}), $
其中$ Bar{t}_{\rm{enc}}(\cdot ) $表示Bart编码器映射函数,$ {U_i} = {u_1} [{\rm{SEP}}] {u_2}[{\rm{SEP}}]…{u_k} $表示对定义1描述的对话历史添加符号$ [{\rm{SEP}}] $作为分隔标志;而$ {\boldsymbol{h}}{{\boldsymbol{u}}_i} \in {\mathbb{R}^{{T_k} \times d}} $表示Bart编码器最后的隐层输出,其中$ {T_k} $为$ {U_i} $中词的个数,d为隐层输出的维度,d=768.
类似地,Bart编码器对$ K_i^{\rm{sel}} $的处理过程为
$ {\boldsymbol{h}}{{\boldsymbol{k}}_i} = Bar{t_{\rm{enc}}}(K_i^{\rm{sel}}), $
其中$ K_i^{\rm{sel}} = {k_1}[{\rm{SEP}}]{k_2}[{\rm{SEP}}]…{k_m}[{\rm{SEP}}] $表示对选择文本知识添加$ [{\rm{SEP}}] $符号;而$ {\boldsymbol{h}}{{\boldsymbol{k}}_i} \in {\mathbb{R}^{{T_m} \times d}} $表示Bart编码器的隐层输出,其中$ {T_m} $为$ K_i^{\rm{sel}} $中词的个数,d为隐层输出的维度,d=768.
最终Bart编码器的隐层输出可以表示为
$ {{\boldsymbol{h}}_i} = [{\boldsymbol{h}}{{\boldsymbol{u}}_i},{\boldsymbol{h}}{{\boldsymbol{k}}_i}] \in {\mathbb{R}^{({T_k} + {T_m}) \times d}}, $
其中$ {{\boldsymbol{h}}_i} $为$ {\boldsymbol{h}}{{\boldsymbol{u}}_i} $与$ {\boldsymbol{h}}{{\boldsymbol{k}}_i} $的连接,后续将作为Bart解码器的输入参与模型的解码.
与GPT模型类似,Bart解码器采用了自回归解码方式. 从图2可以看出,Bart解码器的输入有2个:Bart编码器的隐层输出$ {{\boldsymbol{h}}_i} $(由式(12)计算得到);对话回复$ {r_i} = \langle {\rm s}\rangle {w_1}{w_2}…{w_l} $,是由定义2描述的对话回复通过整体向右移动1个字符,然后在最左端添加一个开始标志$ \langle {\rm s}\rangle $得到的字符序列. Bart解码器输出可表示为
$ {\hat r_i} = Bar{t_{{\mathrm{dec}}}}({{\boldsymbol{h}}_i},{r_i}), $
其中$ {\hat r_i} = {\tilde w_1}{\tilde w_2}…{\tilde w_l}\langle /{\rm s}\rangle $,$ {\tilde w_i} $表示对应回复$ {r_i} $中的字符$ {w_{i - 1}} $模型输出的字符,$ \langle /{\rm s}\rangle $为结束标志.
模型的训练过程是最小化以下目标损失函数:
$ \mathcal{L}(\theta ;U,K,R) = {\mathcal{L}_{\rm{nll}}}(R,\hat R) + \lambda \cdot {D_{\rm{KL}}}(q||p), $
其中$ \theta $表示整个模型的参数,U表示训练数据中的对话历史,K表示训练数据中的文本知识,R表示训练数据中真实的对话回复,$\hat R$表示模型预测输出的回复. $ {\mathcal{L}_{\rm{nll}}}(R,\hat R) = - \displaystyle\sum_{i = 1}^n {\log P({r_i}|{U_i},{K_i})} $表示负对数似然损失函数,$ {D_{\rm{KL}}}(q||p) $表示CVAE的条件先验分布与条件后验分布的KL散度(参见2.1节),$ \lambda $表示权重系数.
本文采用了2个常用的基于知识的对话数据集Wizard [11]和CMU_DoG[25]. Wizard数据集基于维基百科数据,属于一个多轮次闲聊数据集,每轮对话包含导师(wizard)和学徒(apprentice)两个角色,双方围绕某一个主题展开对话. 数据集提供了基于多个主题的文本知识. 对原始的Wizard数据集进行预处理,得到如下3个不同规模的子数据集:训练集、校验集和测试集大小分别为:74 092,4 057,7 789. 其中,测试集又被划分成Test Seen和Test Unseen两个:Test Seen包含的主题均为训练集中已出现的主题,大小为3865. Test Unseen包含的主题为训练集和校验集中均未出现的主题,大小为3924. 本文使用的Wizard数据集中,平均每轮次对话包含约40条文本知识,而每条文本知识的平均长度(单词的个数)约为22. 与Wizard数据集包含多个主题不同,CMU_DoG数据集主要集中在电影评论,也属于多轮次对话数据集,每轮对话包含2个对话者(worker). 数据集提供了额外的文本知识,包括电影名、年份、导演、简介、评论等信息,双方以此作为背景知识进行对话. 经过预处理后得到的3个子数据集的大小分别为:66332(训练集)、3946(校验集)、3269(测试集). 本文使用的CMU_DoG数据集中,平均每轮次对话包含的文本知识为3条,每条知识的平均长度约为26.
沿用多个相关研究的做法,本文采用了多个基于自动评价的度量指标BLEU[26],Rouge,Distinct[4]来评估对话系统生成回复的性能. 其中, BLEU-n指标度量了模型生成的回复与真实回复之间的n-gram词覆盖度,是一种准确率指标,具体包括BLEU-1~BLEU-4.与BLEU指标不同,Rouge指标则主要用于召回率的计算,其中Rouge-n表示基于n-gram词的召回率. 而Distinct-n指标则计算了对话模型在测试数据集上,生成的所有回复中不重复的n-gram词数量占所有n-gram词数量的比率,属于一种多样性和信息性指标,常用的指标有Dist1,Dist2.
为了验证本文提出方法的有效性以及研究显式知识选择方法适用的场景,本文采用了5种代表性对话方法作为基线模型,与本文提出的CVAKSPLM模型进行对比实验.
1)ITDD为文献[9]提出的一种基于Transformer的对话模型,属于隐式知识选择对话方法. 论文源代码可从github1下载.
2)PostKS为一种基于双向RNN 结构的encoder-decoder神经对话模型[12],属于显式知识选择对话方法. 编码器分别用于编码话语和文本知识,采用一个知识管理器用于知识的选择,而解码器则通过合并选择的知识和输入数据生成对话回复. 论文源代码可从github2下载.
3)SKT为一种采用序列潜变量模型的encoder-decoder神经对话模型[13],属于显式知识选择对话方法. 编码器使用了BERT模型,而解码器采用了Transformer的模型. 由于该模型需要人工标记知识标签,因此本文在实验中去除了原模型中的知识损失,使其成为非监督知识选择方法. 论文源代码可从github3下载.
4)KAT_TSLF是一种3阶段学习框架,用于低资源环境下的对话生成[10],属于隐式知识选择对话方法. 该模型使用知识感知的Transformer 模型作为编码器和解码器,用于编码对话历史和文本知识,并解码生成对话回复. 论文源代码可从github4下载.
5)GenKS为文献[20]提出的一种基于预训练模型Bart的神经对话模型,属于有监督隐式知识选择对话方法. 该模型的架构采用了类似提示学习(prompt learning)的范式[27]. 在Wizard数据集上GenKS模型的输入数据需要手工标注的知识. 论文源代码可从github5下载.
本文采用的预训练模型Bart的版本为bart-base(参数量约140×106). BERT模型的版本为bert-base-uncased(参数量约110×106). 模型使用了AdamW最优化器(β1=0.9,β2=0.999,ε=1E−8). 针对学习率,模型对不同模块采用了不同策略:Bart和BERT预训练模型的学习率设为4E−5,而其他模块为2E−2. 在训练过程中,冻结了Bart模型中除layers-4,layers-5,layernorm embedding之外的其他所有层. 在Wizard数据集上设置选择的文本知识数量m=2,而 CMU_DoG上则设置m=1.
表2~4给出了6种模型在3个测试数据集上的相关实验结果.
1)Wizard数据集上的评估结果. 从表2和表3可以看到,本文提出的CVAKSPLM模型除了在Wizard Test Seen数据集的ROUGE-1指标上略低于PostKS模型外,在其他所有指标上均超过了其他5种基线模型. 例如,从衡量文本生成的准确率和连贯性的BLEU-4指标来看,表2中CVAKSPLM模型的值达到了0.1180,GenKS模型达到了0.0586,PostKS模型达到了0.0566,KAT_TSLF模型达到了0.0546,而SKT模型达到了0.0477,ITDD模型分值最低,仅为0.0101. 而从衡量文本生成的多样性和信息性指标Dist1,Dist2来看,CVAKSPLM模型也取得了最优的结果. 这表明在Wizard数据集上,与基线模型相比本文提出的CVAKSPLM模型取得了较明显的优势. 其原因可能是:一方面,知识选择方法起到了关键作用. 例如,从模型结构上看,尽管ITDD 模型和KAT_TSLF模型均采用了基于Transformer的架构,而PostKS模型采用了较老的RNN架构,但由于其引入了知识选择方法,因此在Wizard数据集上,其BLEU-1至BLEU-4指标反而优于ITDD和KAT_TSLF. 这说明知识选择方法是有效的. 而CVAKSPLM模型采用了更好的知识选择算法,这可能是CVAKSPLM模型指标较优的原因之一. 另一方面,除GenKS模型外,与其他4个基线模型相比,CVAKSPLM模型采用了更强大的预训练模型Bart作为基本架构,使其能有效利用Bart较强的文本生成能力. 这也可能是CVAKSPLM模型效果较好的另一原因. 此外,本文由于去除了SKT模型的知识损失,使其无法利用知识标签进行学习,所以其实验结果不理想. GenKS模型整体上的指标仅次于CVAKSPLM模型,其原因主要在于:一方面在Wizard数据集上GenKS模型的输入数据采用了手工标注的知识[20],另一方面GenKS模型也采用了Bart模型,使得GenKS模型在该任务上取得了较好的表现.
2)CMU_DoG数据集上的评估结果. 表4的结果与表2和表3相比呈现了一定的差异. 从表4可以发现,本文提出的CVAKSPLM模型在CMU_DoG数据集上,仅在BLEU-1至BLEU-4指标上优于其他5种基线模型,但也仅略好于GenKS和KAT_TSLF模型. 而KAT_TSLF模型则取得了在其他4个指标上的最优值. 分析其原因可能是本文使用的Wizard数据集和CMU_DoG数据集在各自提供的附加文本知识集方面存在一定差异. 如前所述,Wizard数据集中每轮次对话提供的附加文本知识集平均包含约40条文本知识,每条知识的平均长度约为22;而在CMU_DoG数据集中对应的是3条文本知识,每条知识的平均长度约为26. 由此可见,在训练过程中,Wizard数据集提供的文本知识数量大大超过了CMU_DoG数据集,这使得其文本知识中所包含的ground truth知识被其他冗余和噪声知识“淹没”的概率,也大大超过了CMU_DoG数据集. 在这种场景下,知识选择算法的重要性显得比较突出,这也使得CVAKSPLM模型的性能在Wizard数据集上明显优于CMU_DoG数据集. 而在CMU_DoG数据集上,由于每轮次对话提供的文本知识数量较少(仅为3条),这使得CVAKSPLM模型的知识选择算法优势并不十分明显. 观察表2~4,作为另外2个带有知识选择算法的PostKS模型和SKT模型也存在类似现象.
另外,KAT_TSLF模型主要用于低资源环境下的对话生成. 该模型在微调之前,使用了较大规模的非知识对话数据集和非结构的知识库进行了预训练[10],这可能也是KAT_TSLF模型在4个指标上取得最优值的另一原因. 而与Wizard数据集上不同,GenKS模型在CMU_DoG数据集上,其输入数据并未采用手工标注的知识[20],这也使得其指标逊色于在Wizard数据集上的结果.
根据以上分析,可以得到如下结论:总体上看,本文提出的CVAKSPLM模型能够有效提高对话模型的性能,特别是在训练数据提供了数量较多且包含一定噪声的文本知识的场景下, CVAKSPLM模型的作用非常显著. 此外,从整个实验结果来看,显式知识选择方法在训练文本知识数量较多的情况下,指标要明显优于隐式知识选择方法. 而在训练文本知识数据较少的情况下,显式知识选择方法的优势并不明显,采用更强大的模型如Bart,GPT等可能效果更好.
3)人工评价结果. 除了自动评价方法外,本文分别在Wizard Test Seen和Wizard Test Unseen数据集上随机抽取了300个样本,聘请了3个人员对各模型生成的回复质量进行了人工评价. 遵循文献[10,18]的方法,每个测试数据样本提供了对话历史、对话回复和文本知识集. 评价者对模型生成的回复从3个方面进行评分:语言流利性(language fluency)、知识相关性(knowledge relevance)和上下文连贯性(context coherence). 其中,每个指标的分值取自集合{0,1,2},分别对应于“差”、“中性”和“好”. 而评价者的评价一致性度量使用了Fleiss’ Kappa指标,最后取3个指标的Kappa平均值. 表5给出了人工评价结果,可以看到,在Wizard Test Seen和Wizard Test Unseen数据集上,本文提出的CVAKSPLM模型在3个评价指标上均优于其他4个模型. 另外,表5所有的Kappa值均大于0.6,表明3个评价者在人工评价过程中取得了较好的一致性.
4)消融研究. 为了进一步评估本文提出的模型中各个组件对模型性能的影响,在Wizard Test Unseen数据集上,本文进行了相应的消融实验,实验结果如表6所示. 其中CVAKSPLM为本文提出原始方法,而CVAKSPLM with KAT表示对CVAKSPLM模型进行了一些修改:将预训练Bart编码器-解码器替换为基于知识感知的Transformer(KAT)[10]编码器-解码器,其他组件保持不变. CVKASPLM with random selection表示为去除CVAKSPLM模型中的CVAKS组件,并使用随机化知识选择方法来替代,其他组件保持不变.
从表6可以看到,去除各相应组件后,与原始模型相比,其他2个模型在BLEU-3/BLEU-4指标、Dist1/Dist2指标和ROUGE-1/ROUGE-2指标上均出现了一定下降. 其中CVKASPLM with random selection性能下降得较多,与原始模型相比,其BLEU-3和BLEU-4指标下降了约29.29%和32.1%;Dist1和Dist2指标下降了约9.91%和8.49%;而ROUGE-1和ROUGE-2指标下降了约14.4%和29.64%. CVKASPLM with KAT模型其BLEU-3和BLEU-4指标下降了约20.61%和25.45%;Dist1和Dist2指标下降了约6.66%和3.64%;而ROUGE-1和ROUGE-2指标下降了约9.23%和24.02%. 实验结果表明,使用CVAKS组件和预训练模型Bart均能有效改善神经对话模型的性能,而CVAKS组件对神经对话模型性能的影响最为显著.
5)知识选择准确率评测.为了证明本文提出的CVAKSPLM模型在提高知识选择准确率方面的有效性,本文对3种显式知识选择对话方法:PostKS,SKT,CVAKSPLM进行了对比实验. 其操作为:分别在Wizard Test Seen和Wizard Test Unseen数据集上随机抽取100个样本,并手工标注每个样本知识集合中的ground truth知识. 设置模型超参数m=1,即每个样本仅选择1个文本知识. 在这100个样本数据集上运行3种模型,然后手工统计各模型选择出的知识命中率作为知识选择准确率:
其中,N为样本的容量,$ k_i^{\rm{sel}} $表示模型在第i个样本上选择的知识,$ K_i^* $表示在第i个样本上标注的ground truth知识集;而$ l\{ k_i^{\rm{sel}} \in K_i^*\} $为示性函数,如果$ k_i^{\rm{sel}} \in K_i^* $,函数取1,否则函数取0. 实验结果如表7所示.
从表7可以发现,CVAKSPLM模型在2个测试数据上的指标均优于其他2个模型,验证了本文提出的方法能够有效提高模型知识选择的准确率.
6)模型计算复杂度、参数量及推理速度比较. 为了从复杂度、计算资源和时间消耗等方面对模型进行分析和比较,本文采用计算量、参数量以及推理速度3个指标对4种模型进行了比较. 其中,计算量和参数量的计算采用了thop库6来实现. 实验结果如图3所示.
从图3 可以看到4个模型中,ITDD 模型需要的计算量和参数量最少,推理速度最快. 其次是PostKS模型,而KAT_TSLF 模型需要的计算量和参数量最多,推理速度最慢. 本文提出CVAKSPLM在推理速度指标上不如ITDD 模型和PostKS模型,但要优于KAT_TSLF 模型;而在计算量和参数量指标上,与PostKS模型非常接近. 上述提到的3个指标与模型的大小、算法的设计息息相关. 当前为了提高模型的性能指标一般会倾向采用参数较多的大模型,从而造成模型时间和空间复杂度增加,往往需要提供更多的计算资源来支撑. 因此在模型的效率和复杂度之间需要寻求一定的平衡.
此外,PostKS,KAT_TSLF,CVAKSPLM模型参数量都超过了140×106,但CVAKSPLM模型的计算量却大大低于KAT_TSLF模型,仅为24.4 GFLOPS,与PostKS模型的21.4 GFLOPS相当. 在推理速度方面也比KAT_TSLF快了约104 ms. 为了追求性能上的显著提升,本文采用了较大的Bart和BERT模型,设计了较复杂的知识选择算法,但从前面的实验结果上看,这种设计所付出的代价显然是值得的. 而CVAKSPLM相对较小的计算量可以保证模型在给定的计算机资源(GPU,CPU,内存)下,能够在较短的时间内完成模型的训练.
7)CVAKS在大语言模型上的实验结果. 为了验证本文提出的CVAKS方法在大语言模型上的有效性,在Wizard数据集上,本文进行了关于大模型的初步研究,其实验结果如表8所示. 本文选取了3个基础大模型ChatGPT-2-Medium,Llamma-2-7b,Llamma-2-13b进行对话生成任务的对比. 模型的架构采用了提示学习的范式[27],即使用提示模板(prompt template)将对话历史、对话回复和选择的文本知识填充到模板中作为模型的输入,然后采用自回归解码进行基于QLORA(quantized low-rank adapter)的微调训练和推断. 此外,3种大模型均采用2种知识选择算法:CVAKS和Random(随机化知识选择)用作对比. 模型超参数m=2,即每个样本选择2个文本知识. 对话生成的质量采用了5个度量指标:BERTScore[28]中的精度(precision)和召回率(recall)以及嵌入平均(embedding average)、向量极值(vector extrema)、贪心匹配(greedy matching)等基于嵌入的度量(embedding-based metrics)[18,29]指标.
本文中BERTScore的计算采用了开源包bert_score7,而基于嵌入的度量指标计算则采用了开源包nlgeval8.
从表8可以看到,在2个数据集上,采用本文提出的CVAKS知识选择算法的3种大模型在性能指标上均优于采用随机化知识选择方法的模型. 这表明与随机化知识选择策略相比,使用CVAKS算法能显著提高大模型在对话任务上的性能. 此外还可以看到,随着大模型参数规模的增长,对话生成任务的性能也随之得到提升.
8)案例研究. 图4给出了CVAKSPLM模型与PostKS 和KAT_TSLF模型在Wizard测试数据集上生成的对话回复实例. 其中,使用A和B区分不同对话者的身份, ground truth表示真实的回复. 图4中“选择的知识”行给出了2个CVAKSPLM模型和1个PostKS模型自动选择出的最相关文本知识,分别使用K1和K2表示. 这里需要注意的是:由于KAT_TSLF模型属于隐式知识选择方法. 因此图4没有给出该模型选择的知识.
通过对图4的数据分析,大致可以得到如下结论:本文提出的CVAKSPLM模型能够生成语句流畅、逻辑连贯、基本无语法错误的对话回复. 如实例1中,针对之前的对话历史,模型从文本知识集中选择了2个知识,并生成了肯定的回复:“我观看NFL”,接着给出了理由:“我喜欢NFL因为它是最流行的”,并使用全称“National Football League”替换了这里的缩写“NFL”. 而这种替换,也可以看作是模型对知识K2中全称和缩写的一种理解. 作为对比,KAT_TSLF模型生成的回复,尽管语句流畅、无语法错误,但似乎存在逻辑上的矛盾:“我观看NFL,但不观看NCAA;我不是NFL的粉丝,却是NCAA的粉丝”. PostKS模型生成的回复尽管也是有关“橄榄球”的,但与ground truth的主题存在一定偏差. 其选择的知识与生成的回复在主题上是一致的,且生成的句子流畅,无语法错误. 实例2中,围绕主题“爆米花”,针对之前的对话历史,CVAKSPLM模型从文本知识集中选择了2个相关知识,并生成了回复:“我喜欢爆米花,它们是如此的香甜和美味”. 而KAT_TSLF模型生成的回复中存在句子的重复. PostKS模型在知识的选择方面还是不错的, 生成的回复“我喜欢买爆米花”在语义上没有问题,但语法上存在错误. 从图4中还可以看到,CVAKSPLM模型还能有效利用选择的知识,如直接复制相关的知识片段来生成对话回复. 如实例3中,CVAKSPLM模型生成的对话回复中,主语直接来自对知识K1的复制.
总体上看,CVAKSPLM模型在知识的选择和回复的生成方面,质量最好;KAT_TSLF模型生成的回复质量次之;而PostKS模型在知识的选择上还是令人满意的,但在生成的回复方面,质量相对较差,即生成的回复相对较短,信息性较低. 有些回复包含多个冗余词甚至是不完整的句子,如实例3. 其原因可能是其采用的模型为较早的RNN模型,导致其语言生成能力不足.
基于知识的对话能够有效利用外部知识改善对话生成的多样性和信息性,目前已经成为对话系统研究的一个热点. 而针对当前基于知识的神经对话研究常常面临外部文本知识包含冗余甚噪声信息,从而导致对话系统性能下降的问题,本文提出了一个基于变分注意力知识选择和预训练模型的神经对话方法. 使用一个基于条件变分自编码和注意力机制的知识选择算法,自动选择出与当前对话最相关文本知识集合,同时利用了预训练模型Bart较强的文本生成能力. 实验结果表明,与现有的一些代表性研究方法相比,本文提出的模型能生成多样性和连贯性更好、准确率更高的对话回复. 同时,本文也将条件变分注意力知识选择模型与大模型相结合进行了初步的研究,实验结果证明了该模型在大语言模型上也是有效的. 下一步,将考虑条件变分注意力知识选择模型与大语言模型的深入融合,如研究如何利用条件变分注意力知识选择模型改善大模型提示学习的效率等问题,以进一步提升基于知识的神经对话系统的性能.
作者贡献声明:张乃洲提出论文的主要思想、完成整体模型构思和实验设计,并撰写论文;曹薇和张啸剑负责收集参考文献和论文校对,承担了部分实验的设计和实现,并对实验结果进行分析和整理;李石君对论文进行总体的审阅和修改.

参考文献 引证文献
排序方式:
1
Ni Jinjie, Young T, Pandelea V, et al. Recent advances in deep learning based dialogue systems: A systematic survey[J]. Artificial Intelligence Review, 2023, 56(4): 3055−3155
2
Serban I V, Sordoni A, Bengio Y, et al. Building end-to-end dialogue systems using generative hierarchical neural network models[C]//Proc of the 30th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2016: 3776−3784
3
Serban I V, Sordoni A, Lowe R, et al. A hierarchical latent variable encoder-decoder model for generating dialogues[C]//Proc of the 31st AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2017: 3295–3301
4
Li Jiwei, Galley M, Brocket C, et al. A diversity-promoting objective function for neural conversation models[C]//Proc of the 2016 Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, PA: ACL, 2016: 110−119
5
Zhao Tiancheng, Zhao Ran, Eskénazi M. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders[C]//Proc of the 55th Conf of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2017: 654−664
6
Zhang Rongjunchen, Wu Tingmin, Chen Xiao, et al. A transformer-based dialogue system with dynamic attention[C]//Proc of the Web Conf 2023. New York: ACM, 2023: 1604−1615
7
Golovanov S, Kurbanov R, Sergey I N, et al. Large-scale transfer learning for natural language generation[C]//Proc of the 57th Conf of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2019: 6053−6058
8
Ghazvininejad M, Brockett C, Chang Mingwei, et al. A knowledge-grounded neural conversation model[C]//Proc of the 32nd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2018: 5110−5117
9
Li Zekang, Niu Cheng, Meng Fandong, et al. Incremental transformer with deliberation decoder for document grounded conversations[C]//Proc of the 57th Conf of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2019: 12−21
10
Liu Shilei, Zhao Xiaofeng, Li Bochao, et al. A three-stage learning framework for low-resource knowledge-grounded dialogue generation[C]//Proc of the 2021 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2021: 2262−2272
11
Dinan E, Roller S, Shuster K, et al. Wizard of Wikipedia: Knowledge-powered conversational agents[C/OL]//Proc of the 7th Int Conf on Learning Representations. Ithaca, NY: Cornell University, 2019[2024-09-25]. https://openreview.net/forum?id=HtLWEQ69sf9
12
Lian Rongzhong, Xie Min, Wang Fan, et al. Learning to select knowledge for response generation in dialog systems[C]//Proc of the 28th Int Joint Conf on Artificial Intelligence. San Francisco, CA: Morgan Kaufmann, 2019: 5081−5087
13
Kim B, Ahn J, Kim G. Sequential latent knowledge selection for knowledge-grounded dialogue[C/OL]//Proc of the 8th Int Conf on Learning Representations. Ithaca, NY: Cornell University, 2020[2024-09-25]. https://openreview.net/forum?id=pm6GlJQp4Oo
14
Chen Xiuyi, Meng Fandong, Li Peng, et al. Bridging the gap between prior and posterior knowledge selection for knowledge-grounded dialogue generation[C]//Proc of the 2020 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2020: 3426−3437
15
Chen Xiuyi, Chen Feilong, Meng Fandong, et al. Unsupervised knowledge selection for dialogue generation[C]//Proc of the Joint Conf of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int Joint Conf on Natural Language Processing: Findings. Stroudsburg, PA: ACL, 2021: 1230−1244
16
Liu Zhibin, Niu Zhengyu, Wu Hua, et al. Knowledge aware conversation generation with explainable reasoning over augmented graphs[C]//Proc of the 2019 Conf on Empirical Methods in Natural Language Processing and the 9th Int Joint Conf on Natural Language Processing. Stroudsburg, PA: ACL, 2019: 1782−1792
17
Meng Chuan, Ren Pengjie, Chen Zhumin, et al. Initiative-aware self-supervised learning for knowledge-grounded conversations[C]//Proc of the 44th Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2021: 522−532
18
Zhao Xueliang, Wu Wei, Xu Can, et al. Knowledge-grounded dialogue generation with pre-trained language models[C]//Proc of the 2020 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2020: 115−125
19
Zheng Chujie, Cao Yunbo, Jiang Daxin, et al. Difference-aware knowledge selection for knowledge-grounded conversation generation[C]//Proc of the 2020 Conf on Empirical Methods in Natural Language Processing: Findings. Stroudsburg, PA: ACL, 2020: 115−125
20
Sun Weiwei, Ren Pengjie, Ren Zhaochun. Generative knowledge selection for knowledge-grounded dialogues[C]//Proc of the 17th Conf of the European Chapter of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2023: 2032−2043
21
孙润鑫,马龙轩,张伟男,等. 基于文档的对话研究[J]. 计算机研究与发展,2021,58(9):1915−1924
Sun Runxin, Ma Longxuan, Zhang Weinan, et al. Research on document grounded conversations[J]. Journal of Computer Research and Development, 2021, 58(9): 1915−1924 (in Chinese)
22
Yang Yizhe, Huang Heyan, Liu Yuhang, et al. Graph vs. sequence: An empirical study on knowledge forms for knowledge-grounded dialogue[C]//Proc of the 2023 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2023: 15846−15858
23
Sukhbaatar S, Szlam A, Weston J, et al. End-to-end memory networks[C]//Proc of the 28th Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2015: 2440−2448
24
Lewis M, Liu Yinhan, Goyal N, et al. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[C]//Proc of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2020: 7871−7880
25
Zhou Kangyan, Prabhumoye S, Black A W. A dataset for document grounded conversations[C]//Proc of the 2018 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2018: 708−713
26
Papineni K, Roukos S, Ward T, et al. Bleu: A method for automatic evaluation of machine translation[C]//Proc of the 40th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: ACL, 2002: 311−318
27
Liu Pengfei, Yuan Weizhe, Fu Jinlan, et al. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing[J]. ACM Computing Surveys, 2023, 55(9): 195: 1−195: 35
28
Zhang Tianyi, Kishore V, Wu F, et al. BERTScore: Evaluating text generation with BERT[C/OL]//Proc of the 8th Int Conf on Learning Representations. Ithaca, NY: Cornell University, 2020[2024-09-25]. https://openreview.net/forum?id=SkeHuCVFDr
29
Liu Chiawei, Lowe R, Serban I, et al. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation[C]//Proc of the 2016 Conf on Empirical Methods in Natural Language Processing. Stroudsburg, PA: ACL, 2016: 2122−2132
2025年第62卷第8期
102
35
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202440551
  • 接收时间:2024-06-20
  • 首发时间:2026-11-10
  • 出版时间:2025-08-01
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2024-06-20
  • 修回日期:2024-10-22
  • 录用日期:2024-11-12
基金
作者信息
    1河南财经政法大学计算机与信息工程学院 郑州 450046
    2武汉大学计算机学院 武汉 430072
参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202440551
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏