Article(id=1329458143685357907, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1329458094792356805, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202220970, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=null, receivedDate=1669132800000, receivedDateStr=2022-11-23, revisedDate=1704211200000, revisedDateStr=2024-01-03, acceptedDate=1709654400000, acceptedDateStr=2024-03-06, onlineDate=1794897121592, onlineDateStr=2026-11-17, pubDate=1732982400000, pubDateStr=2024-12-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1794897121592, onlineIssueDateStr=2026-11-17, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1794897121592, creator=13701087609, updateTime=1794897121592, updator=13701087609, issue=Issue{id=1329458094792356805, tenantId=1146029695717560320, journalId=1324759914146529478, year='2024', volume='61', issue='12', pageStart='2937', pageEnd='3187', issueExtLink='null', onlineDate='null', pubDate='1732982400000', pubDateStr='2024-12-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1794897109934, creator='13701087609', updateTime=1794897109934, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext=null, issueFiles=null, downloadFileDto=null}, startPage=3108, endPage=3120, ext={EN=ArticleExt(id=1329458145354690900, articleId=1329458143685357907, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=A Diversity-Enriched Option-Critic Algorithm with Interest Functions, columnId=1325489891789910943, journalTitle=Journal of Computer Research and Development, columnName=Artificial Intelligence, runingTitle=null, highlight=null, articleAbstract=
As a common temporal abstraction method for hierarchical reinforcement learning, Option framework allows agents to learn strategies at different time scales, which can effectively solve sparse reward problems. In order to ensure that options can guide agents to access more state space, some methods improve the diversity of options by introducing mutual information in internal reward and termination functions. However, it will lead to slow algorithm learning speed and low knowledge transfer ability of internal strategy, which seriously affect algorithm performance. To address the above problems, diversity-enriched option-critic algorithm with interest functions(DEOC-IF) is proposed. Based on the diversity-enriched option-critic (DEOC) algorithm, the algorithm constrains the selection of the upper-level strategy on the internal strategy of Option by introducing the interest function, which not only ensures the diversity of the Option set, but also makes the learned internal strategies focus on different regions of the state space, which is conducive to improving the knowledge transfer ability of the algorithm and accelerating the learning speed. In addition, DEOC-IF introduces a new interest function update gradient, which is beneficial to improve the exploration ability of the algorithm. In order to verify the effectiveness and option reusability of the algorithm, the algorithm comparison experiments are carried out in four-room navigation task, Mujoco, and MiniWorld. Experimental results show that DEOC-IF algorithm has better performance and option reusability compared with other algorithms.
, authors=Junwei Li
1, Quan Liu
1, 2, 3, 4, *, Zhigang Huang
1, Yapeng Xu
1, authorsList=Junwei Li, Quan Liu, Zhigang Huang, Yapeng Xu, authorCompany=null, correspAuthors=Quan Liu, authorNote=
born in 1998. Master candidate. His main research interests include reinforcement learning and hierarchical reinforcement learning
born in 1969. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include intelligence information processing, automated reasoning, and machine learning
born in 1993. PhD candidate. His main research interests include reinforcement learning, deep reinforcement learning, and hierarchical reinforcement learning
born in 1996. Master candidate. His main research interests include deep reinforcement learning and hierarchical reinforcement learning
, correspAuthorsNote=null, copyrightStatement=Copyright © 2024 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1329458149486080368, articleId=1329458143685357907, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=基于兴趣函数的多样化Option-Critic算法, columnId=1325489892104483745, journalTitle=计算机研究与发展, columnName=人工智能, runingTitle=null, highlight=null, articleAbstract=
Option框架作为分层强化学习的一种常用时序抽象方法,允许智能体在不同的时间尺度上学习策略,可以有效解决稀疏奖励问题. 为了保证Option可以引导智能体访问更多的状态空间,一些方法通过引入基于互信息的内部奖励和终止函数来提升Option内部策略的多样性. 但这会导致算法学习速度慢和内部策略的知识迁移能力低等问题,严重影响了算法性能. 针对以上问题,提出基于兴趣函数优化的多样化Option-Critic算法(diversity-enriched Option-Critic algorithm with interest functions,DEOC-IF). 该算法在多样化Option-Critic算法(diversity-enriched Option-Critic,DEOC)的基础上,通过引入兴趣函数约束上层策略对Option内部策略的选择,既保证了Option集合的多样性,又使得学习到的内部策略可以关注状态空间的不同区域,有利于提高算法的知识迁移能力,加快学习速度. 此外,DEOC-IF算法引入一种新的兴趣函数更新梯度,有利于提高算法的探索能力. 为了验证算法的有效性和知识迁移能力,分别在4房间导航任务、Mujoco和MiniWorld实验环境中,将DEOC-IF算法与其他最新算法进行对比实验. 结果表明,DEOC-IF算法具有更好的性能优势和策略迁移能力.
, authors=栗军伟
1, 刘全
1, 2, 3, 4, *, 黄志刚
1, 徐亚鹏
1, authorsList=栗军伟, 刘全, 黄志刚, 徐亚鹏, authorCompany=null, correspAuthors=刘全, authorNote=
 |
1998年生. 硕士研究生. 主要研究方向为强化学习、分层强化学习 |
 |
1969年生. 博士,教授,博士生导师. CCF高级会员. 主要研究方向为智能信息处理、自动推理、机器学习 |
 |
1993年生. 博士研究生. 主要研究方向为强化学习、深度强化学习、分层强化学习 |
 |
1996年生. 硕士研究生. 主要研究方向为深度强化学习、分层强化学习 |
, correspAuthorsNote=
, copyrightStatement=版权所有 © 2024 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=Vc0+LY1Htn+5AkN8UizmTg==, magXml=HVS2vPUXZI7BNLyEJJ988A==, pdfUrl=null, pdf=Pah9+zijMIGTouMkootXhQ==, pdfFileSize=6067100, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=q38oQfDeeWd7Phuu//KXww==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=FmB1kOZTvizqGtzAZviN7Q==, mapNumber=null, fund=null)}, authors=[Author(id=1329458150081671550, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=Li_junwei2021@163.com, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329458150199112064, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, authorId=1329458150081671550, language=EN, stringName=Junwei Li, firstName=Junwei, middleName=null, lastName=Li, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=
1, address=
1School of Computer Science and Technology, Soochow University, Suzhou, Jiangsu 215006, bio={"content":"
born in 1998. Master candidate. His main research interests include reinforcement learning and hierarchical reinforcement learning
"}, bioImg=null, bioContent=
born in 1998. Master candidate. His main research interests include reinforcement learning and hierarchical reinforcement learning
, aboutCorrespAuthor=null), CN=AuthorExt(id=1329458150274609537, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, authorId=1329458150081671550, language=CN, stringName=栗军伟, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=
1, address=
1苏州大学计算机科学与技术学院 江苏苏州 215006, bio={"img":"dIJvoBOnUv5cOLHd4RStTg==","content":"
1998年生. 硕士研究生. 主要研究方向为强化学习、分层强化学习
"}, bioImg=dIJvoBOnUv5cOLHd4RStTg==, bioContent=
1998年生. 硕士研究生. 主要研究方向为强化学习、分层强化学习
, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329458149716767089, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, xref=1, ext=[AuthorCompanyExt(id=1329458149725155698, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149716767089, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
1School of Computer Science and Technology, Soochow University, Suzhou, Jiangsu 215006), AuthorCompanyExt(id=1329458149733544307, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149716767089, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
1苏州大学计算机科学与技术学院 江苏苏州 215006)])]), Author(id=1329458150375272836, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=quanliu@suda.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329458151990079882, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, authorId=1329458150375272836, language=EN, stringName=Quan Liu, firstName=Quan, middleName=null, lastName=Liu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=
1, 2, 3, 4, *, address=
1School of Computer Science and Technology, Soochow University, Suzhou, Jiangsu 215006
2Collaborative Innovation Center of Novel Software Technology and Industrialization (Nanjing University), Nanjing 210023
3Key Laboratory of Symbol Computation and Knowledge Engineering (Jilin University), Ministry of Education, Changchun 130012
4Jiangsu Provincial Key Laboratory for Computer Information Processing Technology (Soochow University), Suzhou, Jiangsu 215006, bio={"content":"
born in 1969. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include intelligence information processing, automated reasoning, and machine learning
"}, bioImg=null, bioContent=
born in 1969. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include intelligence information processing, automated reasoning, and machine learning
, aboutCorrespAuthor=null), CN=AuthorExt(id=1329458152094937484, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, authorId=1329458150375272836, language=CN, stringName=刘全, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=
1, 2, 3, 4, *, address=
1苏州大学计算机科学与技术学院 江苏苏州 215006
2软件新技术与产业化协同创新中心(南京大学) 南京 210023
3符号计算与知识工程教育部重点实验室(吉林大学) 长春 130012
4江苏省计算机信息处理技术重点实验室(苏州大学) 江苏苏州 215006, bio={"img":"M8I8CLfRF9FLTEbdrT8bvw==","content":"
1969年生. 博士,教授,博士生导师. CCF高级会员. 主要研究方向为智能信息处理、自动推理、机器学习
"}, bioImg=M8I8CLfRF9FLTEbdrT8bvw==, bioContent=
1969年生. 博士,教授,博士生导师. CCF高级会员. 主要研究方向为智能信息处理、自动推理、机器学习
, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329458149716767089, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, xref=1, ext=[AuthorCompanyExt(id=1329458149725155698, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149716767089, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
1School of Computer Science and Technology, Soochow University, Suzhou, Jiangsu 215006), AuthorCompanyExt(id=1329458149733544307, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149716767089, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
1苏州大学计算机科学与技术学院 江苏苏州 215006)]), AuthorCompany(id=1329458149804847476, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, xref=2, ext=[AuthorCompanyExt(id=1329458149813236085, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149804847476, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
2Collaborative Innovation Center of Novel Software Technology and Industrialization (Nanjing University), Nanjing 210023), AuthorCompanyExt(id=1329458149817430390, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149804847476, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
2软件新技术与产业化协同创新中心(南京大学) 南京 210023)]), AuthorCompany(id=1329458149892927863, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, xref=3, ext=[AuthorCompanyExt(id=1329458149897122168, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149892927863, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
3Key Laboratory of Symbol Computation and Knowledge Engineering (Jilin University), Ministry of Education, Changchun 130012), AuthorCompanyExt(id=1329458149909705081, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149892927863, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
3符号计算与知识工程教育部重点实验室(吉林大学) 长春 130012)]), AuthorCompany(id=1329458149981008250, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, xref=4, ext=[AuthorCompanyExt(id=1329458149989396859, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149981008250, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
4Jiangsu Provincial Key Laboratory for Computer Information Processing Technology (Soochow University), Suzhou, Jiangsu 215006), AuthorCompanyExt(id=1329458149993591164, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149981008250, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
4江苏省计算机信息处理技术重点实验室(苏州大学) 江苏苏州 215006)])]), Author(id=1329458152208183694, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329458152292069776, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, authorId=1329458152208183694, language=EN, stringName=Zhigang Huang, firstName=Zhigang, middleName=null, lastName=Huang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=
1, address=
1School of Computer Science and Technology, Soochow University, Suzhou, Jiangsu 215006, bio={"content":"
born in 1993. PhD candidate. His main research interests include reinforcement learning, deep reinforcement learning, and hierarchical reinforcement learning
"}, bioImg=null, bioContent=
born in 1993. PhD candidate. His main research interests include reinforcement learning, deep reinforcement learning, and hierarchical reinforcement learning
, aboutCorrespAuthor=null), CN=AuthorExt(id=1329458152359178642, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, authorId=1329458152208183694, language=CN, stringName=黄志刚, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=
1, address=
1苏州大学计算机科学与技术学院 江苏苏州 215006, bio={"img":"kKU1WKFW0F0s48+S41NHMg==","content":"
1993年生. 博士研究生. 主要研究方向为强化学习、深度强化学习、分层强化学习
"}, bioImg=kKU1WKFW0F0s48+S41NHMg==, bioContent=
1993年生. 博士研究生. 主要研究方向为强化学习、深度强化学习、分层强化学习
, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329458149716767089, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, xref=1, ext=[AuthorCompanyExt(id=1329458149725155698, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149716767089, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
1School of Computer Science and Technology, Soochow University, Suzhou, Jiangsu 215006), AuthorCompanyExt(id=1329458149733544307, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149716767089, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
1苏州大学计算机科学与技术学院 江苏苏州 215006)])]), Author(id=1329458152430481812, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329458152501784982, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, authorId=1329458152430481812, language=EN, stringName=Yapeng Xu, firstName=Yapeng, middleName=null, lastName=Xu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=
1, address=
1School of Computer Science and Technology, Soochow University, Suzhou, Jiangsu 215006, bio={"content":"
born in 1996. Master candidate. His main research interests include deep reinforcement learning and hierarchical reinforcement learning
"}, bioImg=null, bioContent=
born in 1996. Master candidate. His main research interests include deep reinforcement learning and hierarchical reinforcement learning
, aboutCorrespAuthor=null), CN=AuthorExt(id=1329458152577282455, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, authorId=1329458152430481812, language=CN, stringName=徐亚鹏, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=
1, address=
1苏州大学计算机科学与技术学院 江苏苏州 215006, bio={"img":"6xh4cI/t2BVrKGhPlNMlaQ==","content":"
1996年生. 硕士研究生. 主要研究方向为深度强化学习、分层强化学习
"}, bioImg=6xh4cI/t2BVrKGhPlNMlaQ==, bioContent=
1996年生. 硕士研究生. 主要研究方向为深度强化学习、分层强化学习
, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329458149716767089, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, xref=1, ext=[AuthorCompanyExt(id=1329458149725155698, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149716767089, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
1School of Computer Science and Technology, Soochow University, Suzhou, Jiangsu 215006), AuthorCompanyExt(id=1329458149733544307, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149716767089, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
1苏州大学计算机科学与技术学院 江苏苏州 215006)])])], keywords=[Keyword(id=1329458152673751448, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=EN, orderNo=1, keyword=reinforcement learning), Keyword(id=1329458152732471705, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=EN, orderNo=2, keyword=temporal abstractions), Keyword(id=1329458152786997658, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=EN, orderNo=3, keyword=Option framework), Keyword(id=1329458152862495131, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=EN, orderNo=4, keyword=interest function), Keyword(id=1329458152917021084, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=EN, orderNo=5, keyword=Option-Critic algorithm), Keyword(id=1329458153013490077, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=CN, orderNo=1, keyword=强化学习), Keyword(id=1329458153080598942, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=CN, orderNo=2, keyword=时序抽象), Keyword(id=1329458153143513503, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=CN, orderNo=3, keyword=Option框架), Keyword(id=1329458153298702752, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=CN, orderNo=4, keyword=兴趣函数), Keyword(id=1329458153411948961, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=CN, orderNo=5, keyword=Option-Critic算法)], refs=[Reference(id=1329458159292363200, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=Sutton R S, Barto A G. Reinforcement Learning: An Introduction[M]. Cambridge, MA: MIT, 2018, articleTitle=null, refAbstract=null), Reference(id=1329458160898781634, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=刘全,翟建伟,章宗长,等. 深度强化学习综述[J]. 计算机学报,2018,41(1):1−27, articleTitle=null, refAbstract=null), Reference(id=1329458160970084804, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Quan, Zhai Jianwei, Zhang Zongzhang, et al. A survey on deep reinforcement learning[J]. Chinese Journal of Computers, 2018, 41(1): 1−27 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1329458161041387973, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=刘建伟,高峰,罗雄麟. 基于值函数和策略梯度的深度强化学习综述[J]. 计算机学报,2019,42(6):1406−1438, articleTitle=null, refAbstract=null), Reference(id=1329458161121079750, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Jianwei, Gao Feng, Luo Xionglin. Survey of deep reinforcement learning based on value function and policy gradient[J]. Chinese Journal of Computers, 2019, 42(6): 1406−1438 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1329458161213354439, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540): 529−533, articleTitle=null, refAbstract=null), Reference(id=1329458161305629129, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Lillicrap T P, Hunt J J, Pritzel A, et al. Continuous control with deep reinforcement learning[J]. arXiv preprint, arXiv: 1509.02971, 2015, articleTitle=null, refAbstract=null), Reference(id=1329458161385320906, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Fujimoto S, Hoof H, Meger D. Addressing function approximation error in actor-critic methods[C]//Proc of the 35th Int Conf on Machine Learning. New York: ACM, 2018: 1582−1591, articleTitle=null, refAbstract=null), Reference(id=1329458161502761419, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Schulman J, Levine S, Abbeel P, et al. Trust region policy optimization[C]//Proc of the 32nd Int Conf on Machine Learning. New York: ACM, 2015: 1889−1897, articleTitle=null, refAbstract=null), Reference(id=1329458161595036108, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Haarnoja T, Zhou A, Abbeel P, et al. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]//Proc of the 35th Int Conf on Machine Learning. New York: ACM, 2018: 1861−1870, articleTitle=null, refAbstract=null), Reference(id=1329458161670533581, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=赖俊,魏竞毅,陈希亮. 分层强化学习综述[J]. 计算机工程与应用,2021,57(3):72−79, articleTitle=null, refAbstract=null), Reference(id=1329458161737642446, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Lai Jun, Wei Jingyi, Chen Xiliang. Overview of hierarchical reinforcement learning[J]. Computer Engineering and Applications, 2021, 57(3): 72−79 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1329458161787974095, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Sutton R S, Precup D, Singh S. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning[J]. Artificial Intelligence, 1999, 112(1/2): 181−211, articleTitle=null, refAbstract=null), Reference(id=1329458161859277264, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Kulkarni T D, Narasimhan K, Saeedi A, et al. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation[C]//Proc of the 29th Advances in Neural Information Processing Systems. Cambridge, MA: MIT, 2016: 3675−3683, articleTitle=null, refAbstract=null), Reference(id=1329458161943163345, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhao Dongyang, Zhang Liang, Zhang Bo, et al. Mahrl: Multi-goals abstraction based deep hierarchical reinforcement learning for recommendations[C]//Proc of the 43rd Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2020: 871−880, articleTitle=null, refAbstract=null), Reference(id=1329458162006077906, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Duan Jingliang, Li Shengbo, Guan Yang, et al. Hierarchical reinforcement learning for self-driving decision-making without reliance on labelled driving data[J]. IET Intelligent Transport Systems, 2020, 14(5): 297−305, articleTitle=null, refAbstract=null), Reference(id=1329458162077381075, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Jianfeng, Pan Feiyang, Luo Ling. Gochat: Goal-oriented chatbots with hierarchical reinforcement learning[C]//Proc of the 43rd Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2020: 1793−1796, articleTitle=null, refAbstract=null), Reference(id=1329458162136101332, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Levy A, Konidaris G, Platt R, et al. Learning multi-level hierarchies with hindsight[J]. arXiv preprint, arXiv: 1712.00948, 2017, articleTitle=null, refAbstract=null), Reference(id=1329458162249347542, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Bacon P L, Harb J, Precup D. The option-critic architecture[C]//Proc of the 31st AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2017: 1726−1734, articleTitle=null, refAbstract=null), Reference(id=1329458162316456407, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Harb J, Bacon P L, Klissarov M, et al. When waiting is not an option: Learning options with a deliberation cost[C]//Proc of the 32nd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2018: 3165−3172, articleTitle=null, refAbstract=null), Reference(id=1329458162396148184, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Klissarov M, Bacon P L, Harb J, et al. Learnings options end-to-end for continuous action tasks[J]. arXiv preprint, arXiv: 1712.00004, 2017, articleTitle=null, refAbstract=null), Reference(id=1329458162467451353, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms[J]. arXiv preprint, arXiv: 1707.06347, 2017, articleTitle=null, refAbstract=null), Reference(id=1329458162521977306, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Chenghao, Ma Xiaoteng, Zhang Chongjie, et al. Soac: The soft option actor-critic architecture[J]. arXiv preprint, arXiv: 2006.14363, 2020, articleTitle=null, refAbstract=null), Reference(id=1329458165994861019, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=Kanagawa Y, Kaneko T. Diverse exploration via infomax options[J]. arXiv preprint, arXiv: 2010.02756, 2020, articleTitle=null, refAbstract=null), Reference(id=1329458166082941405, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=Eysenbach B, Gupta A, Ibarz J, et al. Diversity is all you need: Learning skills without a reward function[J]. arXiv preprint, arXiv: 1802.06070, 2018, articleTitle=null, refAbstract=null), Reference(id=1329458166183604702, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=Gregor K, Rezende D J, Wierstra D. Variational intrinsic control[J]. arXiv preprint, arXiv: 1611.07507, 2016, articleTitle=null, refAbstract=null), Reference(id=1329458166280073695, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=26, authorNames=null, journalName=null, refType=null, unstructuredReference=Harutyunyan A, Dabney W, Borsa D, et al. The termination critic[C]//Proc of the 22nd Int Conf on Artificial Intelligence and Statistics. New York: PMLR, 2019: 2231−2240, articleTitle=null, refAbstract=null), Reference(id=1329458166355571168, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=27, authorNames=null, journalName=null, refType=null, unstructuredReference=Kamat A, Precup D. Diversity-enriched option-critic[J]. arXiv preprint, arXiv: 2011.02565, 2020, articleTitle=null, refAbstract=null), Reference(id=1329458166435262945, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=28, authorNames=null, journalName=null, refType=null, unstructuredReference=Khetarpal K, Klissarov M, Chevalier-Boisvert M, et al. Options of interest: Temporal abstraction with interest functions[C]//Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 4444−4451, articleTitle=null, refAbstract=null), Reference(id=1329458166523343330, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=29, authorNames=null, journalName=null, refType=null, unstructuredReference=Sutton R S, McAllester D A, Singh S P, et al. Policy gradient methods for reinforcement learning with function approximation[C]//Proc of the 14th Advances in Neural Information Processing Systems. Cambridge, MA: MIT, 2000: 1057−1063, articleTitle=null, refAbstract=null), Reference(id=1329458166603035107, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=28, rfOrder=30, authorNames=null, journalName=null, refType=null, unstructuredReference=Brockman G, Cheung V, Pettersson L, et al. OpenAI gym[J]. arXiv preprint, arXiv: 1606.01540, 2016, articleTitle=null, refAbstract=null), Reference(id=1329458166670143972, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=29, rfOrder=31, authorNames=null, journalName=null, refType=null, unstructuredReference=Finn C, Abbeel P, Levine S. Model-agnostic meta-learning for fast adaptation of deep networks[C]//Proc of the 34th Int Conf on Machine Learning. New York: ICML, 2017: 1126−1135, articleTitle=null, refAbstract=null), Reference(id=1329458166745641445, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=30, rfOrder=32, authorNames=null, journalName=null, refType=null, unstructuredReference=Henderson P, Chang W D, Shkurti F, et al. Benchmark environments for multitask learning in continuous domains[J]. arXiv preprint, arXiv: 1708.04352, 2017, articleTitle=null, refAbstract=null), Reference(id=1329458166816944614, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=31, rfOrder=33, authorNames=null, journalName=null, refType=null, unstructuredReference=Chevalier-Boisvert M, Bolun D, Mark T, et al. Gym-miniworld environment for OpenAI Gym[EB/OL]. 2018[2022-11-22]. https://github.com/maximecb/gym-miniworld, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1329458149716767089, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, xref=1, ext=[AuthorCompanyExt(id=1329458149725155698, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149716767089, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
1School of Computer Science and Technology, Soochow University, Suzhou, Jiangsu 215006), AuthorCompanyExt(id=1329458149733544307, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149716767089, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
1苏州大学计算机科学与技术学院 江苏苏州 215006)]), AuthorCompany(id=1329458149804847476, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, xref=2, ext=[AuthorCompanyExt(id=1329458149813236085, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149804847476, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
2Collaborative Innovation Center of Novel Software Technology and Industrialization (Nanjing University), Nanjing 210023), AuthorCompanyExt(id=1329458149817430390, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149804847476, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
2软件新技术与产业化协同创新中心(南京大学) 南京 210023)]), AuthorCompany(id=1329458149892927863, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, xref=3, ext=[AuthorCompanyExt(id=1329458149897122168, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149892927863, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
3Key Laboratory of Symbol Computation and Knowledge Engineering (Jilin University), Ministry of Education, Changchun 130012), AuthorCompanyExt(id=1329458149909705081, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149892927863, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
3符号计算与知识工程教育部重点实验室(吉林大学) 长春 130012)]), AuthorCompany(id=1329458149981008250, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, xref=4, ext=[AuthorCompanyExt(id=1329458149989396859, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149981008250, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
4Jiangsu Provincial Key Laboratory for Computer Information Processing Technology (Soochow University), Suzhou, Jiangsu 215006), AuthorCompanyExt(id=1329458149993591164, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, companyId=1329458149981008250, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=
4江苏省计算机信息处理技术重点实验室(苏州大学) 江苏苏州 215006)])], figs=[ArticleFig(id=1329458153554555298, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=EN, label=Figure 1, caption=
Illustration of DEOC-IF algorithm, figureFileSmall=B52X0Zc19x6NAgctIhdKvQ==, figureFileBig=q38oQfDeeWd7Phuu//KXww==, tableContent=null), ArticleFig(id=1329458153613275555, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=CN, label=图1, caption=
DEOC-IF算法示意图, figureFileSmall=B52X0Zc19x6NAgctIhdKvQ==, figureFileBig=q38oQfDeeWd7Phuu//KXww==, tableContent=null), ArticleFig(id=1329458153827185060, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=EN, label=Figure 2, caption=
Four-room navigation task environment, figureFileSmall=GIYGjFmLDxzgvK3cH6fihg==, figureFileBig=Po1oSs7xv6wsTxCoQLCvWw==, tableContent=null), ArticleFig(id=1329458153936236965, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=CN, label=图2, caption=
4房间导航任务环境, figureFileSmall=GIYGjFmLDxzgvK3cH6fihg==, figureFileBig=Po1oSs7xv6wsTxCoQLCvWw==, tableContent=null), ArticleFig(id=1329458154007540134, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=EN, label=Figure 3, caption=
Learning curve in four-room environment, figureFileSmall=wS9kBuss+th+ngyBqKeusg==, figureFileBig=H0/LgUABxInBnE930YOvhQ==, tableContent=null), ArticleFig(id=1329458154074648999, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=CN, label=图3, caption=
4房间环境学习曲线, figureFileSmall=wS9kBuss+th+ngyBqKeusg==, figureFileBig=H0/LgUABxInBnE930YOvhQ==, tableContent=null), ArticleFig(id=1329458154133369256, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=EN, label=Figure 4, caption=
Mujoco experimental environments, figureFileSmall=ARhrxU0qSqNfZf1CxuxeAg==, figureFileBig=QC3vz3Bd2E6gUxbM/p85qQ==, tableContent=null), ArticleFig(id=1329458154208866729, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=CN, label=图4, caption=
Mujoco实验环境, figureFileSmall=ARhrxU0qSqNfZf1CxuxeAg==, figureFileBig=QC3vz3Bd2E6gUxbM/p85qQ==, tableContent=null), ArticleFig(id=1329458154275975594, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=EN, label=Figure 5, caption=
Learning curve of algorithms in Mujoco environments, figureFileSmall=ZY2Ds8+39WUnygLvYSpd1A==, figureFileBig=RZd56vOb40/wCRyZuPtD1g==, tableContent=null), ArticleFig(id=1329458154334695851, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=CN, label=图5, caption=
Mujoco环境中算法学习曲线, figureFileSmall=ZY2Ds8+39WUnygLvYSpd1A==, figureFileBig=RZd56vOb40/wCRyZuPtD1g==, tableContent=null), ArticleFig(id=1329458154410193324, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=EN, label=Figure 6, caption=
Learning curves of algorithms and environments in transfer tasks, figureFileSmall=UpXqwRbTnL0VzIAEZv0eKQ==, figureFileBig=Qgz23dSFzLS8Tgfx152GPw==, tableContent=null), ArticleFig(id=1329458154464719277, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=CN, label=图6, caption=
迁移任务中算法的学习曲线和环境, figureFileSmall=UpXqwRbTnL0VzIAEZv0eKQ==, figureFileBig=Qgz23dSFzLS8Tgfx152GPw==, tableContent=null), ArticleFig(id=1329458154536022446, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=EN, label=Figure 7, caption=
OneRoom environment, figureFileSmall=WYlkFikWo2HhIlQipIH3bg==, figureFileBig=nZoyaSwWZUAY/0x/OQFqig==, tableContent=null), ArticleFig(id=1329458154598937007, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=CN, label=图7, caption=
OneRoom环境, figureFileSmall=WYlkFikWo2HhIlQipIH3bg==, figureFileBig=nZoyaSwWZUAY/0x/OQFqig==, tableContent=null), ArticleFig(id=1329458154674434480, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=EN, label=Figure 8, caption=
Learning curve of algorithms in OneRoom environment, figureFileSmall=GdOWhjDwEQzbZAzIQ6PiqA==, figureFileBig=m6MNVYGi+d69UV9Je5ZLJw==, tableContent=null), ArticleFig(id=1329458154745737649, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=CN, label=图8, caption=
OneRoom环境中算法学习曲线, figureFileSmall=GdOWhjDwEQzbZAzIQ6PiqA==, figureFileBig=m6MNVYGi+d69UV9Je5ZLJw==, tableContent=null), ArticleFig(id=1329458154808652210, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=EN, label=Figure 9, caption=
Ablation experiment, figureFileSmall=dmST56pWLImguuMGc53CYQ==, figureFileBig=6OVnADoxKnBKyMbTGzpWUA==, tableContent=null), ArticleFig(id=1329458156410876340, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=CN, label=图9, caption=
消融实验, figureFileSmall=dmST56pWLImguuMGc53CYQ==, figureFileBig=6OVnADoxKnBKyMbTGzpWUA==, tableContent=null), ArticleFig(id=1329458156490568118, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=EN, label=Table 1, caption=
Hyper-Parameters of Four-Room Navigation Task
, figureFileSmall=null, figureFileBig=null, tableContent=
| 超参数 | 参数值 |
|---|
| $ {\alpha _\vartheta } $ | 0.25 |
| $ {\alpha _\theta } $ | 0.25 |
| $ \gamma $ | 0.99 |
| $ {\alpha _z} $ | 0.15 |
), ArticleFig(id=1329458156557676983, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=CN, label=表1, caption=
4房间导航任务超参数
, figureFileSmall=null, figureFileBig=null, tableContent=
| 超参数 | 参数值 |
|---|
| $ {\alpha _\vartheta } $ | 0.25 |
| $ {\alpha _\theta } $ | 0.25 |
| $ \gamma $ | 0.99 |
| $ {\alpha _z} $ | 0.15 |
), ArticleFig(id=1329458156616397240, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=EN, label=Table 2, caption=
Introduction to Mujoco Experimental Environments
, figureFileSmall=null, figureFileBig=null, tableContent=
| 环境名称 | 动作空间维度 | 状态空间维度 |
|---|
| Humanoid-v2 | 17 | 376 |
| Hopper-v2 | 3 | 11 |
| Walker2d-v2 | 6 | 17 |
| HalfCheetah-v2 | 6 | 17 |
), ArticleFig(id=1329458156683506105, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=CN, label=表2, caption=
Mujoco实验环境介绍
, figureFileSmall=null, figureFileBig=null, tableContent=
| 环境名称 | 动作空间维度 | 状态空间维度 |
|---|
| Humanoid-v2 | 17 | 376 |
| Hopper-v2 | 3 | 11 |
| Walker2d-v2 | 6 | 17 |
| HalfCheetah-v2 | 6 | 17 |
), ArticleFig(id=1329458156738032058, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=EN, label=Table 3, caption=
Hyper-Parameters of Mujoco Experimental Environment
, figureFileSmall=null, figureFileBig=null, tableContent=
| 超参数 | Humanoid-v2 | Hopper-v2 | Walker2d-v2 | HalfCheetah-v2 |
|---|
| $ \tau $ | 0.4 | 0.2 | 0.2 | 0.7 |
| $ c $ | 0.01 | 0.01 | 0.01 | 0.01 |
| $ {\alpha _\vartheta } $ | 5E–7 | 5E–7 | 5E–7 | 5E–7 |
| $ {\alpha _\theta } $ | 3E–4 | 1E–4 | 1E–4 | 1E–4 |
| $ {\alpha _z} $ | 1E–4 | 1E–4 | 1E–4 | 5E–4 |
), ArticleFig(id=1329458156800946619, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=CN, label=表3, caption=
Mujoco实验环境超参数
, figureFileSmall=null, figureFileBig=null, tableContent=
| 超参数 | Humanoid-v2 | Hopper-v2 | Walker2d-v2 | HalfCheetah-v2 |
|---|
| $ \tau $ | 0.4 | 0.2 | 0.2 | 0.7 |
| $ c $ | 0.01 | 0.01 | 0.01 | 0.01 |
| $ {\alpha _\vartheta } $ | 5E–7 | 5E–7 | 5E–7 | 5E–7 |
| $ {\alpha _\theta } $ | 3E–4 | 1E–4 | 1E–4 | 1E–4 |
| $ {\alpha _z} $ | 1E–4 | 1E–4 | 1E–4 | 5E–4 |
), ArticleFig(id=1329458156884832700, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=EN, label=Table 4, caption=
Running Time of Algorithm in Mujoco Environments h
, figureFileSmall=null, figureFileBig=null, tableContent=
| 环境 | OC | PPO | TDEOC | DEOC-IF(本文) |
|---|
| Hopper-v2 | 8.7 | 6.9 | 10.2 | 9.5 |
| Walker2d-v2 | 13.6 | 10.3 | 17.2 | 15.1 |
| Humanoid-v2 | 19.1 | 16.2 | 23.7 | 21.3 |
| HalfCheetah-v2 | 8.3 | 7.1 | 9.4 | 8.9 |
), ArticleFig(id=1329458156985495997, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=CN, label=表4, caption=
Mujoco环境中算法运行时间
, figureFileSmall=null, figureFileBig=null, tableContent=
| 环境 | OC | PPO | TDEOC | DEOC-IF(本文) |
|---|
| Hopper-v2 | 8.7 | 6.9 | 10.2 | 9.5 |
| Walker2d-v2 | 13.6 | 10.3 | 17.2 | 15.1 |
| Humanoid-v2 | 19.1 | 16.2 | 23.7 | 21.3 |
| HalfCheetah-v2 | 8.3 | 7.1 | 9.4 | 8.9 |
), ArticleFig(id=1329458158927458750, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=EN, label=Table 5, caption=
The Table of Mathematical Symbols
, figureFileSmall=null, figureFileBig=null, tableContent=
| 符号 | 符号含义 | | 符号 | 符号含义 |
|---|
| $ S $ | 状态集合 | | $ \varOmega $ | Option集合 |
| $ A $ | 动作集合 | | $ o $ | Option下层策略 |
| $ \gamma $ | 奖励折扣系数 | | $ \theta $ | 下层策略参数 |
| $ r $ | 奖励 | | $ U $ | 到达值函数 |
| $ P $ | 状态转移概率 | | $ \mu $ | 轨迹折扣权重 |
| $ \pi $ | 行为策略 | | $ {R_{{\text{bonus}}}} $ | 伪奖励函数 |
| $ V $ | 状态值函数 | | $ H $ | 香农熵 |
| $ Q $ | 动作值函数 | | $ \tau $ | 伪奖励超参数 |
| $ {I_o} $ | Option初始状态集合 | | $ c $ | 正则项系数 |
| $ {\beta _o} $ | Option终止函数 | | $ z $ | 兴趣函数参数 |
), ArticleFig(id=1329458159049093567, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329458143685357907, language=CN, label=表5, caption=
数学符号表
, figureFileSmall=null, figureFileBig=null, tableContent=
| 符号 | 符号含义 | | 符号 | 符号含义 |
|---|
| $ S $ | 状态集合 | | $ \varOmega $ | Option集合 |
| $ A $ | 动作集合 | | $ o $ | Option下层策略 |
| $ \gamma $ | 奖励折扣系数 | | $ \theta $ | 下层策略参数 |
| $ r $ | 奖励 | | $ U $ | 到达值函数 |
| $ P $ | 状态转移概率 | | $ \mu $ | 轨迹折扣权重 |
| $ \pi $ | 行为策略 | | $ {R_{{\text{bonus}}}} $ | 伪奖励函数 |
| $ V $ | 状态值函数 | | $ H $ | 香农熵 |
| $ Q $ | 动作值函数 | | $ \tau $ | 伪奖励超参数 |
| $ {I_o} $ | Option初始状态集合 | | $ c $ | 正则项系数 |
| $ {\beta _o} $ | Option终止函数 | | $ z $ | 兴趣函数参数 |
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202220970, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202220970, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202220970, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202220970, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1732982400000, fullTextJson=null, articleText=null, reference=null)