收藏切换
A Diversity-Enriched Option-Critic Algorithm with Interest Functions
收藏切换
Junwei Li1, Quan Liu1, 2, 3, 4, *, Zhigang Huang1, Yapeng Xu1
Journal of Computer Research and Development | 2024, 61(12) : 3108 - 3120
Less
收藏切换
Journal of Computer Research and Development | 2024, 61(12): 3108-3120
• Artificial Intelligence •
A Diversity-Enriched Option-Critic Algorithm with Interest Functions
Full
Junwei Li1, Quan Liu1, 2, 3, 4, *, Zhigang Huang1, Yapeng Xu1
Affiliations
  • 1School of Computer Science and Technology, Soochow University, Suzhou, Jiangsu 215006
  • 2Collaborative Innovation Center of Novel Software Technology and Industrialization (Nanjing University), Nanjing 210023
  • 3Key Laboratory of Symbol Computation and Knowledge Engineering (Jilin University), Ministry of Education, Changchun 130012
  • 4Jiangsu Provincial Key Laboratory for Computer Information Processing Technology (Soochow University), Suzhou, Jiangsu 215006
  • born in 1998. Master candidate. His main research interests include reinforcement learning and hierarchical reinforcement learning

    born in 1969. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include intelligence information processing, automated reasoning, and machine learning

    born in 1993. PhD candidate. His main research interests include reinforcement learning, deep reinforcement learning, and hierarchical reinforcement learning

    born in 1996. Master candidate. His main research interests include deep reinforcement learning and hierarchical reinforcement learning

About Author:

born in 1998. Master candidate. His main research interests include reinforcement learning and hierarchical reinforcement learning

born in 1969. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include intelligence information processing, automated reasoning, and machine learning

born in 1993. PhD candidate. His main research interests include reinforcement learning, deep reinforcement learning, and hierarchical reinforcement learning

born in 1996. Master candidate. His main research interests include deep reinforcement learning and hierarchical reinforcement learning

Published: 2024-12-01 doi: 10.7544/issn1000-1239.202220970
Outline
收藏切换

As a common temporal abstraction method for hierarchical reinforcement learning, Option framework allows agents to learn strategies at different time scales, which can effectively solve sparse reward problems. In order to ensure that options can guide agents to access more state space, some methods improve the diversity of options by introducing mutual information in internal reward and termination functions. However, it will lead to slow algorithm learning speed and low knowledge transfer ability of internal strategy, which seriously affect algorithm performance. To address the above problems, diversity-enriched option-critic algorithm with interest functions(DEOC-IF) is proposed. Based on the diversity-enriched option-critic (DEOC) algorithm, the algorithm constrains the selection of the upper-level strategy on the internal strategy of Option by introducing the interest function, which not only ensures the diversity of the Option set, but also makes the learned internal strategies focus on different regions of the state space, which is conducive to improving the knowledge transfer ability of the algorithm and accelerating the learning speed. In addition, DEOC-IF introduces a new interest function update gradient, which is beneficial to improve the exploration ability of the algorithm. In order to verify the effectiveness and option reusability of the algorithm, the algorithm comparison experiments are carried out in four-room navigation task, Mujoco, and MiniWorld. Experimental results show that DEOC-IF algorithm has better performance and option reusability compared with other algorithms.

reinforcement learning  /  temporal abstractions  /  Option framework  /  interest function  /  Option-Critic algorithm
栗军伟, 刘全, 黄志刚, 徐亚鹏. 基于兴趣函数的多样化Option-Critic算法. 计算机研究与发展, 2024 , 61 (12) : 3108 -3120 . DOI: 10.7544/issn1000-1239.202220970
Junwei Li, Quan Liu, Zhigang Huang, Yapeng Xu. A Diversity-Enriched Option-Critic Algorithm with Interest Functions[J]. Journal of Computer Research and Development, 2024 , 61 (12) : 3108 -3120 . DOI: 10.7544/issn1000-1239.202220970
Year 2024 volume 61 Issue 12
91
45
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202220970
  • Receive Date:2022-11-23
  • Online Date:2026-11-17
  • Published:2024-12-01
Article Data
Affiliations
History
  • Received:2022-11-23
  • Revised:2024-01-03
  • Accepted:2024-03-06
Affiliations
    1School of Computer Science and Technology, Soochow University, Suzhou, Jiangsu 215006
    2Collaborative Innovation Center of Novel Software Technology and Industrialization (Nanjing University), Nanjing 210023
    3Key Laboratory of Symbol Computation and Knowledge Engineering (Jilin University), Ministry of Education, Changchun 130012
    4Jiangsu Provincial Key Laboratory for Computer Information Processing Technology (Soochow University), Suzhou, Jiangsu 215006
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202220970
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT