电源学报

基于改进PPO的HCSY-MG并网系统分布式混合储能充放电优化控制

PDF下载

李锦键中国电源学会学生会员 , 王兴贵 , 丁颖杰中国电源学会学生会员

电源学报 | 电池与储能 2025,23(4): 255-264

收起

电源学报 |电池与储能 2025 , 23 (4) : 255 -264

基于改进PPO的HCSY-MG并网系统分布式混合储能充放电优化控制

全屏

[Author(id=1321485249683501200, tenantId=1146029695717560320, journalId=1146031654075715584, articleId=1321485248488124551, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=lijinjian0326@163.com, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1321485249750610066, tenantId=1146029695717560320, journalId=1146031654075715584, articleId=1321485248488124551, authorId=1321485249683501200, language=EN, stringName=Jinjian LI, firstName=Jinjian, middleName=null, lastName=LI, prefix=null, suffix=Student Member, CPSS, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=College of Electrical and Information Engineering, Lanzhou University of Technology, Lanzhou 730050, China, bio=null, bioImg=null, bioContent=null, aboutCorrespAuthor=null), CN=AuthorExt(id=1321485249821913235, tenantId=1146029695717560320, journalId=1146031654075715584, articleId=1321485248488124551, authorId=1321485249683501200, language=CN, stringName=李锦键中国电源学会学生会员, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=兰州理工大学电气工程与信息工程学院，兰州　730050, bio={"content":"

李锦键（1995—　），男，中国电源学会学生会员，博士研究生。研究方向：可再生能源发电系统与控制，微电网技术。E-mail：lijinjian0326@163.com。

"}, bioImg=null, bioContent=

李锦键（1995—　），男，中国电源学会学生会员，博士研究生。研究方向：可再生能源发电系统与控制，微电网技术。E-mail：lijinjian0326@163.com。

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1321485249608003724, tenantId=1146029695717560320, journalId=1146031654075715584, articleId=1321485248488124551, xref=null, ext=[AuthorCompanyExt(id=1321485249616392333, tenantId=1146029695717560320, journalId=1146031654075715584, articleId=1321485248488124551, companyId=1321485249608003724, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Electrical and Information Engineering, Lanzhou University of Technology, Lanzhou 730050, China), AuthorCompanyExt(id=1321485249624780942, tenantId=1146029695717560320, journalId=1146031654075715584, articleId=1321485248488124551, companyId=1321485249608003724, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=兰州理工大学电气工程与信息工程学院，兰州　730050)])]), Author(id=1321485249889022101, tenantId=1146029695717560320, journalId=1146031654075715584, articleId=1321485248488124551, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=wangxg8201@163.com, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1321485249964519575, tenantId=1146029695717560320, journalId=1146031654075715584, articleId=1321485248488124551, authorId=1321485249889022101, language=EN, stringName=Xinggui WANG, firstName=Xinggui, middleName=null, lastName=WANG, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=College of Electrical and Information Engineering, Lanzhou University of Technology, Lanzhou 730050, China, bio=null, bioImg=null, bioContent=null, aboutCorrespAuthor=null), CN=AuthorExt(id=1321485250040017048, tenantId=1146029695717560320, journalId=1146031654075715584, articleId=1321485248488124551, authorId=1321485249889022101, language=CN, stringName=王兴贵, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=兰州理工大学电气工程与信息工程学院，兰州　730050, bio=null, bioImg=null, bioContent=null, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1321485249608003724, tenantId=1146029695717560320, journalId=1146031654075715584, articleId=1321485248488124551, xref=null, ext=[AuthorCompanyExt(id=1321485249616392333, tenantId=1146029695717560320, journalId=1146031654075715584, articleId=1321485248488124551, companyId=1321485249608003724, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Electrical and Information Engineering, Lanzhou University of Technology, Lanzhou 730050, China), AuthorCompanyExt(id=1321485249624780942, tenantId=1146029695717560320, journalId=1146031654075715584, articleId=1321485248488124551, companyId=1321485249608003724, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=兰州理工大学电气工程与信息工程学院，兰州　730050)])]), Author(id=1321485250111320218, tenantId=1146029695717560320, journalId=1146031654075715584, articleId=1321485248488124551, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=dingyj0820@126.com, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1321485250186817692, tenantId=1146029695717560320, journalId=1146031654075715584, articleId=1321485248488124551, authorId=1321485250111320218, language=EN, stringName=Yingjie DING, firstName=Yingjie, middleName=null, lastName=DING, prefix=null, suffix=Student Member, CPSS, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=College of Electrical and Information Engineering, Lanzhou University of Technology, Lanzhou 730050, China, bio=null, bioImg=null, bioContent=null, aboutCorrespAuthor=null), CN=AuthorExt(id=1321485250258120861, tenantId=1146029695717560320, journalId=1146031654075715584, articleId=1321485248488124551, authorId=1321485250111320218, language=CN, stringName=丁颖杰中国电源学会学生会员, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=兰州理工大学电气工程与信息工程学院，兰州　730050, bio={"content":"

丁颖杰（1992—　），女，中国电源学会学生会员，博士研究生。研究方向：可再生能源发电系统与控制，微电网技术。E-mail：dingyj0820@126.com。

"}, bioImg=null, bioContent=

丁颖杰（1992—　），女，中国电源学会学生会员，博士研究生。研究方向：可再生能源发电系统与控制，微电网技术。E-mail：dingyj0820@126.com。

李锦键中国电源学会学生会员, 王兴贵, 丁颖杰中国电源学会学生会员

作者信息

兰州理工大学电气工程与信息工程学院，兰州　730050

通讯作者:

王兴贵（1963—　），男，通信作者，硕士，教授。研究方向：可再生能源发电系统与控制，微电网技术，电力电子与电力传动。E-mail：wangxg8201@163.com。

作者简介:

李锦键（1995—　），男，中国电源学会学生会员，博士研究生。研究方向：可再生能源发电系统与控制，微电网技术。E-mail：lijinjian0326@163.com。

丁颖杰（1992—　），女，中国电源学会学生会员，博士研究生。研究方向：可再生能源发电系统与控制，微电网技术。E-mail：dingyj0820@126.com。

Optimal Control of Distributed Hybrid Energy Storage Charge-discharge for HCSY-MG Grid-connected System Based on Improved PPO

Jinjian LI Student Member, CPSS, Xinggui WANG, Yingjie DING Student Member, CPSS

Affiliations

College of Electrical and Information Engineering, Lanzhou University of Technology, Lanzhou 730050, China

出版时间: 2025-07-30 doi: 10.13234/j.issn.2095-2805.2025.4.255

文章导航

摘要

收起

为平抑微源半桥变流器串联星型结构微电网HCSY-MG（half-bridge converter series Y-connection microgrids）并网系统中微源出力的波动，保证各相直流侧电压之和相等，与并网电流三相平衡，提出1种基于改进近端策略优化PPO（proximal policy optimization）的分布式混合储能系统HESS（hybrid energy storage system）充、放电优化控制策略。在考虑HCSY-MG系统并网电流与分布式HESS特性的条件下，确定影响并网电流的主要系统变量，以及HESS接入系统的最佳拓扑结构。然后结合串联系统的特点，将分布式HESS的充、放电问题转换为深度强化学习的Markov决策过程。同时针对PPO算法中熵损失权重难以确定的问题，提出1种改进的PPO算法，兼顾智能体的收敛性和探索性。最后以某新能源发电基地的典型运行数据为算例，验证所提控制策略的可行性和有效性。

关键词

串联微电网 / 分布式混合储能系统 / 近端策略优化 / 充放电功率 / 深度强化学习

Abstract

收起

To mitigate the fluctuations in the output from micro-sources within a half-bridge converter series Y-connection microgrid (HCSY-MG) grid-connected system and ensure both the balance of the sum of DC-side voltages in each phase and the balance of three-phase grid-connected current, a distributed hybrid energy storage system (HESS) charge-discharge optimal control strategy based on improved proximal policy optimization (PPO) is put forward. With the consideration of grid-connected current in the HCSY-MG system and the characteristics of the distributed HESS, the key system variables affecting the grid-connected current are identified, and the optimal topology for integrating the HESS into the system is determined. Subsequently, by combining the characteristics of the series-connected system, the charge-discharge problem of the distributed HESS is transformed into a Markov decision process within the framework of deep reinforcement learning. Meanwhile, to address the challenge of determining the entropy loss weight in the PPO method, an improved PPO method is proposed, which balances the agent’s convergence and exploration capabilities. Finally, the typical operational data of a renewable energy generation site is used as a case study, and the feasibility and effectiveness of the proposed control strategy is validated.

Key words

Series-connected microgrid / distributed hybrid energy storage system (HESS) / proximal policy optimization (PPO) / charge-discharge power / deep reinforcement learning

引用本文

李锦键中国电源学会学生会员, 王兴贵, 丁颖杰中国电源学会学生会员. 基于改进PPO的HCSY-MG并网系统分布式混合储能充放电优化控制. 电源学报, 2025 , 23 (4) : 255 -264 . DOI: 10.13234/j.issn.2095-2805.2025.4.255

Jinjian LI, Xinggui WANG, Yingjie DING. Optimal Control of Distributed Hybrid Energy Storage Charge-discharge for HCSY-MG Grid-connected System Based on Improved PPO[J]. Journal of Power Supply, 2025 , 23 (4) : 255 -264 . DOI: 10.13234/j.issn.2095-2805.2025.4.255

基金

收起

国家自然科学基金资助项目(51967011)

参考文献

收起

文献

收起

参考文献引证文献

排序方式：

[1]

Wang Xinggui, Ding Yingjie, Li Jinjian, et al. Characteristics analysis of micro-source half-bridge converter series Y-connection based microgrid systems[J]. Journal of Power Electronics, 2023, 23(10): 1483-1495.

[2]

海涛, 刘昱麟, 王钧, 等. 基于模型预测控制的光伏混合储能系统控制策略[J/OL]. 电源学报, 1-12. (2024-04-08) [2024-09-18]. https://link.cnki.net/urlid/12.1420.TM.20240407.1045.002. Hai Tao, Liu Yulin, Wang Jun, et al. Control strategy of photovoltaic hybrid energy storage system based on model predictive control[J/OL]. Journal of Power Supply, 1-12.(2024-04-08) [2024-09-18]. https://link.cnki.net/urlid/12.1420.TM.20240407.1045.002. (in Chinese)

[3]

胡维昊, 曹迪, 黄琦, 等. 深度强化学习在配电网优化运行中的应用[J]. 电力系统自动化, 2023, 47(14): 174-191. Hu Weihao, Cao Di, Huang Qi, et al. Application of deep reinforcement learning in optimal operation of distribution network[J]. Automation of Electric Power Systems, 2023, 47(14): 174-191. (in Chinese)

[4]

张自东, 邱才明, 张东霞, 等. 基于深度强化学习的微电网复合储能协调控制方法[J]. 电网技术, 2019, 43(6): 1914-1921. Zhang Zidong, Qiu Caiming, Zhang Dongxia, et al. A coordinated control method for hybrid energy storage system in microgrid based on deep reinforcement learning[J]. Power System Technology, 2019, 43(6): 1914-1921. (in Chinese)

[5]

李鹏, 钟瀚明, 马红伟, 等. 基于深度强化学习的有源配电网多时间尺度源荷储协同优化调控[J/OL]. 电工技术学报, 1-18. (2024-07-19) [2024-09-18]. https://doi.org/10.19595/j.cnki.1000-6753.tces.240316. Li Peng, Zhong Hanming, Ma Hongwei, et al. Multi-timescale optimal dispatch of source-load-storage coordination in active distribution network based on deep reinforcement learning[J/OL]. Transactions of China Electrotechnical Society, 1-18. (2024-07-19) [2024-09-18]. https://doi.org/10.19595/j.cnki.1000-6753.tces.240316. (in Chinese)

[6]

Bui V H, Hussain A, Kim H M. Double deep Q-learning-based distributed operation of battery energy storage system considering uncertainties[J]. IEEE Transactions on Smart Grid, 2020, 11(1): 457-469.

[7]

刘俊峰, 陈剑龙, 王晓生, 等. 基于深度强化学习的微能源网能量管理与优化策略研究[J]. 电网技术, 2020, 44(10): 3794-3803. Liu Junfeng, Chen Jianlong, Wang Xiaosheng, et al. Energy management and optimization of multi-energy grid based on deep reinforcement learning[J]. Power System Technology, 2020, 44(10): 3794-3803. (in Chinese)

[8]

Ding Tao, Zeng Ziyu, Bai Jiawen, et al. Optimal electric vehicle charging strategy with Markov decision process and reinforcement learning technique[J]. IEEE Transactions on Industry Applications, 2020, 56(5): 5811-5823.

[9]

Cao Di, Hu Weihao, Xu Xiao, et al. Deep reinforcement learning based approach for optimal power flow of distribution networks embedded with renewable energy and storage devices[J]. Journal of Modern Power Systems and Clean Energy, 2021, 9(5): 1101-1110.

[10]

Qiu Xin, Nguyen T A, Crow M L. Heterogeneous energy storage optimization for microgrids[J]. IEEE Transactions on Smart Grid, 2016, 7(3): 1453-1461.

[11]

Kuznetsova E, Li Yanfu, Ruiz C, et al. Reinforcement learning for microgrid energy management[J]. Energy, 2013, 59: 133-146.

[12]

Wang Shengyi, Du Liang, Fan Xiaoyuan, et al. Deep reinforcement scheduling of energy storage systems for real-time voltage regulation in unbalanced LV networks with high PV penetration[J]. IEEE Transactions on Sustainable Energy, 2021, 12(4): 2342-2352.

[13]

Williams R J. Simple statistical gradient-following algorithms for connectionist reinforcement learning[J]. Machine Learning, 1992, 8(3): 229-256.

[14]

Schulman J, Levine S, Moritz P, et al. Trust region policy optimization[EB/OL]. (2015-02-19) [2024-07-21]. https://arxiv.org/abs/1502.05477v5.

[15]

Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms[EB/OL]. (2017-07-20) [2024-07-21]. https://arxiv.org/abs/1707.06347v2.

[16]

Wang Xinggui, Li Jinjian, Guo Qun, et al. Parameter design of half-bridge converter series Y-connection microgrid grid-connected filter based on improved PSOLSSVM[J]. International Transactions on Electrical Energy Systems, 2023: 9534004.

[17]

邹伟, 鬲玲, 刘昱杓. 强化学习[M]. 北京: 清华大学出版社, 2020. Zou Wei, Ge Ling, Liu Yubiao. Reinforcement Learning[M]. Beijing: Tsinghua University Press, 2020. (in Chinese)

[18]

秦湖程, 黄炎焱, 陈天德, 等. 基于PPO算法的集群多目标火力规划方法[J]. 系统工程与电子技术, 2024(11): 3764-3773. Qin Hucheng, Huang Yanyan, Chen Tiande, et al. Cluster multi-target fire planning method based on PPO algorithm[J]. Systems Engineering and Electronics, 2024(11): 3764-3773. (in Chinese)

[19]

刘畅, 卓建坤, 赵东明, 等. 利用储能系统实现可再生能源微电网灵活安全运行的研究综述[J]. 中国电机工程学报, 2020, 40(1): 1-18. Liu Chang, Zhuo Jiankun, Zhao Dongming, et al. A review on the utilization of energy storage system for the flexible and safe operation of renewable energy microgrids[J]. Proceedings of the CSEE, 2020, 40(1): 1-18. (in Chinese)

2025年第23卷第4期

PDF下载

引用本文

BibTeX

文章信息

doi: 10.13234/j.issn.2095-2805.2025.4.255

接收时间：2024-08-14
首发时间：2026-10-26
出版时间：2025-07-30

补充材料

相关文章

文章信息

作者

出版历史

收稿日期：2024-08-14
修回日期：2024-09-18
录用日期：2024-10-07

基金

National Natural Science Foundation of China(51967011)

国家自然科学基金资助项目(51967011)

作者信息

兰州理工大学电气工程与信息工程学院，兰州　730050

通讯作者:

王兴贵（1963—　），男，通信作者，硕士，教授。研究方向：可再生能源发电系统与控制，微电网技术，电力电子与电力传动。E-mail：wangxg8201@163.com。

参考文献

分享链接

https://castjournals.cast.org.cn/joweb/dyxb/CN/10.13234/j.issn.2095-2805.2025.4.255

分享至

全文二维码

扫描看全文

引用本文

BibTeX

本文的引用情况

2种不同金属材料的力学参数

科 Family	属数 Number of genus	种数 Number of species	占总种数比例 Percentage of total species (%)	属 Genus	种数 Number of species	占总种数比例 Percentage of total species (%)
鹅膏菌科Amanitaceae	2	11	5.26	鹅膏菌属 Amanita	10	4.78
小菇科 Mycenaceae	2	12	5.74	丝盖伞属 Inocybe	5	2.39
多孔菌科 Polyporaceae	8	14	6.70	蜡蘑属 Laccaria	5	2.39
红菇科 Russulaceae	3	23	11.00	小皮伞属 Marasmius	6	2.87
				小菇属 Mycena	11	5.26
				光柄菇属 Pluteus	5	2.39
				红菇属 Russula	17	8.13
				栓菌属 Trametes	5	2.39

关闭全屏

BibTeX
EndNote
RefWorks
TxT