Article(id=1329420193664709085, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1329420000110161984, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202550701, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=null, receivedDate=1758384000000, receivedDateStr=2025-09-21, revisedDate=1772553600000, revisedDateStr=2026-03-04, acceptedDate=null, acceptedDateStr=null, onlineDate=1794888073602, onlineDateStr=2026-11-17, pubDate=1782835200000, pubDateStr=2026-07-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1794888073602, onlineIssueDateStr=2026-11-17, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1794888073602, creator=13701087609, updateTime=1794888073602, updator=13701087609, issue=Issue{id=1329420000110161984, tenantId=1146029695717560320, journalId=1324759914146529478, year='2026', volume='63', issue='7', pageStart='1639', pageEnd='1903', issueExtLink='null', onlineDate='null', pubDate='1782835200000', pubDateStr='2026-07-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1794888027455, creator='13701087609', updateTime=1794892162905, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext={EN=IssueExt(id=1329437345918595829, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1329420000110161984, language=EN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=), CN=IssueExt(id=1329437345918595830, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1329420000110161984, language=CN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=)}, issueFiles=null, downloadFileDto=null}, startPage=1854, endPage=1868, ext={EN=ArticleExt(id=1329420193853452766, articleId=1329420193664709085, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=Active Communication Masked Reconstruction Method for Partial Observability in Multi-Agent Reinforcement Learning, columnId=1325489891789910943, journalTitle=Journal of Computer Research and Development, columnName=Artificial Intelligence, runingTitle=null, highlight=null, articleAbstract=

In multi-agent reinforcement learning (MARL), partial observability poses a significant challenge to effective collaboration among agents. Although the centralized training with decentralized execution (CTDE) paradigm alleviates this challenge by leveraging global state information during training, agents must still rely solely on local observations during execution. To address this limitation, existing research has primarily followed two directions. The first direction introduces communication mechanisms to enable information sharing among agents; however, these methods often suffer from high communication overhead and redundant information. The second direction leverages mask reconstruction techniques to infer the missing global information; however, due to sample sparsity, the inferred information depends only on local observations, resulting in low inference accuracy and potential interference with decision-making. To overcome these limitations, we propose an active communication masked reconstruction method. During centralized training, the method dynamically groups agents based on global observation trajectories and allows each agent to select the most relevant communication targets from these groups. During decentralized execution, agents proactively communicate according to their selected target sets, enabling accurate reconstruction of global information and ultimately supporting their decision-making. Extensive experiments conducted on the StarCraftⅡ Multi-Agent Challenge (SMAC) demonstrate the effectiveness of the proposed approach in improving coordination and overall performance in partially observable multi-agent scenarios.

, authors=Yueyang Jia, Wenbin Liu, En Wang*, authorsList=Yueyang Jia, Wenbin Liu, En Wang, authorCompany=null, correspAuthors=En Wang, authorNote=

born in 2001. Master. His main research interest includes multi-agent reinforcement learning

born in 1990. PhD, associate professor. His main research interests include multi-agent reinforcement learning, mobile computing, and mobile crowdsensing

born in 1987. PhD, professor. His main research interests include multi-agent reinforcement learning, mobile computing, and mobile crowdsensing

, correspAuthorsNote=null, copyrightStatement=Copyright © 2026 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1329420196571361789, articleId=1329420193664709085, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=面向多智能体强化学习中部分可观测性的主动通信掩码重建方法, columnId=1325489892104483745, journalTitle=计算机研究与发展, columnName=人工智能, runingTitle=null, highlight=null, articleAbstract=

在多智能体强化学习(multi-agent reinforcement learning,MARL)中,部分可观测性问题严重制约了智能体间的协作。尽管集中训练和分散执行(centralized training and decentralized execution,CTDE)范式通过在训练阶段引入全局状态缓解了该问题,但在执行阶段,智能体仍只能依赖局部观测。为进一步解决该限制,目前有2种方案:一种通过引入通信机制实现智能体间的信息共享,但会面临通信开销大、信息冗余等问题;另一种则通过掩码重建推理全局信息,但受限于样本稀疏性,重建信息仅来自于单一智能体的局部观测,导致重建精度低,可能会干扰决策。基于此,提出一种主动通信掩码重建方法。在集中训练阶段,该方法根据全局观测轨迹对智能体进行动态分组,并为每个智能体从各组中选取最相关的通信对象,进而在分散执行阶段,智能体依据通信对象集合进行主动通信,从而实现全局信息的准确重建,最终辅助智能体决策。通过在星际争霸多智能体挑战赛(StarCraftⅡ Multi-Agent Challenge,SMAC)上的一系列实验,证明了该方法在部分可观测场景下对提升多智能体系统协作与整体策略性能的有效性。

, authors=贾月洋, 刘文彬, 王恩*, authorsList=贾月洋, 刘文彬, 王恩, authorCompany=null, correspAuthors=王恩, authorNote=

作者贡献声明

贾月洋负责提出方法思路,完成实验并撰写论文;刘文彬负责论文审定和修改;王恩提出指导意见并修改论文。

2001年生。硕士。主要研究方向为多智能体强化学习

1990年生。博士,副教授。主要研究方向为多智能体强化学习、移动计算、移动群智感知

1987年生。博士,教授。主要研究方向为多智能体强化学习、移动计算、移动群智感知

, correspAuthorsNote=
王恩()
, copyrightStatement=版权所有 © 2026 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=6S981VrJ6czfklH5rIzrjA==, magXml=s8hfzOGfTItf/h1sRL7/bg==, pdfUrl=null, pdf=SaO+uYPT2HPGhASRB5svaw==, pdfFileSize=2172271, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=Ho+fwkele/rXHDh1hUomYQ==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=WX8LMkkFNint+NIc0fUSNQ==, mapNumber=null, fund=null)}, authors=[Author(id=1329420196940460549, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=yyjia24@mails.jlu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329420197003375110, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, authorId=1329420196940460549, language=EN, stringName=Yueyang Jia, firstName=Yueyang, middleName=null, lastName=Jia, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"content":"

born in 2001. Master. His main research interest includes multi-agent reinforcement learning

"}, bioImg=null, bioContent=

born in 2001. Master. His main research interest includes multi-agent reinforcement learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329420197074678279, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, authorId=1329420196940460549, language=CN, stringName=贾月洋, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"img":"t4mjzUs6AKVLqv/+krJDAQ==","content":"

2001年生。硕士。主要研究方向为多智能体强化学习

"}, bioImg=t4mjzUs6AKVLqv/+krJDAQ==, bioContent=

2001年生。硕士。主要研究方向为多智能体强化学习

, aboutCorrespAuthor=null)}, companyList=null), Author(id=1329420197141787145, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329420197204701706, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, authorId=1329420197141787145, language=EN, stringName=Wenbin Liu, firstName=Wenbin, middleName=null, lastName=Liu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"content":"

born in 1990. PhD, associate professor. His main research interests include multi-agent reinforcement learning, mobile computing, and mobile crowdsensing

"}, bioImg=null, bioContent=

born in 1990. PhD, associate professor. His main research interests include multi-agent reinforcement learning, mobile computing, and mobile crowdsensing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329420197267616267, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, authorId=1329420197141787145, language=CN, stringName=刘文彬, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"img":"zS/+bTmFmRxWjCniL5tkOA==","content":"

1990年生。博士,副教授。主要研究方向为多智能体强化学习、移动计算、移动群智感知

"}, bioImg=zS/+bTmFmRxWjCniL5tkOA==, bioContent=

1990年生。博士,副教授。主要研究方向为多智能体强化学习、移动计算、移动群智感知

, aboutCorrespAuthor=null)}, companyList=null), Author(id=1329420197338919437, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=wangen@jlu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329420197406028302, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, authorId=1329420197338919437, language=EN, stringName=En Wang, firstName=En, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=null, bio={"content":"

born in 1987. PhD, professor. His main research interests include multi-agent reinforcement learning, mobile computing, and mobile crowdsensing

"}, bioImg=null, bioContent=

born in 1987. PhD, professor. His main research interests include multi-agent reinforcement learning, mobile computing, and mobile crowdsensing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329420197477331471, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, authorId=1329420197338919437, language=CN, stringName=王恩, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=null, bio={"img":"4IuF5HrG+ryn331f101tlg==","content":"

1987年生。博士,教授。主要研究方向为多智能体强化学习、移动计算、移动群智感知

"}, bioImg=4IuF5HrG+ryn331f101tlg==, bioContent=

1987年生。博士,教授。主要研究方向为多智能体强化学习、移动计算、移动群智感知

, aboutCorrespAuthor=null)}, companyList=null)], keywords=[Keyword(id=1329420197603160592, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, orderNo=1, keyword=multi-agent reinforcement learning), Keyword(id=1329420197678658065, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, orderNo=2, keyword=partial observability), Keyword(id=1329420197745766930, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, orderNo=3, keyword=dynamic grouping), Keyword(id=1329420197917733395, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, orderNo=4, keyword=multi-agent communication), Keyword(id=1329420197980647956, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, orderNo=5, keyword=masked autoencoder), Keyword(id=1329420198098088469, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, orderNo=6, keyword=StarCraftⅡ Multi-Agent Challenge (SMAC)), Keyword(id=1329420198161003030, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, orderNo=1, keyword=多智能体强化学习), Keyword(id=1329420198257472023, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, orderNo=2, keyword=部分可观测性), Keyword(id=1329420198488158744, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, orderNo=3, keyword=动态分组), Keyword(id=1329420198613987865, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, orderNo=4, keyword=多智能体通信), Keyword(id=1329420198689485338, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, orderNo=5, keyword=掩码自编码器), Keyword(id=1329420198764982811, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, orderNo=6, keyword=SMAC)], refs=[Reference(id=1329420201734550081, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=Ning Zepeng, Xie Lihua. A survey on multi-agent reinforcement learning and its application[J]. Journal of Automation and Intelligence, 2024, 3(2): 73−91, articleTitle=null, refAbstract=null), Reference(id=1329420201831019074, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Canese L, Cardarilli G C, Nunzio L D, et al. Multi-agent reinforcement learning: A review of challenges and applications[J]. Applied Sciences, 2021, 11(11): 4948, articleTitle=null, refAbstract=null), Reference(id=1329420201902322243, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Nguyen T T, Nguyen N D, Nahavandi S. Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications[J]. IEEE Transactions on Cybernetics, 2020, 50(9): 3826−3839, articleTitle=null, refAbstract=null), Reference(id=1329420201977819716, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhou Ming, Luo Jun, Villella J, et al. Smarts: An open-source scalable multi-agent RL training school for autonomous driving[C]//Proc of the 4th Conf on Robot Learning. New York: PMLR, 2021: 264−285, articleTitle=null, refAbstract=null), Reference(id=1329420202040734277, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Yang Yaodong, Luo Rui, Li Minne, et al. Mean field multi-agent reinforcement learning[C]//Proc of the 35th Int Conf on Machine Learning. New York: ACM, 2018: 5571−5580, articleTitle=null, refAbstract=null), Reference(id=1329420202103648838, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Di Yuanzhu, Deng Libao, Wang Ling, et al. A review of deep reinforcement learning methods for production scheduling[J/OL]. Tsinghua Science and Technology, 2025[2026-01-23]. https://doi.org/10.26599/TST.2024.9010247, articleTitle=null, refAbstract=null), Reference(id=1329420202183340615, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Yu Chao, Yang Xinyi, Gao Jiaxuan, et al. Asynchronous multi-agent reinforcement learning for efficient real-time multi-robot cooperative exploration[C]//Proc of the 2023 Int Conf on Autonomous Agents and Multi-agent Systems. Berlin: Springer, 2023: 1107−1115, articleTitle=null, refAbstract=null), Reference(id=1329420202237866568, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Han Sheng, Zhang Hengrui, Wu Hao, et al. Multi-constraint reinforcement learning in complex robot environments[J]. Frontiers of Computer Science, 2025, 19(8) : 198353, articleTitle=null, refAbstract=null), Reference(id=1329420202330141257, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Peng Peng, Wen Ying, Yang Yaodong, et al. Multiagent bidirectionally-coordinated nets: Emergence of human-level coordination in learning to play starcraft combat games[J]. arXiv preprint, arXiv: 1703.10069, 2017, articleTitle=null, refAbstract=null), Reference(id=1329420202393055818, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=路文浩, 赵勇, 季雅泰, 等. 面向应急响应群智感知的异构群体协作任务分配方法[J]. 物联网学报, 2025, 9(4): 77−90, articleTitle=null, refAbstract=null), Reference(id=1329420202481136203, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Lu Wenhao, Zhao Yong, Ji Yatai, et al. Task allocation method for collaborative crowdsensing of heterogeneous entities in emergency response[J]. Chinese Journal on Internet of Things, 2025, 9(4): 77−90 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1329420202569216588, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Gupta J K, Egorov M, Kochenderfer M. Cooperative multi-agent control using deep reinforcement learning[C]//Proc of the 2017 Int Conf on Autonomous Agents and Multi-agent Systems. Berlin: Springer, 2017: 66−83, articleTitle=null, refAbstract=null), Reference(id=1329420202632131149, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Kraemer L, Banerjee B. Multi-agent reinforcement learning as a rehearsal for decentralized planning[J]. Neurocomputing, 2016, 190(19): 82−94, articleTitle=null, refAbstract=null), Reference(id=1329420202695045710, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhu Changxi, Dastani M, Wang Shihan. A survey of multi-agent deep reinforcement learning with communication[J]. arXiv preprint, arXiv: 2203.08975, 2024, articleTitle=null, refAbstract=null), Reference(id=1329420202774737487, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Seo Y, Hafner D, Liu Hao, et al. Masked world models for visual control[C]//Proc of the 6th Conf on Robot Learning. New York: PMLR, 2023: 1332−1344, articleTitle=null, refAbstract=null), Reference(id=1329420202850234960, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Ng A, Jordan M, Weiss Y. On spectral clustering: Analysis and an algorithm[C]//Proc of the 15th Int Conf on Neural Information Processing Systems: Natural and Synthetic. Cambridge, MA: MIT, 2001: 849−856, articleTitle=null, refAbstract=null), Reference(id=1329420202967675473, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proc of the 31st Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2017: 6000−6010, articleTitle=null, refAbstract=null), Reference(id=1329420203043172946, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=He Kaiming, Chen Xinlei, Xie Saining, et al. Masked autoencoders are scalable vision learners[C]//Proc of the 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 16000−16009, articleTitle=null, refAbstract=null), Reference(id=1329420203106087507, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Samvelyan M, Rashid T, De Witt C S, et al. The StarCraft multi-agent challenge[J]. arXiv preprint, arXiv: 1902.04043, 2019, articleTitle=null, refAbstract=null), Reference(id=1329420203198362196, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Lauer M, Riedmiller M A. An algorithm for distributed reinforcement learning in cooperative multi-agent systems[C]//Proc of the 17th Int Conf on Machine Learning. New York: ACM, 2000: 535−542, articleTitle=null, refAbstract=null), Reference(id=1329420203273859669, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Foerster J, Nardelli N, Farquhar G, et al. Stabilising experience replay for deep multi-agent reinforcement learning[C]//Proc of the 34th Int Conf on Machine Learning. New York: ACM, 2017: 1146−1155, articleTitle=null, refAbstract=null), Reference(id=1329420203336774230, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Sunehag P, Lever G, Gruslys A, et al. Value-decomposition networks for cooperative multi-agent learning[J]. arXiv preprint, arXiv: 1706.05296, 2017, articleTitle=null, refAbstract=null), Reference(id=1329420203408077399, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Rashid T, Samvelyan M, De Witt C S, et al. Monotonic value function factorisation for deep multi-agent reinforcement learning[J]. Journal of Machine Learning Research, 2020, 21(1): 1−51, articleTitle=null, refAbstract=null), Reference(id=1329420203470991960, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=Mnih V, Badia A P, Mirza M, et al. Asynchronous methods for deep reinforcement learning[C]//Proc of the 33rd Int Conf on Machine Learning. New York: ACM, 2016: 1928−1937, articleTitle=null, refAbstract=null), Reference(id=1329420203538100825, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=Lillicrap T P, Hunt J J, Pritzel A, et al. Continuous control with deep reinforcement learning[J]. arXiv preprint, arXiv: 1509.02971, 2015, articleTitle=null, refAbstract=null), Reference(id=1329420203596821082, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=Russell S J, Zimdars A. Q-decomposition for reinforcement learning agents[C]//Proc of the 20th Int Conf on Machine Learning. New York: ACM, 2003: 656−663, articleTitle=null, refAbstract=null), Reference(id=1329420203663929947, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=26, authorNames=null, journalName=null, refType=null, unstructuredReference=Schroeder de Witt C, Foerster J, Farquhar G, et al. Multi-agent common knowledge reinforcement learning[C]//Proc of the 33rd Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2019: 9927−9939, articleTitle=null, refAbstract=null), Reference(id=1329420203722650204, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=27, authorNames=null, journalName=null, refType=null, unstructuredReference=Shu Tianmin, Tian Yuandong. M3RL: Mind-aware multi-agent management reinforcement learning[J]. arXiv preprint, arXiv: 1810.00147, 2018, articleTitle=null, refAbstract=null), Reference(id=1329420203798147677, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=28, rfOrder=28, authorNames=null, journalName=null, refType=null, unstructuredReference=Iqbal S, Costales R, Sha F. ALMA: Hierarchical learning for composite multi-agent tasks[C]//Proc of the 36th Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2022: 7155−7166, articleTitle=null, refAbstract=null), Reference(id=1329420203869450846, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=29, rfOrder=29, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Tonghan, Dong Heng, Lesser V, et al. ROMA: Multi-agent reinforcement learning with emergent roles[C]//Proc of the 37th Int Conf on Machine Learning. New York: ACM, 2020: 9876–9886, articleTitle=null, refAbstract=null), Reference(id=1329420203932365407, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=30, rfOrder=30, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Tonghan, Gupta T, Mahajan A, et al. RODE: Learning roles to decompose multi-agent tasks[J]. arXiv preprint, arXiv: 2010.01523, 2020, articleTitle=null, refAbstract=null), Reference(id=1329420204016251488, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=31, rfOrder=31, authorNames=null, journalName=null, refType=null, unstructuredReference=Zang Yifan, He Jinmin, Li Kai, et al. Automatic grouping for efficient cooperative multi-agent reinforcement learning[C]//Proc of the 37th Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2023: 46105−46121, articleTitle=null, refAbstract=null), Reference(id=1329420204087554657, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=32, rfOrder=32, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Chiqiang, Li Dazi. HYGMA: Hypergraph coordination networks with dynamic grouping for multi-agent reinforcement learning[J]. arXiv preprint, arXiv: 2505.07207, 2025, articleTitle=null, refAbstract=null), Reference(id=1329420204158857826, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=33, rfOrder=33, authorNames=null, journalName=null, refType=null, unstructuredReference=Caliński T, Harabasz J. A dendrite method for cluster analysis[J]. Communications in Statistics-theory and Methods, 1974, 3(1): 1−27, articleTitle=null, refAbstract=null), Reference(id=1329420204221772387, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=34, rfOrder=34, authorNames=null, journalName=null, refType=null, unstructuredReference=Foerster J, Assael I A, De Freitas N, et al. Learning to communicate with deep multi-agent reinforcement learning[C]//Proc of the 30th Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2016: 2145−2153, articleTitle=null, refAbstract=null), Reference(id=1329420204288881252, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=35, rfOrder=35, authorNames=null, journalName=null, refType=null, unstructuredReference=Sukhbaatar S, Fergus R. Learning multiagent communication with backpropagation[C]//Proc of the 30th Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2016: 2252−2260, articleTitle=null, refAbstract=null), Reference(id=1329420204385350246, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=36, rfOrder=36, authorNames=null, journalName=null, refType=null, unstructuredReference=Jiang Jiechuan, Lu Zongqing. Learning attentional communication for multi-agent cooperation[C]//Proc of the 32nd Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2018: 7265−7275, articleTitle=null, refAbstract=null), Reference(id=1329420204465042023, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=37, rfOrder=37, authorNames=null, journalName=null, refType=null, unstructuredReference=Singh A, Jain T, Sukhbaatar S. Learning when to communicate at scale in multiagent cooperative and competitive tasks[J]. arXiv preprint, arXiv: 1812.09755, 2018, articleTitle=null, refAbstract=null), Reference(id=1329420204536345192, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=38, rfOrder=38, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Yong, Wang Weixun, Hu Yujing, et al. Multi-agent game abstraction via graph attention neural network[C]//Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 7211−7218, articleTitle=null, refAbstract=null), Reference(id=1329420204624425577, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=39, rfOrder=39, authorNames=null, journalName=null, refType=null, unstructuredReference=Das A, Gervet T, Romoff J, et al. TarMAC: Targeted multi-agent communication[C]//Proc of the 36th Int Conf on Machine Learning. New York: ACM, 2019: 1538−1546, articleTitle=null, refAbstract=null), Reference(id=1329420204758643306, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=40, rfOrder=40, authorNames=null, journalName=null, refType=null, unstructuredReference=Guan Cong, Chen Feng, Yuan Lei, et al. Efficient multi-agent communication via self-supervised information aggregation[C]//Proc of the 36th Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2022: 1020−1033, articleTitle=null, refAbstract=null), Reference(id=1329420204934804075, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=41, rfOrder=41, authorNames=null, journalName=null, refType=null, unstructuredReference=Devlin J, Chang Mingwei, Lee K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]//Proc of the 2019 Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, PA: ACL, 2019: 4171−4186, articleTitle=null, refAbstract=null), Reference(id=1329420205014495852, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=42, rfOrder=42, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Fangchen, Liu Hao, Grover A, et al. Masked autoencoding for scalable and generalizable decision making[C]//Proc of the 36th Int Conf on Machine Learning. New York: ACM, 2022: 12608−12618, articleTitle=null, refAbstract=null), Reference(id=1329420205085799021, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=43, rfOrder=43, authorNames=null, journalName=null, refType=null, unstructuredReference=Wu P, Majumdar A, Stone K, et al. Masked trajectory models for prediction, representation, and control[C]//Proc of the 40th Int Conf on Machine Learning. New York: ACM, 2023: 37607−37623, articleTitle=null, refAbstract=null), Reference(id=1329420205152907886, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=44, rfOrder=44, authorNames=null, journalName=null, refType=null, unstructuredReference=Song Haolin, Feng Mingxiao, Zhou Wengang, et al. MA2CL: Masked attentive contrastive learning for multi-agent reinforcement learning[J]. arXiv preprint, arXiv: 2306.02006, 2023, articleTitle=null, refAbstract=null), Reference(id=1329420205232599663, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=45, rfOrder=45, authorNames=null, journalName=null, refType=null, unstructuredReference=Kang S, Lee Y, Kim G, et al. MA2E: Addressing partial observability in multi-agent reinforcement learning with masked auto-encoder[C]//Proc of the 13rd Int Conf Learning Representations. Mountain View, CA: OpenReview. net, 2025: 51818−51838, articleTitle=null, refAbstract=null), Reference(id=1329420205417149040, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=46, rfOrder=46, authorNames=null, journalName=null, refType=null, unstructuredReference=Oliehoek F A, Amato C. A Concise Introduction to Decentralized POMDPs[M]. Berlin: Springer, 2016, articleTitle=null, refAbstract=null), Reference(id=1329420205555561073, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=47, rfOrder=47, authorNames=null, journalName=null, refType=null, unstructuredReference=Shi Jianbo, Malik J. Normalized cuts and image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(8): 888−905, articleTitle=null, refAbstract=null), Reference(id=1329420205622669938, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=48, rfOrder=48, authorNames=null, journalName=null, refType=null, unstructuredReference=Zha Hongyuan, He Xiaofeng, Ding C, et al. Spectral relaxation for k-means clustering[C]//Proc of the 15th Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2001: 1057−1064, articleTitle=null, refAbstract=null), Reference(id=1329420205719138932, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=49, rfOrder=49, authorNames=null, journalName=null, refType=null, unstructuredReference=Hu Jian, Jiang Siyang, Harding S A, et al. Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning[J]. arXiv preprint, arXiv: 2102.03479, 2021, articleTitle=null, refAbstract=null), Reference(id=1329420205794636405, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=50, rfOrder=50, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Jianhao, Ren Zhizhou, Liu T, et al. QPLEX: Duplex dueling multi-agent Q-learning[C]//Proc of the 9th Int Conf on Learning Representations. Mountain View, CA: OpenReview.net, 2021: 11609−11635, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1329420196768494078, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, xref=null, ext=[AuthorCompanyExt(id=1329420196781076991, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, companyId=1329420196768494078, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=College of Software, Jilin University, Changchun 130012), AuthorCompanyExt(id=1329420196789465600, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, companyId=1329420196768494078, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=吉林大学软件学院 长春 130012)]), AuthorCompany(id=1329420196856574465, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, xref=null, ext=[AuthorCompanyExt(id=1329420196864963074, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, companyId=1329420196856574465, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Key Laboratory of Symbol Computation and Knowledge Engineering (Jilin University), Ministry of Education, Changchun 130012), AuthorCompanyExt(id=1329420196873351683, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, companyId=1329420196856574465, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=符号计算与知识工程教育部重点实验室(吉林大学) 长春 130012)])], figs=[ArticleFig(id=1329420198886617628, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, label=Figure 1, caption=Overall framework of our method, figureFileSmall=1KvRtaO31DbweE6u03jkTQ==, figureFileBig=Ho+fwkele/rXHDh1hUomYQ==, tableContent=null), ArticleFig(id=1329420198957920797, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, label=图1, caption=本文方法整体框架, figureFileSmall=1KvRtaO31DbweE6u03jkTQ==, figureFileBig=Ho+fwkele/rXHDh1hUomYQ==, tableContent=null), ArticleFig(id=1329420199230550558, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, label=Figure 2, caption=Agent-level masked autoencoder combined with active communication, figureFileSmall=2a97OYUtqRmUhJYNK/dK/Q==, figureFileBig=iVXXpysxV6l1DgpXq/apOw==, tableContent=null), ArticleFig(id=1329420199301853727, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, label=图2, caption=结合主动通信的智能体级掩码自编码器, figureFileSmall=2a97OYUtqRmUhJYNK/dK/Q==, figureFileBig=iVXXpysxV6l1DgpXq/apOw==, tableContent=null), ArticleFig(id=1329420199373156896, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, label=Figure 3, caption=Average win rate comparison on SMAC map scenarios, figureFileSmall=P5n0xzEaP+EjPIX9X0KqyA==, figureFileBig=G9KtnZlyYVjjmTD61dxVGw==, tableContent=null), ArticleFig(id=1329420199452848674, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, label=图3, caption=SMAC地图场景下的平均胜率对比, figureFileSmall=P5n0xzEaP+EjPIX9X0KqyA==, figureFileBig=G9KtnZlyYVjjmTD61dxVGw==, tableContent=null), ArticleFig(id=1329420199524151843, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, label=Figure 4, caption=Mean return comparison on SMAC map scenarios, figureFileSmall=9G89YjsdHcavdmNK7G3h8Q==, figureFileBig=iMX2L3dHlloUuWj55+RmFw==, tableContent=null), ArticleFig(id=1329420199587066404, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, label=图4, caption=SMAC地图场景下的平均回报对比, figureFileSmall=9G89YjsdHcavdmNK7G3h8Q==, figureFileBig=iMX2L3dHlloUuWj55+RmFw==, tableContent=null), ArticleFig(id=1329420199679341093, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, label=Figure 5, caption=Impact of information integrity on performance, figureFileSmall=tKCmpfJZuU2J4qAo50o3Qw==, figureFileBig=OvJr4O7Iha1UObJa4Fc7Uw==, tableContent=null), ArticleFig(id=1329420199759032870, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, label=图5, caption=信息完整性对性能的影响, figureFileSmall=tKCmpfJZuU2J4qAo50o3Qw==, figureFileBig=OvJr4O7Iha1UObJa4Fc7Uw==, tableContent=null), ArticleFig(id=1329420199826141735, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, label=Figure 6, caption=Performance under different information integrity, figureFileSmall=tGPqb+SltCDkOBTp0eRGWg==, figureFileBig=IYfgSFCd2bTokRjxKSoyrg==, tableContent=null), ArticleFig(id=1329420199893250600, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, label=图6, caption=不同信息完整性下的性能表现, figureFileSmall=tGPqb+SltCDkOBTp0eRGWg==, figureFileBig=IYfgSFCd2bTokRjxKSoyrg==, tableContent=null), ArticleFig(id=1329420199956165161, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, label=Figure 7, caption=Comparison between QMIX and the proposed method, figureFileSmall=w9ieXmwgCd0W8LmIvfx3fw==, figureFileBig=uxzNdGKpr5S19WyBaS4HSg==, tableContent=null), ArticleFig(id=1329420200031662634, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, label=图7, caption=QMIX与本文方法的对比, figureFileSmall=w9ieXmwgCd0W8LmIvfx3fw==, figureFileBig=uxzNdGKpr5S19WyBaS4HSg==, tableContent=null), ArticleFig(id=1329420200094577195, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, label=Figure 8, caption=Comparison between MA2E and our method, figureFileSmall=93gUdRjLi8e+eajUsQOdlQ==, figureFileBig=c4x2FL3wMXE+lsUI83cnfA==, tableContent=null), ArticleFig(id=1329420200165880364, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, label=图8, caption=MA2E与本文方法的对比, figureFileSmall=93gUdRjLi8e+eajUsQOdlQ==, figureFileBig=c4x2FL3wMXE+lsUI83cnfA==, tableContent=null), ArticleFig(id=1329420200228794925, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, label=Figure 9, caption=Comparison between different communication mechanisms, figureFileSmall=hgW6L7WhbOCZh3M90qIyRA==, figureFileBig=31JgopAS+Stm1IC4wuUgtA==, tableContent=null), ArticleFig(id=1329420200287515182, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, label=图9, caption=不同通信机制之间的对比, figureFileSmall=hgW6L7WhbOCZh3M90qIyRA==, figureFileBig=31JgopAS+Stm1IC4wuUgtA==, tableContent=null), ArticleFig(id=1329420200354624047, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, label=Figure 10, caption=Comparison between VDN and our method, figureFileSmall=cHpfdOIAenvO2qd5Uzo6gA==, figureFileBig=UW7850DiC2Cqsns/T1q/Dg==, tableContent=null), ArticleFig(id=1329420200421732912, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, label=图10, caption=VDN与本文方法的对比, figureFileSmall=cHpfdOIAenvO2qd5Uzo6gA==, figureFileBig=UW7850DiC2Cqsns/T1q/Dg==, tableContent=null), ArticleFig(id=1329420200488841777, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, label=Figure 11, caption=Comparison between QPLEX and our method, figureFileSmall=bBwvzcMZWF5AgafIIzUmPw==, figureFileBig=Goczvbh8cJ6iuwnDLilJOA==, tableContent=null), ArticleFig(id=1329420200551756338, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, label=图11, caption=QPLEX与本文方法的对比, figureFileSmall=bBwvzcMZWF5AgafIIzUmPw==, figureFileBig=Goczvbh8cJ6iuwnDLilJOA==, tableContent=null), ArticleFig(id=1329420200623059507, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, label=Figure 12, caption=Visualization of QMIX and our method, figureFileSmall=hXoPtCJD4zDdW8QRRp9FUw==, figureFileBig=ZPfw/iyvBNmtZQUBfeXjAQ==, tableContent=null), ArticleFig(id=1329420200685974068, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, label=图12, caption=QMIX与本文方法的可视化, figureFileSmall=hXoPtCJD4zDdW8QRRp9FUw==, figureFileBig=ZPfw/iyvBNmtZQUBfeXjAQ==, tableContent=null), ArticleFig(id=1329420200748888629, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, label=Table 1, caption=

The Map Settings

, figureFileSmall=null, figureFileBig=null, tableContent=
难度
等级
地图名称友军单位
类型及数量
敌军单位
类型及数量
特点
Easy2s3z2名狂战士,
3名潜行者
2名狂战士,
3名潜行者
异构,对称
Hard5m_vs_6m5名海军陆战队6名海军陆战队同构,不对称
Super
Hard
6h_vs_8z6名水蛛
(远程)
8名狂战士
(近战)
异构,不对称,
策略凸显
), ArticleFig(id=1329420200845357622, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, label=表1, caption=

地图设置

, figureFileSmall=null, figureFileBig=null, tableContent=
难度
等级
地图名称友军单位
类型及数量
敌军单位
类型及数量
特点
Easy2s3z2名狂战士,
3名潜行者
2名狂战士,
3名潜行者
异构,对称
Hard5m_vs_6m5名海军陆战队6名海军陆战队同构,不对称
Super
Hard
6h_vs_8z6名水蛛
(远程)
8名狂战士
(近战)
异构,不对称,
策略凸显
), ArticleFig(id=1329420200925049399, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, label=Table 2, caption=

The Hyperparameter Settings

, figureFileSmall=null, figureFileBig=null, tableContent=
超参数名称变量符号取值
缓冲区大小buffer_size1000
批量大小batch_size128
批量运行大小batch_size_run8
运行器runnerparallel
优化器optimizerAdam
学习率lr0.001
TD($ \lambda $)td_lambda0.6/0.3
隐藏状态维度rnn_hidden_dim128
混合嵌入维度mixing_embed_dim32
超网络嵌入维度hypernet_embed64
$ \gamma $gamma0.99
动作选择器action_selectorepsilon_greedy
退火时间epsilon_anneal_time100000/500000
), ArticleFig(id=1329420201063461432, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, label=表2, caption=

超参数设置

, figureFileSmall=null, figureFileBig=null, tableContent=
超参数名称变量符号取值
缓冲区大小buffer_size1000
批量大小batch_size128
批量运行大小batch_size_run8
运行器runnerparallel
优化器optimizerAdam
学习率lr0.001
TD($ \lambda $)td_lambda0.6/0.3
隐藏状态维度rnn_hidden_dim128
混合嵌入维度mixing_embed_dim32
超网络嵌入维度hypernet_embed64
$ \gamma $gamma0.99
动作选择器action_selectorepsilon_greedy
退火时间epsilon_anneal_time100000/500000
), ArticleFig(id=1329420201164124729, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, label=Table 3, caption=

Final Win Rate Comparison on SMAC Map Scenarios

, figureFileSmall=null, figureFileBig=null, tableContent=
方法2s3z5m_vs_6m6h_vs_8z
QMIX0.970.470.39
MA2E0.970.480.31
本文0.980.670.43
), ArticleFig(id=1329420201222844986, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, label=表3, caption=

SMAC地图场景下的最终胜率对比

, figureFileSmall=null, figureFileBig=null, tableContent=
方法2s3z5m_vs_6m6h_vs_8z
QMIX0.970.470.39
MA2E0.970.480.31
本文0.980.670.43
), ArticleFig(id=1329420201298342459, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, label=Table 4, caption=

Final Return Comparison on SMAC Map Scenarios

, figureFileSmall=null, figureFileBig=null, tableContent=
方法2s3z5m_vs_6m6h_vs_8z
QMIX19.8514.7716.59
MA2E19.8314.8915.94
本文19.8716.7716.90
), ArticleFig(id=1329420201357062716, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, label=表4, caption=

SMAC地图场景下的最终回报对比

, figureFileSmall=null, figureFileBig=null, tableContent=
方法2s3z5m_vs_6m6h_vs_8z
QMIX19.8514.7716.59
MA2E19.8314.8915.94
本文19.8716.7716.90
), ArticleFig(id=1329420201436754493, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, label=Table 5, caption=

Comparison of Reconstruction Accuracy

, figureFileSmall=null, figureFileBig=null, tableContent=
方法预训练前预训练后微调后
MA2E0.54270.47200.2777
随机通信(通信3个)0.45890.39290.2732
本文0.45820.35260.2645
), ArticleFig(id=1329420201499669054, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, label=表5, caption=

重建精度对比

, figureFileSmall=null, figureFileBig=null, tableContent=
方法预训练前预训练后微调后
MA2E0.54270.47200.2777
随机通信(通信3个)0.45890.39290.2732
本文0.45820.35260.2645
), ArticleFig(id=1329420201570972223, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=EN, label=Table 6, caption=

Cost Comparison of Different Methods on the Map of 5m_vs_6m

, figureFileSmall=null, figureFileBig=null, tableContent=
方法单步通信开销/b收敛时间步模型参数量
本文303680.767×106699342
随机通信(通信3个)350401.049×106683574
MA2E0(无通信)1.332×106680017
QMIX0(无通信)1.423×106141485
), ArticleFig(id=1329420201629692480, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, language=CN, label=表6, caption=

5m_vs_6m地图下不同方法的代价对比

, figureFileSmall=null, figureFileBig=null, tableContent=
方法单步通信开销/b收敛时间步模型参数量
本文303680.767×106699342
随机通信(通信3个)350401.049×106683574
MA2E0(无通信)1.332×106680017
QMIX0(无通信)1.423×106141485
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202550701, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202550701, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202550701, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202550701, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1782835200000, fullTextJson=null, articleText=null, reference=null)
收藏切换
面向多智能体强化学习中部分可观测性的主动通信掩码重建方法
贾月洋 , 刘文彬 , 王恩 *
计算机研究与发展 | 人工智能 2026,63(7): 1854-1868
收起
收藏切换
计算机研究与发展 |人工智能 2026 , 63 (7) : 1854 -1868
面向多智能体强化学习中部分可观测性的主动通信掩码重建方法
全屏
[Author(id=1329420196940460549, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=yyjia24@mails.jlu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329420197003375110, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, authorId=1329420196940460549, language=EN, stringName=Yueyang Jia, firstName=Yueyang, middleName=null, lastName=Jia, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"content":"

born in 2001. Master. His main research interest includes multi-agent reinforcement learning

"}, bioImg=null, bioContent=

born in 2001. Master. His main research interest includes multi-agent reinforcement learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329420197074678279, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, authorId=1329420196940460549, language=CN, stringName=贾月洋, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"img":"t4mjzUs6AKVLqv/+krJDAQ==","content":"

2001年生。硕士。主要研究方向为多智能体强化学习

"}, bioImg=t4mjzUs6AKVLqv/+krJDAQ==, bioContent=

2001年生。硕士。主要研究方向为多智能体强化学习

, aboutCorrespAuthor=null)}, companyList=null), Author(id=1329420197141787145, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329420197204701706, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, authorId=1329420197141787145, language=EN, stringName=Wenbin Liu, firstName=Wenbin, middleName=null, lastName=Liu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"content":"

born in 1990. PhD, associate professor. His main research interests include multi-agent reinforcement learning, mobile computing, and mobile crowdsensing

"}, bioImg=null, bioContent=

born in 1990. PhD, associate professor. His main research interests include multi-agent reinforcement learning, mobile computing, and mobile crowdsensing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329420197267616267, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, authorId=1329420197141787145, language=CN, stringName=刘文彬, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"img":"zS/+bTmFmRxWjCniL5tkOA==","content":"

1990年生。博士,副教授。主要研究方向为多智能体强化学习、移动计算、移动群智感知

"}, bioImg=zS/+bTmFmRxWjCniL5tkOA==, bioContent=

1990年生。博士,副教授。主要研究方向为多智能体强化学习、移动计算、移动群智感知

, aboutCorrespAuthor=null)}, companyList=null), Author(id=1329420197338919437, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=wangen@jlu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329420197406028302, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, authorId=1329420197338919437, language=EN, stringName=En Wang, firstName=En, middleName=null, lastName=Wang, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=null, bio={"content":"

born in 1987. PhD, professor. His main research interests include multi-agent reinforcement learning, mobile computing, and mobile crowdsensing

"}, bioImg=null, bioContent=

born in 1987. PhD, professor. His main research interests include multi-agent reinforcement learning, mobile computing, and mobile crowdsensing

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329420197477331471, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329420193664709085, authorId=1329420197338919437, language=CN, stringName=王恩, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=null, bio={"img":"4IuF5HrG+ryn331f101tlg==","content":"

1987年生。博士,教授。主要研究方向为多智能体强化学习、移动计算、移动群智感知

"}, bioImg=4IuF5HrG+ryn331f101tlg==, bioContent=

1987年生。博士,教授。主要研究方向为多智能体强化学习、移动计算、移动群智感知

, aboutCorrespAuthor=null)}, companyList=null)]
贾月洋 , 刘文彬, 王恩*
作者信息
  • 吉林大学软件学院 长春 130012
  • 符号计算与知识工程教育部重点实验室(吉林大学) 长春 130012
通讯作者:
王恩()
作者简介:

作者贡献声明

贾月洋负责提出方法思路,完成实验并撰写论文;刘文彬负责论文审定和修改;王恩提出指导意见并修改论文。

2001年生。硕士。主要研究方向为多智能体强化学习

1990年生。博士,副教授。主要研究方向为多智能体强化学习、移动计算、移动群智感知

1987年生。博士,教授。主要研究方向为多智能体强化学习、移动计算、移动群智感知

Active Communication Masked Reconstruction Method for Partial Observability in Multi-Agent Reinforcement Learning
Yueyang Jia , Wenbin Liu, En Wang*
Affiliations
  • College of Software, Jilin University, Changchun 130012
  • Key Laboratory of Symbol Computation and Knowledge Engineering (Jilin University), Ministry of Education, Changchun 130012
  • born in 2001. Master. His main research interest includes multi-agent reinforcement learning

    born in 1990. PhD, associate professor. His main research interests include multi-agent reinforcement learning, mobile computing, and mobile crowdsensing

    born in 1987. PhD, professor. His main research interests include multi-agent reinforcement learning, mobile computing, and mobile crowdsensing

出版时间: 2026-07-01 doi: 10.7544/issn1000-1239.202550701
文章导航
收藏切换

在多智能体强化学习(multi-agent reinforcement learning,MARL)中,部分可观测性问题严重制约了智能体间的协作。尽管集中训练和分散执行(centralized training and decentralized execution,CTDE)范式通过在训练阶段引入全局状态缓解了该问题,但在执行阶段,智能体仍只能依赖局部观测。为进一步解决该限制,目前有2种方案:一种通过引入通信机制实现智能体间的信息共享,但会面临通信开销大、信息冗余等问题;另一种则通过掩码重建推理全局信息,但受限于样本稀疏性,重建信息仅来自于单一智能体的局部观测,导致重建精度低,可能会干扰决策。基于此,提出一种主动通信掩码重建方法。在集中训练阶段,该方法根据全局观测轨迹对智能体进行动态分组,并为每个智能体从各组中选取最相关的通信对象,进而在分散执行阶段,智能体依据通信对象集合进行主动通信,从而实现全局信息的准确重建,最终辅助智能体决策。通过在星际争霸多智能体挑战赛(StarCraftⅡ Multi-Agent Challenge,SMAC)上的一系列实验,证明了该方法在部分可观测场景下对提升多智能体系统协作与整体策略性能的有效性。

多智能体强化学习  /  部分可观测性  /  动态分组  /  多智能体通信  /  掩码自编码器  /  SMAC

In multi-agent reinforcement learning (MARL), partial observability poses a significant challenge to effective collaboration among agents. Although the centralized training with decentralized execution (CTDE) paradigm alleviates this challenge by leveraging global state information during training, agents must still rely solely on local observations during execution. To address this limitation, existing research has primarily followed two directions. The first direction introduces communication mechanisms to enable information sharing among agents; however, these methods often suffer from high communication overhead and redundant information. The second direction leverages mask reconstruction techniques to infer the missing global information; however, due to sample sparsity, the inferred information depends only on local observations, resulting in low inference accuracy and potential interference with decision-making. To overcome these limitations, we propose an active communication masked reconstruction method. During centralized training, the method dynamically groups agents based on global observation trajectories and allows each agent to select the most relevant communication targets from these groups. During decentralized execution, agents proactively communicate according to their selected target sets, enabling accurate reconstruction of global information and ultimately supporting their decision-making. Extensive experiments conducted on the StarCraftⅡ Multi-Agent Challenge (SMAC) demonstrate the effectiveness of the proposed approach in improving coordination and overall performance in partially observable multi-agent scenarios.

multi-agent reinforcement learning  /  partial observability  /  dynamic grouping  /  multi-agent communication  /  masked autoencoder  /  StarCraftⅡ Multi-Agent Challenge (SMAC)
贾月洋, 刘文彬, 王恩. 面向多智能体强化学习中部分可观测性的主动通信掩码重建方法. 计算机研究与发展, 2026 , 63 (7) : 1854 -1868 . DOI: 10.7544/issn1000-1239.202550701
Yueyang Jia, Wenbin Liu, En Wang. Active Communication Masked Reconstruction Method for Partial Observability in Multi-Agent Reinforcement Learning[J]. Journal of Computer Research and Development, 2026 , 63 (7) : 1854 -1868 . DOI: 10.7544/issn1000-1239.202550701
多智能体强化学习(multi-agent reinforcement learning,MARL)通过学习多智能体与环境之间的交互,以优化个体或整体的策略表现[1-3]。近年来,该算法在自动驾驶[4]、订单调度[5-6]、机器人系统[7-8]、游戏AI系统[9]等多个领域得到了广泛应用。然而,由于环境的部分可观测性,每个智能体只能获取局部观测信息,难以感知全局状态,这在很大程度上制约了智能体之间的协作与整体策略性能。因此,解决部分观测性问题是多智能体强化学习中提升智能体间协作与整体策略性能的关键挑战[3,10]。
为了应对该挑战,最直接有效的方式是采用中心化思想对所有智能体进行统一训练与控制,即采用集中训练和集中执行(centralized training and centralized execution,CTCE)框架[11]。然而,随着智能体数量的增加,状态-动作联合空间的维度将呈指数级增长,进而引发维度灾难,导致系统面临计算成本高、扩展性低等问题,难以适应复杂的多智能体场景[2]。集中训练和分散执行(centralized training and decentralized execution,CTDE)框架的提出缓解了上述问题[12]。虽然该框架允许在训练阶段访问全局状态,但在执行阶段,智能体仍只能依赖局部观测。因此,部分可观测性所带来的挑战仍未彻底解决。
现有2种针对缓解分散执行阶段缺乏全局信息的策略:一种通过引入通信机制,允许智能体之间在执行阶段共享观测信息,通过消息传递的方式直接获取其他智能体的局部观测,从而模拟全局状态以获取全局视角[13]。该方法虽然有利于智能体间的协作,但往往面临带宽有限、通信冗余等问题。另一种则通过掩码重建,根据智能体局部观测推理全局信息[14]。此方法虽然不依赖于通信,但受样本稀疏性的影响,推理过程仅依赖单一智能体的局部观测,导致重建精度存在偏差,使得智能体决策容易陷入局部最优。尽管可以利用融合思想,在现有掩码重建框架中引入通信机制以缓解样本稀疏性问题,但由于环境是动态变化的,因此难以满足智能体对于全局信息重建的动态通信需求。
为此,本文提出了一种主动通信掩码重建方法。在保持低通信成本的前提下,在训练阶段利用基于谱聚类(spectral clustering)[15]的动态分组机制对智能体进行自适应的多样化分组,实现对智能体间关系的动态建模,并通过掩码交叉注意力(masked cross-attention)机制[16]从各组中挑选最相关的通信对象,从而为每个智能体构建通信对象集合;在执行阶段,每个智能体依据通信对象集合与其他智能体建立通信,从而为智能体引入少量且高度相关的多样化观测信息。随后,结合掩码自编码器(masked autoencoder,MAE)[17]的推理能力,利用整合后的局部观测信息来重建全局信息,从而为智能体构建更准确的全局视角,进一步提升多智能体系统的协作与整体决策性能。本文的主要贡献有4个方面:
1) 提出了一种基于谱聚类的动态分组机制。在集中训练阶段,能够根据全局观测轨迹自适应调整分组,并为智能体确定通信对象,从而为后续智能体在执行阶段进行主动通信提供有效指导。
2) 提出了一种融合主动通信与掩码重建的重建思想。在保持低通信成本的前提下,突破传统仅依赖局部观测的限制,通过引入少量且相关的多样化外部信息辅助智能体构建全局信息,有效缓解传统掩码重建中因样本稀疏导致的重建误差问题,以提升多智能体系统的协作与整体决策性能。
3) 设计了一种主动通信掩码重建框架。该框架利用基于动态分组的主动通信模块,为智能体级掩码自编码器提供多样化样本,以实现全局信息的精确重建,进而通过自注意力机制从重建的全局信息中提取全局特征,从而帮助智能体有效决策。
4) 在星际争霸多智能体挑战赛(StarCraft Ⅱ Multi-Agent Challenge,SMAC)[18]上进行了一系列实验评估。综合结果显示,所提方法在多个场景下取得了良好的性能提升,验证了该方法在部分可观测场景下的有效性。
在多智能体强化学习中,如何在部分可观测环境下实现高效的协同决策一直是研究的主要问题。现有研究主要从缓解部分可观测性出发,围绕多智能体间通信机制与掩码重建2种方法进行展开。同时,分组机制作为一种辅助手段,可为实现高效协作提供支持。因此,本文将从以下4个方面进行详细介绍。
在多智能体系统中,环境通常表现为部分可观测的,每个智能体只能获取自身观测范围内的有限信息。因此,在早期的分散训练和分散执行(decentralized training and decentralized execution,DTDE)框架中,每个智能体通过将其他智能体视为环境的一部分进行独立学习[19]。但由于其他智能体的策略是动态变化的,因此系统是不稳定的[20]。为了缓解非平稳问题,在这之后的许多研究开始采用中心化的思想,即集中训练和集中执行(centralized training and centralized execution,CTCE)框架对所有智能体进行统一训练和控制[11]。然而,智能体数量的增加会引发维度灾难,导致系统面临计算成本高、扩展性低等问题[2]。为了提高系统的适应性,CTDE框架被广泛应用,并已成为如今最流行的框架[12]。该框架下有2种主要的MARL算法:基于值函数的方法和基于策略的方法。基于值函数的方法以值函数分解为导向,将联合值函数因式分解为多个单独的局部值函数,经典方法有VDN[21],QMIX[22]等;基于策略的方法以学习参数化策略为导向,其基于Actor-Critic结构,经典方法有A3C[23],DDPG[24]等。尽管基于策略的方法能够处理连续动作空间,但与基于值函数的方法相比,它通常表现出更高的方差和更低的样本效率,并且可能收敛于局部最优值。因此,为了方便设计,本文基于值函数方法进行框架设计。
在多智能体强化学习研究过程中,分组机制作为提升智能体间信息交互效率以及捕获和建模智能体间关系的重要手段,近年来受到了广泛的关注。该类方法通过将智能体划分为若干子集,在保证组间多样性的同时,使组内智能体之间能够高效的信息共享。早期研究如Q-decomposition[25],MACKRL[26]等方法通常依赖于先验知识实现智能体划分。随后,一部分研究如M3RL[27],ALMA[28]等方法尝试通过任务分配实现隐式动态分组。此外,以ROMA[29]和RODE[30]为代表的角色分解方法通过学习智能体在不同情境下的角色策略,实现上下文相关的功能划分,这类方法虽然能提高多智能体间协作的灵活性和适应性,但需要手动设定角色数量或动作空间划分,难以充分适应复杂多变的动态环境。近年来,以数据为驱动的方法如GoMARL[31]利用智能体之间的相似性度量实现自动分组,尽管该方法能够在动态环境下进行自适应调整,但通常缺乏可解释性。因此,为了兼顾自适应性与可解释性,受HYGMA[32]工作的启发,本文提出一种基于谱聚类的动态分组机制。通过使用具有严格数学基础的谱聚类方法,对智能体观测轨迹进行谱分析,可以发现智能体间复杂的潜在关系,从而实现有效的划分。同时,通过引入CH(Calinski-Harabasz)指数[33],可以实现在动态环境下评估组内紧密度与组间多样性,从而实现对聚类数的自适应调整,能够为后续主动通信提供多样化且最优的分组参考。
为了缓解智能体在分散执行阶段的信息缺失问题,现有研究尝试利用通信机制学习有效的通信策略来实现多智能体之间信息的共享,从而辅助独立决策。早期方法如RIAL和DIAL[34]引入了可学习的通信信道,为通信建模奠定了基础。CommNet[35]则提出了基于消息聚合的共享机制,通过广播实现状态信息的融合。随后,ATOC[36],IC3Net[37]等方法进一步引入门控机制和注意力结构,在减轻通信冗余的同时增强了通信选择的灵活性。近年来,部分工作开始关注有向通信结构与消息选择内容的建模,如G2ANet[38]利用图神经网络动态指定通信目标,TarMAC[39],MASIA[40]等则试图提升消息的表达性与接收端的信息利用效率。虽然这些方法能够有效缓解部分可观测性,但由于训练过程中高度依赖稀疏的强化学习信号,导致通信内容与策略联合优化存在训练不稳定的问题,并且大规模通信会增加系统的复杂性与计算负担。与上述方法不同,本文采用了一种相对轻量的主动通信机制,该机制依据训练阶段所获得的分组结果,通过掩码交叉注意力机制为每个智能体从各组中选取最相关的其他智能体建立通信,从而在执行阶段为智能体提供少量且高相关性的多样化观测信息。这样不仅避免了过多的通信开销,同时能为后续的掩码重建提供多样化的样本,从而有效地缓解样本稀疏性问题。
受自然语言处理[41]、计算机视觉[17]等领域中自监督学习掩码重建思想的启发,近年来,许多研究将掩码重建思想引入到了强化学习任务中,使得智能体在分散执行阶段能够通过推理全局信息获取全局视野。在早期的单智能体强化学习中,MaskDP[42],MTM[43]等方法通过对输入轨迹进行随机遮蔽并重建原始信息,有效提升了策略的泛化能力。而在多智能体场景下,部分研究也尝试将掩码重建用于缓解部分可观测性问题。例如,MA2CL[44]利用对比损失联合训练策略网络与重建模块,以增强智能体间协作;MA2E [45]则采用独立训练的掩码自编码器,将局部观测重建为全局观测信息以辅助策略学习。尽管上述方法在全局信息重建方面取得了一定成效,但由于输入样本的稀疏性,即每个智能体仅依赖自身的局部观测进行推理,导致难以准确重建复杂环境中的全局信息。为此,本文在MA2E工作的基础上,提出了结合主动通信机制的掩码重建方法,通过为模型引入少量且高度相关的多样化的外部观测信息,引导模型更准确地重建全局信息,从而更有效地辅助决策。
在本节中,我们将系统性地介绍所提出的主动通信掩码重建方法。首先,我们对多智能体强化学习问题进行了形式化定义。随后,我们详细介绍了该方法的整体框架,重点阐述了框架的整体结构组成和处理流程。此外,我们对框架中关键的基于谱聚类的动态分组机制、主动通信模块和智能体级掩码自编码器进行了深入解析。最后,我们讲解了整个训练过程的具体步骤,并介绍了所采用的损失函数与优化目标。
我们将多智能体强化学习问题建模为一个分布式部分可观测的马尔可夫决策过程(decentralized partially observable Markov decision process,Dec-POMDP)[46],具体来讲,Dec-POMDP可以被定义为一个元组$ \mathcal{G}= ( \mathcal{N},\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\mathcal{Z},\mathcal{O},\gamma ) $,其中$ \mathcal{N} $表示$ n $个智能体的集合,$ \mathcal{S} $表示全局状态信息的集合,$ \mathcal{A} $表示智能体动作的集合,$ \mathcal{P}\text{:}\mathcal{S}\times \mathcal{A}\rightarrow \Delta (\mathcal{S}) $是状态转移函数,其中$ \Delta (\mathcal{S}) $是$ \mathcal{S} $上概率分布的集合,$ \mathcal{R}\text{:}\mathcal{S}\times \mathcal{A}\rightarrow \mathbb{R} $表示奖励函数,$ \mathcal{Z} $表示智能体的观测集合,$ \mathcal{O}\text{:}\mathcal{S}\times \mathcal{N}\rightarrow \mathcal{Z} $表示观测函数,$ \gamma \in \text{[}0,1\text{)} $表示折扣因子。在每个时间步$ t $,每个智能体$ i\in \mathcal{N} $从观测函数$ \mathcal{O}({\boldsymbol{s}}_{t},i) $接收一个观测$ \boldsymbol{o}_{t}^{i}\in \mathcal{Z} $,其中$ {\boldsymbol{s}}_{t}\in \mathcal{S} $ 表示在时间步$ t $下的全局状态,每个智能体有自己的策略$ {\pi }^{i}(\boldsymbol{a}_{t}^{i}|\boldsymbol{\tau }_{t}^{i}) $,其中$ \boldsymbol{a}_{t}^{i} $是每个智能体的动作,$ \boldsymbol{\tau }_{t}^{i}=(\boldsymbol{o}_{1}^{i},\boldsymbol{a}_{1}^{i},… ,\boldsymbol{o}_{t-1}^{i},\boldsymbol{a}_{t-1}^{i},\boldsymbol{o}_{t}^{i}) $是每个智能体的观测-动作的历史轨迹,环境会执行联合动作$ \boldsymbol{a}_{t} $以进入新的状态$ \boldsymbol{s}_{t+1}\sim \mathcal{P}({\boldsymbol{s}}_{t+1}|{\boldsymbol{s}}_{t},{\boldsymbol{a}}_{t}) $,并且多智能体系统会收到一个整体的全局奖励$ {r}_{t}=\mathcal{R}({\boldsymbol{s}}_{t},{\boldsymbol{a}}_{t}) $,最终的目标是找到一个联合策略$ \pi = ( {\pi }^{1},{\pi }^{2},… ,{\pi }^{n} ) $ 以最大化期望折扣总奖励$ {\mathbb{E}}_{\pi }\left[\displaystyle\sum_{t=0}^{\infty }{\gamma }^{t}r\right] $。在本文中,我们使用基于值函数的方法去更新策略,在分散执行阶段,每个智能体都会学习一个个体Q网络$ {Q}^{i}({\boldsymbol{\tau }}^{i},{\boldsymbol{a}}^{i}) $,之后在集中训练阶段,会通过一个混合网络$ {\mathcal{M}}_{\theta } $将每个智能体的个体Q值聚合为一个联合Q值,即$ {Q}_{\text{tot}}(\boldsymbol{\tau },\boldsymbol{a})={\mathcal{M}}_{\theta }({Q}^{1}({\boldsymbol{\tau }}^{1},{\boldsymbol{a}}^{1}),{Q}^{2}({\boldsymbol{\tau }}^{2}, {\boldsymbol{a}}^{2}),… ,{Q}^{n}({\boldsymbol{\tau }}^{n},{\boldsymbol{a}}^{n})\text{;}\boldsymbol{s}) $,混合网络的参数$ \theta $通过最小化时序差分(temporal difference,TD)损失函数进行学习:
$ \mathcal{L}(\theta )=\displaystyle\sum_{i=1}^{b}\left[{({y_{i}^{\rm{tot}}}-{{Q}_{\rm{tot}}}({{\boldsymbol{\tau }}_{{t}}},{{\boldsymbol{a}}_{{t}}},{{\boldsymbol{s}}_{t}}\text{;}\theta ))}^{2}\right] \;, $
其中$ {y}^{\text{tot}}=r+\gamma \;\underset{{\boldsymbol{a}}_{t+1}}{\text{max}}\;{Q}_{\rm{tot}}({\boldsymbol{\tau }}_{{t}+1} {,}{\boldsymbol{a}}_{t+1},{\boldsymbol{s}}_{t+1}\text{;}{\theta }^{-}) $,$ {\theta }^{-} $是目标网络的参数,$ b $是重放缓冲区$ \mathcal{D} $中采样的批量大小。
本文方法在QMIX框架的基础上进行构建,在继承CTDE范式的同时,保证了值函数分解的单调性IGM(individual-global-max)条件[21]。方法的整体框架如图1所示,其中框架的整体结构包含n个智能体的个体Q网络和1个混合网络。在混合网络中,超网络会根据全局状态为混合网络生成权重和偏置,用于混合个体Q值,从而得到联合Q值。对于个体Q网络,该网络依据由主动通信模块所生成的通信对象集合$ C_{t}^{i} $进行通信,从而为智能体级掩码自编码器提供指导,帮助智能体重建全局信息。
然而,由于重建的全局信息中并不全是有效信息,因此我们通过自注意(self-attention)机制[16]从重建得到的全局信息中筛选重要信息,以获取全局特征,从而帮助智能体学习一个跨智能体的全局协作模式。在这之前,为避免重建误差对注意力权重分配的干扰,还需要将当前智能体重建表示替换为真实值,以确保特征查询过程的准确性。首先,通过自注意力机制计算智能体之间的注意力权重:
$ {\alpha }_{i,j}=\frac{\text{exp}(\tilde{\boldsymbol{\tau }}_{i}^{\mathrm{T}}\boldsymbol{W}_{\rm k}^{\mathrm{T}}{\boldsymbol{W}}_{\rm q}{\tilde{\boldsymbol{\tau }}}_{j})}{\displaystyle\sum_{j=1}^{n}\text{exp}(\tilde{\boldsymbol{\tau }}_{i}^{\mathrm{T}}\boldsymbol{W}_{\rm k}^{\mathrm{T}}{\boldsymbol{W}}_{\rm q}{\tilde{\boldsymbol{\tau }}}_{j})} \;, $
其中$ {\boldsymbol{W}}_{\rm q} $和$ {\boldsymbol{W}}_{\rm k} $分别是查询和键的可学习权重矩阵。然后再计算注意力权重和值的加权和得到加权值:
$ Attn(\tilde{\boldsymbol{\tau }}_{t}^{1: n})=\displaystyle\sum_{j=1}^{n}{\alpha }_{i,j}{\boldsymbol{W}}_{\rm v}\tilde{\boldsymbol{\tau }}_{t}^{j} \;, $
其中$ {\boldsymbol{W}}_{\rm v} $是值的可学习权重矩阵。最终,将该权重加权值与智能体自身观测相结合,用于最终的决策。
在集中训练阶段,我们将$ n $个智能体在$ T $个时间步的观测轨迹$ \{\boldsymbol{o}_{t-T+1: t}^{i}\}_{i=1}^{n} $作为输入,以此构成轨迹矩阵$ {\boldsymbol{O}}_{t}\in {\mathbb{R}}^{b\times T\times d} $,其中$ b $表示批量大小,$ T $表示轨迹的时间步长,$ d $表示轨迹维度。为了在保持组内紧密度的同时保持组间多样性,会通过最小化规范化切割(normalized cut,$ cu{t}_{\text{N}} $)[47]来实现智能体分组,从而得到分组$ {G}_{t}=\{g_{t}^{1},g_{t}^{2},\ldots ,g_{t}^{k}\} $:
$ cu{t}_{\text{N}}(g_{t}^{1},g_{t}^{2},\ldots ,g_{t}^{k})=\displaystyle\sum_{i=1}^{k}\frac{cut(g_{t}^{i},\bar{g}_{t}^{i})}{vol(g_{t}^{i})} \;, $
其中$ k $表示聚类数目,我们预定义初始聚类数目为$ k=\left\lceil n/2\right\rceil $,而$ n $表示智能体总数;$ g_{t}^{i} $表示在第$ t $个时间步下的第$ i $个分组;$ cut(g_{t}^{i},\bar{g}_{t}^{i}) $表示分组$ g_{t}^{i} $到其他群体的边的总权重,间接反映组间多样性;$ vol(g_{t}^{i}) $则表示群体$ g_{t}^{i} $内所有节点的度数之和,间接反映组内紧密度。但在离散划分空间里求最优解是NP难问题,因此需要采用谱松弛(spectral relaxation)[48]将问题转化为连续优化问题:
$ \mathop{\text{arg min}}\limits_{\boldsymbol{H}\in {\mathbb{R}}^{n\times k}}\;\text{tr}({\boldsymbol{H}}^{\text{T}}\boldsymbol{L}\boldsymbol{H}),\;\text{s.t.}\;{\boldsymbol{H}}^{\text{T}}\boldsymbol{H}=\boldsymbol{I} \;, $
其中$ \boldsymbol{H}\in {\mathbb{R}}^{n\times k} $表示特征矩阵,$ \boldsymbol{L}=\boldsymbol{D}-\boldsymbol{W} $表示归一化后的拉普拉斯矩阵,$ \boldsymbol{D} $是度矩阵,而$ \boldsymbol{W} $是邻接矩阵,用于表示智能体观测轨迹之间的相似度。具体而言,W是根据欧几里得距离计算K近邻所得到的,若智能体$ j $是智能体$ i $的K近邻,则$ {W}_{ij} \gt 0 $;否则$ {W}_{ij}=0 $。最终,谱聚类得到的最优分组$ G_{t}^{*} $满足:
$ cu{t}_{\text{N}}({G}_{t})\leq O(\text{log}k)\cdot cu{t}_{\text{N}}(G_{t}^{*}) \;。 $
为了能够适应环境的动态变化,找到最终的最佳分组,我们引入CH(Calinski-Harabasz)指数[33]用于评估聚类后的组内紧密度与组间多样性,从而不断调整聚类数目$ k $以寻找最佳智能体分组:
$ CH_{t}^{k}=\frac{\displaystyle\sum_{i=1}^{k} | g_{t}^{i} | {({\mu _{t}^{i}}-{{\mu }_{t}})}^{\text{2}}}{\displaystyle\sum_{i=1}^{k}\displaystyle\sum_{x_{t}^{j}\in g_{t}^{i}}{({x_{t}^{j}}-{\mu _{t}^{i}})}^{2}}\cdot \frac{n-k}{k-1} \;, $
其中$ x_{t}^{j} $是分组$ g_{t}^{i} $中的第$ j $个样本,$ \mu _{t}^{i} $是分组$ g_{t}^{i} $的质心,表示组内样本均值,$ x_{t}^{j}-\mu _{t}^{i} $则表示样本与质心之间的偏差,而$ {\mu }_{t} $表示在第$ t $个时间步下$ n $个智能体的样本均值。至此,根据计算得到的CH值的大小可以评估聚类数目$ k $的优劣,从而在集中训练阶段对聚类数目$ k $以及分组$ {G}_{t} $进行更新:
$ \begin{aligned} &{k}_{\text{best}}= \mathop{\text{arg max}}\limits_{\begin{subarray}{c} k'\in \{k-1,k,k+1\}\\{k}_{\text{min}}\leq k' \lt {k}_{\text{max}}\end{subarray}}CH_{t}^{{k'}}\;,\\& \text{s.t.}\;CH_{t}^{{k'}}-CH_{t}^{{k}_{\text{current}}} \gt \delta 且CH_{t}^{{k'}} \gt CH_{t}^{\text{best}}\;,\end{aligned} $
其中$ k $有2种选择:$ k-1 $和$ k+1 $。分别计算2种选择下的分组CH值,以此寻找最佳的CH值,即$ CH_{t}^{\text{best}} $,从而得到对应的最佳分组$ G_{t}^{*} $。
通过动态分组机制获得分组结果$ {G}_{t} $(最终固定为最佳分组$ G_{t}^{*} $)以后,为后续智能体级掩码自编码器提供多样化的样本,在训练阶段,我们还将采用掩码交叉注意力(masked cross-attention)机制[16]为每个智能体从各组中选取最相关的其他智能体对象,从而确立通信。对于掩码交叉注意力机制,令当前智能体观测作为查询,每个分组中的智能体观测作为键和值,从而计算注意力权重加权值:
$ CrossAttn(\boldsymbol{o}_{t}^{i},g_{t}^{j})=softmax\left(\frac{\boldsymbol{Q}{\boldsymbol{K}}^{\text{T}}}{\sqrt{{d}_{k}}}+\boldsymbol{M}\right)\cdot \boldsymbol{V} \;, $
其中$ \boldsymbol{o}_{t}^{i} $表示第$ i $个智能体的局部观测,$ g_{t}^{j} $表示第$ j $组的智能体观测,同时我们加入掩码矩阵$ \boldsymbol{M} $用以防止智能体在确定通信对象时选择自身。在计算得到权重加权值以后,我们利用Top-1选择器选取每组中权重值最高的智能体作为通信对象,从而为每个智能体构建通信对象集合$ C_{t}^{i}\subseteq \{1,2,… ,n\}\backslash \{i\},\left| C_{t}^{i}\right| =c $,进而在执行阶段为智能体主动通信提供有效参考,缓解智能体级掩码自编码器所面临的样本稀疏性问题。
为实现对其他智能体信息的有效重建,本文引入了MA2E[45]工作中的智能体级掩码自编码器结构。其中,掩码自编码器的输入为每个智能体在$ T $个时间步内的观测-动作历史轨迹,其包含的编码器和解码器由多头注意力机制(multi-head attention,MHA)和前馈神经网络(feed-forward network,FFN)组成,从而能够有效捕捉轨迹中的智能体之间的依赖关系与相关性。需要注意的是,该掩码自编码器的解码器中并未使用传统Transformer[16]的掩码多头注意力机制(masked MHA),而是在输入阶段额外引入了独立的智能体级掩码层来实现遮蔽处理。该掩码层能够在输入轨迹中对智能体的整个信息进行选择性的遮蔽,从而能够更好地模拟部分可观测环境。考虑到集中训练阶段与分散执行阶段在可访问信息上的差异,本文针对这2个阶段分别设计了不同的掩码策略,以适应训练与执行的需求。
在集中训练阶段,系统可以访问所有智能体的观测-动作历史轨迹$ \boldsymbol{\tau }_{t-T+1: t}^{1: n}=\left(\boldsymbol{\tau }_{t-T+1}^{1: n},\boldsymbol{\tau }_{t-T}^{1: n},… ,\boldsymbol{\tau }_{t}^{1: n}\right) $,其中$ \boldsymbol{\tau }_{t}^{i}=(\boldsymbol{o}_{t}^{i},\boldsymbol{a}_{t-1}^{i}) $表示单个智能体的观测-动作历史轨迹。我们采用随机掩码策略[43],即随机选择$ m $个智能体,并将其全部的历史轨迹进行掩蔽,迫使模型从未遮蔽的轨迹中恢复完整信息$ \tilde{\boldsymbol{\tau }}_{t-T+1: t}^{1: n} $,以便在分散执行阶段为智能体推断出更准确的信息。
在分散执行阶段,智能体仅能获取自身的局部观测信息,无法直接访问其他智能体的观测轨迹。为了缓解样本稀疏性,我们通过基于动态分组的主动通信模块所生成的通信对象集合$ C_{t}^{i} $进行通信,从而为智能体级掩码自编码器提供多样化的样本,帮助智能体级掩码自编码器更好地推断全局信息,智能体级掩码自编码器的具体设计如图2所示。在构建输入轨迹时,除了包含当前智能体的历史轨迹以外,还应该包含通信对象的历史轨迹$ \tau _{t-T+1: t}^{C_{t}^{i}}=\{\boldsymbol{\tau }_{t-T+1: t}^{j}|j\in C_{t}^{i}\} $,其中,通信对象的历史轨迹会在每次通信后进行更新,以确保当前时间步为最新信息。最终,为了在掩码层能够模拟智能体分散执行时的部分可观测性,仅保留当前智能体及其通信智能体的历史轨迹,其余$ z $个未观测智能体的历史轨迹信息均被掩蔽。
同时,为了能够区分多个智能体的历史轨迹,该掩码自编码器在构建位置编码时,同时引入了时间和智能体索引以更好地捕捉多智能体的时序交互特征。
本文方法中的智能体级掩码自编码器与智能体的训练是相互独立的,其中掩码自编码器模型的训练遵循预训练和微调[17]2个阶段。在预训练阶段,我们的目的是为每个智能体预先训练出一个可以直接使用的重建模型,因此在智能体策略训练开始之前,会先与环境交互收集初始样本,以进行随机掩码,从而确保自编码器能够正确推断全局信息。该模型的参数通过最小化真实历史轨迹与重建轨迹之间的均方误差(mean squared error,MSE)来进行学习:
$ {\mathcal{L}}_{\text{rec}}=\frac{1}{nT}\displaystyle\sum_{t=1}^{T}\displaystyle\sum_{i=1}^{n}{({\boldsymbol{\tau }_{t}^{i}}-{\tilde{\boldsymbol{\tau }}_{t}^{i}})}^{2} \;。 $
预训练结束以后,在智能体策略训练的过程中,模型会周期性地进行微调以实现进一步的优化,微调操作具体参考2.5节中的集中训练阶段。
本节从时间复杂度和空间复杂度2个角度对方法进行了定量分析。为了方便表达,设$ n $表示智能体的数量,$ T $表示每个智能体的历史轨迹长度,$ k \lt n-1 $表示聚类数目,$ d $表示自编码器中嵌入向量维度,$ f $表示每个时间步的观测-动作向量维度,$ h $表示自编码器中多头注意力头数,$ b $表示批量大小。
时间复杂度主要包含基于谱聚类的动态分组机制、主动通信、智能体级掩码自编码器以及自注意力机制4部分。在训练阶段,由于引入了基于谱聚类的动态分组机制,需要对智能体轨迹构建K近邻相似图并进行谱松弛,其复杂度约为$ O({n}^{2}Td+{n}^{2}k) $,之后,每个智能体需通过主动通信模块构建通信对象集合,其复杂度约为$ O(n{d}^{2}) $;同时,智能体级掩码自编码器在集中可见的条件下可处理的序列长度为$ nT $,则单层复杂度为$ O(hnT{d}^{2}) $,又因为自编码器中包含多层多头注意力机制,因此,当自编码器中的编码器和解码器总共有$ L $层时,其复杂度为$ O(LhnT{d}^{2}) $。而在执行阶段,每个智能体会依据通信对象集合与其他智能体建立通信,则最终输入到智能体级掩码自编码器中的序列长度为$ (r+1)\cdot T $,其中$ r\leq k $,则掩码自编码在执行阶段的复杂度为$ O(nLh(r+1)T{d}^{2}) $;此外,重建后的全局信息需经过自注意力机制筛选和混合Q网络更新,复杂度分别为$ O(n{d}^{2}) $和$ O(nd) $。综上所述,相较于传统全通信方法,本文方法在保证通信有效性的同时,将全局信息的重建规模从与$ n $成正比下降至与$ r $成正比,因此更适用于大多数中等规模的多智能体任务。
对于空间复杂度,本文方法的主要开销来自于智能体级掩码自编码器的模型参数存储$ O(Lh{d}^{2}) $、历史轨迹缓存$ O(bnTf) $、训练阶段谱聚类中矩阵与向量缓存$ O(nK+nk) $(其中$ K $为近邻数),以及执行阶段通信样本缓存$ O(n(r+1)Tf) $。总体而言,相比于全通信方法,本文方法在保证全局信息重建精度的同时,有效降低了执行阶段的通信所需的存储开销。
为了验证本文方法在部分可观测场景下的有效性,我们选择在SMAC平台上进行实证评估。SMAC是基于暴雪公司旗下的经典即时战略游戏《星际争霸Ⅱ》开发的多智能体强化学习测试平台,它提供了多个专门的多智能体协作场景,能够有效评估方法在部分可观测性下的性能表现[18]。
我们分别在Easy,Hard,Super Hard这3个难度等级的场景地图上进行了测试,选择的地图及地图设置如表1所示。其中Easy难度选取了具有异构特性的地图2s3z,该地图中的友军单位和敌军单位包含2名狂战士和3名潜行者2种异构单位,体现了不同单位属性的组合,用于验证方法在智能体间存在异构差异时的协同与微观操作能力;Hard难度地图选取了具有代表性的地图5m_vs_6m,该地图包含5名海军陆战队友军单位和6名海军陆战队敌军单位,具有一定的不对称性与协同难度,可用于评估方法在同质单位条件下的协作效果;而Super Hard难度选取的地图是6h_vs_8z,该地图由6名水蛛友军远程单位和8名狂战士敌军近战单位组成,具备敌我异构与不对称2个特性。该场景的策略方向较为凸显,可用于评估方法在高难度条件下的策略调整能力,并以此侧面评估整体性能。
我们将本文方法与经典的CTDE基线算法QMIX和仅利用单个智能体的局部观测进行掩码重建的多智能体重建方法MA2E进行比较。每组实验运行3次,时间步数设置为2E6。为了全面评估各方法的性能表现,我们采用平均胜率和平均回报作为评估指标。平均胜率作为多智能体强化学习中被广泛使用,且具备较强可比性和通用性的指标,能够充分反映方法的整体表现,在SMAC平台中,若在限定时间步内,友方单位能成功击败所有敌方单位,则判定为胜利,否则判定为失败,最终统计获胜回合数占总实验回合数的比值作为胜率。同时,我们还在胜率曲线上通过设置下25%和上75%的分位数区域用以表示每种方法的鲁棒性。而平均回报则是智能体在若干评估回合中获得的累计奖励的平均值,相比于平均胜率,该指标能够更好地衡量智能体在整个过程中的综合表现,展现策略的优劣。
需要注意的是,由于许多MARL方法都能从调整的超参数中表现出优异的性能[49],因此,为了确保实验的可对比性和严谨性,我们采用了对齐超参数的实验策略来确保所有方法使用相同的超参数进行对比,超参数设置如表2所示。其中,考虑到6h_vs_8z的地图特性,为了能更加公平地评估各方法在该地图上的性能表现,我们对部分超参数进行了相应的调整,具体而言,我们将TD($ \lambda $)设置为0.3,同时将退火时间设置为500000,以确保各方法能够在该地图上发挥良好。
平均胜率对比结果如图3所示。本文方法在整个训练过程中表现出较强的性能优势,在3种不同难度等级场景下的最终平均胜率均高于MA2E和QMIX这2种方法。其中,在5m_vs_6m地图中,本文方法在结果上的提升最为明显,最终平均胜率能够稳定在约67%,相较于MA2E方法性能直接提升了19%;而在6h_vs_8z地图中,本文方法依旧能保持稳定的性能,在整个训练过程中,其平均胜率整体上高于性能较优的QMIX方法。对于MA2E方法,受样本稀疏性的影响,智能体仅能依赖自身局部观测重建全局信息,使得重建结果准确性偏低,这种对全局信息认知的偏差导致其在各个场景中的平均胜率均低于本文方法。然而,本文方法在此基础上引入了主动通信机制,为每个智能体的掩码自编码器提供了额外的多样化信息,从而缓解了样本稀疏问题,帮助重建更准确的全局信息,进而帮助智能体实现更有效的决策,最终提升作战胜率。对于鲁棒性,从胜率曲线的下25%及上75%的分位数可观察到本文方法在保证稳定性的前提下能够实现更高的胜率。
相比之下,传统的QMIX方法在执行阶段仅允许智能体依赖自身局部观测进行决策,因而缺乏对全局信息的重建能力,使得QMIX容易陷入局部最优,在5m_vs_6m上表现相对较弱。虽然在6h_vs_8z地图中因策略凸显使其性能表现较好,但由于缺乏对全局的感知能力,使得QMIX不能更有效地学习,导致综合性能弱于本文方法。
为了更加直观地展示各方法在不同场景下的性能差异,我们统计并比较了各方法在后1E6时间步内的平均胜率作为最终胜率,结果如表3所示。从结果中可以看出,本文方法的最终胜率均优于MA2E和QMIX这2种方法,说明了本文方法能够帮助智能体学到更有效的策略,从而提高作战胜率,进一步验证了本文方法在性能提升方面的有效性。
为了更好地评估智能体在整个过程中的综合表现,我们测试了各方法在不同难度等级场景下的平均回报表现,对比结果如图4所示。可以看出,本文方法在整个过程中始终保持较高的平均回报,在3种场景中综合优于MA2E和QMIX这2种方法。
具体而言,在Easy场景中,3种方法之间的差距相对较小,在2s3z地图中,由于QMIX结构相对简单,因此在前期平均回报的提升相对较快,而本文方法由于在前期需要广泛探索以寻找最佳分组,因此提升相对较慢,但最终表现与2种方法基本持平,并没有因此造成损失。
而在难度等级为Hard的5m_vs_6m地图中,由于QMIX方法仅依赖智能体自身的局部观测信息,缺乏对全局环境状态的充分把握,导致其策略调整能力受限,难以有效应对复杂的环境变化,从而影响最终的奖励回报。相较之下,本文方法与MA2E表现更为稳定,主要原因在于二者均在智能体执行阶段对全局信息进行了重建,使智能体能够更全面地感知全局,进而根据当前局势动态调整和优化中间策略,提升决策质量,获得更高的奖励回报。同时,本文方法引入的主动通信模块为全局信息重建提供了多样化的样本,使得智能体能够感知更为准确的全局信息,从而做出更有效的决策,使奖励回报得到了进一步提升。
对于6h_vs_8z地图,由于该地图本身的策略特征较为凸显,QMIX虽然仅依赖局部观测,但仍能够较快地捕捉关键战术,从而在训练前期展现出较优的性能。然而,受限于缺乏全局感知能力,QMIX在复杂局势下的策略调整效率不足,难以持续优化,因而在训练中后期逐渐被本文方法反超。相比之下,MA2E虽然尝试通过掩码重建全局信息,但受样本稀疏性的影响,重建结果往往存在误差,从而干扰决策调整,因此整体表现未能优于QMIX。由此可见,即便在具有较强策略信号的前提下,提升全局信息重建的精度仍然是优化策略并进一步增强整体性能的关键。
为了直观地反映各方法在不同场景下的策略调整能力,我们统计并比较了各方法在后1E6时间步内的平均回报作为最终回报,结果如表4所示。结果表明,本文方法在多个场景下均取得了更高的最终回报。这主要得益于本文方法在执行阶段对全局信息的有效重建与利用,从而使智能体能够在复杂环境下实现了更有效的决策。
为深入探究信息完整性对智能体策略协作性能的影响,我们在对比实验基础上进一步扩展了实验。按照全局观测信息的不同完整程度,分别选取了具有代表性的4种情况:局部观测(QMIX)、单独局部观测信息推断全局信息(MA2E)、随机信息推断全局信息(用随机通信代替主动通信,且随机通信的智能体数量设置为3)和高相关的多样化信息推断全局信息(本文方法)。实验选择在地图5m_vs_6m上进行,这是因为该场景为同构单位且具有一定不对称性,能有效放大信息完整性对协作性能的影响;同时其难度适中,不会像地图2s3z那样差异不显著,也避免了地图6h_vs_8z中因策略特征过于凸显而掩盖方法差异的情况,因此更适合用于验证不同信息完整程度下性能的变化趋势。我们依旧采用平均胜率作为评估指标,用以对比信息完整性对智能体策略协作性能的影响,最终的实验结果如图5所示。结果表明,提高信息完整程度对于提升平均胜率具有积极作用。
为了更加清楚地反映信息完整性对智能体策略协作性能的影响,我们取后1E6步的平均胜率,以反映不同信息完整性下的性能表现,最终的性能表现柱状图如图6所示。结果表明,随着信息完整程度的提升,平均胜率呈现上升趋势,验证了增强信息完整性对提升智能体策略协作性能的重要性。
局部观测(QMIX)与本文方法的对比结果如图7所示,实验选择在地图5m_vs_6m上进行。QMIX作为传统方法,仅使用智能体的局部观测信息进行独立决策,由于缺乏全局视野,导致智能体无法感知其他智能体的行为,使得不能对于智能体之间的协作做出针对性的策略调整。因此,验证了部分可观测场景下局部观测信息在复杂协作任务中的固有局限性。
单独局部观测信息推断全局信息(MA2E)与本文方法的对比结果如图8所示,实验选择在地图5m_vs_6m上进行。MA2E作为多智能体掩码重建的基础方法,其利用掩码重建方法,根据智能体的局部观测信息来推断全局观测信息,从而辅助智能体的决策过程。相较于QMIX方法,虽然在一定程度缓解了缺乏全局视野的问题,但由于受样本稀疏性的影响,仅依靠单独的局部观测推断全局信息准确性仍然不高,相比于本文方法存在一定差距,因此仍具有局限性。
为了深入分析通信机制对全局信息推断以及性能的影响,我们还测试了随机通信方法在不同通信数量配置下的结果,并与本文方法进行了对比,实验选择在地图5m_vs_6m上进行,实验结果如图9所示。实验结果表明,随机通信下的智能体协作性能并不会随通信数量的增加而单调提升,这是因为智能体可能接收到关联度较低的信息,导致形成“虚假全局”,从而干扰智能体决策。相比之下,本文方法使智能体能提供更加相关且多样化的观测信息,从而增强掩码自编码器的全局推断能力,提升决策效果。因此,该结果验证了主动通信机制在通信层面的有效性与先进性。
综上所述,提升智能体间协作性能的关键在于增强信息完整性。因此,为了验证本文方法对信息完整性的影响,我们额外测试了不同方法下智能体级掩码自编码器在预训练前、预训练后和微调后对所有智能体信息的重建精度。具体而言,我们通过计算3次实验中所有智能体观测真实值与重建值之间的平均均方误差(MSE),来对比MA2E、随机通信(通信3个)与本文方法在观测值重建精度上的差异。需要说明的是,为确保本文方法能够实现有效的通信调度,并进一步评估模型在动态复杂环境下的重建稳健性,表5所示的测试结果均在训练模式下获得。结果表明,相比于MA2E和随机通信方法(通信3个)这2种方法,本文方法在预训练阶段对于重建精度的提升最为明显,并在最终微调后取得了最高的重建精度,原因在于本文方法能够利用主动通信机制为智能体级掩码自编码器提供重要且多样化的样本,从而帮助智能体更准确地重建全局信息,验证了本文方法在提升信息重建精度方面的有效性。
本文方法虽然基于QMIX框架进行设计,但同样适用于所有基于值函数的方法。因此,为了测试本文方法的适用性,我们将本文方法移植到了VDN[21]和QPLEX[50]这2种基于值函数的方法中,测试了移植前后在地图5m_vs_6m上的平均胜率,实验设置与3.1节保持一致,最终VDN与本文方法的对比结果如图10所示,QPLEX与本文方法的对比结果如图11所示。观察结果可知,2种方法在移植本文方法以后,平均胜率均得到了一定程度的提升,验证了本文方法在基于值函数方法层面的适用性。
同时,为了保证方法的有效性和适用性,本文方法强调在集中训练和分散执行(CTDE)范式下进行实现。在集中训练阶段,智能体要求能够共享观测信息进行动态分组并确定通信对象集合;在分散执行阶段,每个智能体应具备通信能力,从而获取其他智能体的信息,帮助重建更准确的全局信息以辅助自身决策。保证以上条件才能确保本文方法能够在部分可观测场景下提升多智能体系统性能。
由于模型在实际运行过程中受模型规模和通信开销等因素的影响,因此,我们进一步对不同方法进行了代价分析,主要从训练阶段的模型参数总量、收敛所需时间步以及执行阶段的单步通信开销3个方面进行了评估,所有数据均在5m_vs_6m地图上进行收集,最终的统计结果如表6所示。表中的结果表明,本文方法由于引入主动通信机制,确实导致模型参数量有所增加。然而,与随机通信方法相比,本文方法在通过基于谱聚类的动态分组机制寻找到最佳分组并确定通信对象以后,智能体能够在执行阶段通信少量且相关的智能体信息,从而在保证重建效果的同时降低通信开销。与此同时,机制所带来的高效信息利用能力也使得整体训练过程更易收敛,从而缩短了达到稳定性能所需的时间步。
综上所述,本文方法在增加少量模型代价的同时,有效降低了通信开销并加快了收敛速度,实现了更优的综合性能。
为了更加直观地展现本文方法对于智能体间协作表现的影响,我们在SMAC环境中实现了可视化。具体来讲,我们截取了本文方法与QMIX在SMAC地图5m_vs_6m中的实机运行画面,详细观察并记录了这2种方法下盟军用来对抗敌军的协作策略,最终得到的可视化结果如图12所示。
图12中,QMIX由于缺乏全局视角,使得每个智能体只能根据自身的观测信息进行决策,导致未能集火单一目标,而是呈现出各自为战的局面,且智能体后撤并返回时,由于局部观测只能观测到最近可见目标,导致无法与其他智能体协同攻击同一目标,最终以只有2个智能体存活的局面险胜。
而对于本文方法,智能体不仅能根据血量动态调整站位,以保证自身存活与整体阵型稳定,同时还得益于全局视角,智能体能在后撤并返回时优先攻击集火目标,而非最近可见目标,从而实现了有效的集火策略,最终以仅牺牲1个智能体的局面获胜。
综上所述,本文方法通过为智能体提供一种全局视角,从而帮助它们学习得到一个跨智能体的全局协作策略,进而促进协作,体现了本文方法对于增强智能体间协作表现的有效性。
本文针对多智能体强化学习中的部分可观测性问题,提出了一种主动通信掩码重建方法。该方法在集中训练阶段利用谱聚类实现智能体的动态分组,并通过掩码交叉注意力机制为每个智能体选择少量且高度相关的智能体来确定通信关系,进而在分散执行阶段,每个智能体依据通信对象集合主动与对应智能体建立通信,从而为智能体级掩码自编码器提供多样化信息,实现全局信息的准确重建,之后再利用自注意力机制构建全局特征以辅助智能体决策。在SMAC平台上的实验结果表明,本文方法能够显著提高多个协作任务中的性能,验证了方法在提升协作多智能体系统性能上的可行性与有效性。
在未来的研究中,我们将优化通信策略,进一步考虑智能体间的距离以及通信拓扑结构等因素对全局状态推断与协作性能的影响;同时,还将继续探索如何降低计算复杂度的同时,提高掩码重建的精确度;最后,我们将进一步把该方法扩展至完全分布式的智能体系统,并应用于更加复杂的任务环境中。

参考文献 引证文献
排序方式:
1
Ning Zepeng, Xie Lihua. A survey on multi-agent reinforcement learning and its application[J]. Journal of Automation and Intelligence, 2024, 3(2): 73−91
2
Canese L, Cardarilli G C, Nunzio L D, et al. Multi-agent reinforcement learning: A review of challenges and applications[J]. Applied Sciences, 2021, 11(11): 4948
3
Nguyen T T, Nguyen N D, Nahavandi S. Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications[J]. IEEE Transactions on Cybernetics, 2020, 50(9): 3826−3839
4
Zhou Ming, Luo Jun, Villella J, et al. Smarts: An open-source scalable multi-agent RL training school for autonomous driving[C]//Proc of the 4th Conf on Robot Learning. New York: PMLR, 2021: 264−285
5
Yang Yaodong, Luo Rui, Li Minne, et al. Mean field multi-agent reinforcement learning[C]//Proc of the 35th Int Conf on Machine Learning. New York: ACM, 2018: 5571−5580
6
Di Yuanzhu, Deng Libao, Wang Ling, et al. A review of deep reinforcement learning methods for production scheduling[J/OL]. Tsinghua Science and Technology, 2025[2026-01-23]. https://doi.org/10.26599/TST.2024.9010247
7
Yu Chao, Yang Xinyi, Gao Jiaxuan, et al. Asynchronous multi-agent reinforcement learning for efficient real-time multi-robot cooperative exploration[C]//Proc of the 2023 Int Conf on Autonomous Agents and Multi-agent Systems. Berlin: Springer, 2023: 1107−1115
8
Han Sheng, Zhang Hengrui, Wu Hao, et al. Multi-constraint reinforcement learning in complex robot environments[J]. Frontiers of Computer Science, 2025, 19(8) : 198353
9
Peng Peng, Wen Ying, Yang Yaodong, et al. Multiagent bidirectionally-coordinated nets: Emergence of human-level coordination in learning to play starcraft combat games[J]. arXiv preprint, arXiv: 1703.10069, 2017
10
路文浩, 赵勇, 季雅泰, 等. 面向应急响应群智感知的异构群体协作任务分配方法[J]. 物联网学报, 2025, 9(4): 77−90
Lu Wenhao, Zhao Yong, Ji Yatai, et al. Task allocation method for collaborative crowdsensing of heterogeneous entities in emergency response[J]. Chinese Journal on Internet of Things, 2025, 9(4): 77−90 (in Chinese)
11
Gupta J K, Egorov M, Kochenderfer M. Cooperative multi-agent control using deep reinforcement learning[C]//Proc of the 2017 Int Conf on Autonomous Agents and Multi-agent Systems. Berlin: Springer, 2017: 66−83
12
Kraemer L, Banerjee B. Multi-agent reinforcement learning as a rehearsal for decentralized planning[J]. Neurocomputing, 2016, 190(19): 82−94
13
Zhu Changxi, Dastani M, Wang Shihan. A survey of multi-agent deep reinforcement learning with communication[J]. arXiv preprint, arXiv: 2203.08975, 2024
14
Seo Y, Hafner D, Liu Hao, et al. Masked world models for visual control[C]//Proc of the 6th Conf on Robot Learning. New York: PMLR, 2023: 1332−1344
15
Ng A, Jordan M, Weiss Y. On spectral clustering: Analysis and an algorithm[C]//Proc of the 15th Int Conf on Neural Information Processing Systems: Natural and Synthetic. Cambridge, MA: MIT, 2001: 849−856
16
Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proc of the 31st Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2017: 6000−6010
17
He Kaiming, Chen Xinlei, Xie Saining, et al. Masked autoencoders are scalable vision learners[C]//Proc of the 2022 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 16000−16009
18
Samvelyan M, Rashid T, De Witt C S, et al. The StarCraft multi-agent challenge[J]. arXiv preprint, arXiv: 1902.04043, 2019
19
Lauer M, Riedmiller M A. An algorithm for distributed reinforcement learning in cooperative multi-agent systems[C]//Proc of the 17th Int Conf on Machine Learning. New York: ACM, 2000: 535−542
20
Foerster J, Nardelli N, Farquhar G, et al. Stabilising experience replay for deep multi-agent reinforcement learning[C]//Proc of the 34th Int Conf on Machine Learning. New York: ACM, 2017: 1146−1155
21
Sunehag P, Lever G, Gruslys A, et al. Value-decomposition networks for cooperative multi-agent learning[J]. arXiv preprint, arXiv: 1706.05296, 2017
22
Rashid T, Samvelyan M, De Witt C S, et al. Monotonic value function factorisation for deep multi-agent reinforcement learning[J]. Journal of Machine Learning Research, 2020, 21(1): 1−51
23
Mnih V, Badia A P, Mirza M, et al. Asynchronous methods for deep reinforcement learning[C]//Proc of the 33rd Int Conf on Machine Learning. New York: ACM, 2016: 1928−1937
24
Lillicrap T P, Hunt J J, Pritzel A, et al. Continuous control with deep reinforcement learning[J]. arXiv preprint, arXiv: 1509.02971, 2015
25
Russell S J, Zimdars A. Q-decomposition for reinforcement learning agents[C]//Proc of the 20th Int Conf on Machine Learning. New York: ACM, 2003: 656−663
26
Schroeder de Witt C, Foerster J, Farquhar G, et al. Multi-agent common knowledge reinforcement learning[C]//Proc of the 33rd Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2019: 9927−9939
27
Shu Tianmin, Tian Yuandong. M3RL: Mind-aware multi-agent management reinforcement learning[J]. arXiv preprint, arXiv: 1810.00147, 2018
28
Iqbal S, Costales R, Sha F. ALMA: Hierarchical learning for composite multi-agent tasks[C]//Proc of the 36th Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2022: 7155−7166
29
Wang Tonghan, Dong Heng, Lesser V, et al. ROMA: Multi-agent reinforcement learning with emergent roles[C]//Proc of the 37th Int Conf on Machine Learning. New York: ACM, 2020: 9876–9886
30
Wang Tonghan, Gupta T, Mahajan A, et al. RODE: Learning roles to decompose multi-agent tasks[J]. arXiv preprint, arXiv: 2010.01523, 2020
31
Zang Yifan, He Jinmin, Li Kai, et al. Automatic grouping for efficient cooperative multi-agent reinforcement learning[C]//Proc of the 37th Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2023: 46105−46121
32
Liu Chiqiang, Li Dazi. HYGMA: Hypergraph coordination networks with dynamic grouping for multi-agent reinforcement learning[J]. arXiv preprint, arXiv: 2505.07207, 2025
33
Caliński T, Harabasz J. A dendrite method for cluster analysis[J]. Communications in Statistics-theory and Methods, 1974, 3(1): 1−27
34
Foerster J, Assael I A, De Freitas N, et al. Learning to communicate with deep multi-agent reinforcement learning[C]//Proc of the 30th Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2016: 2145−2153
35
Sukhbaatar S, Fergus R. Learning multiagent communication with backpropagation[C]//Proc of the 30th Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2016: 2252−2260
36
Jiang Jiechuan, Lu Zongqing. Learning attentional communication for multi-agent cooperation[C]//Proc of the 32nd Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2018: 7265−7275
37
Singh A, Jain T, Sukhbaatar S. Learning when to communicate at scale in multiagent cooperative and competitive tasks[J]. arXiv preprint, arXiv: 1812.09755, 2018
38
Liu Yong, Wang Weixun, Hu Yujing, et al. Multi-agent game abstraction via graph attention neural network[C]//Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 7211−7218
39
Das A, Gervet T, Romoff J, et al. TarMAC: Targeted multi-agent communication[C]//Proc of the 36th Int Conf on Machine Learning. New York: ACM, 2019: 1538−1546
40
Guan Cong, Chen Feng, Yuan Lei, et al. Efficient multi-agent communication via self-supervised information aggregation[C]//Proc of the 36th Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2022: 1020−1033
41
Devlin J, Chang Mingwei, Lee K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]//Proc of the 2019 Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, PA: ACL, 2019: 4171−4186
42
Liu Fangchen, Liu Hao, Grover A, et al. Masked autoencoding for scalable and generalizable decision making[C]//Proc of the 36th Int Conf on Machine Learning. New York: ACM, 2022: 12608−12618
43
Wu P, Majumdar A, Stone K, et al. Masked trajectory models for prediction, representation, and control[C]//Proc of the 40th Int Conf on Machine Learning. New York: ACM, 2023: 37607−37623
44
Song Haolin, Feng Mingxiao, Zhou Wengang, et al. MA2CL: Masked attentive contrastive learning for multi-agent reinforcement learning[J]. arXiv preprint, arXiv: 2306.02006, 2023
45
Kang S, Lee Y, Kim G, et al. MA2E: Addressing partial observability in multi-agent reinforcement learning with masked auto-encoder[C]//Proc of the 13rd Int Conf Learning Representations. Mountain View, CA: OpenReview. net, 2025: 51818−51838
46
Oliehoek F A, Amato C. A Concise Introduction to Decentralized POMDPs[M]. Berlin: Springer, 2016
47
Shi Jianbo, Malik J. Normalized cuts and image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(8): 888−905
48
Zha Hongyuan, He Xiaofeng, Ding C, et al. Spectral relaxation for k-means clustering[C]//Proc of the 15th Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2001: 1057−1064
49
Hu Jian, Jiang Siyang, Harding S A, et al. Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning[J]. arXiv preprint, arXiv: 2102.03479, 2021
50
Wang Jianhao, Ren Zhizhou, Liu T, et al. QPLEX: Duplex dueling multi-agent Q-learning[C]//Proc of the 9th Int Conf on Learning Representations. Mountain View, CA: OpenReview.net, 2021: 11609−11635
2026年第63卷第7期
84
36
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202550701
  • 接收时间:2025-09-21
  • 首发时间:2026-11-17
  • 出版时间:2026-07-01
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2025-09-21
  • 修回日期:2026-03-04
基金
作者信息
    吉林大学软件学院 长春 130012
    符号计算与知识工程教育部重点实验室(吉林大学) 长春 130012

通讯作者:

王恩()
参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202550701
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏