MeanField MultiAgent Reinforcement Learning: A Decentralized Network Approach
Abstract
One of the challenges for multiagent reinforcement learning (MARL) is designing efficient learning algorithms for a large system in which each agent has only limited or partial information of the entire system. While exciting progress has been made to analyze decentralized MARL with the network of agents for social networks and team video games, little is known theoretically for decentralized MARL with the network of states for modeling selfdriving vehicles, ridesharing, and data and traffic routing. This paper proposes a framework of localized training and decentralized execution to study MARL with network of states. Localized training means that agents only need to collect local information in their neighboring states during the training phase; decentralized execution implies that agents can execute afterwards the learned decentralized policies, which depend only on agents' current states. The theoretical analysis consists of three key components: the first is the reformulation of the MARL system as a networked Markov decision process with teams of agents, enabling updating the associated team Qfunction in a localized fashion; the second is the Bellman equation for the value function and the appropriate Qfunction on the probability measure space; and the third is the exponential decay property of the team Qfunction, facilitating its approximation with efficient sample efficiency and controllable error. The theoretical analysis paves the way for a new algorithm LTDENeuralAC, where the actorcritic approach with overparameterized neural networks is proposed. The convergence and sample complexity is established and shown to be scalable with respect to the sizes of both agents and states. To the best of our knowledge, this is the first neural network based MARL algorithm with network structure and provably convergence guarantee.
 Publication:

arXiv eprints
 Pub Date:
 August 2021
 DOI:
 10.48550/arXiv.2108.02731
 arXiv:
 arXiv:2108.02731
 Bibcode:
 2021arXiv210802731G
 Keywords:

 Computer Science  Machine Learning;
 Mathematics  Optimization and Control;
 49N80;
 90C40;
 68T05;
 68T07