Learning ZeroSum SimultaneousMove Markov Games Using Function Approximation and Correlated Equilibrium
Abstract
We develop provably efficient reinforcement learning algorithms for twoplayer zerosum finitehorizon Markov games with simultaneous moves. To incorporate function approximation, we consider a family of Markov games where the reward function and transition kernel possess a linear structure. Both the offline and online settings of the problems are considered. In the offline setting, we control both players and aim to find the Nash Equilibrium by minimizing the duality gap. In the online setting, we control a single player playing against an arbitrary opponent and aim to minimize the regret. For both settings, we propose an optimistic variant of the leastsquares minimax value iteration algorithm. We show that our algorithm is computationally efficient and provably achieves an $\tilde O(\sqrt{d^3 H^3 T} )$ upper bound on the duality gap and regret, where $d$ is the linear dimension, $H$ the horizon and $T$ the total number of timesteps. Our results do not require additional assumptions on the sampling model. Our setting requires overcoming several new challenges that are absent in Markov decision processes or turnbased Markov games. In particular, to achieve optimism with simultaneous moves, we construct both upper and lower confidence bounds of the value function, and then compute the optimistic policy by solving a generalsum matrix game with these bounds as the payoff matrices. As finding the Nash Equilibrium of a generalsum game is computationally hard, our algorithm instead solves for a Coarse Correlated Equilibrium (CCE), which can be obtained efficiently. To our best knowledge, such a CCEbased scheme for optimism has not appeared in the literature and might be of interest in its own right.
 Publication:

arXiv eprints
 Pub Date:
 February 2020
 DOI:
 10.48550/arXiv.2002.07066
 arXiv:
 arXiv:2002.07066
 Bibcode:
 2020arXiv200207066X
 Keywords:

 Computer Science  Machine Learning;
 Computer Science  Computer Science and Game Theory;
 Computer Science  Multiagent Systems;
 Mathematics  Optimization and Control;
 Statistics  Machine Learning
 EPrint:
 Accepted for presentation at COLT 2020