Online Reinforcement Learning in Markov Decision Process Using Linear Programming
Abstract
We consider online reinforcement learning in episodic Markov decision process (MDP) with unknown transition function and stochastic rewards drawn from some fixed but unknown distribution. The learner aims to learn the optimal policy and minimize their regret over a finite time horizon through interacting with the environment. We devise a simple and efficient model-based algorithm that achieves $\widetilde{O}(LX\sqrt{TA})$ regret with high probability, where $L$ is the episode length, $T$ is the number of episodes, and $X$ and $A$ are the cardinalities of the state space and the action space, respectively. The proposed algorithm, which is based on the concept of ``optimism in the face of uncertainty", maintains confidence sets of transition and reward functions and uses occupancy measures to connect the online MDP with linear programming. It achieves a tighter regret bound compared to the existing works that use a similar confidence set framework and improves computational effort compared to those that use a different framework but with a slightly tighter regret bound.
- Publication:
-
arXiv e-prints
- Pub Date:
- March 2023
- DOI:
- 10.48550/arXiv.2304.00155
- arXiv:
- arXiv:2304.00155
- Bibcode:
- 2023arXiv230400155L
- Keywords:
-
- Computer Science - Machine Learning;
- Electrical Engineering and Systems Science - Systems and Control;
- Mathematics - Optimization and Control
- E-Print:
- 2023 62nd IEEE Conference on Decision and Control (CDC)