Taming "data-hungry" reinforcement learning? Stability in continuous state-action spaces
Abstract
We introduce a novel framework for analyzing reinforcement learning (RL) in continuous state-action spaces, and use it to prove fast rates of convergence in both off-line and on-line settings. Our analysis highlights two key stability properties, relating to how changes in value functions and/or policies affect the Bellman operator and occupation measures. We argue that these properties are satisfied in many continuous state-action Markov decision processes, and demonstrate how they arise naturally when using linear function approximation methods. Our analysis offers fresh perspectives on the roles of pessimism and optimism in off-line and on-line RL, and highlights the connection between off-line RL and transfer learning.
- Publication:
-
arXiv e-prints
- Pub Date:
- January 2024
- DOI:
- 10.48550/arXiv.2401.05233
- arXiv:
- arXiv:2401.05233
- Bibcode:
- 2024arXiv240105233D
- Keywords:
-
- Computer Science - Machine Learning;
- Computer Science - Information Theory;
- Electrical Engineering and Systems Science - Systems and Control;
- Mathematics - Optimization and Control;
- Statistics - Machine Learning