The Statistical Complexity of Interactive Decision Making
Abstract
A fundamental challenge in interactive learning and decision making, ranging from bandit problems to reinforcement learning, is to provide sampleefficient, adaptive learning algorithms that achieve nearoptimal regret. This question is analogous to the classical problem of optimal (supervised) statistical learning, where there are wellknown complexity measures (e.g., VC dimension and Rademacher complexity) that govern the statistical complexity of learning. However, characterizing the statistical complexity of interactive learning is substantially more challenging due to the adaptive nature of the problem. The main result of this work provides a complexity measure, the DecisionEstimation Coefficient, that is proven to be both necessary and sufficient for sampleefficient interactive learning. In particular, we provide: 1. a lower bound on the optimal regret for any interactive decision making problem, establishing the DecisionEstimation Coefficient as a fundamental limit. 2. a unified algorithm design principle, EstimationtoDecisions (E2D), which transforms any algorithm for supervised estimation into an online algorithm for decision making. E2D attains a regret bound matching our lower bound, thereby achieving optimal sampleefficient learning as characterized by the DecisionEstimation Coefficient. Taken together, these results constitute a theory of learnability for interactive decision making. When applied to reinforcement learning settings, the DecisionEstimation Coefficient recovers essentially all existing hardness results and lower bounds. More broadly, the approach can be viewed as a decisiontheoretic analogue of the classical Le Cam theory of statistical estimation; it also unifies a number of existing approaches  both Bayesian and frequentist.
 Publication:

arXiv eprints
 Pub Date:
 December 2021
 arXiv:
 arXiv:2112.13487
 Bibcode:
 2021arXiv211213487F
 Keywords:

 Computer Science  Machine Learning;
 Mathematics  Optimization and Control;
 Mathematics  Statistics Theory;
 Statistics  Machine Learning
 EPrint:
 Updated with new lower bound