Algorithms with Logarithmic or Sublinear Regret for Constrained Contextual Bandits
Abstract
We study contextual bandits with budget and time constraints, referred to as constrained contextual bandits.The time and budget constraints significantly complicate the exploration and exploitation tradeoff because they introduce complex coupling among contexts over time.Such coupling effects make it difficult to obtain oracle solutions that assume known statistics of bandits. To gain insight, we first study unitcost systems with known context distribution. When the expected rewards are known, we develop an approximation of the oracle, referred to AdaptiveLinearProgramming (ALP), which achieves nearoptimality and only requires the ordering of expected rewards. With these highly desirable features, we then combine ALP with the upperconfidencebound (UCB) method in the general case where the expected rewards are unknown {\it a priori}. We show that the proposed UCBALP algorithm achieves logarithmic regret except for certain boundary cases. Further, we design algorithms and obtain similar regret analysis results for more general systems with unknown context distribution and heterogeneous costs. To the best of our knowledge, this is the first work that shows how to achieve logarithmic regret in constrained contextual bandits. Moreover, this work also sheds light on the study of computationally efficient algorithms for general constrained contextual bandits.
 Publication:

arXiv eprints
 Pub Date:
 April 2015
 arXiv:
 arXiv:1504.06937
 Bibcode:
 2015arXiv150406937W
 Keywords:

 Computer Science  Machine Learning;
 Statistics  Machine Learning
 EPrint:
 36 pages, 4 figures