Top$k$ eXtreme Contextual Bandits with Arm Hierarchy
Abstract
Motivated by modern applications, such as online advertisement and recommender systems, we study the top$k$ extreme contextual bandits problem, where the total number of arms can be enormous, and the learner is allowed to select $k$ arms and observe all or some of the rewards for the chosen arms. We first propose an algorithm for the nonextreme realizable setting, utilizing the Inverse Gap Weighting strategy for selecting multiple arms. We show that our algorithm has a regret guarantee of $O(k\sqrt{(Ak+1)T \log (\mathcal{F}T)})$, where $A$ is the total number of arms and $\mathcal{F}$ is the class containing the regression function, while only requiring $\tilde{O}(A)$ computation per time step. In the extreme setting, where the total number of arms can be in the millions, we propose a practicallymotivated arm hierarchy model that induces a certain structure in mean rewards to ensure statistical and computational efficiency. The hierarchical structure allows for an exponential reduction in the number of relevant arms for each context, thus resulting in a regret guarantee of $O(k\sqrt{(\log Ak+1)T \log (\mathcal{F}T)})$. Finally, we implement our algorithm using a hierarchical linear function class and show superior performance with respect to wellknown benchmarks on simulated bandit feedback experiments using extreme multilabel classification datasets. On a dataset with three million arms, our reduction scheme has an average inference time of only 7.9 milliseconds, which is a 100x improvement.
 Publication:

arXiv eprints
 Pub Date:
 February 2021
 arXiv:
 arXiv:2102.07800
 Bibcode:
 2021arXiv210207800S
 Keywords:

 Statistics  Machine Learning;
 Computer Science  Artificial Intelligence;
 Computer Science  Machine Learning