On the Complexity of Estimating Renyi Divergences
Abstract
This paper studies the complexity of estimating Renyi divergences of discrete distributions: $p$ observed from samples and the baseline distribution $q$ known \emph{a priori}. Extending the results of Acharya et al. (SODA'15) on estimating Renyi entropy, we present improved estimation techniques together with upper and lower bounds on the sample complexity. We show that, contrarily to estimating Renyi entropy where a sublinear (in the alphabet size) number of samples suffices, the sample complexity is heavily dependent on \emph{events occurring unlikely} in $q$, and is unbounded in general (no matter what an estimation technique is used). For any divergence of order bigger than $1$, we provide upper and lower bounds on the number of samples dependent on probabilities of $p$ and $q$. We conclude that the worstcase sample complexity is polynomial in the alphabet size if and only if the probabilities of $q$ are nonnegligible. This gives theoretical insights into heuristics used in applied papers to handle numerical instability, which occurs for small probabilities of $q$. Our result explains that small probabilities should be handled with care not only because of numerical issues, but also because of a blow up in sample complexity.
 Publication:

arXiv eprints
 Pub Date:
 February 2017
 arXiv:
 arXiv:1702.01666
 Bibcode:
 2017arXiv170201666S
 Keywords:

 Computer Science  Information Theory;
 Computer Science  Computational Complexity;
 H.1.1
 EPrint:
 some typos fixed and references added (comparing to the previous version)