Estimating Functions of Distributions Defined over Spaces of Unknown Size
Abstract
We consider Bayesian estimation of informationtheoretic quantities from data, using a Dirichlet prior. Acknowledging the uncertainty of the event space size $m$ and the Dirichlet prior's concentration parameter $c$, we treat both as random variables set by a hyperprior. We show that the associated hyperprior, $P(c, m)$, obeys a simple "Irrelevance of Unseen Variables" (IUV) desideratum iff $P(c, m) = P(c) P(m)$. Thus, requiring IUV greatly reduces the number of degrees of freedom of the hyperprior. Some informationtheoretic quantities can be expressed multiple ways, in terms of different event spaces, e.g., mutual information. With all hyperpriors (implicitly) used in earlier work, different choices of this event space lead to different posterior expected values of these informationtheoretic quantities. We show that there is no such dependence on the choice of event space for a hyperprior that obeys IUV. We also derive a result that allows us to exploit IUV to greatly simplify calculations, like the posterior expected mutual information or posterior expected multiinformation. We also use computer experiments to favorably compare an IUVbased estimator of entropy to three alternative methods in common use. We end by discussing how seemingly innocuous changes to the formalization of an estimation problem can substantially affect the resultant estimates of posterior expectations.
 Publication:

Entropy
 Pub Date:
 October 2013
 DOI:
 10.3390/e15114668
 arXiv:
 arXiv:1311.4548
 Bibcode:
 2013Entrp..15.4668W
 Keywords:

 Mathematics  Statistics Theory;
 Physics  Data Analysis;
 Statistics and Probability;
 Quantitative Biology  Quantitative Methods
 EPrint:
 33 pages, 3 figures. Matches published version