On Empirical Entropy
Abstract
We propose a compression-based version of the empirical entropy of a finite string over a finite alphabet. Whereas previously one considers the naked entropy of (possibly higher order) Markov processes, we consider the sum of the description of the random variable involved plus the entropy it induces. We assume only that the distribution involved is computable. To test the new notion we compare the Normalized Information Distance (the similarity metric) with a related measure based on Mutual Information in Shannon's framework. This way the similarities and differences of the last two concepts are exposed.
- Publication:
-
arXiv e-prints
- Pub Date:
- March 2011
- DOI:
- 10.48550/arXiv.1103.5985
- arXiv:
- arXiv:1103.5985
- Bibcode:
- 2011arXiv1103.5985V
- Keywords:
-
- Computer Science - Information Theory;
- Computer Science - Machine Learning;
- 68;
- 94;
- H.1;
- F.1;
- J.1
- E-Print:
- 14 pages, LaTeX