Extrapolating Expected Accuracies for Large Multi-Class Problems
Abstract
The difficulty of multi-class classification generally increases with the number of classes. Using data from a subset of the classes, can we predict how well a classifier will scale with an increased number of classes? Under the assumptions that the classes are sampled identically and independently from a population, and that the classifier is based on independently learned scoring functions, we show that the expected accuracy when the classifier is trained on k classes is the (k-1)st moment of a certain distribution that can be estimated from data. We present an unbiased estimation method based on the theory, and demonstrate its application on a facial recognition example.
- Publication:
-
arXiv e-prints
- Pub Date:
- December 2017
- DOI:
- 10.48550/arXiv.1712.09713
- arXiv:
- arXiv:1712.09713
- Bibcode:
- 2017arXiv171209713Z
- Keywords:
-
- Statistics - Machine Learning;
- Computer Science - Computer Vision and Pattern Recognition;
- Computer Science - Machine Learning
- E-Print:
- Submitted to JMLR