Regret Bounds for Nondecomposable Metrics with Missing Labels
Abstract
We consider the problem of recommending relevant labels (items) for a given data point (user). In particular, we are interested in the practically important setting where the evaluation is with respect to nondecomposable (over labels) performance metrics like the $F_1$ measure, and the training data has missing labels. To this end, we propose a generic framework that given a performance metric $\Psi$, can devise a regularized objective function and a threshold such that all the values in the predicted score vector above and only above the threshold are selected to be positive. We show that the regret or generalization error in the given metric $\Psi$ is bounded ultimately by estimation error of certain underlying parameters. In particular, we derive regret bounds under three popular settings: a) collaborative filtering, b) multilabel classification, and c) PU (positiveunlabeled) learning. For each of the above problems, we can obtain precise nonasymptotic regret bound which is small even when a large fraction of labels is missing. Our empirical results on synthetic and benchmark datasets demonstrate that by explicitly modeling for missing labels and optimizing the desired performance metric, our algorithm indeed achieves significantly better performance (like $F_1$ score) when compared to methods that do not model missing label information carefully.
 Publication:

arXiv eprints
 Pub Date:
 June 2016
 arXiv:
 arXiv:1606.02077
 Bibcode:
 2016arXiv160602077J
 Keywords:

 Computer Science  Machine Learning;
 Statistics  Machine Learning