Generalization for Adaptivelychosen Estimators via Stable Median
Abstract
Datasets are often reused to perform multiple statistical analyses in an adaptive way, in which each analysis may depend on the outcomes of previous analyses on the same dataset. Standard statistical guarantees do not account for these dependencies and little is known about how to provably avoid overfitting and false discovery in the adaptive setting. We consider a natural formalization of this problem in which the goal is to design an algorithm that, given a limited number of i.i.d.~samples from an unknown distribution, can answer adaptivelychosen queries about that distribution. We present an algorithm that estimates the expectations of $k$ arbitrary adaptivelychosen realvalued estimators using a number of samples that scales as $\sqrt{k}$. The answers given by our algorithm are essentially as accurate as if fresh samples were used to evaluate each estimator. In contrast, prior work yields error guarantees that scale with the worstcase sensitivity of each estimator. We also give a version of our algorithm that can be used to verify answers to such queries where the sample complexity depends logarithmically on the number of queries $k$ (as in the reusable holdout technique). Our algorithm is based on a simple approximate median algorithm that satisfies the strong stability guarantees of differential privacy. Our techniques provide a new approach for analyzing the generalization guarantees of differentially private algorithms.
 Publication:

arXiv eprints
 Pub Date:
 June 2017
 DOI:
 10.48550/arXiv.1706.05069
 arXiv:
 arXiv:1706.05069
 Bibcode:
 2017arXiv170605069F
 Keywords:

 Computer Science  Machine Learning;
 Computer Science  Data Structures and Algorithms;
 Statistics  Machine Learning
 EPrint:
 To appear in Conference on Learning Theory (COLT) 2017