Conformal Risk Control
Abstract
We extend conformal prediction to control the expected value of any monotone loss function. The algorithm generalizes split conformal prediction together with its coverage guarantee. Like conformal prediction, the conformal risk control procedure is tight up to an $\mathcal{O}(1/n)$ factor. We also introduce extensions of the idea to distribution shift, quantile risk control, multiple and adversarial risk control, and expectations of Ustatistics. Worked examples from computer vision and natural language processing demonstrate the usage of our algorithm to bound the false negative rate, graph distance, and tokenlevel F1score.
 Publication:

arXiv eprints
 Pub Date:
 August 2022
 DOI:
 10.48550/arXiv.2208.02814
 arXiv:
 arXiv:2208.02814
 Bibcode:
 2022arXiv220802814A
 Keywords:

 Statistics  Methodology;
 Computer Science  Artificial Intelligence;
 Computer Science  Machine Learning;
 Mathematics  Statistics Theory;
 Statistics  Machine Learning
 EPrint:
 Code available at https://github.com/aangelopoulos/conformalrisk