Generalization in portfoliobased algorithm selection
Abstract
Portfoliobased algorithm selection has seen tremendous practical success over the past two decades. This algorithm configuration procedure works by first selecting a portfolio of diverse algorithm parameter settings, and then, on a given problem instance, using an algorithm selector to choose a parameter setting from the portfolio with strong predicted performance. Oftentimes, both the portfolio and the algorithm selector are chosen using a training set of typical problem instances from the application domain at hand. In this paper, we provide the first provable guarantees for portfoliobased algorithm selection. We analyze how large the training set should be to ensure that the resulting algorithm selector's average performance over the training set is close to its future (expected) performance. This involves analyzing three key reasons why these two quantities may diverge: 1) the learningtheoretic complexity of the algorithm selector, 2) the size of the portfolio, and 3) the learningtheoretic complexity of the algorithm's performance as a function of its parameters. We introduce an endtoend learningtheoretic analysis of the portfolio construction and algorithm selection together. We prove that if the portfolio is large, overfitting is inevitable, even with an extremely simple algorithm selector. With experiments, we illustrate a tradeoff exposed by our theoretical analysis: as we increase the portfolio size, we can hope to include a wellsuited parameter setting for every possible problem instance, but it becomes impossible to avoid overfitting.
 Publication:

arXiv eprints
 Pub Date:
 December 2020
 arXiv:
 arXiv:2012.13315
 Bibcode:
 2020arXiv201213315B
 Keywords:

 Computer Science  Artificial Intelligence
 EPrint:
 AAAI 2021