Analysis of boosting algorithms using the smooth margin function
Abstract
We introduce a useful tool for analyzing boosting algorithms called the ``smooth margin function,'' a differentiable approximation of the usual margin for boosting algorithms. We present two boosting algorithms based on this smooth margin, ``coordinate ascent boosting'' and ``approximate coordinate ascent boosting,'' which are similar to Freund and Schapire's AdaBoost algorithm and Breiman's arc-gv algorithm. We give convergence rates to the maximum margin solution for both of our algorithms and for arc-gv. We then study AdaBoost's convergence properties using the smooth margin function. We precisely bound the margin attained by AdaBoost when the edges of the weak classifiers fall within a specified range. This shows that a previous bound proved by Rätsch and Warmuth is exactly tight. Furthermore, we use the smooth margin to capture explicit properties of AdaBoost in cases where cyclic behavior occurs.
- Publication:
-
arXiv e-prints
- Pub Date:
- March 2008
- DOI:
- 10.48550/arXiv.0803.4092
- arXiv:
- arXiv:0803.4092
- Bibcode:
- 2008arXiv0803.4092R
- Keywords:
-
- Statistics - Machine Learning;
- Mathematics - Statistics;
- 68W40;
- 68Q25 (Primary) 68Q32 (Secondary)
- E-Print:
- Published in at http://dx.doi.org/10.1214/009053607000000785 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)