Quasi-Newton Methods: A New Direction
Abstract
Four decades after their invention, quasi-Newton methods are still state of the art in unconstrained numerical optimization. Although not usually interpreted thus, these are learning algorithms that fit a local quadratic approximation to the objective function. We show that many, including the most popular, quasi-Newton methods can be interpreted as approximations of Bayesian linear regression under varying prior assumptions. This new notion elucidates some shortcomings of classical algorithms, and lights the way to a novel nonparametric quasi-Newton method, which is able to make more efficient use of available information at computational cost similar to its predecessors.
- Publication:
-
arXiv e-prints
- Pub Date:
- June 2012
- DOI:
- 10.48550/arXiv.1206.4602
- arXiv:
- arXiv:1206.4602
- Bibcode:
- 2012arXiv1206.4602H
- Keywords:
-
- Computer Science - Numerical Analysis;
- Computer Science - Machine Learning;
- Statistics - Machine Learning
- E-Print:
- ICML2012