An iterative KFAC algorithm for Deep Learning
Abstract
Kroneckerfactored Approximate Curvature (KFAC) method is a high efficiency second order optimizer for the deep learning. Its training time is less than SGD(or other firstorder method) with same accuracy in many largescale problems. The key of KFAC is to approximates Fisher information matrix (FIM) as a blockdiagonal matrix where each block is an inverse of tiny Kronecker factors. In this short note, we present CGFAC  an new iterative KFAC algorithm. It uses conjugate gradient method to approximate the nature gradient. This CGFAC method is matrixfree, that is, no need to generate the FIM matrix, also no need to generate the Kronecker factors A and G. We prove that the time and memory complexity of iterative CGFAC is much less than that of standard KFAC algorithm.
 Publication:

arXiv eprints
 Pub Date:
 January 2021
 arXiv:
 arXiv:2101.00218
 Bibcode:
 2021arXiv210100218C
 Keywords:

 Computer Science  Machine Learning;
 Mathematics  Numerical Analysis;
 Statistics  Machine Learning
 EPrint:
 5 pages