Training Convolutional ReLU Neural Networks in Polynomial Time: Exact Convex Optimization Formulations
Abstract
We study training of Convolutional Neural Networks (CNNs) with ReLU activations and introduce exact convex optimization formulations with a polynomial complexity with respect to the number of data samples, the number of neurons and data dimension. Particularly, we develop a convex analytic framework utilizing semiinfinite duality to obtain equivalent convex optimization problems for several CNN architectures. We first prove that twolayer CNNs can be globally optimized via an $\ell_2$ norm regularized convex program. We then show that certain threelayer CNN training problems are equivalent to an $\ell_1$ regularized convex program. We also extend these results to multilayer CNN architectures. Furthermore, we present extensions of our approach to different pooling methods.
 Publication:

arXiv eprints
 Pub Date:
 June 2020
 arXiv:
 arXiv:2006.14798
 Bibcode:
 2020arXiv200614798E
 Keywords:

 Computer Science  Machine Learning;
 Computer Science  Computational Complexity;
 Statistics  Machine Learning