Representation Benefits of Deep Feedforward Networks
Abstract
This note provides a family of classification problems, indexed by a positive integer $k$, where all shallow networks with fewer than exponentially (in $k$) many nodes exhibit error at least $1/6$, whereas a deep network with 2 nodes in each of $2k$ layers achieves zero error, as does a recurrent network with 3 distinct nodes iterated $k$ times. The proof is elementary, and the networks are standard feedforward networks with ReLU (Rectified Linear Unit) nonlinearities.
- Publication:
-
arXiv e-prints
- Pub Date:
- September 2015
- DOI:
- 10.48550/arXiv.1509.08101
- arXiv:
- arXiv:1509.08101
- Bibcode:
- 2015arXiv150908101T
- Keywords:
-
- Computer Science - Machine Learning;
- Computer Science - Neural and Evolutionary Computing