Data-dependent Pruning to find the Winning Lottery Ticket
Abstract
The Lottery Ticket Hypothesis postulates that a freshly initialized neural network contains a small subnetwork that can be trained in isolation to achieve similar performance as the full network. Our paper examines several alternatives to search for such subnetworks. We conclude that incorporating a data dependent component into the pruning criterion in the form of the gradient of the training loss -- as done in the SNIP method -- consistently improves the performance of existing pruning algorithms.
- Publication:
-
arXiv e-prints
- Pub Date:
- June 2020
- DOI:
- 10.48550/arXiv.2006.14350
- arXiv:
- arXiv:2006.14350
- Bibcode:
- 2020arXiv200614350L
- Keywords:
-
- Computer Science - Machine Learning;
- Statistics - Machine Learning