Regularizing activations in neural networks via distribution matching with the Wasserstein metric
Abstract
Regularization and normalization have become indispensable components in training deep neural networks, resulting in faster training and improved generalization performance. We propose the projected error function regularization loss (PER) that encourages activations to follow the standard normal distribution. PER randomly projects activations onto onedimensional space and computes the regularization loss in the projected space. PER is similar to the PseudoHuber loss in the projected space, thus taking advantage of both $L^1$ and $L^2$ regularization losses. Besides, PER can capture the interaction between hidden units by projection vector drawn from a unit sphere. By doing so, PER minimizes the upper bound of the Wasserstein distance of order one between an empirical distribution of activations and the standard normal distribution. To the best of the authors' knowledge, this is the first work to regularize activations via distribution matching in the probability distribution space. We evaluate the proposed method on the image classification task and the wordlevel language modeling task.
 Publication:

arXiv eprints
 Pub Date:
 February 2020
 arXiv:
 arXiv:2002.05366
 Bibcode:
 2020arXiv200205366J
 Keywords:

 Computer Science  Machine Learning;
 Statistics  Machine Learning
 EPrint:
 ICLR 2020