Controlling Covariate Shift using Balanced Normalization of Weights
Abstract
We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs. The proposed technique keeps the contribution of positive and negative weights to the layer output balanced. We validate our method on a set of standard benchmarks including CIFAR-10/100, SVHN and ILSVRC 2012 ImageNet.
- Publication:
-
arXiv e-prints
- Pub Date:
- December 2018
- DOI:
- 10.48550/arXiv.1812.04549
- arXiv:
- arXiv:1812.04549
- Bibcode:
- 2018arXiv181204549D
- Keywords:
-
- Computer Science - Machine Learning;
- Statistics - Machine Learning