Deriving Differential Target Propagation from Iterating Approximate Inverses
Abstract
We show that a particular form of target propagation, i.e., relying on learned inverses of each layer, which is differential, i.e., where the target is a small perturbation of the forward propagation, gives rise to an update rule which corresponds to an approximate GaussNewton gradientbased optimization, without requiring the manipulation or inversion of large matrices. What is interesting is that this is more biologically plausible than backpropagation yet may turn out to implicitly provide a stronger optimization procedure. Extending difference target propagation, we consider several iterative calculations based on local autoencoders at each layer in order to achieve more precise inversions for more accurate target propagation and we show that these iterative procedures converge exponentially fast if the autoencoding function minus the identity function has a Lipschitz constant smaller than one, i.e., the autoencoder is coarsely succeeding at performing an inversion. We also propose a way to normalize the changes at each layer to take into account the relative influence of each layer on the output, so that larger weight changes are done on more influential layers, like would happen in ordinary backpropagation with gradient descent.
 Publication:

arXiv eprints
 Pub Date:
 July 2020
 DOI:
 10.48550/arXiv.2007.15139
 arXiv:
 arXiv:2007.15139
 Bibcode:
 2020arXiv200715139B
 Keywords:

 Computer Science  Machine Learning;
 Statistics  Machine Learning