A Taxonomy of Recurrent Learning Rules
Abstract
Backpropagation through time (BPTT) is the de facto standard for training recurrent neural networks (RNNs), but it is non-causal and non-local. Real-time recurrent learning is a causal alternative, but it is highly inefficient. Recently, e-prop was proposed as a causal, local, and efficient practical alternative to these algorithms, providing an approximation of the exact gradient by radically pruning the recurrent dependencies carried over time. Here, we derive RTRL from BPTT using a detailed notation bringing intuition and clarification to how they are connected. Furthermore, we frame e-prop within in the picture, formalising what it approximates. Finally, we derive a family of algorithms of which e-prop is a special case.
- Publication:
-
arXiv e-prints
- Pub Date:
- July 2022
- DOI:
- 10.48550/arXiv.2207.11439
- arXiv:
- arXiv:2207.11439
- Bibcode:
- 2022arXiv220711439M
- Keywords:
-
- Computer Science - Machine Learning
- E-Print:
- Lecture Notes in Computer Science, 13529 (2022) 478-490