RNN-based Online Learning: An Efficient First-Order Optimization Algorithm with a Convergence Guarantee
Abstract
We investigate online nonlinear regression with continually running recurrent neural network networks (RNNs), i.e., RNN-based online learning. For RNN-based online learning, we introduce an efficient first-order training algorithm that theoretically guarantees to converge to the optimum network parameters. Our algorithm is truly online such that it does not make any assumption on the learning environment to guarantee convergence. Through numerical simulations, we verify our theoretical results and illustrate significant performance improvements achieved by our algorithm with respect to the state-of-the-art RNN training methods.
- Publication:
-
arXiv e-prints
- Pub Date:
- March 2020
- DOI:
- 10.48550/arXiv.2003.03601
- arXiv:
- arXiv:2003.03601
- Bibcode:
- 2020arXiv200303601M
- Keywords:
-
- Computer Science - Machine Learning;
- Statistics - Machine Learning
- E-Print:
- This paper was an early draft of the presented results. We have written and published another paper (arXiv:2005.08948) where we have improved the material in this paper. The published paper covers most of the material presented in this paper as well. Therefore, we remove this paper from Arxiv and kindly refer the interested readers to arXiv:2005.08948