Training Recurrent Neural Networks against Noisy Computations during Inference
Abstract
We explore the robustness of recurrent neural networks when the computations within the network are noisy. One of the motivations for looking into this problem is to reduce the high power cost of conventional computing of neural network operations through the use of analog neuromorphic circuits. Traditional GPU/CPUcentered deep learning architectures exhibit bottlenecks in powerrestricted applications, such as speech recognition in embedded systems. The use of specialized neuromorphic circuits, where analog signals passed through memorycell arrays are sensed to accomplish matrixvector multiplications, promises large power savings and speed gains but brings with it the problems of limited precision of computations and unavoidable analog noise. In this paper we propose a method, called {\em Deep Noise Injection training}, to train RNNs to obtain a set of weights/biases that is much more robust against noisy computation during inference. We explore several RNN architectures, such as vanilla RNN and longshortterm memories (LSTM), and show that after convergence of Deep Noise Injection training the set of trained weights/biases has more consistent performance over a wide range of noise powers entering the network during inference. Surprisingly, we find that Deep Noise Injection training improves overall performance of some networks even for numerically accurate inference.
 Publication:

arXiv eprints
 Pub Date:
 July 2018
 arXiv:
 arXiv:1807.06555
 Bibcode:
 2018arXiv180706555Q
 Keywords:

 Computer Science  Machine Learning;
 Statistics  Machine Learning
 EPrint:
 10 pages