Empirical Analysis of the Inductive Bias of Recurrent Neural Networks by Discrete Fourier Transform of Output Sequences
Abstract
A unique feature of Recurrent Neural Networks (RNNs) is that it incrementally processes input sequences. In this research, we aim to uncover the inherent generalization properties, i.e., inductive bias, of RNNs with respect to how frequently RNNs switch the outputs through time steps in the sequence classification task, which we call output sequence frequency. Previous work analyzed inductive bias by training models with a few synthetic data and comparing the model's generalization with candidate generalization patterns. However, when examining the output sequence frequency, previous methods cannot be directly applied since enumerating candidate patterns is computationally difficult for longer sequences. To this end, we propose to directly calculate the output sequence frequency for each model by regarding the outputs of the model as discretetime signals and applying frequency domain analysis. Experimental results showed that Long ShortTerm Memory (LSTM) and Gated Recurrent Unit (GRU) have an inductive bias towards lowerfrequency patterns, while Elman RNN tends to learn patterns in which the output changes at high frequencies. We also found that the inductive bias of LSTM and GRU varies with the number of layers and the size of hidden layers.
 Publication:

arXiv eprints
 Pub Date:
 May 2023
 DOI:
 10.48550/arXiv.2305.09178
 arXiv:
 arXiv:2305.09178
 Bibcode:
 2023arXiv230509178I
 Keywords:

 Computer Science  Machine Learning