Empirical Analysis of the Inductive Bias of Recurrent Neural Networks by Discrete Fourier Transform of Output Sequences

Taiga Ishii, Ryo Ueda, Yusuke MiyaoarXiv 2023 · OpenReview

Abstract

A unique feature of Recurrent Neural Networks (RNNs) is that they incrementally process input sequences. In this research, we aim to uncover the inherent generalization properties, i.e., inductive bias, of RNNs with respect to how frequently RNNs switch their outputs through time steps in the sequence classification task—a property we call output sequence frequency. Previous work analyzed inductive bias by training models with sparse synthetic data and comparing the model's generalization with candidate patterns. However, for longer sequences, enumerating candidate patterns is computationally difficult. To this end, we propose to directly calculate the output sequence frequency for each model by regarding the outputs as discrete-time signals and applying frequency domain analysis. Experimental results show that Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) have an inductive bias towards lower-frequency patterns, while Elman RNNs tend to learn patterns in which the output changes at high frequencies. We also found that the inductive bias of LSTM and GRU varies with the number of layers and the size of hidden layers.

Introduction: The Inductive Bias Problem

Deep learning models are often viewed as "black boxes." While they generalize remarkably well to unseen data, the underlying reasons for their specific choices—their inductive bias—remain elusive. When multiple functions can fit the training data, why does a model choose one over another? For Recurrent Neural Networks (RNNs), which process data sequentially, a critical dimension of this bias is "output sequence frequency": the rate at which a model switches its predictions over time.

Generalization patterns in RNNs: Low vs. High Frequency.

Previous research into inductive bias required manually defining candidate patterns to compare against, a method that fails to scale as sequences get longer. This paper introduces a direct, scalable framework to measure these rhythmic preferences using digital signal processing techniques.

Methodology: Frequency Domain Analysis

The researchers propose treating the probability of an RNN's prediction at each time step as a discrete-time signal. By applying a Discrete Fourier Transform (DFT), they decompose these output signals into constituent sine waves of various frequencies. The frequency with the highest amplitude, termed the Dominant Frequency ($\omega_{dom}$), serves as a metric for the model's "natural rhythm" when generalizing from sparse data.

To isolate architectural bias from real-world data complexities, the study uses synthetic datasets of binary sequences (length 100). Models are trained on extremely sparse "label change" points—providing only 1–10 training points for a 100-step sequence. This forces the models to "fill in the blanks," revealing their inherent tendencies.

Key Results: Architectural "Personalities"

The analysis reveals distinct inductive biases across common RNN architectures:

  • LSTM and GRU: These models are "smooth" operators, exhibiting a strong bias toward lower-frequency patterns. They naturally prefer stable, slowly changing output sequences.
  • Elman RNN: In contrast, the classic Elman RNN is "jittery," gravitating toward high-frequency patterns where outputs oscillate rapidly between states.
Dominant frequency distributions for LSTM, GRU, and Elman RNNs across various configurations of layers and hidden sizes.

Interestingly, model capacity does not dictate bias in a straightforward way. While LSTMs are theoretically more expressive than Elman RNNs, they consistently prefer simpler, lower-frequency functions in practice. Furthermore, adding layers to an LSTM counter-intuitively increases its bias toward low-frequency stability, even as the model's complexity grows.

Discussion and Future Directions

A notable finding is the failure of these models to naturally "connect the dots" via simple interpolation. Instead, they all impose their own periodic, rhythmic patterns on the data. This suggests that the choice of architecture is a choice of "personality" that must match the temporal nature of the task—for instance, choosing LSTMs for slow-changing video activity recognition versus Elman-like structures for high-frequency audio signal processing.

Future work aims to extend this DFT-based analysis to Transformers and larger vocabularies, potentially establishing output sequence frequency as a formal metric for model complexity and architectural selection.

Summary updated

Abstract

A unique feature of Recurrent Neural Networks (RNNs) is that it incrementally processes input sequences. In this research, we aim to uncover the inherent generalization properties, i.e., inductive bias, of RNNs with respect to how frequently RNNs switch the outputs through time steps in the sequence classification task, which we call output sequence frequency. Previous work analyzed inductive bias by training models with a few synthetic data and comparing the model's generalization with candidate generalization patterns. However, when examining the output sequence frequency, previous methods cannot be directly applied since enumerating candidate patterns is computationally difficult for longer sequences. To this end, we propose to directly calculate the output sequence frequency for each model by regarding the outputs of the model as discrete-time signals and applying frequency domain analysis. Experimental results showed that Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) have an inductive bias towards lower-frequency patterns, while Elman RNN tends to learn patterns in which the output changes at high frequencies. We also found that the inductive bias of LSTM and GRU varies with the number of layers and the size of hidden layers.
BibTeX
@misc{ishii2023empirical,
  author = {Taiga Ishii and Ryo Ueda and Yusuke Miyao},
  title = {{Empirical Analysis of the Inductive Bias of Recurrent Neural Networks by Discrete Fourier Transform of Output Sequences}},
  year = {2023},
  eprint = {2305.09178},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.09178},
}
Figure