Eigenvalue Normalized Recurrent Neural Networks for Short Term Memory

Kyle Helfrich, Qiang YeAAAI 2020 · DOI

Introduction

Standard Recurrent Neural Networks (RNNs) are notoriously difficult to train on long sequences due to the vanishing and exploding gradient problem. While orthogonal and unitary RNNs were developed to mitigate this by keeping weight matrix eigenvalues on the unit circle, they introduced a new challenge: the "perfect memory" problem. Without a mechanism to discard irrelevant information, these models' hidden states become cluttered with noise, limiting their effectiveness in tasks requiring both long-term persistence and short-term transient processing.

The Eigenvalue Normalized Recurrent Neural Network (ENRNN) addresses this by explicitly separating the hidden state into two components: a persistent long-term memory and a dissipative short-term memory.

Methodology: Balancing Persistence and Dissipation

The ENRNN architecture splits the hidden state into two interacting parts:

  1. Long-term state ($h^{(L)}$): Governed by an orthogonal matrix ($W^{(L)}$), this component preserves information across many time steps without decay.
  2. Short-term state ($h^{(S)}$): Governed by a dissipative matrix ($W^{(S)}$) where the spectral radius is constrained to be strictly less than 1. This ensures that information naturally "leaks" or decays over time, preventing the hidden state from becoming saturated with outdated data.

Eigenvalue Normalization and Coupling

To maintain the dissipative property, the authors parameterize $W^{(S)}$ using a normalization technique: $W^{(S)} = \frac{T}{\rho(T) + \epsilon}$, where $\rho(T)$ is the spectral radius of a general matrix $T$. A coupling matrix $W^{(C)}$ allows the short-term state to feed into the long-term state, effectively acting as a hierarchical preprocessor that extracts local features before they are committed to long-term memory.

Experimental Results

The ENRNN was evaluated against standard LSTMs and state-of-the-art orthogonal RNNs (such as scoRNN and Spectral RNN) across several benchmarks.

Synthetic Benchmarks: Copying and Adding Problems

On the Copying Problem ($T=2000$), which requires remembering a sequence across a long blank interval, the ENRNN demonstrated superior convergence speed compared to all other tested architectures.

Figure 1: Performance on the Copying Problem (T=2000). The ENRNN (black line) converges to zero cross-entropy significantly faster than competing architectures.

Real-World Performance: TIMIT Speech Task

In the TIMIT speech prediction task, the ENRNN's ability to manage both local fluctuations and long-term structure resulted in a dramatic performance gap. With approximately 200k trainable parameters, the ENRNN achieved a validation MSE of 0.13, while the standard LSTM and scoRNN lagged significantly behind with much higher error rates.

Figure 3: Validation MSE on the TIMIT dataset. The ENRNN maintains a consistently lower error rate compared to LSTM and scoRNN under similar parameter constraints.

Conclusion

The ENRNN proves that "forgetting" is a vital feature for efficient sequence modeling. By replacing complex gating mechanisms with simple linear algebraic constraints on eigenvalues, the ENRNN achieves state-of-the-art performance with lower computational overhead in many scenarios. While the $O(n^3)$ complexity of eigenvalue decomposition remains a challenge for very large hidden states, the model's structural simplicity and hierarchical processing offer a compelling alternative to traditional gated RNNs.

Summary updated

BibTeX
@inproceedings{helfrich2020eigenvalue,
  author = {Kyle Helfrich and Qiang Ye},
  title = {{Eigenvalue Normalized Recurrent Neural Networks for Short Term Memory}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i04.5831},
}
Figure