Introduction
Standard Recurrent Neural Networks (RNNs) are notoriously difficult to train on long sequences due to the vanishing and exploding gradient problem. While orthogonal and unitary RNNs were developed to mitigate this by keeping weight matrix eigenvalues on the unit circle, they introduced a new challenge: the "perfect memory" problem. Without a mechanism to discard irrelevant information, these models' hidden states become cluttered with noise, limiting their effectiveness in tasks requiring both long-term persistence and short-term transient processing.
The Eigenvalue Normalized Recurrent Neural Network (ENRNN) addresses this by explicitly separating the hidden state into two components: a persistent long-term memory and a dissipative short-term memory.
Methodology: Balancing Persistence and Dissipation
The ENRNN architecture splits the hidden state into two interacting parts:
- Long-term state ($h^{(L)}$): Governed by an orthogonal matrix ($W^{(L)}$), this component preserves information across many time steps without decay.
- Short-term state ($h^{(S)}$): Governed by a dissipative matrix ($W^{(S)}$) where the spectral radius is constrained to be strictly less than 1. This ensures that information naturally "leaks" or decays over time, preventing the hidden state from becoming saturated with outdated data.
Eigenvalue Normalization and Coupling
To maintain the dissipative property, the authors parameterize $W^{(S)}$ using a normalization technique: $W^{(S)} = \frac{T}{\rho(T) + \epsilon}$, where $\rho(T)$ is the spectral radius of a general matrix $T$. A coupling matrix $W^{(C)}$ allows the short-term state to feed into the long-term state, effectively acting as a hierarchical preprocessor that extracts local features before they are committed to long-term memory.
Experimental Results
The ENRNN was evaluated against standard LSTMs and state-of-the-art orthogonal RNNs (such as scoRNN and Spectral RNN) across several benchmarks.
Synthetic Benchmarks: Copying and Adding Problems
On the Copying Problem ($T=2000$), which requires remembering a sequence across a long blank interval, the ENRNN demonstrated superior convergence speed compared to all other tested architectures.
Real-World Performance: TIMIT Speech Task
In the TIMIT speech prediction task, the ENRNN's ability to manage both local fluctuations and long-term structure resulted in a dramatic performance gap. With approximately 200k trainable parameters, the ENRNN achieved a validation MSE of 0.13, while the standard LSTM and scoRNN lagged significantly behind with much higher error rates.
Conclusion
The ENRNN proves that "forgetting" is a vital feature for efficient sequence modeling. By replacing complex gating mechanisms with simple linear algebraic constraints on eigenvalues, the ENRNN achieves state-of-the-art performance with lower computational overhead in many scenarios. While the $O(n^3)$ complexity of eigenvalue decomposition remains a challenge for very large hidden states, the model's structural simplicity and hierarchical processing offer a compelling alternative to traditional gated RNNs.
Summary updated