RNNs Incrementally Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients?

Anil Kag, Ziming Zhang, Venkatesh SaligramaICLR 2020 · OpenReview

TL;DR

The Incremental RNN (iRNN) solves the age-old vanishing and exploding gradient problem by forcing hidden states to evolve toward an "equilibrium manifold." By modeling state changes as increments rather than total values, the network naturally exhibits "identity gradients." This allows it to learn long-term dependencies much faster and with significantly fewer parameters than standard LSTMs.

What Problem Does This Paper Address?

Recurrent Neural Networks (RNNs) are designed for sequential data, but they are notoriously difficult to train on long sequences. This difficulty stems from the Vanishing and Exploding Gradient problem: as errors are backpropagated through time, the gradient is multiplied by the network's weights at every step. If those weights are even slightly less than one, the signal vanishes; if they are greater than one, it explodes.

While architectures like LSTMs and GRUs use "gates" to protect the gradient flow, they don't truly eliminate the problem—they just mitigate it. Other approaches, like Unitary RNNs, impose strict mathematical constraints on weight matrices that can limit the model's expressive power. The authors of this paper seek a "panacea": a structural way to ensure gradients remain stable (near identity) without sacrificing performance or computational efficiency.

Methods

The authors draw inspiration from Continuous-Time RNNs (CTRNNs), originally proposed by Rosenblatt in 1962, which describe neural activity using continuous differential equations.

The core innovation is the Incremental RNN (iRNN). Instead of calculating an entirely new hidden state from scratch at each time step, the iRNN calculates an increment (a small change) and adds it to the previous state.

  1. Equilibrium Manifold: The model is designed so that for any given input, the hidden state naturally wants to settle into a stable "equilibrium."
  2. Inner Recursion ($K$): For each input $x_m$, the model runs $K$ internal iterations to move the hidden state toward this equilibrium.
iRNN Architecture
  1. Identity Gradient: The authors mathematically prove that when the system is at or near this equilibrium, the partial derivative of the current state with respect to the previous state converges to the identity matrix ($I$). This means the gradient signal can pass through hundreds or thousands of time steps without shrinking or growing.
  2. SiRNN: A simplified version where $K=1$ (single-step incremental RNN), which is computationally cheaper and works exceptionally well for slowly varying inputs.

Datasets

The authors evaluated iRNN across a diverse set of challenging benchmarks:

  • Long-Term Dependency (LTD) Tasks: The classic "Addition" and "Copy Memory" tasks.
  • Sequential Vision: Pixel-by-pixel MNIST and Permuted MNIST (where an image is fed to the RNN one pixel at a time).
  • Noise-Padded Tasks: Noisy-MNIST and Noisy-CIFAR, where actual data is followed by a long string of random noise to test if the RNN can "remember" the beginning.
  • Activity Recognition: HAR-2 (Human Activity Recognition) and Google-30 (Speech Commands).
  • Language Modeling: Penn Treebank (PTB).

Key Insights and Results

The "Identity" Breakthrough and Fast Convergence

Most RNNs struggle because their state-to-state Jacobian is rarely identity. By framing the RNN as a root-finding problem (moving toward equilibrium), the iRNN forces the Jacobian toward identity by design. Because the gradients are so stable, the model requires far fewer training iterations to reach a solution compared to LSTMs or other gated variants.

Training Convergence and Gradient Stability

Empirical plots confirm this theory: while standard RNN gradients fluctuate wildly or vanish, iRNN gradients stay remarkably close to 1.0 throughout the entire training process. On the Addition task, iRNN converged roughly 2x faster than FastRNN and significantly faster than LSTMs.

Parameter Efficiency

The authors found that they could use low-rank matrices for the state transitions. This allows the iRNN to achieve state-of-the-art results with a fraction of the parameters used by LSTMs. On Pixel-MNIST, iRNN achieved 98.13% accuracy using only 4k parameters, while an LSTM required 53k parameters to reach a lower 97.81%.

Noise Robustness

Because the model tracks changes, it is remarkably robust to noise. If the input becomes pure noise, the system stays near its last informative equilibrium, effectively "locking in" the memory. In the Noisy-CIFAR task, where LSTMs failed completely (11.6% accuracy), iRNN maintained a strong 54.5% accuracy, demonstrating its ability to ignore irrelevant temporal noise.

Limitations and Future Work

  • Inner Loop Overhead: While iRNN is highly efficient in terms of parameters, setting $K > 1$ increases the computational cost per time step. However, the authors note that $K=1$ (SiRNN) is often sufficient for many tasks.
  • Slowly Varying Assumption: The theoretical "fidelity" of the model relies on the assumption that inputs vary slowly relative to the system's time constant. While it performs well on discrete tasks, very high-frequency signals might challenge the equilibrium approach.
  • Deep Architectures: The paper focuses primarily on single-layer performance to highlight the incremental logic. While the Appendix mentions extensions to deep (multi-layer) RNNs, more extensive testing on massive-scale deep learning tasks remains an exciting avenue for future work.

Summary updated

Abstract

Recurrent neural networks (RNNs) are particularly well-suited for modeling long-term dependencies in sequential data, but are notoriously hard to train because the error backpropagated in time either vanishes or explodes at an exponential rate. While a number of works attempt to mitigate this effect through gated recurrent units, skip-connections, parametric constraints and design choices, we propose a novel incremental RNN (iRNN), where hidden state vectors keep track of incremental changes, and as such approximate state-vector increments of Rosenblatt's (1962) continuous-time RNNs. iRNN exhibits identity gradients and is able to account for long-term dependencies (LTD). We show that our method is computationally efficient overcoming overheads of many existing methods that attempt to improve RNN training, while suffering no performance degradation. We demonstrate the utility of our approach with extensive experiments and show competitive performance against standard LSTMs on LTD and other non-LTD tasks.
BibTeX
@inproceedings{kag2020rnns,
  author = {Anil Kag and Ziming Zhang and Venkatesh Saligrama},
  title = {{RNNs Incrementally Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients?}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=rhuEcwieeB},
}
Figure