Training Recurrent Neural Networks via Forward Propagation Through Time

Anil Kag, Venkatesh Saligrama

This paper introduces a significant shift in how we train Recurrent Neural Networks (RNNs). By moving away from the standard "backwards" approach, the authors demonstrate that we can overcome long-standing issues like gradient instability and high memory usage.

TL;DR

Forward Propagation Through Time (FPTT) is a novel training algorithm that updates RNN parameters at every timestep using a dynamically regularized "instantaneous" loss. It eliminates the need to store long histories of hidden states, solves the vanishing/exploding gradient problem, and allows vanilla LSTMs to outperform much more complex architectures on long-range dependency tasks.

What Problem Does This Paper Address?

The standard way to train RNNs is Back-propagation Through Time (BPTT). While effective, BPTT has two major flaws:

  1. Gradient Instability: As you back-propagate errors through long sequences, gradients tend to either vanish (become zero) or explode (become infinite). This makes it nearly impossible for an RNN to learn "long-range dependencies"—relationships between data points far apart in time.
  2. Memory and Complexity: BPTT requires storing every intermediate hidden state for the entire sequence length ($T$) to calculate the gradient. This leads to $O(T)$ memory overhead and high computational costs for long sequences.

While researchers have tried to fix this by designing complex new RNN architectures, these models are still usually trained with BPTT. This paper argues that the problem isn't necessarily the architecture, but the training algorithm itself.

Methods

The authors propose Forward Propagation Through Time (FPTT). Instead of waiting until the end of a sequence to go backward, FPTT updates the model parameters at every timestep $t$.

Key components include:

  • Instantaneous Risk Function: At each step, the model optimizes a local loss that combines the current error with a dynamic regularizer.
  • Dynamic Regularization: Inspired by consensus methods in distributed optimization, the regularizer uses a state vector that acts as a running average of parameters. This ensures that the frequent updates remain stable and don't "drift" too far.
  • Mathematical Guarantee: The authors provide a proof showing that as these local updates converge, they reach a stationary point of the global empirical risk function. Solving many small local problems eventually solves the big global problem.
  • Surrogate Losses: For "terminal" tasks (where you only get a label at the very end of a sequence), the authors develop a method to create artificial intermediate losses. This guides the model's learning throughout the sequence rather than just at the end.

Datasets

The researchers evaluated FPTT across several classic and difficult benchmarks:

  • Add Task: A toy problem specifically designed to test long-range memory (finding two numbers in a long sequence of noise and adding them).
  • Sequential MNIST & Permute MNIST: Image classification where pixels are fed to the RNN one by one.
  • Sequential CIFAR-10: A harder version of the pixel-by-pixel task.
  • Penn Tree Bank (PTB): A standard language modeling dataset, tested at both the word level and character level.

Key Insights and Ideas

  • Memory Efficiency: Because FPTT only looks forward, it doesn't need to store the full history of hidden states. Its memory storage is $O(1)$ relative to sequence length, compared to $O(T)$ for BPTT.
Algorithm Complexity Comparison
  • Solving the "Add Task": BPTT famously fails on the Add Task when the sequence length $T$ exceeds 200-500. FPTT solves it easily even at $T=1000$, showing it truly handles long-range dependencies better.
Add Task Performance at T=1000
  • Vanilla is Enough: One of the most striking findings is that "vanilla" LSTMs—which often struggle with BPTT—become incredibly powerful when trained with FPTT. They often beat state-of-the-art, specialized architectures.
  • The $K$ Trade-off: The authors introduced FPTT-K, a version that updates every $K$ steps. This allows practitioners to balance computational speed (lower $K$) with training accuracy (higher $K$).

Key Results

  • Accuracy Gains: On CIFAR-10, a vanilla LSTM trained with FPTT achieved 71.03% accuracy, compared to only 60.11% when trained with BPTT.
  • Language Modeling: On the PTB word-level task, FPTT improved the test perplexity of an LSTM by nearly 11 points (from 117.09 to 106.27), outperforming several recent specialized RNN models.
  • Competitive with Complexity: A simple 1-layer LSTM trained with FPTT was competitive with (and sometimes beat) multi-layered, high-capacity models like TrellisNet and IndRNN on Sequential MNIST tasks.

Limitations and Future Work

  • Hyperparameter Tuning: The algorithm introduces a new hyperparameter, $\alpha$ (the regularization weight). Performance is sensitive to this value; if $\alpha$ is too low, the model diverges; if too high, it learns too slowly.
  • Computational Overhead: While memory-efficient, FPTT performs more arithmetic operations per timestep than a single BPTT pass because it updates weights constantly. The authors suggest using FPTT-K to mitigate this.
  • Architecture Scope: While tested on LSTMs and GRUs, extending FPTT to more complex hierarchical or stacked RNN structures is a promising direction for future research.

Source: Kag, A., & Saligrama, V. (2021). Training Recurrent Neural Networks via Forward Propagation Through Time. Proceedings of the 38th International Conference on Machine Learning (ICML).

Summary updated

Abstract

Back-propagation through time (BPTT) has been widely used for training Recurrent Neural Networks (RNNs). BPTT updates RNN parameters on an instance by back-propagating the error in time over the en...
BibTeX
@inproceedings{kag2021training,
  author = {Anil Kag and Venkatesh Saligrama},
  title = {{Training Recurrent Neural Networks via Forward Propagation Through Time}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=ljIOXlZM0H},
}
Figure