Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations

Behnam Neyshabur, Yuhuai Wu, Ruslan Salakhutdinov, Nathan Srebro

Abstract

This paper introduces a "path-normalized" optimization method for simple Recurrent Neural Networks (RNNs) with ReLU activations. By making the optimization process invariant to weight rescaling, the authors enable "plain" RNNs to capture long-term dependencies that usually require much more complex architectures like LSTMs.

Problem Statement: The Challenge of Training RNNs

Recurrent Neural Networks are notoriously difficult to train because of the vanishing and exploding gradient problems. When signals pass through many time steps, gradients can either shrink to zero or grow exponentially, hindering learning or causing instability.

To mitigate this, researchers typically employ complex "gated" architectures like LSTMs or GRUs. While effective, these models are computationally expensive. The authors argue that the issue lies not in the "plain" RNN model itself, but in the optimization method. Standard Stochastic Gradient Descent (SGD) is tied to Euclidean geometry, which fails to respect the internal structure of neural networks—specifically, the fact that weights can be rescaled across layers without changing the network's output.

Proposed Method: Path-SGD for RNNs

The authors extend Path-SGD to networks with shared weights, such as RNNs, focusing on several key components:

  1. Path-Norm: Instead of individual weights, the optimization considers the "path-norm"—the sum of the products of squared weights along every possible path from input to output.
  2. Geometry-Aware Optimization: They derive an update rule that acts as a steepest descent step with respect to this path-norm. This makes the optimizer invariant to node-wise rescaling.
Figure 1: An example of invariances in an RNN with two hidden layers. The network on the left is equivalent to the network on the right for any nonzero scaling factors a, b, c, d.
  1. Handling Shared Weights: In an RNN, the same weight matrix is used at every time step. The authors decompose the path-norm gradient into two terms: $\kappa^{(1)}$ (direct effects) and $\kappa^{(2)}$ (interactions between shared weights on the same path).
  2. Efficiency: Despite the exponential number of paths, updates are calculated efficiently using modified forward and backward passes.

Key Insights and Experimental Results

The researchers evaluated Path-SGD on challenging benchmarks including the Addition Problem, Sequential MNIST, and language modeling (PTB and text8).

Invariance and Stability

By being "blind" to the scale of weights, Path-SGD prevents the exponential explosion of gradients that typically plagues ReLU RNNs. A significant finding was that the complex interaction term ($\kappa^{(2)}$) is often negligible. Using just the simpler $\kappa^{(1)}$ term provides nearly identical performance at a lower computational cost, as shown in the training and test error curves.

Figure 6: Training and test perplexity on language modeling tasks. Path-SGD variants significantly outperform standard SGD, with the first-order term capturing most of the benefit.

Performance Benchmarks

  • Long-Term Memory: On the Addition Problem ($T=750$), Path-SGD achieved 0% error, while LSTMs and other specialized RNNs failed.
  • Sequential MNIST: Path-SGD reached a 3.1% test error, outperforming other plain RNN methods like IRNN (5.0%).
  • Language Modeling: On PTB and text8, Path-SGD closed over 50% of the performance gap between vanilla RNNs and LSTMs. On PTB, it improved Bits-Per-Character (BPC) from 1.55 to 1.47.

Limitations and Future Outlook

While Path-SGD shows promise, it has limitations. Plain RNNs still failed the addition problem at $T=1000$. Additionally, for very large hidden layers, Path-SGD can be slow to converge, requiring integration with optimizers like Adam. Future work may involve applying path-normalization to LSTMs or GRUs to further enhance their performance.

Summary updated

BibTeX
@inproceedings{neyshabur2016path,
  author = {Behnam Neyshabur and Yuhuai Wu and Ruslan Salakhutdinov and Nathan Srebro},
  title = {{Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2016},
  url = {https://openreview.net/forum?id=SkbZiw-u-S},
}
Figure