Abstract
This paper introduces a "path-normalized" optimization method for simple Recurrent Neural Networks (RNNs) with ReLU activations. By making the optimization process invariant to weight rescaling, the authors enable "plain" RNNs to capture long-term dependencies that usually require much more complex architectures like LSTMs.
Problem Statement: The Challenge of Training RNNs
Recurrent Neural Networks are notoriously difficult to train because of the vanishing and exploding gradient problems. When signals pass through many time steps, gradients can either shrink to zero or grow exponentially, hindering learning or causing instability.
To mitigate this, researchers typically employ complex "gated" architectures like LSTMs or GRUs. While effective, these models are computationally expensive. The authors argue that the issue lies not in the "plain" RNN model itself, but in the optimization method. Standard Stochastic Gradient Descent (SGD) is tied to Euclidean geometry, which fails to respect the internal structure of neural networks—specifically, the fact that weights can be rescaled across layers without changing the network's output.
Proposed Method: Path-SGD for RNNs
The authors extend Path-SGD to networks with shared weights, such as RNNs, focusing on several key components:
- Path-Norm: Instead of individual weights, the optimization considers the "path-norm"—the sum of the products of squared weights along every possible path from input to output.
- Geometry-Aware Optimization: They derive an update rule that acts as a steepest descent step with respect to this path-norm. This makes the optimizer invariant to node-wise rescaling.
- Handling Shared Weights: In an RNN, the same weight matrix is used at every time step. The authors decompose the path-norm gradient into two terms: $\kappa^{(1)}$ (direct effects) and $\kappa^{(2)}$ (interactions between shared weights on the same path).
- Efficiency: Despite the exponential number of paths, updates are calculated efficiently using modified forward and backward passes.
Key Insights and Experimental Results
The researchers evaluated Path-SGD on challenging benchmarks including the Addition Problem, Sequential MNIST, and language modeling (PTB and text8).
Invariance and Stability
By being "blind" to the scale of weights, Path-SGD prevents the exponential explosion of gradients that typically plagues ReLU RNNs. A significant finding was that the complex interaction term ($\kappa^{(2)}$) is often negligible. Using just the simpler $\kappa^{(1)}$ term provides nearly identical performance at a lower computational cost, as shown in the training and test error curves.
Performance Benchmarks
- Long-Term Memory: On the Addition Problem ($T=750$), Path-SGD achieved 0% error, while LSTMs and other specialized RNNs failed.
- Sequential MNIST: Path-SGD reached a 3.1% test error, outperforming other plain RNN methods like IRNN (5.0%).
- Language Modeling: On PTB and text8, Path-SGD closed over 50% of the performance gap between vanilla RNNs and LSTMs. On PTB, it improved Bits-Per-Character (BPC) from 1.55 to 1.47.
Limitations and Future Outlook
While Path-SGD shows promise, it has limitations. Plain RNNs still failed the addition problem at $T=1000$. Additionally, for very large hidden layers, Path-SGD can be slow to converge, requiring integration with optimizers like Adam. Future work may involve applying path-normalization to LSTMs or GRUs to further enhance their performance.
Summary updated