Introduction
In the deep learning community, it is common knowledge that gated architectures like Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) generally outperform "vanilla" Recurrent Neural Networks (RNNs). However, the underlying reasons for this performance gap have remained somewhat ambiguous. Is it because gating mechanisms allow the network to represent more complex functions (higher capacity), or is it because they create a smoother optimization landscape (better trainability)?
This paper disentangles these two factors, revealing a surprising truth: most popular RNN architectures have nearly identical information capacities when optimized thoroughly. The perceived superiority of gated models stems almost entirely from their trainability, rather than an inherent advantage in expressiveness.
Disentangling Capacity and Trainability
To ensure a fair comparison, the researchers utilized a massive hyperparameter optimization process based on Gaussian Processes. By running thousands of full training runs for each architecture, they mitigated the risk of attributing poor performance to suboptimal learning rates or initializations.
A critical methodological choice was parameter matching. Instead of comparing models with the same number of hidden units—which gives gated models an unfair advantage due to their higher parameter count per unit—the authors matched models by their total number of parameters.
The study also introduced two novel architectures:
- UGRNN: A "minimally gated" RNN.
- +RNN (Intersection RNN): A model that uses coupled gates to manage both depth and recurrence.
To measure capacity, the authors evaluated "bits per parameter" using a synthetic memorization task, and "per-unit memory" by testing how well a model could reconstruct a random input vector after several time steps.
Key Insights
The "5-Bit" Constant and Linear Scaling
Across almost all architectures and depths, RNNs demonstrate a fundamental empirical limit: they can store approximately 5 bits of task information per parameter. Task capacity scales linearly with the number of parameters across several orders of magnitude. This suggests a hard limit on how much information can be packed into a neural network weight using current optimization techniques.
The Gating Tax
Interestingly, the vanilla RNN actually possesses a slightly higher raw capacity than the LSTM or GRU. Gating mechanisms consume a portion of the "parameter budget" for control logic rather than raw data storage. However, this slight reduction in capacity is a worthwhile "tax" to pay for the massive gains in trainability that gates provide.
The Hidden Unit Trap
Historically, researchers have compared RNNs by matching the number of hidden units. Because an LSTM has roughly four times as many parameters per unit as a vanilla RNN, it naturally performs better in such comparisons. When architectures are matched by total parameter count, the performance gap on standard tasks like language modeling (e.g., text8) almost entirely disappears.
Trainability is the Real Bottleneck
On difficult, "hard-to-learn" tasks—such as counting nested parentheses or performing arithmetic from character strings—vanilla RNNs often failed to converge entirely, whereas gated models succeeded. This confirms that the true advantage of LSTMs and GRUs lies in their optimization landscape. Furthermore, for very deep networks (8+ layers), the authors' novel +RNN architecture proved to be the most trainable, outperforming even the GRU and LSTM in convergence reliability.
Additional Findings and Future Work
- Input Memory: All tested RNNs can store roughly one real number from their input history per hidden unit, regardless of the specific architecture.
- Activation Functions: While ReLU is highly popular in feed-forward networks, the authors found that it significantly reduced the capacity of RNNs in memorization tasks compared to the Tanh activation function.
While the empirical limit of ~5 bits per parameter is robust across these experiments, the authors note the absence of a formal theoretical proof for this specific bound. Future work could explore whether these capacity bounds hold for other sequence models, such as Transformers, or if different precision formats (like FP16 or Int8) alter the bits-per-parameter efficiency.
Summary updated