Introduction
Sequence modeling has long been dominated by the Transformer architecture due to its superior performance and parallel training capabilities. However, Transformers suffer from quadratic ($O(n^2)$) time and memory complexity, which limits their application for extremely long sequences. While Recurrent Neural Networks (RNNs) offer a linear ($O(n)$) alternative, they traditionally face challenges with sequential training bottlenecks and vanishing gradients.
The Hierarchically Gated Recurrent Neural Network (HGRN) addresses these limitations by introducing a novel linear RNN architecture. By incorporating learnable, layer-dependent lower bounds on forget gates, HGRN effectively captures both local and long-range dependencies while maintaining the efficiency of $O(n)$ complexity and parallel training speeds.
Methodology
HGRN revitalizes the concept of the forget gate—a staple of LSTMs—for the modern parallel computing era through three primary innovations.
1. Element-wise Linear Recurrence (ELR)
To overcome the sequential training bottleneck, HGRN utilizes element-wise linear recurrence. Unlike traditional RNNs that rely on full matrix multiplications for hidden state updates, HGRN uses element-wise products. This simplification allows the model to leverage parallel scan algorithms, enabling training speeds on GPUs that are competitive with Transformers.
2. Hierarchical Gating
The core innovation of HGRN is its hierarchical gating mechanism. The forget gate ($\lambda_t$) is lower-bounded by a learnable value $\gamma^k$ specific to each layer $k$:
$$\lambda_t = \gamma^k + (1 - \gamma^k) \odot \mu_t$$
These lower bounds are designed to increase monotonically across layers. In lower layers, smaller bounds allow the model to "forget" more frequently, focusing on local patterns. In upper layers, bounds approach 1, forcing the retention of information to model global, long-term dependencies. This hierarchy prevents gate saturation and smooths the optimization process.
3. Complex-Valued Recurrence
To enhance expressiveness, HGRN operates in the complex domain. It incorporates a data-independent phase argument (rotation angle $\theta$), which functions similarly to Rotary Positional Embeddings (RoPE) by encoding relative positional information directly into the recurrence.
Experimental Results
The HGRN model was evaluated across diverse benchmarks, including language modeling (Wikitext-103, The Pile), the Long Range Arena (LRA), and image classification (ImageNet-1k).
Language Modeling Performance
On the Wikitext-103 dataset, HGRN achieved a perplexity (PPL) of 24.14, outperforming the standard Transformer (24.40) and other modern RNN variants such as RWKV (24.31) and LRU (29.86).
Scaling and Long Context
When scaled to 1 billion parameters on "The Pile," HGRN demonstrated significant gains, reaching a PPL of 4.14, compared to 4.56 for a Transformer of similar scale. Furthermore, on the LRA benchmark—designed to test context lengths up to 16k tokens—HGRN achieved a competitive average score of 86.91, validating its ability to handle long-range dependencies efficiently.
Conclusion
HGRN demonstrates that with a well-designed gating mechanism, linear RNNs can match or exceed the performance of Transformers while offering superior efficiency. By mimicking the human cognitive process of hierarchical information processing—focusing on local details at lower levels and global context at higher levels—HGRN provides a scalable and powerful alternative for modern sequence modeling tasks. Future work will focus on scaling the architecture to the size of frontier large language models and exploring its performance in multi-modal domains like audio and video.
Summary updated