Hierarchically Gated Recurrent Neural Network for Sequence Modeling

Zhen Qin, Songlin Yang, Yiran Zhong

Introduction

Sequence modeling has long been dominated by the Transformer architecture due to its superior performance and parallel training capabilities. However, Transformers suffer from quadratic ($O(n^2)$) time and memory complexity, which limits their application for extremely long sequences. While Recurrent Neural Networks (RNNs) offer a linear ($O(n)$) alternative, they traditionally face challenges with sequential training bottlenecks and vanishing gradients.

The Hierarchically Gated Recurrent Neural Network (HGRN) addresses these limitations by introducing a novel linear RNN architecture. By incorporating learnable, layer-dependent lower bounds on forget gates, HGRN effectively captures both local and long-range dependencies while maintaining the efficiency of $O(n)$ complexity and parallel training speeds.

Methodology

HGRN revitalizes the concept of the forget gate—a staple of LSTMs—for the modern parallel computing era through three primary innovations.

1. Element-wise Linear Recurrence (ELR)

To overcome the sequential training bottleneck, HGRN utilizes element-wise linear recurrence. Unlike traditional RNNs that rely on full matrix multiplications for hidden state updates, HGRN uses element-wise products. This simplification allows the model to leverage parallel scan algorithms, enabling training speeds on GPUs that are competitive with Transformers.

2. Hierarchical Gating

The core innovation of HGRN is its hierarchical gating mechanism. The forget gate ($\lambda_t$) is lower-bounded by a learnable value $\gamma^k$ specific to each layer $k$:

$$\lambda_t = \gamma^k + (1 - \gamma^k) \odot \mu_t$$

These lower bounds are designed to increase monotonically across layers. In lower layers, smaller bounds allow the model to "forget" more frequently, focusing on local patterns. In upper layers, bounds approach 1, forcing the retention of information to model global, long-term dependencies. This hierarchy prevents gate saturation and smooths the optimization process.

The HGRN architecture consists of stacked layers with HGRU (token mixing) and GLU (channel mixing) modules.

3. Complex-Valued Recurrence

To enhance expressiveness, HGRN operates in the complex domain. It incorporates a data-independent phase argument (rotation angle $\theta$), which functions similarly to Rotary Positional Embeddings (RoPE) by encoding relative positional information directly into the recurrence.

Experimental Results

The HGRN model was evaluated across diverse benchmarks, including language modeling (Wikitext-103, The Pile), the Long Range Arena (LRA), and image classification (ImageNet-1k).

Language Modeling Performance

On the Wikitext-103 dataset, HGRN achieved a perplexity (PPL) of 24.14, outperforming the standard Transformer (24.40) and other modern RNN variants such as RWKV (24.31) and LRU (29.86).

Comparison of HGRN against attention-based, MLP-based, and other RNN-based models on Wikitext-103.

Scaling and Long Context

When scaled to 1 billion parameters on "The Pile," HGRN demonstrated significant gains, reaching a PPL of 4.14, compared to 4.56 for a Transformer of similar scale. Furthermore, on the LRA benchmark—designed to test context lengths up to 16k tokens—HGRN achieved a competitive average score of 86.91, validating its ability to handle long-range dependencies efficiently.

Conclusion

HGRN demonstrates that with a well-designed gating mechanism, linear RNNs can match or exceed the performance of Transformers while offering superior efficiency. By mimicking the human cognitive process of hierarchical information processing—focusing on local details at lower levels and global context at higher levels—HGRN provides a scalable and powerful alternative for modern sequence modeling tasks. Future work will focus on scaling the architecture to the size of frontier large language models and exploring its performance in multi-modal domains like audio and video.

Summary updated

BibTeX
@inproceedings{qin2023hierarchically,
  author = {Zhen Qin and Songlin Yang and Yiran Zhong},
  title = {{Hierarchically Gated Recurrent Neural Network for Sequence Modeling}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=P1TCHxJwLB},
}
Figure