Lattice Recurrent Unit: Improving Convergence and Statistical Efficiency for Sequence Modeling

Chaitanya Ahuja, Louis-Philippe Morency

Introduction

Recurrent Neural Networks (RNNs) are theoretically capable of processing sequences of arbitrary length, yet they remain notoriously difficult to optimize in practice. Two primary hurdles have historically limited their effectiveness: the Depth Challenge and Low-Resource Scenarios. As models are stacked to increase depth, gradients often vanish or explode along the vertical dimension, making it difficult for the network to learn hierarchical features. Furthermore, standard architectures like LSTMs and GRUs typically require massive datasets to generalize effectively, often failing to converge to optimal solutions when data is scarce.

To address these limitations, we introduce the Lattice Recurrent Unit (LRU). The LRU is a new family of recurrent architectures designed to decouple the flow of information along the time and depth dimensions. By creating a symmetric "lattice" structure, the LRU achieves faster convergence and superior statistical efficiency compared to traditional gated units.

The Lattice Recurrent Unit

The core innovation of the LRU is the systematic decoupling of internal components. In a standard stacked GRU, the hidden state is pushed both "forward" in time and "upward" in depth simultaneously, forcing a single representation to serve two distinct purposes. The LRU architecture breaks this constraint, allowing information to flow symmetrically across both axes.

Lattice structure of the LRU showing information flow across time and depth.

As shown in the lattice diagram, the model treats the vertical (depth) and horizontal (time) dimensions with equal priority. This symmetry ensures that gradients remain "healthy" even in very deep networks, as there is always a clear path for information to propagate without being bottlenecked by a single shared state.

Architectural Innovations

The LRU family introduces three variants based on the degree of decoupling:

  1. PS-LRU (Projected State): Decouples the candidate hidden state ($\hat{h}$), providing unique projections for the temporal and depth dimensions.
  2. RG-LRU (Reset Gate): Decouples the Reset Gates ($r$), allowing the model to independently "forget" information based on whether it is processing a sequence step or a hierarchical layer.
  3. UG-LRU (Update Gate / Full LRU): Decouples all three components, including the Update Gate ($z$), for maximum flexibility.
Architectural comparison between (a) GRU, (b) LRU, and (c) GridLSTM.

Compared to the standard GRU (a) and the more complex GridLSTM (c), the LRU (b) provides a more balanced and flexible gating mechanism. While GridLSTM attempted a similar two-dimensional flow, it often lacked the symmetry required for efficient cross-dimensional information passing.

Experimental Evaluation

We evaluated the LRU family on character-level language modeling tasks across four datasets:

  • Low-Resource Benchmarks: Penn Treebank (PTB) and War and Peace (WP), containing roughly 5 million characters each.
  • Large-Scale Benchmarks: enwik8 and text8, containing 100 million characters from Wikipedia, to test scalability in data-rich environments.

In all experiments, we maintained a fixed parameter budget (e.g., 10M or 24M parameters) to ensure fair comparisons between architectures.

Key Results and Insights

Superior Accuracy and Convergence

The LRU consistently outperformed standard GRUs and Grid-LSTMs. On the PTB dataset with a 10M parameter budget, the LRU achieved a Categorical Cross Entropy (CCE) loss of 0.965, surpassing the GRU (0.972) and Grid-LSTM (0.970).

More importantly, the LRU demonstrated remarkable convergence speed. On average, LRU variants reached their peak performance in just 12 epochs, whereas LSTMs and GRUs typically required between 22 and 30 epochs. This stability in training reduces the computational overhead and variance often associated with training deep recurrent models.

Statistical Efficiency

A standout finding of this research is the "statistical efficiency" of the LRU. Our results indicate that an LRU trained on only 40% of a dataset can outperform competing models trained on 60% of the same data. This makes the LRU an ideal candidate for niche domains—such as medical or legal text processing—where high-quality labeled data is expensive or difficult to acquire.

Gradient Health

Visualization of gradient norms confirmed that the LRU effectively solves the vanishing gradient problem in the depth dimension. While GRUs often suffer from "dead" gradients in deeper layers, the LRU maintains robust gradient flow across all layers, proving its architectural suitability for very deep sequence modeling.

Conclusion

The Lattice Recurrent Unit represents a significant step forward in making deep recurrent models more practical and efficient. By decoupling temporal and hierarchical information flows, the LRU offers a symmetric, stable, and data-efficient alternative to traditional RNNs. Future work will explore the application of LRUs in multi-modal tasks, where the distinct dimensions of the lattice could represent different data modalities, such as synchronized audio and video streams.

Summary updated

Abstract

Recurrent neural networks have shown remarkable success in modeling sequences. However low resource situations still adversely affect the generalizability of these models. We introduce a new family of models, called Lattice Recurrent Units (LRU), to address the challenge of learning deep multi-layer recurrent models with limited resources. LRU models achieve this goal by creating distinct (but coupled) flow of information inside the units: a first flow along time dimension and a second flow along depth dimension. It also offers a symmetry in how information can flow horizontally and vertically. We analyze the effects of decoupling three different components of our LRU model: Reset Gate, Update Gate and Projected State. We evaluate this family on new LRU models on computational convergence rates and statistical efficiency. Our experiments are performed on four publicly-available datasets, comparing with Grid-LSTM and Recurrent Highway networks. Our results show that LRU has better empirical computational convergence rates and statistical efficiency values, along with learning more accurate language models.
BibTeX
@inproceedings{ahuja2018lattice,
  author = {Chaitanya Ahuja and Louis-Philippe Morency},
  title = {{Lattice Recurrent Unit: Improving Convergence and Statistical Efficiency for Sequence Modeling}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.12025},
}
Figure