Lattice Recurrent Unit: Improving Convergence and Statistical Efficiency for Sequence Modeling

Chaitanya Ahuja, Louis-Philippe Morency

Abstract

Recurrent neural networks have shown remarkable success in modeling sequences. However low resource situations still adversely affect the generalizability of these models. We introduce a new family of models, called Lattice Recurrent Units (LRU), to address the challenge of learning deep multi-layer recurrent models with limited resources. LRU models achieve this goal by creating distinct (but coupled) flow of information inside the units: a first flow along time dimension and a second flow along depth dimension. It also offers a symmetry in how information can flow horizontally and vertically. We analyze the effects of decoupling three different components of our LRU model: Reset Gate, Update Gate and Projected State. We evaluate this family on new LRU models on computational convergence rates and statistical efficiency. Our experiments are performed on four publicly-available datasets, comparing with Grid-LSTM and Recurrent Highway networks. Our results show that LRU has better empirical computational convergence rates and statistical efficiency values, along with learning more accurate language models.

Introduction

Recurrent Neural Networks (RNNs) are fundamental for sequential data processing. However, they face significant challenges when scaled in depth. While stacking layers theoretically enables more complex representations, it often leads to optimization difficulties, such as vanishing or exploding gradients across both time and depth. Existing solutions like Grid-LSTMs or Recurrent Highway Networks (RHNs) attempt to mitigate these issues but often underperform in low-resource settings. Standard stacked architectures force a single hidden state to manage both temporal and hierarchical transformations, limiting the model's capacity to learn distinct features for each dimension.

Methods: The Lattice Architecture

The Lattice Recurrent Unit (LRU) addresses these limitations by decoupling information flow. Unlike standard GRUs, the LRU treats "depth" with the same mathematical priority as "time," creating a symmetric lattice structure.

The Lattice Recurrent Unit (LRU) architecture showing symmetric information flow along time and depth dimensions.

The core innovation lies in the systematic decoupling of GRU components across dimensions:

  1. Projected State LRU (PS-LRU): Decouples the candidate hidden state ($\hat{h}$) for each dimension.
  2. Reset Gate LRU (RG-LRU): Decouples the reset gates ($r$), allowing independent "forgetting" logic for temporal vs. depth-wise information.
  3. Update Gate LRU (LRU): The full model, where update gates ($z$) are also decoupled, ensuring dedicated channels for information flow.

Experimental Setup

The models were evaluated on character-level language modeling tasks using four datasets: Penn Treebank (PTB), War and Peace (WP), and the large-scale enwik8 and text8 datasets. These benchmarks allow for testing across small-scale, long-range dependency, and large-scale Wikipedia-derived data.

Results and Discussion

Our experiments demonstrate that decoupling is a significant advantage. As the model moves from a standard GRU to the fully decoupled LRU, performance consistently improves across all metrics.

Faster Convergence and Accuracy

LRUs achieve peak performance significantly faster than LSTMs or Grid-LSTMs. On the PTB dataset, the LRU converged in approximately 8 epochs, compared to 53 epochs for standard LSTMs. Furthermore, LRUs achieved lower Categorical Cross Entropy (CCE) loss across tasks. For instance, on a 10M parameter PTB task, the LRU reached a loss of 0.965, outperforming both the standard GRU (0.972) and Grid-LSTM (0.970).

Comparison of test loss convergence across different LRU variants and the standard GRU.

Statistical Efficiency and Gradient Health

A key finding is the superior statistical efficiency of LRUs. An LRU trained on 40% of a dataset often outperforms standard models trained on 60%, making it ideal for data-scarce domains. Analysis of gradient norms indicates that while gradients in standard GRUs often vanish before reaching lower layers, LRUs maintain "healthy" gradient flow across the entire depth of the network, enabling effective deep learning.

Conclusion

The Lattice Recurrent Unit offers a robust alternative for deep sequence modeling, particularly when computational resources or data are limited. While this study focused on character-level language modeling without standard regularization techniques like dropout, the results suggest that the LRU's symmetric, decoupled architecture is a powerful tool for building deep recurrent models. Future work will explore its application in broader domains such as machine translation and speech recognition.

Summary updated

Abstract

Recurrent neural networks have shown remarkable success in modeling sequences. However low resource situations still adversely affect the generalizability of these models. We introduce a new family of models, called Lattice Recurrent Units (LRU), to address the challenge of learning deep multi-layer recurrent models with limited resources. LRU models achieve this goal by creating distinct (but coupled) flow of information inside the units: a first flow along time dimension and a second flow along depth dimension. It also offers a symmetry in how information can flow horizontally and vertically. We analyze the effects of decoupling three different components of our LRU model: Reset Gate, Update Gate and Projected State. We evaluate this family on new LRU models on computational convergence rates and statistical efficiency. Our experiments are performed on four publicly-available datasets, comparing with Grid-LSTM and Recurrent Highway networks. Our results show that LRU has better empirical computational convergence rates and statistical efficiency values, along with learning more accurate language models.
BibTeX
@inproceedings{ahuja2018lattice,
  author = {Chaitanya Ahuja and Louis-Philippe Morency},
  title = {{Lattice Recurrent Unit: Improving Convergence and Statistical Efficiency for Sequence Modeling}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.12025},
}
Figure