Lattice Recurrent Unit: Improving Convergence and Statistical Efficiency for Sequence Modeling

Chaitanya Ahuja, Louis-Philippe Morency

Abstract

Recurrent neural networks have shown remarkable success in modeling sequences. However low resource situations still adversely affect the generalizability of these models. We introduce a new family of models, called Lattice Recurrent Units (LRU), to address the challenge of learning deep multi-layer recurrent models with limited resources. LRU models achieve this goal by creating distinct (but coupled) flow of information inside the units: a first flow along time dimension and a second flow along depth dimension. It also offers a symmetry in how information can flow horizontally and vertically. We analyze the effects of decoupling three different components of our LRU model: Reset Gate, Update Gate and Projected State. We evaluate this family on new LRU models on computational convergence rates and statistical efficiency. Our experiments are performed on four publicly-available datasets, comparing with Grid-LSTM and Recurrent Highway networks. Our results show that LRU has better empirical computational convergence rates and statistical efficiency values, along with learning more accurate language models.
BibTeX
@inproceedings{ahuja2018lattice,
  author = {Chaitanya Ahuja and Louis-Philippe Morency},
  title = {{Lattice Recurrent Unit: Improving Convergence and Statistical Efficiency for Sequence Modeling}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.12025},
}
Figure