The Lower The Simpler: Simplifying Hierarchical Recurrent Models

Chao Wang, Hui JiangNAACL 2019 · DOI

Hierarchical recurrent models, such as the Hierarchical Recurrent Encoder-Decoder (HRED) for multi-turn dialogue and R-NET for machine reading comprehension, have become staples in natural language processing. However, these architectures are notoriously computationally expensive. This paper, presented at NAACL-HLT 2019, introduces a refreshingly intuitive strategy for optimization: "the lower the simpler."

The Backpropagation Bottleneck

The primary challenge with deep hierarchical recurrent models isn't just the forward pass; it is the backward propagation. Due to the chain rule, the complexity of computing gradients increases significantly as one moves from the top layers down to the bottom. In a deep stack, the bottom layers often become a massive computational sink.

The authors argue that researchers often over-engineer these lower layers, employing heavy-duty units like Gated Recurrent Units (GRUs) where simpler mechanisms might suffice for basic feature extraction. By simplifying the bottom of the hierarchy, one can alleviate the backpropagation bottleneck without sacrificing the model's reasoning capabilities at the top.

Tiered Simplification Strategy

The proposed strategy involves keeping the top layer (usually the decoder or output layer) complex while aggressively simplifying the layers beneath it using two primary mechanisms:

  1. Scalar Gated Unit (SGU): A "diet" version of the standard GRU. While a standard GRU uses vector-based update and reset gates requiring large weight matrices, the SGU reduces these gates to scalars. This replacement swaps expensive element-wise multiplications for simple scalar multiplications, drastically reducing the number of trainable parameters.
  2. Fixed-size Ordinally-Forgetting Encoding (FOFE): For the very bottom layer, the authors replace recurrent units entirely with FOFE. FOFE is a parameter-free encoding method that uses a simple "forgetting factor" to model word order. Because it has no weights to learn, it is incredibly fast and serves as a highly efficient base for the layers above it.

Experimental Results

The researchers evaluated this approach across two distinct domains: dialogue modeling and machine reading comprehension (MRC).

Dialogue Modeling

Using the MovieTriples and Ubuntu datasets, the authors modified HRED by replacing the sentence encoder with FOFE and the dialogue encoder with SGU. On the MovieTriples dataset, the simplified model achieved a 25%–35% reduction in trainable parameters and slashed training time by over 50%.

Performance comparison of Baseline vs. Simplified HRED on the MovieTriples dataset.

As shown in the table above, the simplified HRED not only trained faster but also achieved slightly better (lower) perplexity scores across various embedding sizes, suggesting that the simplification may have provided a beneficial regularizing effect.

Machine Reading Comprehension

For the MRC task using the SQuAD dataset, the authors modified the R-NET architecture. They replaced the embedding BiGRU with BiFOFE and the encoding BiGRU with BiSGU.

Performance comparison of Baseline vs. Simplified R-NET on the SQuAD dataset.

The results on SQuAD mirrored the dialogue tasks: the simplified R-NET reduced parameters by 13% and training time by 21%, while slightly improving the F1 score from 79.5 to 79.7.

Conclusion

The "the lower the simpler" philosophy provides a practical blueprint for training large-scale NLP models on limited hardware budgets. By identifying that the return on investment for complexity diminishes as one goes deeper into a model's hierarchy, the authors demonstrate that you don't need "recurrency all the way down." Parameter-free encodings like FOFE can capture sufficient sequential information at the word level to feed more complex reasoning layers above, resulting in models that are faster, leaner, and often more robust.

Summary updated

BibTeX
@inproceedings{wang2019lower,
  author = {Chao Wang and Hui Jiang},
  title = {{The Lower The Simpler: Simplifying Hierarchical Recurrent Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/n19-1402},
}
Figure