Hierarchical recurrent models, such as the Hierarchical Recurrent Encoder-Decoder (HRED) for multi-turn dialogue and R-NET for machine reading comprehension, have become staples in natural language processing. However, these architectures are notoriously computationally expensive. This paper, presented at NAACL-HLT 2019, introduces a refreshingly intuitive strategy for optimization: "the lower the simpler."
The Backpropagation Bottleneck
The primary challenge with deep hierarchical recurrent models isn't just the forward pass; it is the backward propagation. Due to the chain rule, the complexity of computing gradients increases significantly as one moves from the top layers down to the bottom. In a deep stack, the bottom layers often become a massive computational sink.
The authors argue that researchers often over-engineer these lower layers, employing heavy-duty units like Gated Recurrent Units (GRUs) where simpler mechanisms might suffice for basic feature extraction. By simplifying the bottom of the hierarchy, one can alleviate the backpropagation bottleneck without sacrificing the model's reasoning capabilities at the top.
Tiered Simplification Strategy
The proposed strategy involves keeping the top layer (usually the decoder or output layer) complex while aggressively simplifying the layers beneath it using two primary mechanisms:
- Scalar Gated Unit (SGU): A "diet" version of the standard GRU. While a standard GRU uses vector-based update and reset gates requiring large weight matrices, the SGU reduces these gates to scalars. This replacement swaps expensive element-wise multiplications for simple scalar multiplications, drastically reducing the number of trainable parameters.
- Fixed-size Ordinally-Forgetting Encoding (FOFE): For the very bottom layer, the authors replace recurrent units entirely with FOFE. FOFE is a parameter-free encoding method that uses a simple "forgetting factor" to model word order. Because it has no weights to learn, it is incredibly fast and serves as a highly efficient base for the layers above it.
Experimental Results
The researchers evaluated this approach across two distinct domains: dialogue modeling and machine reading comprehension (MRC).
Dialogue Modeling
Using the MovieTriples and Ubuntu datasets, the authors modified HRED by replacing the sentence encoder with FOFE and the dialogue encoder with SGU. On the MovieTriples dataset, the simplified model achieved a 25%–35% reduction in trainable parameters and slashed training time by over 50%.
As shown in the table above, the simplified HRED not only trained faster but also achieved slightly better (lower) perplexity scores across various embedding sizes, suggesting that the simplification may have provided a beneficial regularizing effect.
Machine Reading Comprehension
For the MRC task using the SQuAD dataset, the authors modified the R-NET architecture. They replaced the embedding BiGRU with BiFOFE and the encoding BiGRU with BiSGU.
The results on SQuAD mirrored the dialogue tasks: the simplified R-NET reduced parameters by 13% and training time by 21%, while slightly improving the F1 score from 79.5 to 79.7.
Conclusion
The "the lower the simpler" philosophy provides a practical blueprint for training large-scale NLP models on limited hardware budgets. By identifying that the return on investment for complexity diminishes as one goes deeper into a model's hierarchy, the authors demonstrate that you don't need "recurrency all the way down." Parameter-free encodings like FOFE can capture sufficient sequential information at the word level to feed more complex reasoning layers above, resulting in models that are faster, leaner, and often more robust.
Summary updated