Multi-Granular Sequence Encoding via Dilated Compositional Units for Reading Comprehension

Yi Tay, Anh Tuan Luu, Siu Cheung HuiEMNLP 2018 · OpenReview

Introduction

Modern Reading Comprehension (RC) models predominantly rely on bidirectional Recurrent Neural Networks (RNNs) like LSTMs or GRUs to encode text. While effective, these sequential processors face significant challenges: computational bottlenecks during training on long documents and "contextual blindness" regarding long-range dependencies. This paper introduces Dilated Compositional Units (DCU), a novel sequence encoder designed to address these limitations by explicitly modeling text at multiple granularities—from individual words to entire paragraphs—through a parallelizable "fold-unfold" mechanism.

The DCU Architecture

The core innovation of the DCU is its "divide-and-conquer" approach to sequence encoding, which avoids the sequential overhead of standard recurrence.

The DCU Encoder architecture, illustrating the Fold and Unfold operations used to capture multi-granular context.

The architecture consists of several key stages:

  1. The Fold Operation: The input sequence is partitioned into blocks of varying sizes (e.g., 1, 2, 4, 10, 25 words). For each block, a "Neural Bag-of-Words" (NBOW) representation is created by summing word embeddings and applying a feed-forward layer.
  2. The Unfold Operation: These compressed representations are projected back to the original sequence length, ensuring every word token has access to a summary of its surrounding phrase, sentence, or paragraph.
  3. Multi-Granular Reasoning: Summaries from different scales are concatenated and processed by a two-layer network to produce a gating vector. This gating mechanism allows the model to dynamically determine which contextual scales are most relevant for a given word.

The authors propose three variants: Sim-DCU (a simple gated approach), Recurrent DCU (incorporating multi-granular gates into a lightweight recurrent cell), and DCU-LSTM (a hybrid stacking a DCU layer on a BiLSTM).

Experimental Evaluation

The DCU was evaluated on three challenging RC benchmarks:

  • RACE: English exam questions requiring complex, multi-sentence reasoning.
  • SearchQA: A large-scale dataset involving snippets from search engine results.
  • NarrativeQA: A dataset based on full-story summaries with human-written answers.

Key Results and Efficiency

The DCU demonstrated significant performance gains and efficiency improvements over traditional models. On the RACE-H (High School) subset, the DCU achieved a 6% improvement over standard LSTM/GRU encoders.

Performance comparison on the RACE dataset, showing DCU variants achieving state-of-the-art results with significantly reduced training times.

As shown in the results table, the DCU outperformed the complex Dynamic Fusion Network (DFN). Crucially, while the DFN required approximately one week to train, the DCU model converged in less than 12 hours. Each training epoch for the DCU took only 4–5 minutes, compared to several hours for the DFN.

In span-prediction tasks like SearchQA, the model matched the accuracy of the state-of-the-art AMANDA model while being four times more efficient. The hybrid DCU-LSTM performed best in these scenarios, suggesting that DCUs serve as excellent complementary building blocks for existing architectures where precise word positioning is critical.

Conclusion

The Dilated Compositional Unit offers a powerful alternative to standard RNNs for reading comprehension. By explicitly calculating multi-scale context and utilizing efficient parallelization, the DCU achieves superior results with a fraction of the computational cost. While simpler versions like Sim-DCU excel at multiple-choice reasoning, hybrid models remain effective for tasks requiring fine-grained sequence modeling. Future work will explore the application of DCU encoders to other NLP domains such as machine translation and summarization.

Summary updated

BibTeX
@inproceedings{tay2018multi,
  author = {Yi Tay and Anh Tuan Luu and Siu Cheung Hui},
  title = {{Multi-Granular Sequence Encoding via Dilated Compositional Units for Reading Comprehension}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=SyNmRZGdbS},
}
Figure