Abstract
We propose MRU (Multi-Range Reasoning Units), a new fast compositional encoder for machine comprehension (MC). Our proposed MRU encoders are characterized by multi-ranged gating, executing a series of parameterized contract-and-expand layers for learning gating vectors that benefit from long and short-term dependencies. The aims of our approach are as follows: (1) learning representations that are concurrently aware of long and short-term context, (2) modeling relationships between intra-document blocks and (3) fast and efficient sequence encoding. We show that our proposed encoder demonstrates promising results both as a standalone encoder and as well as a complementary building block. We conduct extensive experiments on three challenging MC datasets, namely RACE, SearchQA and NarrativeQA, achieving highly competitive performance on all. On the RACE benchmark, our model outperforms DFN (Dynamic Fusion Networks) by 1.5%-6% without using any recurrent or convolution layers. Similarly, we achieve competitive performance relative to AMANDA on the SearchQA benchmark and BiDAF on the NarrativeQA benchmark without using any LSTM/GRU layers. Finally, incorporating MRU encoders with standard BiLSTM architectures further improves performance, achieving state-of-the-art results.
Introduction
Machine Comprehension (MC) tasks require models to process extensive passages to answer complex questions. Traditionally, Bidirectional Recurrent Neural Networks (RNNs), such as LSTMs or GRUs, have been the standard for encoding these sequences. However, RNNs suffer from two primary limitations: computational bottlenecks due to their sequential nature, which prevents parallelization, and a limited "contextual vision" that makes it difficult to capture long-range dependencies across distant sentences in a single layer.
To address these challenges, we introduce Multi-Range Reasoning Units (MRU). MRU is a novel compositional encoder designed to replace or augment traditional RNNs. By utilizing a "contract-and-expand" mechanism, MRU can simultaneously analyze text across multiple window sizes, enabling it to capture both local and global context efficiently.
Methods: The MRU Architecture
The core of our approach is the MRU Encoder, which processes input sequences through a series of parameterized layers designed for multi-range reasoning.
Contract-and-Expand Operation
The MRU encoder operates through two main phases:
- Contraction: The input sequence is compressed $k$ times using various "ranges" or window sizes (e.g., 1, 2, 4, 10, and 25 words). This process creates a "neural bag-of-words" for each window, effectively summarizing local information at different scales.
- Expansion: These compressed representations are passed through a linear layer and then expanded back to the original sequence length.
Multi-range Gating
After expansion, the model concatenates these different views. A small neural network then learns a gating vector, which determines the relative influence of each hierarchy (e.g., the 1-word view vs. the 25-word view) on the final representation of each word. This allows the model to dynamically focus on the most relevant context range for a given token.
We propose two variants:
- Simple MRU: A non-recurrent version using highway-network style gates. It is highly parallelizable and extremely fast.
- Recurrent MRU: Inspired by Quasi-Recurrent Neural Networks (QRNNs), this version includes a lightweight recurrence for tasks where strict word order is critical, such as span prediction.
Experimental Setup
We evaluated the MRU encoder on three diverse benchmarks:
- RACE: Middle and High school English exams requiring complex multi-sentence reasoning.
- SearchQA: A large-scale dataset using snippets from search engines.
- NarrativeQA: Based on full stories and movie scripts, requiring human-written answers rather than simple span extraction.
Key Results
Our experiments demonstrate that MRU encoders are not only faster but often more accurate than traditional sequential models.
Performance on RACE
On the RACE dataset, the MRU model outperformed the previous state-of-the-art Dynamic Fusion Network (DFN) by 1.5% to 6%. Most notably, the MRU model achieved these results with a dramatic reduction in training time. While DFN required approximately one week to train, the MRU model converged in under 12 hours.
SearchQA and NarrativeQA
In SearchQA, the MRU model achieved performance competitive with the AMANDA model while running five times faster. For NarrativeQA, we found that a hybrid MRU-LSTM architecture—stacking an MRU layer on top of a BiLSTM—yielded state-of-the-art results. This suggests that multi-range gating and sequential modeling capture complementary linguistic features.
Discussion and Conclusion
The success of MRU highlights that hierarchy and efficiency are critical features for modern MC models. By looking at multiple n-gram blocks simultaneously, MRU captures intra-document relationships that sequential models often miss.
While the "Simple MRU" showed some weakness in pinpointing exact answer spans in NarrativeQA—suggesting that some form of recurrence remains necessary for fine-grained span prediction—the hybrid MRU-LSTM approach successfully mitigated this. Future work will explore the scalability of MRU to even longer documents, such as entire books, where its computational efficiency will be even more advantageous.
Summary updated