Abstract
To answer the question in machine comprehension (MC) task, the models need to establish the interaction between the question and the context. To tackle the problem that the single-pass model cannot reflect on and correct its answer, we present Ruminating Reader. Ruminating Reader adds a second pass of attention and a novel information fusion component to the Bi-Directional Attention Flow model (BIDAF). We propose novel layer structures that construct a query aware context vector representation and fuse encoding representation with intermediate representation on top of BIDAF model. We show that a multi-hop attention mechanism can be applied to a bi-directional attention structure. In experiments on SQuAD, we find that the Reader outperforms the BIDAF baseline by 2.1 F1 score and 2.7 EM score. Our analysis shows that different hops of the attention have different responsibilities in selecting answers.
Introduction
Machine Reading Comprehension (MRC) is a fundamental task in natural language processing where a model must answer questions based on a given text passage. Most state-of-the-art models employ a "single-pass" reasoning strategy, processing the question and context once to identify the answer. However, this approach lacks the iterative refinement characteristic of human cognition. When humans encounter complex questions, they often "ruminate"—re-evaluating the text with a refined focus based on their initial understanding. Single-pass models are susceptible to errors because they cannot revisit or correct their initial interpretations. To address this limitation, researchers from NYU introduced the Ruminating Reader, a model that incorporates a second pass of attention and a novel information fusion component to "think twice" before committing to an answer.
Methodology
The Ruminating Reader extends the well-established Bi-Directional Attention Flow (BiDAF) architecture. The primary innovation is the introduction of a rumination phase that enables the model to refine its internal representations through multi-hop attention.
Architecture and Rumination
The model's architecture follows several key stages:
- Initial Encoding: The text is processed using standard word and character-level embeddings to create foundational representations.
- Summarization Layer: After the first pass of bi-directional attention, a Bi-directional LSTM is used to generate a "summarization vector." This vector encapsulates the initial interaction between the query and the context.
- Rumination Layers: This core component applies a gating mechanism (similar to LSTM gates) to both query and context representations. These layers fuse the original encodings with the information from the summarization vector, allowing the model to update its understanding with "query-aware" context representations.
- Second Hop Attention: The model performs a second "hop" of attention using the refined, ruminated representations. This allows the system to focus on candidate answers that might have been overlooked or undervalued during the first pass.
- Answer-Question Similarity Loss: To prevent the model from simply echoing the question, a specific loss function was implemented to penalize excessive lexical overlap between the predicted answer and the query.
Experimental Evaluation
The Ruminating Reader was evaluated on the Stanford Question Answering Dataset (SQuAD), a large-scale benchmark for extractive question answering consisting of over 100,000 question-answer pairs.
Results
The model demonstrated a clear performance advantage over the BiDAF baseline:
- Performance Gain: The Ruminating Reader outperformed the baseline by 2.1 points in F1 score and 2.7 points in Exact Match (EM) score.
- Benchmark Scores: On the hidden SQuAD test set, the model achieved an F1 of 79.5 and an EM of 70.6.
- Error Analysis: The model was particularly effective at handling "Why" questions, which are typically more challenging due to their length and lack of direct lexical cues.
Analysis of Multi-Hop Attention
A significant finding of this work is the functional specialization of the attention hops. Visualization of the attention weights revealed that:
- The first hop is primarily responsible for identifying key informative words within the query.
- The second hop focuses on locating the specific candidate answer spans within the context.
The gating mechanism proved vital for this process, as it allowed the model to integrate new insights from the rumination phase without losing the essential context provided by the initial encodings. Additionally, the researchers observed that an LSTM layer could produce useful gating representations even when fed a repeated (tiled) summary vector, suggesting the LSTM's internal state effectively tracks positional information.
Conclusion
The Ruminating Reader demonstrates that a multi-hop attention mechanism can significantly improve machine reading performance by mimicking human-like reflection. By allowing the model to "think twice," it achieves superior accuracy on complex benchmarks like SQuAD. Future research will explore simplifying the architecture for better efficiency and investigating the potential of additional reasoning hops for multi-document comprehension tasks.
Summary updated