Proceedings of the Workshop on Machine Reading for Question Answering

Yichen Gong, Samuel Bowman, Martı́n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Cor- Rado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafał Józefowicz, Łukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek G. Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Tal- War, Paul A. Tucker, Vincent Vanhoucke, Vijay Vasude- Van, Fernanda Viégas, Oriol Vinyals, Pete Warden, Mir Rosenberg, Song Xia, Jianfeng Gao, Saurabh Tiwary, Marc-Antoine Rondeau, Timothy J. Hazen, Seunghak Yu, Sathish Reddy Indurthi, Seohyun Back, Haejun Lee, Jörg Franke, Jan Niehues, Alex Waibel, Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, Pritish Yuvraj, Rajarshi Das, Andrew McCallum, Maria Chang, Achille Fokoue-Nkoutche, Pavan Kapanipathi, Nicholas Mattei, Ryan Musa, Kartik Talamadupula, Michael Witbrock, Boratko Michael, Fokoue-Nkoutche Achille, Franke Jörg, Wei He 0017, Sathish Reddy, Xuan Liu, Wadhwa Soumya, Samuel, Chandu, Khyathi, Rajarshi, Du, Ye, Franke Jrg, Yichen, Wei He, Pavan, Kai, Yuan, Lyu, Yajuan, Nicholas, Divyendra, Nyberg, Lukasz KaiserOpenalex Unknown 2018 · DOI · OpenAlex

Abstract

To answer the question in machine comprehension (MC) task, the models need to establish the interaction between the question and the context. To tackle the problem that the single-pass model cannot reflect on and correct its answer, we present Ruminating Reader. Ruminating Reader adds a second pass of attention and a novel information fusion component to the Bi-Directional Attention Flow model (BIDAF). We propose novel layer structures that construct a query aware context vector representation and fuse encoding representation with intermediate representation on top of BIDAF model. We show that a multi-hop attention mechanism can be applied to a bi-directional attention structure. In experiments on SQuAD, we find that the Reader outperforms the BIDAF baseline by 2.1 F1 score and 2.7 EM score. Our analysis shows that different hops of the attention have different responsibilities in selecting answers.

Introduction

Machine Reading Comprehension (MRC) is a fundamental task in natural language processing where a model must answer questions based on a given text passage. Most state-of-the-art models employ a "single-pass" reasoning strategy, processing the question and context once to identify the answer. However, this approach lacks the iterative refinement characteristic of human cognition. When humans encounter complex questions, they often "ruminate"—re-evaluating the text with a refined focus based on their initial understanding. Single-pass models are susceptible to errors because they cannot revisit or correct their initial interpretations. To address this limitation, researchers from NYU introduced the Ruminating Reader, a model that incorporates a second pass of attention and a novel information fusion component to "think twice" before committing to an answer.

Methodology

The Ruminating Reader extends the well-established Bi-Directional Attention Flow (BiDAF) architecture. The primary innovation is the introduction of a rumination phase that enables the model to refine its internal representations through multi-hop attention.

Architecture and Rumination

The model's architecture follows several key stages:

  1. Initial Encoding: The text is processed using standard word and character-level embeddings to create foundational representations.
  2. Summarization Layer: After the first pass of bi-directional attention, a Bi-directional LSTM is used to generate a "summarization vector." This vector encapsulates the initial interaction between the query and the context.
  3. Rumination Layers: This core component applies a gating mechanism (similar to LSTM gates) to both query and context representations. These layers fuse the original encodings with the information from the summarization vector, allowing the model to update its understanding with "query-aware" context representations.
  4. Second Hop Attention: The model performs a second "hop" of attention using the refined, ruminated representations. This allows the system to focus on candidate answers that might have been overlooked or undervalued during the first pass.
  5. Answer-Question Similarity Loss: To prevent the model from simply echoing the question, a specific loss function was implemented to penalize excessive lexical overlap between the predicted answer and the query.

Experimental Evaluation

The Ruminating Reader was evaluated on the Stanford Question Answering Dataset (SQuAD), a large-scale benchmark for extractive question answering consisting of over 100,000 question-answer pairs.

Results

The model demonstrated a clear performance advantage over the BiDAF baseline:

  • Performance Gain: The Ruminating Reader outperformed the baseline by 2.1 points in F1 score and 2.7 points in Exact Match (EM) score.
  • Benchmark Scores: On the hidden SQuAD test set, the model achieved an F1 of 79.5 and an EM of 70.6.
  • Error Analysis: The model was particularly effective at handling "Why" questions, which are typically more challenging due to their length and lack of direct lexical cues.

Analysis of Multi-Hop Attention

A significant finding of this work is the functional specialization of the attention hops. Visualization of the attention weights revealed that:

  • The first hop is primarily responsible for identifying key informative words within the query.
  • The second hop focuses on locating the specific candidate answer spans within the context.

The gating mechanism proved vital for this process, as it allowed the model to integrate new insights from the rumination phase without losing the essential context provided by the initial encodings. Additionally, the researchers observed that an LSTM layer could produce useful gating representations even when fed a repeated (tiled) summary vector, suggesting the LSTM's internal state effectively tracks positional information.

Conclusion

The Ruminating Reader demonstrates that a multi-hop attention mechanism can significantly improve machine reading performance by mimicking human-like reflection. By allowing the model to "think twice," it achieves superior accuracy on complex benchmarks like SQuAD. Future research will explore simplifying the architecture for better efficiency and investigating the potential of additional reasoning hops for multi-document comprehension tasks.

Summary updated

Abstract

To answer the question in machine comprehension (MC) task, the models need to establish the interaction between the question and the context.To tackle the problem that the single-pass model cannot reflect on and correct its answer, we present Ruminating Reader.Ruminating Reader adds a second pass of attention and a novel information fusion component to the Bi-Directional Attention Flow model (BIDAF).We propose novel layer structures that construct a query aware context vector representation and fuse encoding representation with intermediate representation on top of BIDAF model.We show that a multi-hop attention mechanism can be applied to a bi-directional attention structure.In experiments on SQuAD, we find that the Reader outperforms the BIDAF baseline by 2.1 F1 score and 2.7 EM score.Our analysis shows that different hops of the attention have different responsibilities in selecting answers.
BibTeX
@inproceedings{gong2018proceedings,
  author = {Yichen Gong and Samuel Bowman and Martı́n Abadi and Ashish Agarwal and Paul Barham and Eugene Brevdo and Zhifeng Chen and Craig Citro and Greg Cor- Rado and Andy Davis and Jeffrey Dean and Matthieu Devin and Sanjay Ghemawat and Ian Goodfellow and Andrew Harp and Geoffrey Irving and Michael Isard and Yangqing Jia and Rafał Józefowicz and Łukasz Kaiser and Manjunath Kudlur and Josh Levenberg and Dan Mané and Rajat Monga and Sherry Moore and Derek G. Murray and Chris Olah and Mike Schuster and Jonathon Shlens and Benoit Steiner and Ilya Sutskever and Kunal Tal- War and Paul A. Tucker and Vincent Vanhoucke and Vijay Vasude- Van and Fernanda Viégas and Oriol Vinyals and Pete Warden and Mir Rosenberg and Song Xia and Jianfeng Gao and Saurabh Tiwary and Marc-Antoine Rondeau and Timothy J. Hazen and Seunghak Yu and Sathish Reddy Indurthi and Seohyun Back and Haejun Lee and Jörg Franke and Jan Niehues and Alex Waibel and Michael Boratko and Harshit Padigela and Divyendra Mikkilineni and Pritish Yuvraj and Rajarshi Das and Andrew McCallum and Maria Chang and Achille Fokoue-Nkoutche and Pavan Kapanipathi and Nicholas Mattei and Ryan Musa and Kartik Talamadupula and Michael Witbrock and Boratko Michael and Fokoue-Nkoutche Achille and Franke Jörg and Wei He and Sathish Reddy and Xuan Liu and Wadhwa Soumya and Samuel and Chandu and Khyathi and Chang and Das and Rajarshi and Du and Ye and Fokoue-Nkoutche Achille and Franke Jrg and Gong and Yichen and Wei He and Sathish Reddy and Kapanipathi and Pavan and Lee and Liu and Kai and Xuan Liu and Yuan and Lyu and Yajuan and Mattei and Nicholas and Mikkilineni and Divyendra and Nyberg and Padigela and Lukasz Kaiser and Michael J. Witbrock},
  title = {{Proceedings of the Workshop on Machine Reading for Question Answering}},
  booktitle = {Openalex Unknown},
  year = {2018},
  doi = {10.18653/v1/w18-26},
}
Figure