Abstract
Cloze-style question answering (QA) requires a model to fill a missing word in a query based on a provided document. While existing multi-hop architectures use the query to attend to the document, they often treat the query representation as static. We present Dependent Gated Reading (DGR), a novel neural architecture that models the deep interdependency between document and query. By allowing representations to mutually influence each other across multiple reading iterations, DGR achieves state-of-the-art results on the Children’s Book Test (CBT) and Who Did What (WDW) benchmarks.
Introduction
In cloze-style QA, the objective is to predict a missing token in a query given a context document. Recent advances have leveraged attention mechanisms and iterative "multi-hop" reading strategies. However, a limitation in many current models is the unidirectional flow of information: the query guides the document reading, but the document rarely informs the query's own representation.
Researchers at Oregon State University argue that human comprehension is inherently bidirectional; as we uncover clues in a text, our understanding of the question itself evolves. DGR was designed to bridge this gap by modeling the deep interdependency between the document and the query through a document-dependent query reading mechanism.
Methodology: The DGR Architecture
The DGR model utilizes Bidirectional Gated Recurrent Units (Bi-GRUs) to process text through an iterative, multi-hop process.
The process consists of three main stages:
- Initial Encoding: Independent Bi-GRUs generate base representations for the document and query.
- Iterative Gated Reading (Hops): In each iteration, the model calculates attention weights between every token in the document and query. It extracts relevant information from the query to update the document representation, and crucially, extracts relevant information from the document to update the query representation. These features are merged via element-wise multiplication (gating).
- Dependent Query Reading: Unlike previous models (like the GA Reader) that restart query encoding from scratch in each hop, DGR initializes the query's Bi-GRU with the hidden state from the previous iteration. This allows the model to maintain a "summary understanding" of the query that evolves over time.
Finally, the model uses a "pointer-sum" attention mechanism to rank candidate words and select the most likely answer.
Experimental Evaluation
The model was evaluated on two primary large-scale benchmarks: the Children’s Book Test (CBT), specifically the Named Entities (CBT-NE) and Common Nouns (CBT-CN) subsets, and the Who Did What (WDW) dataset, which includes both "Strict" and "Relaxed" versions.
Results and Ablation Study
DGR achieved new state-of-the-art (SOTA) performance on three out of four major test settings: CBT-NE (75.4%), WDW-Strict (72.0%), and WDW-Relaxed (72.9%). To understand the contribution of each component, the authors conducted an ablation study on the development sets.
The ablation study revealed that bidirectionality is key. Removing the document's influence on the query (variant (a) in the table) led to the most significant performance drops. Furthermore, visualization of attention weights showed that DGR produces much more "focused" attention maps than its predecessors, zeroing in on specific clues rather than spreading focus across irrelevant words.
Discussion and Conclusion
DGR demonstrates that document-dependent query reading is a critical element for machine comprehension. By preserving query state across hops and allowing the document to refine the query representation, the model outperforms complex systems that rely on external linguistic toolkits like POS tagging or NER.
While highly effective for Named Entities, the model did not achieve SOTA on the CBT-CN (Common Nouns) dataset, suggesting that common nouns may require different types of linguistic reasoning. Future work will investigate how these bidirectional dependencies scale to longer documents and explore mechanisms to handle common noun tasks more effectively.
Summary updated