Dependent Gated Reading for Cloze-Style Question Answering

Reza Ghaeini, Xiaoli Z. Fern, Hamed Shahbazi, Prasad Tadepalli

Abstract

Cloze-style question answering (QA) requires a model to fill a missing word in a query based on a provided document. While existing multi-hop architectures use the query to attend to the document, they often treat the query representation as static. We present Dependent Gated Reading (DGR), a novel neural architecture that models the deep interdependency between document and query. By allowing representations to mutually influence each other across multiple reading iterations, DGR achieves state-of-the-art results on the Children’s Book Test (CBT) and Who Did What (WDW) benchmarks.

Introduction

In cloze-style QA, the objective is to predict a missing token in a query given a context document. Recent advances have leveraged attention mechanisms and iterative "multi-hop" reading strategies. However, a limitation in many current models is the unidirectional flow of information: the query guides the document reading, but the document rarely informs the query's own representation.

Researchers at Oregon State University argue that human comprehension is inherently bidirectional; as we uncover clues in a text, our understanding of the question itself evolves. DGR was designed to bridge this gap by modeling the deep interdependency between the document and the query through a document-dependent query reading mechanism.

Methodology: The DGR Architecture

The DGR model utilizes Bidirectional Gated Recurrent Units (Bi-GRUs) to process text through an iterative, multi-hop process.

The DGR architecture illustrating the iterative multi-hop process and bidirectional information flow between query and document.

The process consists of three main stages:

  1. Initial Encoding: Independent Bi-GRUs generate base representations for the document and query.
  2. Iterative Gated Reading (Hops): In each iteration, the model calculates attention weights between every token in the document and query. It extracts relevant information from the query to update the document representation, and crucially, extracts relevant information from the document to update the query representation. These features are merged via element-wise multiplication (gating).
  3. Dependent Query Reading: Unlike previous models (like the GA Reader) that restart query encoding from scratch in each hop, DGR initializes the query's Bi-GRU with the hidden state from the previous iteration. This allows the model to maintain a "summary understanding" of the query that evolves over time.

Finally, the model uses a "pointer-sum" attention mechanism to rank candidate words and select the most likely answer.

Experimental Evaluation

The model was evaluated on two primary large-scale benchmarks: the Children’s Book Test (CBT), specifically the Named Entities (CBT-NE) and Common Nouns (CBT-CN) subsets, and the Who Did What (WDW) dataset, which includes both "Strict" and "Relaxed" versions.

Results and Ablation Study

DGR achieved new state-of-the-art (SOTA) performance on three out of four major test settings: CBT-NE (75.4%), WDW-Strict (72.0%), and WDW-Relaxed (72.9%). To understand the contribution of each component, the authors conducted an ablation study on the development sets.

Development accuracy of DGR and its variants on CBT and WDW datasets, highlighting the impact of different architectural components.

The ablation study revealed that bidirectionality is key. Removing the document's influence on the query (variant (a) in the table) led to the most significant performance drops. Furthermore, visualization of attention weights showed that DGR produces much more "focused" attention maps than its predecessors, zeroing in on specific clues rather than spreading focus across irrelevant words.

Discussion and Conclusion

DGR demonstrates that document-dependent query reading is a critical element for machine comprehension. By preserving query state across hops and allowing the document to refine the query representation, the model outperforms complex systems that rely on external linguistic toolkits like POS tagging or NER.

While highly effective for Named Entities, the model did not achieve SOTA on the CBT-CN (Common Nouns) dataset, suggesting that common nouns may require different types of linguistic reasoning. Future work will investigate how these bidirectional dependencies scale to longer documents and explore mechanisms to handle common noun tasks more effectively.

Summary updated

BibTeX
@inproceedings{ghaeini2018dependent,
  author = {Reza Ghaeini and Xiaoli Z. Fern and Hamed Shahbazi and Prasad Tadepalli},
  title = {{Dependent Gated Reading for Cloze-Style Question Answering}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure