EA Reader: Enhance Attentive Reader for Cloze-Style Question Answering via Multi-Space Context Fusion

Chengzhen Fu, Yan ZhangAAAI 2019 · DOI

Abstract

Cloze-style question answering (QA) requires machines to predict a missing word in a query based on a provided document. Traditional attention mechanisms often operate at a coarse token-wise level, assigning a single weight to entire word embeddings. This paper introduces the EA Reader, which utilizes Multi-Space Context Fusion (MSCF) to perform fine-grained, bit-level attention. By projecting embeddings into multiple semantic subspaces and employing a multi-hop architecture with dynamic memory updates, the EA Reader achieves state-of-the-art results on major benchmarks including CNN and Daily Mail.

Introduction

In Natural Language Processing, cloze-style QA serves as a benchmark for machine reading comprehension. Most existing models follow an "encoder-interaction-prediction" framework where interaction is governed by token-wise attention. However, word embeddings contain diverse semantic features (bits) representing syntax, sentiment, or factual categories. Standard attention mechanisms force all these features to share a single weight, limiting the model's ability to filter noise or capture complex query-document relationships.

The EA Reader addresses these limitations through two primary innovations: Multi-Space Context Fusion (MSCF) for granular feature weighting and a multi-hop architecture that iteratively updates its internal representation of the document.

Methodology

Multi-Space Context Fusion (MSCF)

The MSCF module shifts from token-level to bit-level attention. It first projects query and document embeddings into $L$ distinct latent semantic subspaces. Within each subspace, instead of a scalar weight, the model computes an attention score map. This allows for an element-wise product between the query and document representations, effectively weighting individual dimensions (bits) of the embedding vectors differently. This granularity acts as a fine-grained feature filter, allowing the model to ignore irrelevant parts of a word that do not relate to the specific query.

Architecture of the Multi-Space Context Fusion (MSCF) module showing projection into subspaces and bit-wise attention.

Multi-Hop Reasoning Architecture

The EA Reader extends MSCF into a multi-hop framework to simulate iterative human reading. The process involves three core operations:

  1. Read: Extracts "evidence" from the document using MSCF based on the current query state.
  2. Update: Refines the query representation using the retrieved evidence.
  3. Write: Updates the document "memory" using a gating mechanism. Unlike static models, the EA Reader writes new evidence back into document slots, allowing it to filter out irrelevant information in subsequent hops and "clean" its understanding as it approaches the answer.
The multi-hop reasoning framework of the EA Reader, highlighting the 'Write' operation and gating mechanism.

Experimental Evaluation

The model was evaluated on four standard large-scale benchmarks: CNN, Daily Mail, and the Children’s Book Test (CBT-NE and CBT-CN). In these datasets, entities are often anonymized to force the model to rely on contextual reasoning rather than external knowledge.

Key Results

The EA Reader demonstrated significant improvements over previous state-of-the-art (SOTA) models:

  • Single-Hop Performance: The MSCF module alone outperformed the popular BiDAF model by 2.4% on CNN and 3.1% on Daily Mail test sets.
  • State-of-the-Art Benchmarks: The full multi-hop EA Reader reached 81.4% accuracy on CNN and 84.3% on Daily Mail, surpassing complex models like the GA Reader and NSE.
  • Ablation Study: Removing the "Write" operation resulted in a 1.0-1.5% drop in accuracy, confirming that dynamic memory updates are crucial for effective multi-hop reasoning.

Discussion and Conclusion

The success of the EA Reader highlights that granularity in attention matters. Bit-wise attention allows the model to isolate relevant features within word embeddings, while multiple subspaces enable specialization—where different spaces focus on distinct aspects like entities, actions, or temporal markers.

While the model excels at single-word cloze tasks, future work will explore adapting bit-wise attention to span-extractive QA tasks (e.g., SQuAD) and developing more sophisticated memory architectures for broader language understanding tasks.

Summary updated

BibTeX
@inproceedings{fu2019ea,
  author = {Chengzhen Fu and Yan Zhang},
  title = {{EA Reader: Enhance Attentive Reader for Cloze-Style Question Answering via Multi-Space Context Fusion}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33016375},
}
Figure