Reinforced Mnemonic Reader for Machine Reading Comprehension

Minghao Hu, Yuxing Peng, Zhen Huang, Xipeng Qiu, Furu Wei, Ming Zhou

Abstract

In this paper, we introduce the Reinforced Mnemonic Reader for machine reading comprehension tasks, which enhances previous attentive readers in two aspects. First, a reattention mechanism is proposed to refine current attentions by directly accessing to past attentions that are temporally memorized in a multi-round alignment architecture, so as to avoid the problems of attention redundancy and attention deficiency. Second, a new optimization approach, called dynamic-critical reinforcement learning, is introduced to extend the standard supervised method. It always encourages to predict a more acceptable answer so as to address the convergence suppression problem occurred in traditional reinforcement learning algorithms. Extensive experiments on the Stanford Question Answering Dataset (SQuAD) show that our model achieves state-of-the-art results. Meanwhile, our model outperforms previous systems by over 6% in terms of both Exact Match and F1 metrics on two adversarial SQuAD datasets.

Introduction

Machine Reading Comprehension (MRC) has seen explosive growth, yet even high-performing models struggle with maintaining focus across complex texts or becoming distracted by irrelevant information. Most models use attention mechanisms to align question words with context. However, multi-round alignment models often suffer from attention redundancy (repeatedly focusing on the same words) or attention deficiency (missing crucial information) because current attention rounds lack awareness of previous ones. Additionally, standard Reinforcement Learning (RL) training can suffer from "convergence suppression," where partially correct answers are penalized if they fall short of a greedy baseline.

Methods

Overall Architecture

The Reinforced Mnemonic Reader follows an "Encoder-Aligner-Pointer" framework. It utilizes ELMo and GloVe embeddings for rich word representations. The core of the model is the Iterative Reattention Aligner, which consists of multiple blocks (typically three) that refine the search for the answer using a novel reattention mechanism.

Overall architecture of the Reinforced Mnemonic Reader, showing the Encoder, Iterative Reattention Aligner, and Answer Pointer.

Reattention Mechanism

To address attention redundancy and deficiency, the reattention mechanism explicitly stores attention distributions from previous rounds. When calculating new attention, the model performs a dot product between the current state and these stored "past" attentions. This allows the model to either concentrate focus or shift it to un-scanned regions, acting as a "mnemonic" device that prevents the model from getting stuck in repetitive loops.

Illustration of the reattention mechanism, showing how past attention distributions are used to refine current focus via dot products.

Dynamic-Critical Reinforcement Learning (DCRL)

Standard RL training (like Self-Critical Sequence Training) can be overly punitive, discouraging the model from exploring partially correct answers. DCRL is more "encouraging." It takes two samples (one random, one greedy) and always treats the higher-scoring one as the reward and the lower-scoring one as the baseline. This ensures the normalized reward is always positive, constantly nudging the model toward better performance rather than just punishing less-than-perfect attempts.

Results and Discussion

The model was evaluated on the SQuAD v1.1 benchmark and adversarial datasets (AddSent and AddOneSent).

  • SQuAD Benchmark: The single model achieved 79.5% Exact Match (EM) and 86.6% F1 score. The ensemble reached 82.3% EM / 88.5% F1, outperforming the human baseline in F1 at the time of publication.
  • Adversarial Robustness: The model showed significant resilience to "distractor" sentences. On the AddSent dataset, it outperformed the previous best (FusionNet) by over 6%, jumping from 46.2 to 53.0 EM. This suggests the reattention mechanism is highly effective at ignoring noise and staying focused on true evidence.

Conclusion

The Reinforced Mnemonic Reader demonstrates that memory-aware attention and more encouraging reinforcement learning objectives can significantly boost MRC performance and robustness. While increasing the number of alignment blocks beyond three or four led to performance drops due to depth-related degradation, the core innovations of Reattention and DCRL offer promising directions for broader natural language tasks like Natural Language Inference (NLI).

Summary updated

Abstract

In this paper, we introduce the Reinforced Mnemonic Reader for machine reading comprehension tasks, which enhances previous attentive readers in two aspects. First, a reattention mechanism is proposed to refine current attentions by directly accessing to past attentions that are temporally memorized in a multi-round alignment architecture, so as to avoid the problems of attention redundancy and attention deficiency. Second, a new optimization approach, called dynamic-critical reinforcement learning, is introduced to extend the standard supervised method. It always encourages to predict a more acceptable answer so as to address the convergence suppression problem occurred in traditional reinforcement learning algorithms. Extensive experiments on the Stanford Question Answering Dataset (SQuAD) show that our model achieves state-of-the-art results. Meanwhile, our model outperforms previous systems by over 6% in terms of both Exact Match and F1 metrics on two adversarial SQuAD datasets.
BibTeX
@inproceedings{hu2018reinforced,
  author = {Minghao Hu and Yuxing Peng and Zhen Huang and Xipeng Qiu and Furu Wei and Ming Zhou},
  title = {{Reinforced Mnemonic Reader for Machine Reading Comprehension}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/570},
}
Figure