A Deep Reinforcement Learning Based Multi-Step Coarse to Fine Question Answering (MSCQA) System.

Yu Wang, Hongxia JinAAAI 2019 · OpenReview

Abstract

In this paper, we present a multi-step coarse to fine question answering (MSCQA) system which can efficiently processes documents with different lengths by choosing appropriate actions. The system is designed using an actor-critic based deep reinforcement learning model to achieve multistep question answering. Compared to previous QA models targeting on datasets mainly containing either short or long documents, our multi-step coarse to fine model takes the merits from multiple system modules, which can handle both short and long documents. The system hence obtains a much better accuracy and faster trainings speed compared to the current state-of-the-art models. We test our model on four QA datasets, WIKEREADING, WIKIREADING LONG, CNN and SQuAD, and demonstrate 1.3%-1.7% accuracy improvements with 1.5x-3.4x training speed-ups in comparison to the baselines using state-of-the-art models.

Introduction

Traditional Machine Reading Comprehension (MRC) models often face a "Goldilocks" problem regarding document length. High-performance models like QANet excel at short documents but become computationally expensive and less accurate when applied to long contexts. Conversely, "coarse-to-fine" models—which first filter for relevant sentences before zooming in—work well for long documents but often underperform on short ones due to the loss of context or the introduction of "false positives" (selecting a plausible but incorrect answer).

The Multi-Step Coarse to Fine Question Answering (MSCQA) system bridges this gap. By employing a Deep Reinforcement Learning (DRL) agent, the system dynamically decides the most efficient way to process a document, mimicking human reading strategies such as skimming long articles or intently reading short paragraphs.

Methodology

The core of the MSCQA system is an Actor-Critic Reinforcement Learning framework that functions as an "Action Selector." For any given question and document pair, the agent evaluates the state and chooses one of three distinct actions:

  1. Generate Answer ($a_1$): A terminal state where the model extracts the final answer from the current context.
  2. Sentence Selection ($a_2$): A non-terminal state where the model filters the document down to the $K$ most relevant sentences, narrowing the search space for subsequent steps.
  3. False Positive Rejection ($a_3$): A unique self-correction mechanism. If the model identifies a predicted answer as a likely false positive, it excludes that sub-context and re-evaluates the remaining document.
The MSCQA Action Selector framework showing the decision flow between sentence selection, answer generation, and sub-context rejection.

To support these actions efficiently, the system utilizes an advanced embedding and encoding pipeline. It combines word-level (GloVe) and character-level embeddings, which are then processed by an Embedding Encoder Layer. This layer replaces traditional, slower Recurrent Neural Networks (RNNs) with a combination of depth-wise separable convolutions for speed and self-attention mechanisms to capture long-range dependencies.

The Embedding Encoder Layer architecture, which uses convolutions and self-attention to process input embeddings in parallel.

Experimental Setup

The model was evaluated across four diverse benchmarks to demonstrate its flexibility and robustness:

  • WIKIREADING: A large-scale dataset (18.58M pairs) derived from Wikipedia and Wikidata.
  • WIKIREADING LONG: A subset of documents longer than ten sentences where the answer is not found in the initial sentence.
  • CNN: A cloze-style dataset based on news stories.
  • SQuAD 1.1: The standard benchmark for short-document reading comprehension, with an average document length of 122 tokens.

Results and Discussion

Key Insights

  • Dynamic Strategy Adaptation: The RL agent learns to adapt its strategy based on the dataset. For short documents (SQuAD), it chooses direct answer generation ($a_1$) 76% of the time. For longer documents (WIKIREADING LONG), it utilizes coarse sentence selection ($a_2$) in 39% of cases.
  • The Power of Rejection: The inclusion of the sub-context selection action ($a_3$) proved critical. By allowing the model to "erase" a mistake and look elsewhere, the system consistently outperformed versions without this self-correction capability.
  • Efficiency via Architecture: The parallel nature of the CNN and self-attention layers, combined with the RL agent's ability to skip unnecessary processing, led to significant performance gains.

Performance Metrics

MSCQA outperformed state-of-the-art baselines across all tested datasets. Notably, it achieved 83.6% Exact Match (EM) accuracy on the SQuAD dev set, surpassing the QANet baseline of 82.9%. Furthermore, the system demonstrated a 1.5x to 3.4x training speedup, validating the efficiency of the multi-step approach.

Conclusion

The MSCQA system demonstrates that a reinforcement learning-based dynamic action selection strategy can significantly improve both the accuracy and efficiency of question answering systems. While the model shows substantial improvements, a gap remains between current performance and a theoretical "Oracle" sentence selector, particularly on complex datasets like CNN and SQuAD.

Future research directions include refining the action space with more granular states, exploring more complex reward functions to guide the RL agent, and extending this multi-step reasoning framework to tasks requiring multi-hop reasoning across multiple documents.

Summary updated

Abstract

In this paper, we present a multi-step coarse to fine question answering (MSCQA) system which can efficiently processes documents with different lengths by choosing appropriate actions. The system is designed using an actor-critic based deep reinforcement learning model to achieve multistep question answering. Compared to previous QA models targeting on datasets mainly containing either short or long documents, our multi-step coarse to fine model takes the merits from multiple system modules, which can handle both short and long documents. The system hence obtains a much better accuracy and faster trainings speed compared to the current state-of-the-art models. We test our model on four QA datasets, WIKEREADING, WIKIREADING LONG, CNN and SQuAD, and demonstrate 1.3%-1.7% accuracy improvements with 1.5x-3.4x training speed-ups in comparison to the baselines using state-of-the-art models.
BibTeX
@inproceedings{wang2019deep,
  author = {Yu Wang and Hongxia Jin},
  title = {{A Deep Reinforcement Learning Based Multi-Step Coarse to Fine Question Answering (MSCQA) System.}},
  booktitle = {AAAI},
  year = {2019},
  url = {https://openreview.net/forum?id=SQKEsWMlu6S},
}
Figure