Active Reading Comprehension: A Dataset for Learning the Question-Answer Relationship Strategy

Diana Galván-SosaACL 2019 · DOI

Abstract

Current Reading Comprehension (RC) benchmarks primarily evaluate the "product" of understanding—the final answer—rather than the "process" of reasoning. This focus can lead to models that rely on superficial heuristics rather than true comprehension. We propose a framework based on the Question-Answer Relationship (QAR) strategy, a meta-cognitive technique from educational psychology. By repurposing existing datasets through weak supervision, we create a benchmark that requires models to identify the source and complexity of the information needed to answer a question.

Introduction

A significant challenge in modern NLP is the "brittleness" of reading comprehension models. While models may achieve high accuracy on standard datasets, they often fail when faced with adversarial examples or when the reasoning requirements shift. This suggests that models lack an understanding of the relationship between a question and the text. Humans use active reading strategies to determine if an answer is explicitly stated, requires synthesis of multiple points, or depends on external knowledge.

This paper introduces a methodology to formalize these strategies into a machine learning task. By categorizing questions into four QAR classes—Right There, Think and Search, Author and Me, and On My Own—we can evaluate and train models on the reasoning process itself.

The Question-Answer Relationship (QAR) Framework

The QAR framework categorizes the relationship between a question, the text, and the reader's background knowledge:

  1. Right There: The answer is found in a single sentence in the text.
  2. Think and Search: The answer is found in the text but requires combining information from multiple sentences.
  3. Author and Me: The answer requires combining information from the text with the reader's background knowledge.
  4. On My Own: The answer can be provided using background knowledge alone.

Methodology: Weak Supervision via Metadata

To construct a large-scale QAR dataset, we leverage weak supervision. Instead of manual labeling, we use the existing metadata of well-known RC datasets. Specifically, the number of "supporting sentences" or "supporting facts" ($n$) provided in datasets like bAbI, HotpotQA, and MultiRC serves as a direct proxy for the QAR category.

Supporting sentence metadata used for labeling via weak supervision.

As illustrated in the figure above, the "Sentence needed" column identifies the specific sentences required to answer each question. We apply a mapping rule where $n=1$ corresponds to "Right There" (Category 1) and $n>1$ corresponds to "Think and Search" (Category 2).

Example of QAR category assignment in the bAbI dataset.

The resulting labels, shown in the figure above, allow us to transform a standard RC task into a process-aware classification task. For instance, question 9 requires two sentences (4 and 8) to determine that Mary is carrying "none" objects, correctly identifying it as a "Think and Search" problem.

Dataset Construction and Recycling

We identify several datasets that can be effectively mapped to the QAR framework:

  • bAbI: Tasks 7 and 8 provide clear sentence-level support for Categories 1 and 2.
  • SQuAD: Primarily used for Category 1 ("Right There").
  • HotpotQA & MultiRC: These multi-hop datasets are rich sources for Category 2 ("Think and Search").
  • MCScript: Used for the "In My Head" categories (3 and 4) by leveraging "Commonsense" vs. "Text-based" tags.

Conclusion

By shifting the focus from product-based to process-based evaluation, we can build more robust and interpretable reading comprehension systems. The proposed QAR dataset, constructed through weak supervision of existing benchmarks, provides a scalable way to train models in active reading strategies. Future work will focus on integrating this QAR classification as an auxiliary task in state-of-the-art RC architectures.

Summary updated

BibTeX
@inproceedings{galvansosa2019active,
  author = {Diana Galván-Sosa},
  title = {{Active Reading Comprehension: A Dataset for Learning the Question-Answer Relationship Strategy}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-2014},
}
Figure