Inquisitive Question Generation for High Level Text Comprehension

Wei-Jen Ko, Te-Yuan Chen, Yiyan Huang, Greg Durrett, Junyi Jessy Li

Abstract

Inquisitive probing questions come naturally to humans in a variety of settings, but is a challenging task for automatic systems. One natural type of question to ask tries to fill a gap in knowledge during text comprehension, like reading a news article: we might ask about background information, deeper reasons behind things occurring, or more. Despite recent progress with data-driven approaches, generating such questions is beyond the range of models trained on existing datasets. We introduce INQUISITIVE, a dataset of ~19K questions that are elicited while a person is reading through a document. Compared to existing datasets, INQUISITIVE questions target more towards high-level (semantic and discourse) comprehension of text. We show that readers engage in a series of pragmatic strategies to seek information. Finally, we evaluate question generation models based on GPT-2 and show that our model is able to generate reasonable questions although the task is challenging, and highlight the importance of context to generate INQUISITIVE questions.

Introduction

Most existing Question Generation (QG) systems are trained on datasets like SQuAD, where the goal is essentially to "reverse-engineer" a question from a known answer found in the text. While useful for reading comprehension tests, this doesn't reflect how humans actually process information. When we read, we naturally ask questions to fill gaps: Why did this happen? What does this technical term mean? What is the historical context here? These are "inquisitive" questions.

The challenge is that current models struggle to generate these because they require a deep understanding of discourse, semantics, and "common ground" between the reader and writer. This paper addresses the lack of data and modeling techniques for this more natural, information-seeking behavior.

The INQUISITIVE Dataset

Researchers at UT Austin have introduced INQUISITIVE, a dataset of 19,131 questions elicited from 1,500 news articles sourced from the WSJ, Associated Press, and Newsela. Unlike traditional datasets that ask questions about what is already in the text, INQUISITIVE captures the gaps in knowledge readers feel while they are reading. Questions were collected sentence-by-sentence to capture the reader's evolving state of mind and information needs.

Examples of inquisitive questions categorized by pragmatic strategy.

Comparative Analysis

A key finding is that human curiosity isn't monolithic. Questions in the dataset fall into several pragmatic categories, such as Causal (Why?), Elaboration (How?), Definition (What is X?), and Forward-looking (What happens next?).

Compared to datasets like SQuAD, QuAC, or NewsQA, INQUISITIVE contains a much higher percentage of "Why" and "How" questions. While SQuAD is dominated by "What" questions aimed at fact retrieval, INQUISITIVE questions require synthesizing new concepts; only 22% of words in the questions appeared in the highlighted text spans. Furthermore, the dataset exhibits high lexical diversity, with a "distinct bigram" metric of 0.41, significantly higher than QuAC (0.26) or NewsQA (0.29).

Distribution of question types across INQUISITIVE and other popular QA datasets.

Modeling Inquisitive Questions

The authors developed a GPT-2 based model called Inquirer to generate these questions. The model is conditioned on the entire article context up to the current sentence, the current sentence itself, and a specific "target span" (a phrase the reader is curious about).

Experiments proved that context is king: models perform significantly better when they have access to the full prior context of the article. Without it, models often ask questions that have already been answered or are semantically shallow. Human evaluators found that the Inquirer model generated "valid" questions 75.8% of the time. Interestingly, questions generated by the model were rated as highly "salient" (relevant to the story) with a score of 3.65/5, slightly higher than the human average of 3.18.

Conclusion and Future Work

One of the biggest hurdles for the model remains redundancy—asking questions that were already answered earlier in the text (~10% of the time for the model vs. 5% for humans). Additionally, predicting which part of a sentence is "question-worthy" is highly subjective, making automated span-prediction a difficult target. Despite these challenges, INQUISITIVE provides a new benchmark for curiosity-driven AI, with potential applications in guided writing and educational tools for active reading.

Summary updated

Abstract

Inquisitive probing questions come naturally to humans in a variety of settings, but is a challenging task for automatic systems. One natural type of question to ask tries to fill a gap in knowledge during text comprehension, like reading a news article: we might ask about background information, deeper reasons behind things occurring, or more. Despite recent progress with data-driven approaches, generating such questions is beyond the range of models trained on existing datasets. We introduce INQUISITIVE, a dataset of ~19K questions that are elicited while a person is reading through a document. Compared to existing datasets, INQUISITIVE questions target more towards high-level (semantic and discourse) comprehension of text. We show that readers engage in a series of pragmatic strategies to seek information. Finally, we evaluate question generation models based on GPT-2 and show that our model is able to generate reasonable questions although the task is challenging, and highlight the importance of context to generate INQUISITIVE questions.
BibTeX
@inproceedings{ko2020inquisitive,
  author = {Wei-Jen Ko and Te-Yuan Chen and Yiyan Huang and Greg Durrett and Junyi Jessy Li},
  title = {{Inquisitive Question Generation for High Level Text Comprehension}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=JWv899hKDs3},
}
Figure