Abstract: Inquisitive probing questions come naturally to humans in a variety of settings, but is a challenging task for automatic systems. One natural type of question to ask tries to fill a gap in knowledge during text comprehension, like reading a news article: we might ask about background information, deeper reasons behind things occurring, or more. Despite recent progress with data-driven approaches, generating such questions is beyond the range of models trained on existing datasets. We introduce INQUISITIVE, a dataset of ~19K questions that are elicited while a person is reading through a document. Compared to existing datasets, INQUISITIVE questions target more towards high-level (semantic and discourse) comprehension of text. We show that readers engage in a series of pragmatic strategies to seek information. Finally, we evaluate question generation models based on GPT-2 and show that our model is able to generate reasonable questions although the task is challenging, and highlight the importance of context to generate INQUISITIVE questions.
Shifting the Focus: From Finding Answers to Asking Questions
Traditional natural language processing (NLP) has long prioritized "finding answers" within a text. Most question generation (QG) research focuses on factoid questions where the answer is explicitly contained in the provided passage. However, human reading is a dynamic, inquisitive process. As we read, we identify gaps in our understanding—missing causal links, jargon that needs defining, or background information—and we ask questions to resolve these gaps.
Presented at EMNLP 2020, this research introduces INQUISITIVE, a dataset and modeling framework designed to simulate this human-like curiosity. Unlike "backward-looking" factoid datasets like SQuAD, INQUISITIVE focuses on "forward-looking" questions that probe for information not yet revealed in the text.
The INQUISITIVE Dataset
The authors curated a dataset of approximately 19,000 curiosity-driven questions elicited from readers as they progressed through news articles (from the Wall Street Journal, Associated Press, and Newsela) sentence-by-sentence. To ensure the questions were truly inquisitive, crowdsourced workers were only shown text up to the current sentence, forcing them to ask questions based on their evolving mental model of the story.
Compared to standard benchmarks like SQuAD, QuAC, and NewsQA, the INQUISITIVE dataset is linguistically richer. It features a significantly higher proportion of "Why" and "How" questions, reflecting a deeper level of semantic and discourse-level engagement with the text.
Modeling Curiosity with GPT-2
To automate the generation of these questions, the researchers developed the "Inquirer" model based on GPT-2 (medium). The model is designed to take the entire preceding context of an article plus the current sentence—with a specific target span highlighted—to generate a relevant question.
Key Findings from Modeling:
- Context is King: Ablation studies revealed that models seeing only the current sentence or the highlighted span performed significantly worse. Effective inquisitive QG requires understanding the "common ground" established in the preceding text.
- Factoid Knowledge Doesn't Transfer: Pre-training the model on SQuAD 2.0 did not improve performance on INQUISITIVE. This suggests that the pragmatic structures of inquisitive questions are fundamentally different from those used to probe for existing facts.
- Subjectivity: The task is inherently creative. Even when different humans highlight the same text span, they often ask entirely different questions, making this a challenging "one-to-many" mapping problem for AI.
Results and Performance
In human evaluations, the "Inquirer" model generated valid questions 75.8% of the time. While this is a strong result for a generative model, it still trails the human baseline of 95.8%. Furthermore, the dataset's lexical diversity (distinct bigram metric of 0.41) is notably higher than that of QuAC (0.26) or NewsQA (0.29), confirming that INQUISITIVE is a more complex and less repetitive benchmark for the NLP community.
One significant challenge remains in span prediction. A BERT-based span predictor achieved an exact match with human-selected spans only 12.1% of the time, illustrating the difficulty in predicting exactly what a reader will find curious.
Future Directions
The authors envision INQUISITIVE models being used for guided writing—helping authors identify where their explanations might be lacking—or as educational tools to enhance reading comprehension. Future work will need to address model redundancy (generating questions already answered in the text) and develop better automatic metrics, as standard n-gram overlap measures like BLEU are insufficient for capturing the quality of creative, inquisitive thought.
Summary updated