Covering Uncommon Ground: Gap-Focused Question Generation for Answer Assessment

Roni Rabin, Alexandre Djerbetian, Roee Engelberg, Lidan Hackmon, Gal Elidan, Reut Tsarfaty, Amir Globerson

Abstract

Human communication often involves information gaps between the interlocutors. For example, in an educational dialogue, a student often provides an answer that is incomplete, and there is a gap between this answer and the perfect one expected by the teacher. Successful dialogue then hinges on the teacher asking about this gap in an effective manner, thus creating a rich and interactive educational experience. We focus on the problem of generating such gap-focused questions (GFQs) automatically. We define the task, highlight key desired aspects of a good GFQ, and propose a model that satisfies these. Finally, we provide an evaluation by human annotators of our generated questions compared against human generated ones, demonstrating competitive performance.

Introduction

In any educational or collaborative dialogue, the exchange of information is rarely symmetrical. A teacher typically possesses a comprehensive understanding of a subject—the "source text"—while a student may provide only a partial or incomplete summary. The pedagogical challenge lies in bridging this "information gap." To guide a student toward a complete understanding, a teacher must identify what is missing and formulate a question that targets that specific void.

This research introduces a framework for generating Gap-Focused Questions (GFQs). Unlike traditional Question Generation (QG) models that focus on summarizing or querying existing content, GFQ generation requires identifying the "uncommon ground"—the information present in a reference text but absent from a student's response.

The Challenge of Gap-Focused Questions

Generating questions about missing information is significantly more complex than standard QG. A high-quality GFQ must satisfy three primary criteria to be effective in a dialogue:

  1. Answerability: The question must be answerable using the source text (the teacher's knowledge).
  2. Novelty: The question should not address information the student has already provided (the "common ground").
  3. Pragmatics: The question must not "leak" the answer. For instance, if a student mentions a man playing guitar and the source text specifies he is wearing a "blue hat," asking "What color is his blue hat?" is a pragmatic failure because it reveals the very information the student lacked.

Methodology: A Gricean Pipeline

To address these challenges, the researchers developed a three-step pipeline inspired by Grice’s Maxims of Conversation, operationalizing philosophical concepts into an algorithmic workflow.

Step 1: Generating Answerable Questions (Maxim of Relevance)

The model first identifies all potential "answer spans" within the source text using a constituency parser. It then employs a T5-xxl model, fine-tuned on the SQuAD dataset, to generate candidate questions for every identified span. This ensures that every potential question is grounded in the source material.

Step 2: Filtering Common Ground (Maxim of Quantity)

To ensure novelty, the model utilizes a Question Answering (QA) system to test the candidate questions against the student's partial response. If the QA system can successfully answer a question using only the student's input, that question is discarded. This step ensures the model only targets the information gap.

Step 3: Minimizing Information Leakage (Maxim of Manner)

Finally, the model ranks the remaining questions to ensure they are natural and do not reveal the answer within the prompt. The ranking algorithm prefers questions that contain the fewest "new" entities or words not already present in the student's text, thereby maintaining the integrity of the assessment.

Experimental Setup and Datasets

Because specialized dialogue datasets for teacher-student gaps are rare, the researchers repurposed the Stanford Natural Language Inference (SNLI) dataset. In this setup, the "premise" serves as the teacher’s complete knowledge, while the "entailed hypothesis" represents the student’s partial description.

To evaluate the model, the team released a new benchmark consisting of 200 instances from the SNLI test set. These instances include human-generated gap-focused questions to serve as a gold standard for comparison.

Results and Evaluation

Human annotators rated the quality of the generated questions on a scale of 1 to 5. The evaluation revealed a clear "staircase" of improvement as each stage of the pipeline was implemented:

  • Step 1 (Basic QG): 3.72
  • Step 2 (Filtering): 3.86
  • Step 3 (Ranking/Pragmatics): 3.94
  • Human Performance: 4.06

The final model (Step 3) achieved a score of 3.94, performing remarkably close to the human gold standard. Statistical analysis confirmed that both the filtering and ranking steps provided significant, measurable boosts in question quality.

Discussion and Future Work

This work demonstrates that abstract linguistic theories like Stalnaker’s "common ground" and Grice’s maxims can be effectively used to constrain and improve Large Language Models (LLMs) for specialized tasks. By focusing on the "inverse of grounding"—identifying what is not there—the model provides a vital tool for automated tutoring and fact-checking.

However, several limitations remain. The current model is designed for single-turn interactions and does not yet support the multi-turn "scaffolding" common in human tutoring. Additionally, the model assumes the student's initial response is factually correct but incomplete; future iterations will need to address student hallucinations or errors. Finally, increasing "uptake"—the active rephrasing of student input within the question—could make the generated dialogue feel more natural and conversational.

Summary updated

Abstract

Human communication often involves information gaps between the interlocutors. For example, in an educational dialogue, a student often provides an answer that is incomplete, and there is a gap between this answer and the perfect one expected by the teacher. Successful dialogue then hinges on the teacher asking about this gap in an effective manner, thus creating a rich and interactive educational experience. We focus on the problem of generating such gap-focused questions (GFQs) automatically. We define the task, highlight key desired aspects of a good GFQ, and propose a model that satisfies these. Finally, we provide an evaluation by human annotators of our generated questions compared against human generated ones, demonstrating competitive performance.
BibTeX
@inproceedings{rabin2023covering,
  author = {Roni Rabin and Alexandre Djerbetian and Roee Engelberg and Lidan Hackmon and Gal Elidan and Reut Tsarfaty and Amir Globerson},
  title = {{Covering Uncommon Ground: Gap-Focused Question Generation for Answer Assessment}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=-S7dAPvbigp},
}
Figure