Abstract: Roni Rabin, Alexandre Djerbetian, Roee Engelberg, Lidan Hackmon, Gal Elidan, Reut Tsarfaty, Amir Globerson. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).
2023.
Introduction
Researchers from Google and several Israeli universities have introduced a novel task: Gap-Focused Question Generation (GFQ). In educational or collaborative settings, communication is driven by information gaps. Instead of simply asking questions about a text, GFQ identifies the specific "information gap" between a complete reference (like a teacher's knowledge) and a partial description (like a student's answer). The goal is to generate follow-up questions that elicit missing details without "spoiling" the answer by revealing information the student hasn't yet acknowledged.
The Information Gap Problem
Traditional Question Generation (QG) models usually focus on generating questions that are answerable from a single piece of text. However, they often struggle with the pragmatics of dialogue. For example, if a student provides an incomplete answer like "A man is playing guitar," and the full fact is "A man in a blue hat is playing guitar," a standard model might ask, "What color is the man's hat?" This is considered a poor pedagogical question because it reveals the existence of a hat—information the student had not yet mentioned.
The goal of GFQ is to generate questions that are natural, relevant, and respect the "common ground" between speakers, ensuring the teacher doesn't ask something the student already knows or reveal too much too soon.
The GFQ Pipeline
The authors propose a three-step pipeline that leverages general-purpose NLP models (specifically T5-xxl and the Berkeley Neural Parser) rather than training a single end-to-end model:
- Step 1: Generate Answerable Questions ($P1$): The model identifies all sub-phrases (constituents) in the complete source text ($T_C$) and generates questions for each of these spans.
- Step 2: Filter Common Ground ($P2$): To avoid redundancy, a Question Answering (QA) model checks if the student’s partial text ($T_S$) can already answer a generated question. If it can, that question is discarded.
- Step 3: Minimize "New" Information ($P3$): Inspired by Grice’s Maxims of communication, the model ranks the remaining questions. It prefers questions that reveal the least amount of new information. For instance, asking "What is the man wearing?" is better than "What color is the hat?" because the latter "leaks" the fact that there is a hat.
Evaluation and Results
The researchers evaluated their model using the SNLI (Stanford Natural Language Inference) dataset, treating the "premise" as complete teacher knowledge and the "entailed hypothesis" as the student's partial answer. They also released a new benchmark of 200 instances paired with human-generated gold-standard questions.
Human annotators rated the models on a scale of 1 to 5 based on how well the question functioned as a follow-up. The results show a clear "staircase" improvement, where each step of the pipeline significantly added to the quality of the questions.
The final model (3.94) performs remarkably close to the human baseline (4.06), proving that linguistic constraints like filtering for common ground and minimizing information leakage are effective proxies for human-like tutoring behavior.
Insights and Future Work
One of the most significant findings is that a zero-shot pipeline can achieve near-human performance without a massive, specialized dataset. By cleverly chaining constituency parsing, QG, and QA, the model successfully navigates the "common ground" constraint.
However, limitations remain. The current model only addresses missing information and does not handle student errors or hallucinations. Furthermore, it focuses on single-turn dialogue. Future work could involve multi-turn interactions where the common ground shifts dynamically, and improving phrasing sensitivity to better match a student's specific vocabulary.
Summary updated