FOCUS: A Benchmark for Targeted Socratic Question Generation via Source-Span Grounding

Surawat Pothong, Machi Shimmei, Naoya Inoue, Paul Reisert, Ana Brassard, Wenzhi Wang, Shoichi Naito, Jungmin Choi, Kentaro Inui2025 · DOI

Abstract

Surawat Pothong, Machi Shimmei, Naoya Inoue, Paul Reisert, Ana Brassard, Wenzhi Wang, Shoichi Naito, Jungmin Choi, Kentaro Inui. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics.

2025.

Introduction

Socratic Questioning (SQ) is a cornerstone of critical thinking pedagogy. Rather than providing direct answers, a teacher asks questions that guide students to uncover logical inconsistencies or gaps in their own reasoning. While automating this process is a long-standing goal in AI education, current models often produce generic questions such as "Can you explain that further?" Such broad inquiries lack the precision necessary for a student to identify which specific part of their argument requires revision.

The FOCUS benchmark addresses this by introducing "source-span grounding." By linking every question to a specific segment of text (the "where") and a specific logical weakness (the "why"), the framework enables AI to provide actionable, targeted feedback.

The FOCUS Framework

The authors propose a new task setting that moves away from "black-box" generation toward a structured, three-step process:

  1. Classify the specific argumentative weakness.
  2. Extract the relevant text span (the source span) from the argument.
  3. Construct a fluent question by inserting that span into a predefined exemplar template.
The FOCUS workflow demonstrates how a specific text span is identified and used to construct a targeted Socratic question.

Fine-Grained Typology

To achieve this precision, the researchers expanded traditional broad categories (like "Clarification") into a detailed 11-type Focused Socratic Question Type (FSQType) system. This typology includes nuanced logical flaws such as Temporal Contrast, Implicit Existence, and Causality Flipped, allowing for much more specific probing of a student's logic.

The proposed typology expands existing broad categories into 11 specific subcategories with associated question exemplars.

Datasets and Methodology

The benchmark was developed using several key data sources:

  • FOCUS Dataset: 440 manually annotated instances derived from Reddit's r/ChangeMyView. These were filtered for "contextual sufficiency" to ensure the AI has enough information to identify flaws without external knowledge.
  • LogicClimate: A specialized dataset of 50 climate-related sentences used to evaluate the framework's ability to generalize to complex, scientific domains.

Key Results and Insights

  • High Human Reliability: The dataset's quality is validated by high inter-annotator agreement, with PABAK scores ranging from 0.63 to 1.00.
  • LLM Performance Gaps: Even advanced models like GPT-4 and LLaMA-3 struggle with the FOCUS task. While they can identify explicit flaws like "Lacks Evidence," they often fail at "reading between the lines" for categories like "Bias and Subjectivity."
  • Overgeneralization Bias: Error analysis revealed that LLMs tend to label most flaws as general reasoning errors rather than identifying specific nuances like "Implicit Existence."
  • Generalization: The framework successfully transferred to the LogicClimate dataset with an observed agreement of 0.88, suggesting the 11-type typology is robust across different subject matters.

Conclusion

The FOCUS benchmark demonstrates that actionable AI feedback requires precision in both location and logic. By grounding Socratic questions in specific source spans, AI tutors can move beyond vague prompts to provide transformative educational support. Future work will focus on expanding the benchmark to multiple languages and developing "discourse-aware" models capable of handling more complex, multi-claim arguments.

Summary updated

BibTeX
@misc{pothong2025focus,
  author = {Surawat Pothong and Machi Shimmei and Naoya Inoue and Paul Reisert and Ana Brassard and Wenzhi Wang and Shoichi Naito and Jungmin Choi and Kentaro Inui},
  title = {{FOCUS: A Benchmark for Targeted Socratic Question Generation via Source-Span Grounding}},
  year = {2025},
  doi = {10.18653/v1/2025.ijcnlp-long.157},
}
Figure