Human and LLM-based Assessment of Teaching Acts in Expert-led Explanatory Dialogues

Aliki Anagnostopoulou, Nils Feldhus, Yi-Sheng Hsu, Milad Alshomary, Henning Wachsmuth, Daniel Sonntag2025 · DOI

Abstract

Aliki Anagnostopoulou, Nils Feldhus, Yi-Sheng Hsu, Milad Alshomary, Henning Wachsmuth, Daniel Sonntag. Proceedings of the 6th Workshop on Computational Approaches to Discourse, Context and Document-Level Inferences (CODI 2025).

2025.

Introduction

Effective teaching is a dynamic dialogue where experts adjust their strategies based on a learner's prior knowledge. While general conversation datasets are abundant, there is a lack of fine-grained data capturing specific "teaching acts"—the discourse mechanisms that make an explanation effective. This paper addresses how to computationally represent these moves (e.g., checking prior knowledge vs. providing analogies) and how to measure pedagogical quality across varying audience levels.

Methodology: The ReWIRED Framework

The researchers developed ReWIRED, a multi-stage framework for annotating and evaluating explanatory dialogues.

The ReWIRED framework encompasses expert annotation, LLM experimentation, and qualitative evaluation via IXQuisite.

The approach involved several key components:

  • Span-Level Annotation: Education experts marked specific text spans with one of nine Teaching Acts (e.g., Assess Prior Knowledge, Comparison, Generalization), allowing for overlapping intents within a single sentence.
  • Automated Labeling: Various LLMs (GPT-4o, Gemini 1.5) were tested using structured prompting paradigms (JSON, TANL, GoLLIE) and compared against fine-tuned models like BERT and GPT-4o-mini.
  • IXQUISITE Suite: A framework of 13 metrics categorized into "Function" (pedagogical structure) and "Form" (linguistic features like readability).
  • LLM-as-a-Judge: A comparison between static evaluation (tag counting) and prompt-based LLM evaluation where models rated the quality of teaching acts on a scale of 0–10.

The ReWIRED dataset extends the WIRED "5 Levels" series, featuring 130 dialogues across 26 STEM topics explained to five distinct audience levels: Child, Teenager, Undergraduate, Graduate Student, and Colleague.

Key Insights and Results

A primary finding is that pedagogical strategies shift significantly based on the "explainee's" expertise.

Distribution of teaching acts across audience levels shows how strategies like analogies (T06) and checking prior knowledge (T01) decrease as audience expertise increases.
  • Fine-tuning Performance: Fine-tuned models, even smaller ones like GPT-4o-mini, achieved near-perfect performance (99.04% Macro-F1) in identifying teaching acts, significantly outperforming few-shot frontier models which reached only ~74%.
  • Discourse Grammar: Clear patterns emerged in audience adaptation. For instance, Checking Prior Knowledge and Analogies are heavily used for children but taper off as the audience's expertise increases.
  • Evaluation Efficacy: Traditional static metrics (counting keywords or word length) failed to capture functional quality. In contrast, prompt-based LLM evaluations aligned remarkably well with human expert judgments, successfully perceiving the graded relationship between teaching strategy and learner level.

Limitations and Future Directions

The dataset is currently focused on STEM topics from a specific video series, and the authors note that teaching acts may vary in other domains like humanities. Future work aims to incorporate multimodal data—such as tone, prosody, and gestures—and explore real-time feedback systems for AI tutors to self-assess their pedagogical strategies during live interaction.

Summary updated

BibTeX
@misc{anagnostopoulou2025human,
  author = {Aliki Anagnostopoulou and Nils Feldhus and Yi-Sheng Hsu and Milad Alshomary and Henning Wachsmuth and Daniel Sonntag},
  title = {{Human and LLM-based Assessment of Teaching Acts in Expert-led Explanatory Dialogues}},
  year = {2025},
  doi = {10.18653/v1/2025.codi-1.15},
}
Figure