Bringing Pedagogy into Focus: Evaluating Virtual Teaching Assistants’ Question-Answering in Asynchronous Learning Environments

Li Siyan, Zhen Xu, Vethavikashini Chithrra Raghuram, Xuanming Zhang, Renzhe Yu, Yu Zhou2025 · DOI

TL;DR

Researchers from Columbia University have introduced a first-of-its-kind pedagogical evaluation framework for Virtual Teaching Assistants (VTAs) in asynchronous online forums. By moving beyond surface-level language metrics, they demonstrate that while current LLM-based assistants are proficient at clarifying basic misunderstandings, they significantly struggle with higher-order teaching skills like fostering metacognition or collaborative knowledge construction.

What Problem Does This Paper Address?

Asynchronous Learning Environments (ALEs)—think online discussion forums or MOOCs—offer flexibility but often suffer from delayed instructor feedback, which can kill student motivation. Virtual Teaching Assistants (VTAs) powered by Large Language Models (LLMs) are a promising solution, but we have a major "measuring" problem: how do we know if they are actually teaching well?

Most current evaluations rely on generic NLP metrics (like BLEU or ROUGE) that measure how much an AI response looks like a human one, or simple "helpfulness" scores. These don't capture pedagogical effectiveness—the ability to deepen a student's thinking or encourage peer interaction. This paper bridges the gap between AI engineering and learning sciences by creating a theory-grounded rubric to evaluate the "instructional difficulty" of VTA responses.

Methods

The authors developed a two-pronged approach:

  1. The Pedagogical Skill Model: Grounded in the ICAP framework (Interactive, Constructive, Active, Passive), they defined five hierarchical levels of teaching capability:
  2. Level 1: Clarify Misunderstandings.
  3. Level 2: Deepen Disciplinary Understanding.
  4. Level 3: Foster Higher-Order Thinking.
  5. Level 4: Enhance Metacognitive Awareness.
  6. Level 5: Foster Collaborative Knowledge Construction.
  7. Automated Evaluation: To make this scalable, they built LLM-based classifiers. They experimented with:
  8. Prompt Optimization: Using the DSPy framework and the SIMBA optimizer to find the best instructions for GPT-4o and GPT-4o-mini to act as "expert" graders.
  9. Open-Weight Fine-Tuning: They fine-tuned smaller models (Qwen3-4B/8B and ModernBERT) on synthetic data to see if specialized, local models could match the performance of proprietary giants.

Datasets

  • Source Data: A massive dataset of 600,000 real-world forum posts from a large U.S. university. From this, they filtered for 300 diverse academic questions across 85 different courses (from criminal law to physics).
  • Expert Annotations: Two education experts annotated 600 unique post-response pairs using the new rubric, achieving a high Inter-Rater Reliability (ICC of 0.78).
  • Synthetic Data: To train the open-weight classifiers, they used GPT-4.1-nano to generate ~1,200 synthetic examples per pedagogical level, carefully balanced across the scoring rubric.

Key Insights and Ideas

  • The "Context Paradox": Surprisingly, providing the VTA with "forum-level context" (similar posts from other students) actually decreased the pedagogical quality for Levels 2-4. The models often got "lost in the middle" of the long context, shifting from deep engagement with the student's specific problem to providing broader, more superficial summaries.
  • The Ceiling of AI Pedagogy: LLMs are currently "Level 1" masters but "Level 4" amateurs. They are excellent at correcting facts but struggle to prompt students to reflect on their own learning process (metacognition).
  • Markdown Sensitivity: The researchers found that LLM evaluators are highly sensitive to stylistic cues like Markdown formatting. Removing bolding or bullet points changed how the AI "graded" the response, whereas human experts were able to look past the formatting to the underlying pedagogical content.
  • Synthetic Data Works: Fine-tuning smaller models on high-quality synthetic data boosted F-1 scores by up to 41.7%, suggesting that we don't necessarily need "frontier" models to perform specialized educational assessments.

Key Results

  • Human-AI Alignment: Automated classifiers performed "decently" at lower pedagogical levels (Level 1 F-1 scores ~85%) but performance plummeted as instructional difficulty increased.
  • Prompt Optimization: Using DSPy's SIMBA optimizer led to a 28.4% F-1 score improvement for Level 4 assessments in GPT-4o, proving that how you ask the LLM to "grade" is just as important as the model itself.
  • Model Comparison: Fine-tuned ModernBERT models maintained competitive performance across most levels compared to prompt-optimized GPT-4o, offering a path for institutions to run these evaluations privately and locally.

Limitations and Future Work

The authors acknowledge that their current training data comes from a single LLM (Llama-3), which might limit the generalizability of the classifiers to other response styles. Furthermore, while the rubric is theoretically sound, the paper does not yet link these VTA pedagogical scores to actual student learning outcomes (e.g., higher exam scores).

Future work aims to expand the expert-annotated dataset and conduct "in-the-wild" usability studies with actual university instructors to see if this framework helps them manage their digital classrooms more effectively.

Summary updated

BibTeX
@misc{siyan2025bringing,
  author = {Li Siyan and Zhen Xu and Vethavikashini Chithrra Raghuram and Xuanming Zhang and Renzhe Yu and Yu Zhou},
  title = {{Bringing Pedagogy into Focus: Evaluating Virtual Teaching Assistants’ Question-Answering in Asynchronous Learning Environments}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.518},
}
Figure