Bringing Pedagogy into Focus: Evaluating Virtual Teaching Assistants' Question-Answering in Asynchronous Learning Environments

Li Siyan, Zhen Xu, Vethavikashini Chithrra Raghuram, Xuanming Zhang, Renzhe Yu, Zhou YuarXiv 2025 · DOI · arXiv

Introduction

Asynchronous Learning Environments (ALEs), such as online discussion forums (e.g., Piazza, Canvas), are fundamental to modern education. However, managing these platforms is a significant burden for instructors, often resulting in delayed feedback that diminishes student engagement. While AI-powered Virtual Teaching Assistants (VTAs) offer a scalable solution, existing evaluation metrics—often limited to surface-level linguistic scores like BLEU or ROUGE—fail to capture the pedagogical quality of AI responses. This research introduces a rigorous framework to bridge the gap between AI engineering and learning science, defining what constitutes effective teaching in a forum setting.

The Five-Level Pedagogical Framework

To evaluate VTAs, the researchers developed a five-level model grounded in the ICAP (Interactive, Constructive, Active, Passive) theory of learning. These levels represent increasing instructional difficulty:

  1. Level 1: Clarify Misunderstandings – Correcting factual errors or confusion.
  2. Level 2: Deepen Disciplinary Understanding – Providing deeper conceptual context.
  3. Level 3: Foster Higher-Order Thinking – Encouraging analysis, evaluation, or creation.
  4. Level 4: Enhance Metacognitive Awareness – Helping students reflect on their own learning process.
  5. Level 5: Foster Collaborative Knowledge Construction – Facilitating peer-to-peer interaction.
An example of a VTA response evaluated against the pedagogical framework dimensions.

Methodology and Data

The study utilized a dataset of 600,000 real-world forum posts from a large US university, focusing on 300 academic questions across 85 courses. Responses were generated using Llama-3-70B-Instruct and rated by human experts (education graduate students) to establish a "gold standard" dataset of 600 annotated pairs. To scale evaluation, the team developed automated classifiers using GPT-4o, Qwen, and ModernBERT, optimized via DSPy and augmented with synthetic data generated by GPT-4.

Key Results and Insights

The evaluation revealed several critical findings regarding the current state of AI in education:

The Context Paradox

A significant finding was the "context paradox": providing VTAs with "Forum-Level Context" (e.g., similar peer posts) actually degraded performance in Levels 2, 3, and 4. This is attributed to the "lost in the middle" phenomenon, where extra context distracts the model from its core instructional goal. Specifically, 59.7% of responses in Level 2 saw a score decrease when context was added.

The Pedagogical Ceiling

While VTAs perform well at clarification (Level 1), their effectiveness drops sharply as the required pedagogical skill increases. Level 5 remains largely unattainable for current single-post processing systems, as they lack the holistic view of the forum community necessary to foster collaboration.

Distribution of pedagogical scores across different dimensions, showing a decline in performance for higher-order skills.

Automated Evaluation and Model Scaling

Automated evaluators showed promise, with GPT-4o-mini achieving a weighted F1 score of 60.3. Interestingly, fine-tuning smaller models like ModernBERT on high-quality synthetic data boosted their F1 scores by up to 41.7%, suggesting that pedagogical awareness can be effectively distilled into smaller, more efficient architectures. However, these evaluators remain sensitive to formatting, sometimes mistaking professional "style" for pedagogical "substance."

Conclusion

This work highlights the need for a shift from generic "helpfulness" to disciplinary-specific pedagogical evaluation for AI in education. While current LLMs show strength in basic clarification, significant hurdles remain in fostering metacognition and collaboration. Future research will focus on improving classifier robustness across diverse VTA styles and conducting usability studies with live instructors to align AI ratings with professional classroom needs.

Summary updated

BibTeX
@misc{siyan2025bringing,
  author = {Li Siyan and Zhen Xu and Vethavikashini Chithrra Raghuram and Xuanming Zhang and Renzhe Yu and Zhou Yu},
  title = {{Bringing Pedagogy into Focus: Evaluating Virtual Teaching Assistants' Question-Answering in Asynchronous Learning Environments}},
  year = {2025},
  eprint = {2509.17961},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.17961},
}
Figure