Training Turn-by-Turn Verifiers for Dialogue Tutoring Agents: The Curious Case of LLMs as Your Coding Tutors

Jian Wang, Yinpei Dai, Yichi Zhang, Ziqiao Ma, Wenjie Li, Joyce Chai

TL;DR

This paper introduces TRAVER, a novel workflow for training Large Language Models (LLMs) to act as effective coding tutors. By combining knowledge tracing with a turn-by-turn verifier, TRAVER agents can better adapt to students' individual learning needs and guide them more proactively towards completing complex coding tasks. The authors also present DICT, an automated evaluation protocol for assessing these tutoring agents.

What Problem Does This Paper Address?

Intelligent tutoring systems (ITS) have shown great promise in personalized education, but guiding students through complex, real-world tasks like coding has remained a challenge. Off-the-shelf LLMs, while powerful, often lack the grounding and adaptive capabilities needed for effective tutoring. They struggle to understand a student's knowledge state and to proactively offer guidance that truly helps them learn. Assessing the performance of these tutoring agents is also difficult, as traditional methods are time-consuming and costly. This paper aims to bridge this gap by developing more effective LLM-powered coding tutors and a robust evaluation framework.

Methods

The core innovation is the Trace-and-Verify (TRAVER) workflow. It has two key components:

  1. Knowledge Tracing (KT): At each turn of the dialogue, TRAVER explicitly estimates the student's knowledge state regarding specific "knowledge components" (KCs), which are defined as reference dependencies or solution steps for the coding task. This estimation helps the tutor agent tailor its language to address the student's specific knowledge gaps.
  2. Turn-by-Turn Verifier: To select the most effective tutor utterance from a set of candidates, TRAVER employs a verifier model. This verifier, trained using a reward signal, evaluates potential responses based on their contribution to the overall tutoring goal and the current dialogue context. This allows the agent to prioritize utterances that advance the student's learning progress.

The authors also introduce Dialogue for Coding Tutoring (DICT), an automatic evaluation protocol. DICT uses LLM-based simulated students with varying levels of programming knowledge to engage in dialogues with tutor agents. The effectiveness of the tutoring is then assessed through automated unit tests of the code generated by the simulated students after the tutoring session.

Datasets

The primary dataset used is EvoCodeBench (Li et al., 2024), a benchmark for repository-level code generation. The authors preprocessed this dataset to include 100 target coding tasks, along with their associated code contexts, reference dependencies, and solution steps. For simulating students and evaluating tutor agents, they utilized LLMs like Mixtral-8x7B-Instruct and GPT-40.

Key Insights and Ideas

  • LLMs need more than just instructions: Simply using LLMs as tutors with basic prompts is insufficient for effective coding education. They require specialized architectures and training methodologies.
  • Proactive guidance is crucial: The TRAVER workflow's emphasis on turn-by-turn verification and knowledge tracing allows tutors to actively guide students, rather than just passively answering questions.
  • Adaptability is key to success: The performance gap between tutoring low-level and high-level students highlights the need for tutors that can truly adapt their strategies to individual learning needs. TRAVER shows promise in narrowing this gap.
  • Automated evaluation is vital: The DICT protocol offers a scalable and reproducible way to assess tutoring agents, accelerating development cycles.

Key Results

  • TRAVER significantly outperforms baseline LLM tutors in terms of Pass Rate (functional correctness of generated code) and Recall (acquisition of dependency knowledge). For example, TRAVER built on Llama-3.1-70B-Instruct achieved a Pass rate of 39.3% compared to 35.1% for Vanilla Instruct.
  • TRAVER demonstrates substantial improvements across students with different knowledge levels, effectively narrowing the gap with an "Oracle" tutor (which has perfect knowledge).
  • Ablation studies show that both knowledge tracing and the verifier component are critical for TRAVER's performance, with the verifier playing a particularly important role in improving the Pass rate.
  • Human evaluations indicate that TRAVER tutors are significantly better in proactivity and adaptability compared to other methods, while maintaining comparable coherence.

Limitations and Future Work

  • Simulated Students: The study relies on LLM-simulated students, which, while convenient and scalable, may not perfectly replicate the nuances of human learning behavior. Future work should focus on improving the realism of these simulations and conducting studies with actual human students.
  • Real-world Deployment: The current evaluation is primarily conducted with simulated students. Further research is needed to assess how these TRAVER agents perform when interacting directly with human learners.
  • Generalizability: While the paper uses coding tutoring as an example, extending TRAVER to other complex task-tutoring domains would be a valuable next step.

Summary updated

BibTeX
@inproceedings{wang2025training,
  author = {Jian Wang and Yinpei Dai and Yichi Zhang and Ziqiao Ma and Wenjie Li and Joyce Chai},
  title = {{Training Turn-by-Turn Verifiers for Dialogue Tutoring Agents: The Curious Case of LLMs as Your Coding Tutors}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure