TL;DR
Large Language Models (LLMs) show promise as AI tutors, but evaluating their actual teaching ability is tricky. This paper introduces TeachBench, a new framework that assesses LLMs by measuring how much a simulated student improves after multi-turn instruction based on structured knowledge points. Their findings reveal that LLMs' teaching skills vary significantly across subjects and that current models struggle to effectively integrate example problems into their teaching, often defaulting to error correction.
What Problem Does This Paper Address?
While LLMs excel at problem-solving, their ability to teach is less understood. Existing benchmarks often focus on whether an LLM can solve a problem or provide basic guidance, but they don't truly assess pedagogical skills like explaining concepts, guiding learning, and fostering understanding. A key challenge is evaluating teaching effectiveness without "information leakage" – where the LLM might have seen the test questions during training, invalidating the results. The authors aim to fill this gap by creating a robust evaluation framework that isolates and measures the teaching capability of LLMs.
Methods
The core of TeachBench is a syllabus-grounded evaluation framework. Here's how it works:
- Knowledge Structuring: They first convert exam syllabi into a structured knowledge tree. This tree organizes topics hierarchically, providing a curriculum for the LLM teacher.
- LLM-based Question Tagger: An LLM is used to tag existing exam questions with relevant knowledge points from the syllabus tree.
- LLM-based Question Generator: Another LLM generates example problems of varying difficulty for each knowledge point. This is crucial for providing practice material without revealing the actual test questions.
- Teaching Loop: A simulated student agent (another LLM with moderate capability) interacts with the teacher agent (the LLM being evaluated) in a multi-turn dialogue. The teacher agent is only given the knowledge points and example problems, not the target questions.
- Evaluation: The student agent's performance is measured before and after the teaching interaction on held-out questions. The improvement in the student's score is the primary metric for teaching effectiveness.
This setup ensures that the evaluation focuses on the LLM's ability to impart knowledge and guide learning, rather than just its problem-solving prowess or ability to recall answers.
Datasets
The primary dataset used is from the Chinese National College Entrance Examination (Gaokao). This includes:
- Syllabi for seven subjects (Mathematics, Physics, Chemistry, Biology, History, Geography, and Politics).
- Corresponding Gaokao questions from the C-Eval benchmark.
The authors constructed a new knowledge-structured benchmark based on this data, which includes annotated knowledge paths for each question and generated example problems.
Key Insights and Ideas
- Teaching is a distinct skill: The framework successfully differentiates LLMs based on their teaching ability, showing it's a measurable dimension separate from raw problem-solving.
- Example problems can backfire: Current LLMs tend to shift from conceptual teaching to example-specific error correction when given example problems. This can fragment the teaching process and reduce overall effectiveness.
- Domain matters: LLMs are better teachers in subjects where knowledge can be directly applied (like Math, History, Politics) and struggle in domains requiring deeper integration and complex scenario interpretation (like Physics, Chemistry).
- Structured knowledge is key: Even without explicit teaching, access to structured knowledge points provides some benefit, but multi-turn instruction significantly amplifies learning gains.
Key Results
- Qwen3-235B-A22B-Instruct showed the strongest teaching performance, achieving a 7.63-point improvement in Pass@1 accuracy on the Gaokao Mathematics subset.
- Teaching effectiveness varied significantly by subject. Mathematics, History, and Politics yielded the highest improvements, while Physics and Chemistry showed the weakest.
- When example problems were included, performance gains were often smaller than when teaching was based solely on knowledge points. For instance, GPT-5 with examples saw a decrease in Pass@1 improvement compared to GPT-5 without examples.
- The average number of dialogue turns for teaching sessions was relatively low (around 30 turns), suggesting that current LLMs can complete instruction cycles efficiently, though the quality of that instruction is the focus.
Limitations and Future Work
The authors acknowledge several limitations:
- Student Agent Proxy: Using LLMs as student agents is convenient for controlled evaluation but may not fully capture the nuances of human learning.
- No Human Baseline: The framework doesn't compare LLM teachers against human teachers, which would provide valuable context.
- Specific Example-Based Teaching Design: The current analysis of example-based teaching is limited to a particular interaction design; alternative approaches might yield different results.
Future work could involve comparing LLM teachers to human teachers, exploring different interaction designs for example-based teaching, and expanding the framework to more diverse subjects and educational contexts.
Summary updated