Abstract Large language models (LLMs) have achieved remarkable success across diverse domains. However, their potential as effective language teachers—particularly in complex pedagogical scenarios like teaching Chinese as a second language—remains inadequately assessed. To address this gap, we propose the first pedagogical competence benchmark for LLMs, rigorously evaluating their performance against international standards for Chinese language teachers. Our framework spans three core dimensions: (1) basic knowledge evaluation, covering 32 subtopics across five major categories; (2) international teacher examination, based on data collected from international Chinese teacher certification exams; and (3) teaching practice evaluation, where target LLMs summarize knowledge points and design instructional content for student models, followed by testing the student models to assess the LLM's ability to distill and teach key concepts. We conduct a comprehensive evaluation of 13 latest multilingual and Chinese LLMs. While most models demonstrate promising pedagogical potential, there remains substantial room for improvement in their teaching capabilities. This study contributes to the development of AI-assisted language education tools capable of rivaling human teaching excellence.
Introduction
While Large Language Models (LLMs) are often viewed as walking encyclopedias, a high score on general knowledge benchmarks like MMLU does not necessarily equate to teaching effectiveness. Teaching, particularly for a complex language like Chinese as a second language, requires more than factual recall; it demands the ability to simplify concepts, design structured lesson plans, and adapt to a learner's specific needs. Existing benchmarks focus primarily on static knowledge or logical reasoning, failing to capture the "pedagogical competence" required to explain why a grammar point is used or how to scaffold a lesson for a struggling student. This study addresses this gap by introducing CLTE (Chinese Language Teaching Evaluation), a framework that treats the LLM as a professional educator.
Methodology: The Three Dimensions of CLTE
The researchers propose a three-dimensional evaluation framework designed to simulate the professional lifecycle of a language teacher:
- Basic Knowledge Evaluation (BKE): A rigorous test of foundational knowledge across 32 subtopics, including linguistics, pedagogy, and culture.
- International Teacher Examination (ITE): This dimension uses authentic questions from international Chinese teacher certification exams to test "situational" professional literacy and the application of knowledge in classroom settings.
- Teaching Practice Evaluation (TPE): The most novel component of the benchmark, TPE uses a "Teacher-Student" simulation. A target LLM (the "Teacher") is given a teaching guideline and asked to generate instructional content. This content is then fed to a smaller, "weaker" LLM (the "Student"). The Teacher's effectiveness is measured by the improvement in the Student's test scores after consuming the Teacher's lesson.
The CLTE Benchmark Dataset
The CLTE benchmark consists of 2,364 total questions curated from master’s entrance exams, TCSOL (Teaching Chinese to Speakers of Other Languages) mock tests, and official certification exams. Additionally, it includes 120 instructional materials and guidelines specifically for the Teaching Practice Evaluation. The data is categorized into five core domains: Linguistics, Chinese Culture, Pedagogy, World Culture, and Cross-cultural Communication.
Results and Analysis
The evaluation of 13 state-of-the-art LLMs revealed several key insights into the current state of AI in education:
- Reasoning Models Lead the Way: "Thinking" or reasoning models, such as DeepSeek-R1 and OpenAI o1-mini, generally outperformed standard chat models. Their ability to structure logic translates effectively into structured teaching content.
- The Knowledge-Pedagogy Gap: Models are significantly better at memorizing facts (BKE) than they are at applying them in a teaching context (TPE). While DeepSeek-R1 achieved a high BKE score of 0.869, its performance in practical teaching scenarios shows there is still room for growth.
- Linguistics is the Greatest Challenge: Across almost all models, scores in Linguistics were lower than in Culture or Pedagogy. The nuances of language structure and phonetic rules remain a significant hurdle for AI compared to more fact-based subjects.
- Human Comparison: In the BKE category, DeepSeek-R1 (0.869) comfortably outperformed human "laypeople" (0.613) but still lagged behind human "experts" (0.976).
Discussion and Limitations
While the results are promising, the authors identify several areas for future development. The current TPE is a one-way broadcast (Teacher writes, Student reads), whereas real teaching is a dynamic, multi-turn conversation. Future versions of the benchmark aim to include conversational simulations and a wider diversity of "student" models to test instructional adaptability. Furthermore, expanding evaluation beyond multiple-choice questions to include error correction and dialogic scaffolding will provide a more nuanced view of a model's pedagogical prowess. This study serves as a foundational step toward AI-assisted language education tools that can eventually rival human teaching excellence.
Summary updated