When a large language model (LLM) is asked to solve a complex calculus problem or debug a broken piece of Python code, it often delivers a flawless solution in seconds. Yet, if you ask that same model to act as a tutor and help a struggling student solve the exact same problem, it frequently stumbles. It might give away the answer too quickly, fail to diagnose why the student is confused, or provide a hint that is completely unaligned with the student's current knowledge state.
This friction highlights a fundamental realization in educational AI: problem-solving proficiency does not naturally translate into teaching proficiency. This gap is known as the solver-tutor disparity. To build effective automated instructors, we must understand why general-purpose models fail at pedagogical reasoning and how researchers are restructuring them to bridge this gap.
The Paradox of the Fluent Solver
Intuitively, we tend to assume that deep subject-matter expertise is the primary prerequisite for teaching. In the context of AI, however, benchmarks show a stark divergence between academic knowledge and instructional capability.
In TutorBench, Rakshith S. Srinivasa evaluated frontier LLMs specifically on pedagogical tasks like identifying student misconceptions and providing active learning support. Even advanced models struggled significantly, revealing that high general reasoning capabilities do not automatically guarantee effective instructional strategies. Similarly, Jakub Macina, in introducing the MathDial dataset, demonstrated that models capable of solving math problems routinely fail at tutoring because they reveal solutions prematurely or provide incorrect pedagogical feedback.
This phenomenon occurs because conversational fluency is distinct from instructional utility. As shown by Anaïs Tack et al. in The AI Teacher Test, a model can excel at maintaining a fluid, engaging dialogue (conversational uptake) while completely failing to address a student's underlying confusion. Sounding like a teacher is not the same as teaching.
Why Standard Alignment Fails Education
The root cause of the solver-tutor disparity lies in how modern LLMs are trained. Through Reinforcement Learning from Human Feedback (RLHF), standard instruction-tuned models are optimized to be "helpful, honest, and harmless." In practice, "helpfulness" is mathematically aligned with maximizing information delivery—giving the user a direct, comprehensive answer as efficiently as possible.
In education, this objective function is toxic. Effective tutoring relies on scaffolding: the practice of withholding the final answer and offering incremental, strategic hints that force the learner to perform the heavy cognitive lifting.
Standard LLM Objective:
[Student Query] ──> [Optimize for Helpfulness] ──> Give Direct Answer (Spoiler)
Pedagogical LLM Objective:
[Student Query] ──> [Optimize for Scaffolding] ──> Diagnose Misconception ──> Provide Minimal Hint
As Soonwoo Kwon et al. observe in their work on BIPED, general-purpose models prioritize direct answers over guiding a learner's progress because they lack defined pedagogical boundaries. This conflict is deeper than simple prompting can fix. Giulia Bonino et al. note in EULER that standard instruction-tuning objectives directly counteract Socratic interaction; a highly compliant model inherently struggles to withhold information that a student needs to discover independently. Furthermore, general dialogue reward models are typically trained on web data where the goal is information retrieval, creating a fundamental domain mismatch for educational environments (Justin Vasselli et al. in NAISTeacher).
The Cognitive Blind Spot: Modeling Student Errors
An effective tutor does not just know the correct path; they must also understand the incorrect paths a student is likely to take. This requires a cognitive model of human error—a capacity current LLMs largely lack.
This deficiency becomes obvious when attempting to use LLMs to design assessments, specifically multiple-choice question "distractors" (the incorrect but plausible options). When a human teacher designs a distractor, they target a specific, predictable student misconception. For instance, if a student is adding fractions ($\frac{1}{2} + \frac{1}{3}$), a common procedural error is adding the numerators and denominators directly to get $\frac{2}{5}$.
LLMs struggle to generate these high-quality distractors. In Exploring Automated Distractor Generation, Wanyong Feng et al. discovered that while LLMs can easily generate mathematically incorrect answers, they fail to replicate the specific, messy procedural errors (like digit misplacement or decimal slips) common among human learners. The models treat errors as statistical noise or logical deviations rather than the output of a structured, albeit flawed, mental model.
The authors of Math Multiple Choice Question Generation via Human-Large Language Model Collaboration describe this as a lack of pedagogical empathy. They documented a frequent failure mode where an LLM could qualitatively describe a student's misconception perfectly, but then fail to perform the exact flawed calculation required to generate the corresponding numerical distractor. The model's internal solver logic overrides its ability to simulate being "human-wrong."
Architectures for Pedagogical Alignment
To bridge the solver-tutor disparity, researchers are moving away from zero-shot prompting of general models and toward architectures that explicitly enforce educational constraints.
1. Intent-First Decoupling
Instead of mapping a student's input directly to a textual response, modern tutoring systems decouple the pedagogical strategy from the linguistic generation. For example, the framework proposed by Soonwoo Kwon et al. in BIPED uses a two-step process: the model must first explicitly select a "tutor act" (such as offering a hint, validating a partial step, or probing a misconception) from a granular taxonomy before it is allowed to generate the actual response text. This prevents the model's default "answer-provider" instinct from hijacking the interaction.
2. Preference Optimization for Withholding
To train models to resist giving away answers, researchers are shifting the alignment phase using Direct Preference Optimization (DPO). In EULER, Giulia Bonino et al. used DPO to explicitly penalize "spoiler" answers and reward conversational turns that guide students through questioning. By curating preference pairs where the dispreferred response is highly informative but pedagogically passive, they successfully steered smaller open-source models to adopt a sustained Socratic style.
3. Cycle-Consistent Error Modeling
To fix the distractor generation problem, models must learn how to simulate incorrect reasoning chains. The authors of Learning to Make MISTAKEs introduced a framework that models the explicit relationship between a latent misconception and a final incorrect answer. By forcing the model to maintain cycle consistency—generating a wrong answer from a misconception, and conversely inferring the misconception from a wrong answer—they trained models to generate distractors that mirror authentic classroom errors rather than arbitrary mathematical noise.
4. Overgenerate-and-Rank Pipelines
Rather than relying on an LLM to generate the perfect pedagogical response on the first try, researchers are leveraging discriminative ranking models trained on real student data. Alexander Scarlatos et al. demonstrated this in Improving Automated Distractor Generation for Math Multiple-choice Questions with Overgenerate-and-rank. By generating a wide pool of candidate distractors and using a secondary model optimized via DPO on real-world student response logs, they allowed smaller, specialized models to match or exceed the diagnostic utility of frontier models like GPT-4.
Moving Forward
The transition of LLMs from fluent conversationalists to effective intelligent tutoring systems requires moving past the assumption that raw scale solves all downstream tasks. Teaching is not merely the transmission of correct facts; it is the targeted, strategic management of another mind's misconceptions. Resolving the solver-tutor disparity requires that we continue to build architectures that treat human error not as noise to be filtered out, but as a structured logic to be understood and guided.
Key Foundational Work
The Geometry Tutor (1985) — contributed 1 concept elements to this direction.
Extracting Student Models for Intelligent Tutoring Systems (2007) — contributed 1 concept elements to this direction.