A good teacher grading a student's hand-drawn graph or geometry construction does more than mark it wrong. They name the mistaken rule the student followed, for example "plotted cumulative totals instead of per-bin counts" or "drew the perpendicular bisector through the wrong endpoint." Current multimodal models are poor at this kind of diagnosis. The best model on the SketchJudge grading benchmark still trails non-expert humans, and open-source models trail by a wide margin. In text, even GPT-4o loses about thirty points when asked to trace the flawed reasoning behind a wrong answer instead of the correct one. The usual fix is to have a large model write diagnostic labels and fine-tune a small model on them, which is called distillation. That fix runs into two problems. A student trained to imitate a teacher's diagnoses inherits the teacher's preference for correct reasoning. And producing and filtering enough teacher-written data is expensive.
We propose to skip the teacher for the hard part. We write misconceptions as small executable programs, which the education literature calls "mal-rules." We run them on structured, programmatically defined STEM problems, render the resulting wrong student artifacts with hand-drawn-style noise, and train a small vision-language model on the result. Each training label is exact because the program that made the error knows which rule it applied and at which step. No teacher has to recognize the error, so the student is not capped by a teacher's diagnostic accuracy. Generation is cheap rendering with nearly every sample usable, rather than repeated sampling and filtering by a large model.
Core Claim
A 7B-scale multimodal model fine-tuned on mal-rule-executed synthetic artifacts will diagnose errors in real hand-drawn student diagrams better than the same model fine-tuned on teacher-written diagnoses of the identical images. It should approach or exceed the frontier models that would serve as that teacher, at a small fraction of the generation compute.
The key ingredient is the coherence of the errors. A mal-rule produces a systematic, reconstructible mistake, not a random corruption of the image. The model therefore learns to infer the rule from its visible consequences, which is the inverse-planning view of misconceptions. Pattern-matching "something looks off" is not enough to solve these examples.
Where This Sits in Prior Work
Two papers motivate the gap. MalAlgoQA defines malgorithm identification on text multiple-choice items. It finds a strong "correctness bias": GPT-4o reaches 95.7% when identifying correct rationales but only 66.1% on flawed ones, and GPT-3.5 drops from 86.3% to 14.7%. It also finds that chain-of-thought prompting makes malgorithm identification worse. SketchJudge moves diagnosis to hand-drawn geometry, physics, chart, and flowchart responses with a 23-category error taxonomy. It shows that strong solvers are weak graders and that chain-of-thought spreads early misreadings of strokes. Both papers are evaluation-only. Neither trains anything, and neither has a procedural model of where errors come from. We supply that procedural model and use it as a data source.
The method we borrow comes from two sources. Synthesize Step-by-Step builds chart reasoning data from SVG metadata and templates, which yields executable rationales that are correct by construction. It raised human-written ChartQA from 38.2% to 53.9%. Our generator reuses this "structured scene first, then render" idea, but executes wrong procedures instead of right ones. Using Inverse Planning for Personalized Feedback places mal-rules inside the hypothesis space of a model of student problem-solving. We use the same mal-rule library forward, as a simulator, rather than only for inference at diagnosis time.
Several papers constrain the design. Post-Semantic-Thinking shows that errors in teacher rationales corrupt students trained on rationale-then-answer sequences. On the Mechanisms of Weak-to-Strong Generalization argues that capacity-bounded teacher supervision bounds the student. Together they explain why teacher-written diagnoses are the wrong supervision when the teacher itself is the bottleneck.
The closest attempts to get past the teacher ceiling or reduce synthesis cost use text or correct-answer settings. Teaching Small Language Models Reasoning through Counterfactual Distillation trains small models to refute distractors, but it treats wrong options as things to eliminate, not as outputs of coherent student procedures, and it is text-only. DISCO and Selective Reflection-Tuning show that small, well-filtered synthetic sets beat large noisy ones: 75k filtered counterfactual examples in DISCO, and 926 student-selected samples in Selective Reflection-Tuning. Both still rely on a large generator plus a filter. CVPD avoids an external teacher for visual perception, but only 17.2% of its candidates survive filtering and each update needs four forward passes. Our generator's labels need no filter, because they are true by construction.
TrueTeacher is evidence that a student can beat its labeler: a T5-11B student scored 87.8 ROC-AUC versus 84.9 for its 540B teacher. That student still learned from teacher labels on realistic model errors. We remove the teacher from labeling entirely.
Research Program
The generator. The first artifact is a mal-rule library and scene engine covering SketchJudge's four domains:
- Geometry: construction programs over points, lines, and circles.
- Charts: a data table plus a plotting specification.
- Physics: free-body and circuit scenes as typed graphs.
- Flowcharts: control graphs.
For each domain we write a correct solution procedure and then a family of mal-rules as edits to that procedure. Examples include swapping axes, using cumulative instead of per-category values, omitting a normal force, reversing a branch condition, and bisecting the wrong segment. We author mal-rules against SketchJudge's 23-category taxonomy and against the math malgorithm descriptions in MalAlgoQA, so that coverage of real error types is measured rather than assumed. Each executed mal-rule yields three things:
- a wrong final artifact;
- a step-indexed trace marking where it diverges from the correct procedure;
- a label naming the rule and its taxonomy category.
A rendering stage then turns the symbolic artifact into a sketch-like image. It applies stroke jitter, uneven line weight, overshoot, partial erasures, and hand-lettered labels from handwriting fonts. Its purpose is to close the perception gap that SketchJudge identifies.
The student. Next we train an open multimodal student, Qwen2.5-VL-7B (the perceiver used in RAPID). Its input is the problem statement, plus the reference figure where there is one, plus the rendered student response. It is trained to output a correct/incorrect verdict, the error category, and a short localized statement of the divergent step.
Because we have exact traces, we can directly test the open contradiction about rationale ordering. We compare training the model to emit the diagnosis before the trace (post-thinking, following Post-Semantic-Thinking) against emitting the trace first. We also check whether either ordering avoids the chain-of-thought degradation reported by MalAlgoQA and SketchJudge.
Accounting. Throughout, we record the cost of producing one accepted training example in GPU-hours and dollars, and the yield (accepted over generated). Generation-overhead claims are then measured, not asserted.
Baselines
The naive reference point is the untuned student, Qwen2.5-VL-7B, prompted zero-shot and with chain-of-thought on SketchJudge. SketchJudge reports open-source models including Qwen2.5-VL at between 53% and 69% grading accuracy. We re-run our exact checkpoint under the benchmark's protocol so that the reference is our own number, not a range.
The strongest existing system is a frontier model prompted directly. On grading accuracy, SketchJudge reports 78.42% for GPT-5, 77.74% for Gemini-2.5-Flash, and 83.33% for human non-experts. We treat GPT-5's 78.42% as the bar a 7B student must approach. Matching it would be a strong result. Exceeding it would show that the teacher ceiling has been broken, because GPT-5-class models are the natural teachers for the alternative pipeline. No per-category diagnostic accuracy is reported in the file, so for error-type identification we compute frontier numbers ourselves on the same items.
The most important baseline isolates label source: teacher-labeled distillation on identical images. We take exactly the rendered artifacts our generator produces, discard the program labels, and have a frontier model write diagnoses. Here "frontier model" means the strongest one we can afford, with GPT-4o as a floor because MalAlgoQA reports its 66.1% malgorithm identification accuracy. We then train the same student with the same steps, learning-rate sweep, and number of examples. As a stronger variant, we also train on teacher labels filtered by the student's own difficulty scores, following Selective Reflection-Tuning, with the training set matched in size after filtering. Beating this variant on SketchJudge and on held-out mal-rules shows that program-executed supervision, not just the images, is responsible for the gain.
Two ablations isolate the remaining ingredients. In the incoherent-corruption ablation, we replace mal-rules with random perturbations of the same magnitude (moved points, deleted strokes, relabeled axes) that do not correspond to any consistent student procedure. Labels are still exact, but there is no misconception structure. If this matches the full method, the misconception-modeling claim fails and the method reduces to error-detection augmentation. In the clean-render ablation, we remove sketch noise, which tells us how much of any gain comes from closing the perception gap rather than the reasoning gap.
Finally, we compare against a perceiver-plus-reasoner pipeline in the style of RAPID. The student's vision side captions the sketch and a strong frozen text reasoner diagnoses the caption. Its inference compute is matched or exceeded relative to our model. All trained variants share the same base checkpoint, the same tuning budget, and selection on the same validation split drawn from synthetic data only. SketchJudge is never used for model selection.
Why not just…?
Why not just prompt GPT-5 or Gemini directly? For a grading service with no cost or privacy constraints, this is the honest default, and it is reasonably strong at 78.42% on SketchJudge. Three things argue against stopping there. It still trails non-expert humans. Prompting cannot be trained toward the specific error taxonomy a curriculum uses. And the text evidence from MalAlgoQA (66.1% malgorithm identification for GPT-4o, versus 95.7% when identifying correct rationales) suggests the gap is concentrated in naming the misconception rather than detecting that an error exists. We will report frontier grading accuracy and error-category accuracy separately. If frontier models are near-ceiling on categories as well as verdicts, the case for our method shrinks to cost and openness, and we will say so.
Why not just distill from a frontier teacher's diagnoses, perhaps with good filtering? This is the real threat, and the label-source baseline is designed to settle it. The prior reason to expect it to underperform is that correctness bias lives in the teacher, so filtering by student difficulty cannot recover labels the teacher got wrong. Selective Reflection-Tuning selects for compatibility, not truth. There is also a cost side. Every teacher label costs a frontier call, and if we verify those labels against the program anyway, the teacher adds nothing. If filtered teacher labels match program labels on SketchJudge, our contribution reduces to the generator and its cost savings.
Why not just caption the sketch and let a strong text model diagnose it? Prior evidence is mixed. GeoSQA shows that even human-written diagram descriptions left models at chance. That result predates modern language models and concerns solving, not grading. RAPID shows that captions not tuned to the reasoner's needs drop the relational details reasoning depends on. Captioning a wrong drawing is especially risky, because a captioner biased toward correct answers may "fix" the error in its description. The RAPID-style baseline tests this directly, and we will inspect whether its failures happen at the captioning step.
Why not just fine-tune on real student sketches? SketchJudge has only 1,015 responses and serves as our held-out test, so it cannot also be training data. Collecting and expert-labeling real work at the scale our generator produces in hours is exactly the overhead we are trying to remove. As a ceiling check, we will run a k-fold experiment that fine-tunes on part of SketchJudge, to see how much real data our synthetic data is worth.
Proposed Experiments and Decisive Checks
Transfer to real sketches. The headline experiment trains on synthetic data only and evaluates on all of SketchJudge. We report grading accuracy against the 78.42% GPT-5 figure, and error-category accuracy against frontier models we run ourselves, broken down by the four domains.
Held-out mal-rule families. We withhold entire mal-rule families, for example all axis-scaling errors, from training and test diagnosis on artifacts they generate. This checks whether the student infers unfamiliar procedures or only memorizes the rules it saw. It is also the cleanest test of whether the student surpasses a teacher. On held-out families the teacher-labeled student and the program-labeled student see the same images, so any gap comes from label quality.
Coverage. We report how many of SketchJudge's 23 categories our library expresses, and accuracy on covered versus uncovered categories. If gains appear only on covered categories, the method works but requires authoring effort that scales with the curriculum.
Text transfer. We evaluate the student on the math portion of MalAlgoQA to see whether visual mal-rule training reduces correctness bias in text, using GPT-4o's 66.1% as the comparison.
Solving ability. On MathVista and MATH-Vision we check that diagnosis training does not damage problem-solving.
Ordering and cost. The pre- versus post-thinking comparison is reported on SketchJudge. Cost accounting compares GPU-hours per accepted example and yield against the teacher-labeling pipeline, with CVPD's 17.2% yield as a reference point.
Failure criteria. The proposal fails if any of the following holds: the incoherent-corruption ablation matches the full method; teacher labels match program labels on held-out families; or synthetic-only training does not clearly beat our own measured zero-shot number for the base model on SketchJudge.
References
- MalAlgoQA: Pedagogical Evaluation of Counterfactual Reasoning in Large Language Models and Implications for AI in Education
- SketchJudge: A Diagnostic Benchmark for Grading Hand-drawn Diagrams with Multimodal Large Language Models
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- Using Inverse Planning for Personalized Feedback
- Distilling Reasoning Ability From Large Language Models With Adaptive Thinking
- On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective
- Teaching Small Language Models Reasoning through Counterfactual Distillation
- DISCO: Distilling Counterfactuals with Large Language Models
- Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning
- Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spots
- TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models
- Reasoning-Aligned Perception Decoupling for Scalable Multi-modal Reasoning
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset