In a large course, most students who get a problem wrong get it wrong in one of a handful of ways. If a system can group submissions by which mistake was made, an instructor can write one careful explanation per group instead of one per student, so feedback cost grows with the number of distinct mistakes rather than the number of students. For typed text and code this grouping is known to work. For handwritten mathematics it fails for a specific reason. The machine first has to read the page, and when it misreads a minus sign or an exponent, a correct derivation lands in an "error" group. A single misconception can also split into several groups because students write it differently. Recent grading studies on real student exams locate the main failure here. Evaluating AI Grading on Real-World Handwritten College Mathematics reports that grading errors come mostly from transcription, not from the language model's mathematics.
We propose to group handwritten derivations only after separating "what did the student write?" from "what did the student get wrong?". The system produces several plausible transcriptions of each submission instead of one. It diagnoses every transcription with a layered check: legibility, then notation, then step-by-step validity, then choice of theorem and justification. A submission is placed into a misconception group only if all of its plausible readings agree on the first faulty step and its type. If the readings disagree, the system has shown that the disagreement is about reading the page, and that submission goes to a quick human legibility check rather than receiving wrong feedback. The bet is that this "agreement across readings" rule sharply reduces cases where students get feedback for mistakes they did not make, while keeping most of the gain from grouping. We test this on university-level courses, where justification errors are hardest for current models.
Core Claim
Grouping submissions to scale feedback on handwritten STEM work only becomes safe when two conditions hold. First, the grouping key must be a layered diagnosis, meaning the first faulty step together with its tier, rather than a whole-answer score or an embedding of the solution. Second, membership must require that the diagnosis stays the same across alternative transcriptions. We claim that this pair produces far fewer misattributed feedback messages than grouping on a single best transcription, at a comparable cost in instructor time. We also claim it beats the existing human-in-the-loop practice of having a person verify every AI grade, measured in instructor minutes per correct feedback message delivered.
Relation to Prior Work
Three deployed systems motivate the problem, and all three scale per submission rather than per misconception. Evaluating AI Grading on Real-World Handwritten College Mathematics segments calculus pages, transcribes them with a language model that is given the problem statement, and grades with a "max of two rubrics" rule. Human-in-the-Loop LLM Grading for Handwritten Mathematics Assessments grades each paper five times with GPT-5.1 and has an instructor confirm every result. Evaluating GPT-4 at Grading Handwritten Solutions in Math Exams grades probability exams with a single prompt. None of them groups errors across the cohort, and none separates reading errors from reasoning errors in its output. We reuse their transcription and rubric ideas and change the unit of human effort.
Two benchmarks supply the layered evaluation idea. Can Vision-Language Models Evaluate Handwritten Math? (FERMAT) and VEHME stage handwritten-math assessment as detect, then localize, then correct, at K-12 level. VEHME also trains an open 7B model that uses visual box prompts to localize errors. We borrow localization as the grouping key and extend the tiers to separate reading errors from notation, step validity, and theorem use. Both papers evaluate individual items and never cohort feedback.
Two papers constrain the design. TheoremQA finds that pasting theorem statements into the prompt does not substantially help models apply them. For that reason, our theorem tier checks against instructor-written, problem-specific justification criteria and does not rely on retrieval. Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset and VCBench report that step-by-step reasoning can rationalize misperceptions. For that reason, we never let the diagnosing model see only one reading.
The closest methodological ancestor is BeaconChart. It re-extracts chart values with an independent verifier that does not see the first pass, and it stores corrective instructions per chart type. Our multiple-readings check is the same anti-confirmation idea applied to handwriting. The difference is that our stored "corrections" are instructor-authored, student-facing explanations attached to misconception groups, not prompt heuristics.
Research Program
The system has four parts.
The first part is reading. Each segmented solution region is transcribed into LaTeX $M$ times by independently prompted readers, with problem-statement context as in the UC Irvine pipeline. The readers are a large proprietary model, an open vision-language model, and Mathpix. We record the line-level variants and merge readings that are symbolically identical.
The second part is layered diagnosis, applied to each reading. Tier 1 flags lines that the readers could not agree on. Tier 2 checks that each line parses as a well-formed expression, with no dropped bounds or unbalanced terms. Tier 3 checks each transition between consecutive lines, using a computer algebra system to test equivalence wherever the step is algebraic. This is the student-side counterpart of the program-based solving that TheoremQA found useful. Tier 4 asks a language model to judge whether the required theorem or justification is invoked validly, against instructor criteria. The output for each reading is a key: the first failing step, its tier, and a short canonical error label.
The third part is grouping by agreement. Let $R_i$ be the set of plausible readings of submission $i$, and let $d(r)$ be the diagnostic key of reading $r$. Submission $i$ joins group $g$ only if $d(r)=g$ for every $r \in R_i$. Otherwise it is routed to a legibility queue, where a human resolves only the disputed lines, which is quick, and the submission is then re-diagnosed. Keys with the same error label but different surface algebra are merged by a language model, and an instructor can split or merge groups.
The fourth part is propagating feedback. The instructor views a representative of each group alongside two random members, writes or edits one explanation, and approves it for the group. Overrides on a group are recorded and attached to that group key. When the same key appears in a later term or section, the stored explanation is proposed as a starting draft.
The domain is university mathematics with justification-heavy items, such as calculus, probability, and linear algebra. This is where Evaluating GPT-4 at Grading Handwritten Solutions in Math Exams found accuracy falling to 16.67% on justification questions. We will collect consented, anonymized exam scans from at least two courses, and instructors will produce gold step-level error annotations and legibility ratings. Expert-annotated artifacts are scarce, so this collection is itself a contribution. We will release the annotation scheme even where privacy rules out releasing the images.
Baselines
The naive reference point is the per-student human-in-the-loop workflow of Human-in-the-Loop LLM Grading for Handwritten Mathematics Assessments: five model passes, then human verification of every submission, with no grouping. It reports a 23.3% reduction in grading time over digital entry and weighted κ of 0.70–0.96 with humans, with significant model errors in about 3% of cases. This is the practice we must beat on instructor minutes per correct feedback message while matching its accuracy. We will rerun it on our data, using the same model family and the same per-submission budget, and timed instructors.
The strongest automatic grading pipeline is the modular approach of Evaluating AI Grading on Real-World Handwritten College Mathematics. It reports over 90% of scores within 1 point of human reviewers, 79.8% of feedback rated fully correct, and an 84% acceptable transcription rate against 55% for Mathpix. We reimplement it with identical segmentation and readers and give it the same tuning effort on a held-out development split. We will count our system as beating it if the expert-rated fully-correct feedback rate rises meaningfully above its rate on our data, at lower instructor time. On FERMAT, the strongest reported correction result is Gemini-1.5-Pro at a 77% correction rate. VEHME's exact FERMAT numbers are not in the file. We found no reported result on Lacuna for grouping misconceptions in handwritten mathematics, so our own naive grouping baseline defines that reference.
The key ablation is identical grouping on the single highest-confidence transcription, with no agreement requirement. Comparing it with the full system isolates the agreement rule. A second ablation replaces the layered key with an embedding of the model's free-text critique and then clusters those embeddings, which isolates the value of the tier structure.
Why not just…?
Why not just keep per-student grading with a strong model and human verification? It works, and it is our naive baseline. Its human cost still grows linearly with enrollment, because a person reads every grade, and its students receive model-written critiques rather than instructor-written explanations. The timed comparison in this proposal will show whether grouping actually saves minutes. If the legibility queue turns out to be so large that savings vanish, the per-student approach wins on that course, and we will report it.
Why not just group on one good transcription? The UC Irvine readers reach 84% acceptable transcription on hard items, which suggests a single reading might be good enough. However, a 16% misread rate is exactly the rate at which correct work would be grouped as wrong. The controlled degradation experiment below tests whether that leakage matters in practice.
Why not just use disagreement among repeated grading passes as the uncertainty signal? This is the most serious alternative, because the five-pass protocol already produces disagreement for free. Its weakness is that repeated grading passes conflate reading uncertainty with reasoning uncertainty. They can also agree confidently on a wrong reading, which is the confirmation effect that BeaconChart counters with an independent extractor. We will run it as a third grouping variant, where a submission is grouped only if the five diagnoses agree, with the same compute. If it matches our misattribution rate, we will adopt the simpler signal and report that the tier separation was unnecessary.
Why not just add theorem text to close the domain gap? TheoremQA already shows that this does not substantially help models apply theorems. We include a retrieval-augmented tier-4 variant only as a check on whether that finding transfers to grading.
Proposed Experiments
The first decisive check uses FERMAT, whose deliberately perturbed derivations come with known error type and location, so true groups exist. We measure group purity (adjusted Rand index against the true type-and-step labels) and the misattribution rate, defined as the share of correct or differently-wrong derivations given a group's feedback. We then degrade the handwriting images in controlled steps, with blur, slant, and faint strokes. The prediction is that misattribution under single-reading grouping rises with degradation, while under agreement-based grouping it stays roughly flat and the legibility queue grows instead.
The second check uses the university exam data. It reports expert-rated feedback correctness, score agreement (quadratically weighted κ and the within-one-point rate), and timed instructor minutes per correct feedback message, for all baselines and both ablations.
The third check is a fairness analysis. We regress score error and misattribution on instructor legibility ratings, to test whether students with untidy handwriting are penalized. The UC Irvine work names this concern but does not measure it. The proposal fails if agreement-based grouping does not reduce misattribution relative to single-reading grouping at matched instructor time, or if the repeated-pass disagreement variant matches it.
References
- Evaluating AI Grading on Real-World Handwritten College Mathematics: A Large-Scale Study Toward a Benchmark
- Human-in-the-Loop LLM Grading for Handwritten Mathematics Assessments
- Evaluating GPT-4 at Grading Handwritten Solutions in Math Exams
- Can Vision-Language Models Evaluate Handwritten Math?
- VEHME: A Vision-Language Model For Evaluating Handwritten Mathematics Expressions
- TheoremQA: A Theorem-driven Question Answering Dataset
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency
- BeaconChart: Verifier-Guided Corrective Instruction for Chart Understanding