Solve Before You Look: Firewalled Diagnosis for Multimodal AI Tutors

When an AI tutor reads a photo of a student's handwritten math, the student's work is not neutral input. If the student made a mistake, the page is misleading context that the tutor has to reason about without believing it. Language models are known to fail at exactly this. Given plausible but wrong context, they build logical bridges to justify it, and explicit step-by-step "thinking" can make the problem worse. We propose to measure how much multimodal tutors anchor on students' errors, and to remove that anchoring with a simple information firewall. The model first solves the problem from the problem statement and figure alone, without seeing the student's work. It then compares its own solution to the student's work step by step. The model that writes the hint sees only the first step where the two diverge, and never the full solution.

The idea joins three problems that are usually treated separately. The first is the cost of writing tutoring content: human-authored hints take enormous expert labor, and generated hints only pay off if they are reliable. The second is that models degrade when their context contains distracting or misleading material. The third is that vision-language models are much worse at reasoning over diagrams and handwriting than over text. Our two-sentence summary is this: multimodal tutors fail partly because the student's own wrong work acts as adversarial context. Making the tutor commit to an independent solution before it looks, and then limiting what the hint writer can see, should improve error diagnosis and also reduce answer leaking.

Core Claim

We make two claims that can be tested separately. The diagnostic claim is that, holding the problem fixed, showing a vision-language model a student's erroneous handwritten solution measurably pulls the model's own judgment toward the student's wrong values. We call this the anchoring rate, and we expect it to be larger when thinking mode is on. The intervention claim is that blind solving plus step-scoped hint generation improves error localization and hint quality over the strongest training-free alternatives. It should do so without an authored reference solution, which keeps it inside the low-authoring-cost regime.

The motivating evidence comes from several papers. Thinking in a Crowd shows that misleading auxiliary text drops Qwen3-8B from 96.22% to 77.03% with thinking on, below the 74.87% it gets with thinking off. That study covers text snippets only, and the misleading material there is not something the model is supposed to evaluate. FERMAT reports that vision-language models detect errors in handwritten solutions at near-random rates, and that they are better at producing a correction from scratch than at locating the wrong line. We read that asymmetry as a hint that independent solving is easier than judging someone else's page, which is the asymmetry our design exploits. MMTutorBench finds that top models fail at solution scope control in about 29% of cases. MathDial finds that ChatGPT reveals the solution in 66% of tutoring turns. Both are failures of a hint writer that holds, and gives away, too much.

Research Program

The first component is a controlled anchoring study. FERMAT supplies handwritten solutions for 609 problems, with error perturbations whose locations are known. For each perturbed solution we pose the underlying problem to a model under two conditions: alone, and alongside the student's handwritten page. We record the model's answer and its intermediate quantities in both conditions. The anchoring rate is the fraction of cases where the model gets the problem right alone but, with the page visible, either adopts the student's erroneous value or judges the erroneous step correct.

We report the anchoring rate under thinking on and thinking off, and for open models (Qwen2.5-VL, InternVL) and proprietary models. As a noise control, we also overlay irrelevant handwriting such as crossed-out lines and margin notes on correct solutions. This separates two effects that both count as contextual noise: harm from mere clutter, of the kind Plot Twist documented for gridlines, and harm from content that is semantically misleading.

The second component is the firewalled tutor, which runs in three stages using the same frozen model in each.

In the blind-solve stage, the model sees the problem text and any problem figure, but not the student's page. It samples several solutions and takes a majority vote on the answer, following the self-consistency recipe that Pardos and Bhandari used to cut hint errors to near zero in algebra. When the samples disagree too much, the system marks the item low-confidence and falls back to a hint that asks the student a question rather than asserting a correction.

In the alignment stage, the model receives the student's page and the voted reference solution. It returns the first student step that conflicts with the reference, along with that step's location on the page. We mark that location on the image with a box, reusing the region-marking technique from DCoT, which draws a red box on the image rather than cropping it so that surrounding context is kept.

In the hint stage, a fresh call receives the original image with the marked region, the student's query, the student's divergent step, and only the corrected version of that one step. The final answer and later steps are withheld. The idea is that a hint writer cannot leak what it does not hold.

The third component is analysis. We stratify all results by whether the blind solve was correct. This tells us how much of the gain comes from avoiding anchoring and how much is lost when blind solving fails. Failures should be common on hard figures: GPT-4o scores 30.39% on MATH-Vision. This stratification sets the scope of the method honestly. We expect it to help most on K-12 and algebra material, as in FERMAT and MMTutorBench, where blind solving is reliable. We expect it to help little on competition geometry.

Baselines

The naive reference point is direct multimodal tutoring. A single frozen model receives the image, the student's query, and a Socratic system prompt, under the same model, sampling budget, and prompt-tuning effort as our method. On MMTutorBench the strongest reported number of this kind is Gemini-2.5-Pro at 4.77/6, against 5.85/6 for human experts. On FERMAT, Gemini-1.5-Pro reaches 77% on error correction. For error detection, FERMAT reports performance "near random-guess levels" without a single figure we can cite. On TutorBench, Gemini 2.5 Pro scores 55.65% overall, and we found no reported result on Lacuna for TutorBench's multimodal subset specifically. We run this baseline with thinking both on and off.

The strongest existing training-free competitor is a combination of DCoT-style focusing and self-consistency. The model first predicts a box around the relevant region of the student's page, the box is drawn on the image, and then it samples ten tutoring responses and votes, giving the same compute as our multi-stage pipeline. This is the strongest competitor because it already applies the two most effective known tricks, visual focusing and sample voting. What it does not do is stop the model from seeing the student's work before committing to a solution. Beating it means a statistically significant gain in FERMAT error localization and in MMTutorBench solution-scope-control scores at matched token budget.

The OCR-first pipeline transcribes the page to LaTeX and tutors with a text model. MMTutorBench reports that this degrades tutoring quality. We include it because it is the cheapest system a practitioner would build.

The oracle-reference condition gives the tutor an authored correct solution together with the student's page. This is an upper bound on reference quality, but the full solution is visible to the hint writer. It tests whether the benefit of our method comes from having a reference at all or from the firewall.

Two ablations isolate the key ingredients. The first is a non-blind reference, where the model solves with the student's page visible. It tests whether blindness matters. The second is the full-reference hint writer, which uses our pipeline but gives the hint stage the entire solution. It tests whether step scoping drives the reduction in leaking.

Why not just…?

Why not just give the tutor the correct solution? This is the most serious threat, and the oracle-reference condition tests it directly. Even if an authored solution helps, it reintroduces the authoring cost that generated tutoring is meant to remove. A full solution in the hint writer's context should also raise leaking. MathDial's 66% telling rate for ChatGPT suggests that models give away what they hold. If the oracle condition with full solution visible matches our method on scope control, the firewall claim fails. In that case we would report the result as "self-generated references suffice" and drop the scoping claim.

Why not just turn on thinking mode or chain-of-thought? Thinking in a Crowd shows that thinking mode amplifies misleading context. MATH-Vision and MV-MATH report that chain-of-thought often lowers accuracy on multimodal math. Our anchoring study measures directly whether thinking raises the anchoring rate on student work. If it does not, that weakens our motivation but leaves the pipeline comparison intact.

Why not just transcribe the page with OCR? MMTutorBench reports degraded tutoring from OCR-first pipelines because they lose layout and diagram cues. FERMAT finds only marginal OCR gains for top models. More fundamentally, a clean transcription of a wrong derivation is still misleading context, so OCR addresses the handwriting problem but not the anchoring problem.

Why not just tell the model to ignore the student's mistakes, or fine-tune it on clean-versus-distracted pairs? Breaking Focus shows that "ignore irrelevant information" instructions fail, while preference training on clean versus distracted pairs helps. Ignore-prompting is also incoherent here, because the tutor must attend to the student's work. Preference training is a real alternative, but it needs paired data and model access. We treat it as complementary future work and keep the main comparison among frozen models, where the firewall needs no training.

Proposed Experiments

The decisive check comes first, as the anchoring study on FERMAT's paired perturbed and correct solutions. If the anchoring rate is near zero, the core motivation fails and the remaining pipeline is just self-consistency with extra steps. We will also report the false-alarm rate on correct solutions, since FERMAT notes that smaller models hallucinate errors in correct work. A firewall that raises false alarms would be a net loss.

Second, on FERMAT, we compare all systems on error detection, error localization, and error correction. Our main prediction is the largest gain on localization, among items where the blind solve is correct.

Third, on MMTutorBench, we compare total rubric score and the solution-scope-control dimension. Our main prediction is fewer answer leaks for step-scoped hints than for full-reference hints at equal or better correctness.

Fourth, on TutorBench's 828 multimodal samples, we test transfer to conversational feedback across STEM subjects. MathDial serves as a text-only sanity check that scoping lowers the telling rate without hurting student success.

All comparisons use matched model, matched token budget, and paired bootstrap tests. Results are broken out by blind-solve correctness and, for the noise-overlay variant, by distractor type.