Finding the Misread: Root-Cause Localization for Multimodal Reasoning Errors

When a vision-language model gets a chart or geometry question wrong, the visible mistake is usually at the end: a wrong subtraction, a wrong final choice. The actual cause is often earlier. Somewhere the model turned the image into a sentence such as "the 2019 bar is about 40" or "angle ABC is a right angle," and that sentence was wrong. Everything downstream then reasoned correctly from a false premise. Software engineering has a name for this failure: fixing the symptom instead of the root cause. Coding agents show the same pattern when they patch the crash site rather than the faulty line (FaultLoc).

We propose to treat multimodal reasoning errors as a fault-localization problem at the level of single perceptual claims. The method samples several reasoning traces and breaks each into atomic claims about the image. It then finds the claims whose value decides which answer a trace reaches. Only those claims are re-checked against the image, using a short and narrow query rather than a long chain-of-thought. The final answer comes from the traces that agree with the re-checked claims. In short, we put extra compute into localizing and repairing the one misread fact, and we do not simply sample more or reason longer.

Core Claim

Our claim is that most reasoning failures on visual math and chart benchmarks that look like "reasoning errors" fall into two groups. Some come from a small number of decisive perceptual claims that were lost when the image was translated into text. The rest are true deduction errors. The first group can be repaired cheaply if those claims are localized at fine granularity and re-checked in a clean, short context. Three findings in the literature motivate this claim.

The first concerns how errors are classified. MATH-Vision attributes 42.2% of GPT-4V errors to reasoning and 31.9% to vision recognition. That breakdown is made at the level of the whole answer, and a misread value that propagates can look like a reasoning failure at that level. The same paper finds that chain-of-thought sometimes lowers accuracy for multimodal models, which fits the idea that early cross-modal errors compound.

The second concerns what is lost in translation. ChartCoder shows that turning a visual chart into text drops exact values and styling details. Plot Twist shows that models can answer complex chart questions while failing elementary perceptual queries about the same chart. Perceptual claims embedded in reasoning are therefore unreliable, and their errors are hidden behind language priors.

The third concerns context noise. Extra context hurts both text and multimodal models (Improving Zero-shot Reader, DCoT). Programmatic decomposition improves grounding but costs answer accuracy: +9.6 MuRGAt-SCORE against −7.4 accuracy in Multimodal Fact-Level Attribution. Repairs should therefore happen outside the long trace, not by appending more reasoning to it.

What is new here is the unit and the signal. The unit is the single perceptual claim, as opposed to the whole answer or the whole reasoning step. The signal is answer dependence across sampled traces. This combines the decomposition into atomic facts from Multimodal Fact-Level Attribution with the statistical fault-localization logic of DeepCover. DeepCover scores pixels by how often they appear in failing versus passing mutated runs. We score claims by how strongly their value separates the answer clusters of naturally sampled traces. Because the "mutations" are free samples the model already produces, the cost is far below DeepCover's roughly 2,000 masked runs per image.

Research Program

The pipeline has four stages.

In the first stage, we sample $N$ chain-of-thought traces per question. A parser turns each trace into atomic claims: the same model, prompted with a fixed decomposition template, does this, as in MuRGAt. The parser tags each claim as perceptual (a statement about what the image shows) or deductive (a statement derived from earlier claims). Perceptual claims are then normalized to shared "quantities," for example "value of the 2019 bar" or "length of segment AB," so that we can compare what different traces say about the same thing.

In the second stage, we score each quantity $q$ by how much the final answer depends on it:

$$ s(q) = \frac{I(V_q; A)}{H(A)} \cdot \big(1 - p_{\max}(V_q)\big) $$

Here $V_q$ is the value a trace asserts for $q$ and $A$ is the trace's final answer, both treated as random variables over the $N$ samples. $I$ is mutual information and $H$ is entropy. $p_{\max}(V_q)$ is the share of traces holding the most common value of $q$. The first factor is high when knowing a trace's reading of $q$ tells you which answer it reached. The second factor is high when traces disagree about $q$. Together they pick out the divergence point, meaning the claim whose different readings send traces toward different answers.

In the third stage, we re-check only the top-$k$ quantities against the original image. Each query is short and contains just the quantity, the question, and the image with the relevant region marked by a red box, following DCoT's finding that marking beats cropping. The prior reasoning trace and any other context are left out.

In the fourth stage, we aggregate under the re-checked values. We keep the traces whose claims agree with the re-checked readings and take a majority vote among them. If no trace agrees, we run a single new trace conditioned on the verified claims.

The analysis side builds a small claim-level root-cause annotation set. We will annotate about 600 wrong traces from MATH-Vision, ChartQA-H, and MathVista, marking the earliest claim that is false and labeling it as perceptual or deductive. Following FaultLoc's file/hunk/line hierarchy, localization is evaluated at three granularities: trace level ("this trace is wrong"), step level, and claim level. This set also lets us re-estimate the vision-versus-reasoning error split with cause-level labels instead of answer-level ones. We will also test the "symptom versus cause" pattern directly, by measuring how often a text-only critic flags the last wrong step when the annotated root cause is an earlier misread.

A secondary transfer experiment applies the same idea to FaultLoc. Here the "claims" are statements in agent trajectories about which function allocates, frees, or bounds-checks the faulting object. Re-checking means a narrow query over that function's source and the sanitizer report, with the proof-of-concept input left out, since FaultLoc found it acts as a distractor. This tests whether claim-level divergence localization is a general remedy for symptom-site fixation, or something specific to images.

Baselines

The naive reference point is direct answering without chain-of-thought. MATH-Vision reports that chain-of-thought can lower multimodal accuracy, so "just answer" is a real competitor and not a straw man. Standard single-trace chain-of-thought is the second reference. For orientation, MATH-Vision reports GPT-4o at 30.39% accuracy and human performance at 68.82%. MathVista reports GPT-4V at 49.9% accuracy.

The strongest generic competitor is self-consistency, meaning majority voting over sampled traces. It uses exactly the samples our method uses and differs only in aggregation. We found no reported result on Lacuna for self-consistency on MATH-Vision, ChartQA, or MathVista, so we will run it ourselves on the same models and sample pools. Its budget will be matched in model calls and generated tokens by giving it $N + k$ samples, where $k$ is our number of re-check queries. Our method beats it only if it wins at this matched budget, with paired bootstrap confidence intervals over questions.

The strongest domain-specific competitor on charts is the tool-based route of Synthesize Step-by-Step. That method converts the chart to a table with DePlot and reasons in executable Python. It reports a rise on ChartQA's human-written split from 38.2% to 53.9% and 72.6% relaxed accuracy overall for augmented MatCha. Because it is fine-tuned, we will not claim to beat those absolute numbers with a training-free method. We will instead run its DePlot-plus-program pipeline on our evaluation splits as a reference, and we will use DePlot tables as an alternative re-checking source (see below).

A text-only critic baseline prompts the same model to find the first wrong step in a trace and regenerate from there. This is the natural "process verification" alternative. We also run DCoT itself on the same models.

Three ablations isolate the key ingredients. To test localization, we replace the top-$k$ quantities with $k$ randomly chosen perceptual quantities. To test where the re-check looks, we re-check against a caption or a DePlot table instead of the image, which measures whether cross-modal loss actually binds. To test context length, we re-check inside the full trace context instead of in a short query, which measures dilution.

Why not just…?

Why not just use self-consistency? This is the most serious threat. Self-consistency already exploits disagreement across samples, and on easy questions our method may simply reproduce the majority. It can only add value when the majority is wrong because of a contested, decisive perceptual claim that a focused re-read gets right. We therefore report results separately on the subset where the majority answer is wrong but at least one sample is right. If our method shows no gain at matched compute even on that subset, the idea is not useful and we will say so.

Why not just have a critic find the first wrong step? A text-only critic sees the trace and not the image. If the misread claim is internally consistent with the rest of the reasoning, the critic has nothing to object to. This is the multimodal version of the crash-site fixation documented in FaultLoc, and the MuRGAt results suggest larger models even rationalize ungrounded claims. The annotated root-cause set tests this directly by comparing claim-level localization accuracy and the rate at which each method flags symptoms instead of causes.

Why not just caption or linearize the image once and reason in text? GeoSQA's result constrains how far text alone can go: models still failed when diagrams were replaced with human-written descriptions (GeoSQA). ChartCoder documents that linearization drops exact values. Still, if the caption or DePlot re-checking ablation matches image re-checking, then our image-side step is unnecessary. We would report that as a finding that the gain comes from localization rather than from avoiding cross-modal loss.

Why not just crop or filter the context? DCoT found that cropping loses the surrounding context models need, which is why we mark regions instead. Our re-check query is a form of filtering, but it is targeted by the dependence score rather than applied uniformly. The random-claim ablation shows whether targeting matters.

Proposed Experiments

We will use two open-weight models, LLaVA-1.5-13B and InternVL-Chat-V1-2-Plus (both evaluated in the cited benchmarks), and GPT-4o through its API. Each is evaluated on MATH-Vision (3,040 problems), ChartQA's human-written split, and MathVista, with $N \in \{5, 10, 20\}$ and $k \in \{1, 2, 3\}$. To test noise sensitivity, we also build distractor-augmented versions of ChartQA-H questions by adding irrelevant chart panels and unrelated question text. We then measure whether short-context re-checking degrades less than chain-of-thought and self-consistency as distractors grow. Grounding quality of final traces is scored with MuRGAt-SCORE on the WorldSense/Video-MMMU test material, checking whether our method avoids the reported trade-off between better attribution and lower accuracy.

Three results would settle the main claim. The first is a positive matched-compute gain over self-consistency on the majority-wrong subset. The second is a clear gap between top-$k$ and random-$k$ re-checking. The third is higher claim-level root-cause localization than the text-only critic on the annotated set. A secondary result is the image-versus-caption re-checking comparison, which decides whether cross-modal information loss is the mechanism or a bystander.

On FaultLoc, the reference point is GPT-5.3-Codex's reported Line F1 of 24.90 and its drop in Hunk Recall from 55.8% on single-site bugs to 24.3% on multi-site bugs. We will run a 200-instance subset with an open model under identical tool access, with and without divergence-based re-checking. Here the meaningful success criterion is the relative gain in line-level and multi-site recall over the same model's unmodified agent, not beating the frontier number.

The method fails if the gain over self-consistency disappears at matched compute, or if random re-checking performs as well as targeted re-checking. A further failure mode would be that the annotated root causes turn out to be mostly deductive. In that case we would have learned that MATH-Vision's "reasoning error" label is accurate, and that a perceptual repair targets the wrong bottleneck.