Which Mistake Caused the Others? Counterfactual Root-Cause Attribution for Multimodal Reasoning Traces

When a vision-language model fails a geometry or chart problem, current evaluations give each failure one label, such as "vision error" or "reasoning error." The usual way to assign that label is to look at the first wrong step, or to ask a judge model. That label is the basis of a central claim in the field: on MATH-Vision, 42.2% of GPT-4V's failures were reasoning errors and only 31.9% were vision recognition errors (Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset). But a wrong deduction in step five may be an honest consequence of a coordinate misread in step two. In that case the "reasoning error" is a symptom, and the perception error is the cause. Parsing research faced the same confusion two decades ago. It resolved it with a simple experiment: fix one error, re-run the system, and see which other errors disappear (Descriptive and Empirical Approaches to Capturing Underlying Dependencies among Parsing Errors).

We propose to run that experiment on multimodal chains of thought. For each erroneous step in a model's own reasoning trace, we substitute a corrected version and let the model continue from that point. We record whether later errors and the final answer recover. The output is a dependency graph that separates root-cause errors from induced ones, with fractional blame where several fixes each suffice. The practical claim is testable. If attribution is right, repairing the step it identifies as the root cause should recover more answers than repairing the step a judge model labels, or the first wrong step.

Core Claim

Single-label, first-error taxonomies systematically mislabel multimodal failures. We expect a substantial share of what is currently counted as "reasoning error" to be induced by earlier perceptual slips, and a smaller share to be the reverse: steps that look perceptual but are actually caused by a misread linguistic constraint. Counterfactual intervention on the model's own trace measures this directly, and it yields a better guide to repair than judge-based categorization.

This bears on a tension in the file. Some work locates the bottleneck in reasoning even when perception is removed: GeoSQA's human transcriptions do not rescue text models (GeoSQA). Other work finds that visual input corrupts reasoning that would otherwise succeed (MATHVERSE; TIGER). Step-level root-cause attribution can say, per benchmark and per model, which story describes which failures.

Research Program

The method borrows two tools from linguistic error analysis. The first is the empirical "what-if" re-parse of Hara, Miyao and Tsujii. They forced a single correction into a parser's chart and re-parsed, classifying downstream errors that vanished as induced. Using it, they found that prepositional attachment was a frequent root cause, while noun-phrase head errors were almost always symptoms. The second is the fractional labeling of Popovic and Vilar (Automatic error classification with multiple error labels). They spread an ambiguous error's label across all equally good alignments instead of picking one arbitrarily, which raised correlation with human judgments on TERRA from .891 to .936. We reuse the logic of both. Neither was designed for a generative model that writes its own intermediate steps conditioned on an image.

Concretely, a model produces a trace of steps $s_1,\dots,s_n$ for an image–question pair. A step annotator, consisting of a strong judge model audited by humans on a subset, marks the candidate erroneous steps. Each is tagged with a type from an extended version of ErrorRadar's five classes (visual perception, calculation, reasoning, knowledge, misinterpretation of the prompt) (ErrorRadar). For each flagged step $i$, we build a corrected step $\tilde s_i$ from benchmark-provided ground truth. Perception fixes use MathVerse's text-dominant versions that state the essential conditions, ChartQA's underlying data tables, and GeoSQA's expert diagram transcriptions. Reasoning and calculation fixes use reference solutions where the benchmark provides them, as ErrorRadar does. We then splice $\tilde s_i$ into the prefix, sample $m$ continuations, and measure the correction effect

$$c_i = \Pr(\text{correct} \mid s_{<i}, \tilde s_i) - \Pr(\text{correct} \mid s_{<i}, s_i),$$

where both probabilities are estimated over the $m$ sampled continuations. A flagged error $j>i$ counts as induced by $i$ if it disappears in most continuations after fixing $i$. Errors not removed by fixing any earlier error count as independent.

When fixing either of two errors recovers the answer, blame is split in proportion to their correction effects. This is the fractional allocation adapted from Popovic and Vilar. It gives an honest number for ambiguous cases at the perception–reasoning boundary instead of a forced label. Aggregated over a benchmark, the result is a root-cause profile per model: the share of failures whose root is perceptual, linguistic, or deductive, together with the most common one-way dependencies, such as "misread axis value induces wrong comparison."

Two limits from the parsing work carry over and will be reported, not hidden. First, we can only intervene on errors someone flagged, just as Hara et al.'s descriptive patterns missed roughly 60% of parsing errors. Second, a model that has drifted far from the reference path may not use a corrected step at all. We measure both: annotator recall on a human-labeled audit set, and the rate at which corrections are ignored. The ignore rate is an indicator of the "access gap" that TIGER describes.

Baselines

The naive reference point is first-error attribution. The earliest flagged step's type is taken as the failure's cause, which is the implicit procedure behind single-label breakdowns like MATH-Vision's. It needs no extra compute, and any attribution method must beat it at predicting which repair works.

The strongest existing method is judge-based diagnosis in the style of ErrorRadar. A strong multimodal model is given the problem, the reference solution, and the trace, and asked to locate the faulty step and name its category. ErrorRadar reports that GPT-4o reaches 55.1% step-identification accuracy and 53.1% categorization accuracy, against 69.8% and 60.7% for human educators. Those numbers were measured on student mistakes, not model traces. We will run the same judge, with the same prompt budget, on exactly the traces we attribute counterfactually. To keep the comparison fair on compute, the judge may also sample $m$ times and take a majority vote, matching our continuation budget. Counterfactual attribution beats it if the repair it selects recovers significantly more final answers on held-out items. A secondary criterion is agreement with human root-cause labels on the audit set.

A problem-level modality swap is the third baseline. The whole image is replaced with its gold text description and the model is re-run, crediting perception if the answer recovers. This is cheap and close in spirit to TIGER's modality counterfactuals and MathVerse's versions.

Two ablations isolate the key ingredient. One keeps the intervention but drops the dependency analysis, fixing only the first error. The other keeps fractional allocation but replaces model continuation with judge-predicted counterfactuals ("would fixing this help?"). If the second ablation matches the full method, actual re-execution is unnecessary.

Why not just…?

Why not just ask a strong judge model to diagnose the trace? This is the cheapest serious competitor. ErrorRadar's own findings argue against trusting it on visual items. Judges relabel student visual-perception errors as reasoning errors because their own encoder misreads the diagram, and smaller judges fall back on "calculation error" by default. A judge diagnosing a model trace on the same image inherits the same blindness. Counterfactual re-execution sidesteps this by observing whether the corrected percept changes the outcome, not by asking anyone to see it. If the judge's chosen repairs recover answers as often as ours, the judge is sufficient, and we will report that.

Why not just swap the whole image for a gold transcription? This is a real threat for coarse questions such as "is this benchmark perception-bound?". It cannot separate mixed failures, where a percept and a later deduction are both wrong, or cases where the transcription removes the misread constraint along with the image. It also changes the input distribution: MathVerse shows that removing diagrams can raise accuracy by over 5%, so a swap confounds "perception was the cause" with "images act as noise." Our comparison on items where the two methods disagree will show whether step-level resolution adds anything beyond the swap.

Why not just train with final-answer reinforcement learning and skip diagnosis? RAPID's perceiver–reasoner split reaches 53.2% average accuracy on Qwen2.5-VL-7B, an 11.2-point absolute gain (Reasoning-Aligned Perception Decoupling for Scalable Multi-modal Reasoning). TIGER reports a gain of about 15% on MathVerse. These are repair methods, not diagnoses, and we do not compete with them on accuracy. The objection is still fair if diagnosis offers nothing they lack. MathVerse shows that final-answer rewards can reinforce correct answers reached through hallucinated steps. Our attribution can test whether RAPID-trained models actually reduce perception-rooted failures or merely shift them. Running the analysis before and after such training is part of the program.

Proposed Experiments

We attribute traces from two open models, Qwen2.5-VL-7B (RAPID's base) and InternVL2-76B, plus one proprietary model. We use failures on MathVerse testmini, ChartQA's human-authored split, and GeoSQA. These are the benchmarks with gold perceptual ground truth. We add ErrorRadar, whose reference solutions support reasoning fixes. MATH-Vision is used for the re-labeling analysis wherever reference solutions permit. A few hundred traces per benchmark receive human root-cause labels for auditing.

The decisive check is predictive repair on held-out items. For each failed trace, each method nominates a single step to repair. We apply the repair, re-run, and compare final-answer recovery rates across first-error, judge, problem-level swap, and counterfactual attribution, using the same continuation budget for all. If counterfactual attribution does not beat the judge, the proposal's central claim fails.

The second check re-estimates the perception-versus-reasoning split. We report what fraction of judge-labeled reasoning errors turn out to be induced by perceptual roots, separately per benchmark. The prediction is a large share on chart and diagram items, and a small share on GeoSQA, where its own results suggest genuine reasoning failure.

The third check measures the ignore rate. This is how often a correct inserted percept fails to change the continuation, compared between images and text renderings of the same content. A high ignore rate would directly show TIGER's access gap at the step level rather than the problem level.