Perception-Aware Rollback for Visual Math Reasoning

When a vision-language model solves a geometry or chart problem, it usually fails in one of two ways. Either it misreads the picture early on (an axis limit, an angle, which point lies on which line), or it reads the picture correctly and then reasons badly. Today's reasoning chains run forward only, so an early misreading flows into every later step and cannot be undone. Text-only reasoners already handle mistakes by backtracking: when a check fails, they jump back to an earlier step and try again. That backtracking treats every step the same, though. When a multimodal chain fails, the useful question is which kind of step to return to. Returning to a misread fact calls for looking at the image again, harder. Returning to a bad deduction calls for thinking again.

We propose a rollback controller for multimodal models that tells the two cases apart using a signal the model already produces: how much of its attention, at each generated token, goes to image tokens rather than text tokens. Steps where attention leans heavily on the image are recorded as perceptual commitments, meaning claims about what the image shows. When the answer fails a check, these commitments are re-verified one at a time, starting with the earliest. Each re-check is a short, targeted query in which the image tokens are deliberately up-weighted. The chain rolls back to just before the first commitment that does not survive this second look, and it resumes with the corrected fact fixed in context. If every commitment survives, the controller falls back to ordinary reasoning rollback. In two sentences: backtracking helps multimodal models only if it can tell "I misread the picture" from "I reasoned wrongly," and the model's own vision-to-text attention ratio is a cheap, training-free way to tell them apart. When the cause is a misreading, the retry should re-perceive with the visual stream amplified, not merely resample text.

Core Claim

Under a matched token budget, rollback that targets perceptual commitments and re-perceives with amplified visual weight will beat two alternatives on vision-dependent math and chart problems: uniform text-style rollback, and forward-only sampling. The gain should be largest exactly where current models lean on text shortcuts. We therefore predict that it narrows the gap between the text-dominant and vision-only versions of MathVerse problems.

Where This Sits

Three papers motivate the proposal by documenting the gap. MATH-Vision evaluates 3,040 competition problems. Its GPT-4V error analysis splits failures into 31.9% recognition errors and 42.2% reasoning errors, and it reports that multimodal chain-of-thought can hurt accuracy. Both kinds of error are common enough that any recovery method must handle both. MathVerse rewrites problems into versions ranging from text-dominant to vision-only. It finds that some models score more than 5% higher with the diagram removed, which shows that models' implicit weighting of image against text collapses toward text. Plot Twist shows that models answer trend questions about a chart yet fail to read its axis extremes or point coordinates. This gives us a test where the ground-truth perceptual facts are known.

Two papers supply the method. Thought Rollback backtracks text-only reasoning chains to an earlier valid thought when a verifier detects an error, and adds the failure as a negative constraint. It reaches 52.16% on MATH Level 5 with GPT-4. We reuse its rollback-and-constrain loop. It has no notion of perception and no image input. Attention-Guided Switching (AGS) shows that the vision-to-text attention ratio separates perception-heavy steps from symbolic steps, and that token entropy conflates the two. It uses that ratio to switch between latent and explicit thinking. AGS runs strictly forward and never uses the ratio to locate or repair errors. We use the same signal for a different job: as an index of where perceptual commitments were made, and so as the set of candidate rollback targets.

Two lines of work constrain the design. The dynamic modality-weighting literature, including SE-style channel gating for face anti-spoofing and modality-aware fusion for document layout, assumes that learned gates will down-weight the unreliable stream. The modality-collapse findings (backdoor modality collapse, traffic VideoQA auditing) show that these gates often lock onto the easier text stream instead. For that reason we do not learn a gate. We intervene on visual weight only at the moment of re-checking, where the intervention can be audited. R³DAO does topology-level rollback for data agents and states that it has no intrinsic visual reasoning, so it does not cover this setting.

The confidential-computing work on state rollback plays only a conceptual role. REBOUND treats rollback as a sanctioned operation over an append-only, auditable record of states, rather than as something either forbidden or silent. ElephantDP shows the failure mode that comes from losing track of committed state on restart: an attacker restarts the system, cumulative state is reset, and the same leaked query can be replayed. Our commitment ledger borrows both lessons. Commitments are appended and never silently overwritten. A rolled-back commitment is marked as superseded and kept in context, so the retry cannot quietly replay the same misreading. We make no claim about enclave security, and the trusted-execution threat model is out of scope.

Research Program

The controller wraps an open multimodal model at inference time and needs no training. As the model generates, we compute AGS's per-step vision-to-text attention ratio $R_t$. A step whose $R_t$ exceeds a threshold is written to the ledger together with the claim it states, for example "the y-axis maximum is 40" or "segment AB is tangent at B." The threshold is calibrated on a held-out split.

A rollback is triggered when the final answer fails a check. For multiple-choice items the check is self-consistency against a second short sample, and for numeric items it is execution of an extracted computation, following the code-decomposition idea of chart reasoning transfer. The controller then re-verifies ledger entries from earliest to latest. Each entry is turned into a closed question about the image, and the model answers it with attention logits on image tokens shifted upward:

$$a_{ij} \leftarrow a_{ij} + \log\gamma \cdot \mathbb{1}[j \in \mathcal{V}],$$

where $a_{ij}$ is the pre-softmax attention score from query token $i$ to key token $j$, $\mathcal{V}$ is the set of image-token positions, and $\gamma \ge 1$ is the visual amplification factor ($\gamma = 1$ recovers ordinary attention). The first entry whose re-check disagrees becomes the rollback point. The chain is truncated just before it, the corrected fact is pinned in context, the superseded claim is kept as a negative constraint, and generation resumes. If no entry disagrees, the controller rolls back to the last low-$R_t$ step, which reproduces Thought Rollback. The number of rollbacks is capped, and every result is reported at a matched generated-token budget.

The analysis has three parts. The first is a localization study on Chart101, where elementary perceptual facts have ground truth. Here we measure whether the first ledger entry that fails re-checking is the claim that is actually wrong. The second is a sweep over $\gamma$, which tests whether amplification helps re-perception or merely makes the model over-trust spurious visual detail such as gridlines. Plot Twist reports that adding gridlines dropped PaLI-3 from 29.5% to 21.3%, which is exactly this risk. The third is an error-type breakdown, which checks whether perceptual and reasoning rollbacks fix the two error classes identified by MATH-Vision.

Baselines

The naive reference point is plain greedy chain-of-thought with the same backbone, Qwen3-VL-8B and the 4B variant used by AGS. Every method must clear this floor. We also report the published reference points: random guess at about 5.86% and GPT-4o at 30.39% on MATH-Vision (MATH-Vision), and the frequent-guess heuristic at 26.3% and GPT-4V at 49.9% on MathVista (MathVista).

The strongest existing method on this question is AGS, a training-free method that uses the same attention signal, runs on the same backbones, and targets the same benchmarks. On MathVerse it reports a +15.9 percentage-point improvement for its 4B variant, and it cuts generation steps from 2214 to 953 on Qwen3-VL-8B (Attention-Guided Switching). The file gives no absolute MathVerse accuracy for AGS, so we will rerun it ourselves with its published window and switch-budget settings. To beat it, our method must improve MathVerse vision-dominant and vision-only accuracy over rerun AGS at an equal or smaller token budget. Because AGS's efficiency is part of its claim, we also report accuracy per token.

The second strong competitor is Thought Rollback ported to the multimodal model: the same verifier and rollback loop, with rollback targets chosen without reference to perception. The file contains no multimodal number for it ("no reported result found on Lacuna for Thought Rollback on MathVista, MATH-Vision, or MathVerse"). We implement it with the same verifier and rollback cap as our method.

Two ablations isolate the key ingredients. The first keeps perception-indexed targeting but sets $\gamma = 1$, which tests whether amplified re-perception matters. The second keeps amplification but replaces $R_t$ with token entropy for selecting ledger entries, which tests whether the attention signal matters. All methods share decoding settings, the token cap, and the validation set used for any threshold tuning, and each gets the same small grid-search budget.

Why not just…?

Why not just sample N chains and majority-vote? This is the most serious threat, because self-consistency at a large enough N may stumble onto a correct reading of the diagram. Our counterargument is mechanistic. Independent samples share the same text-leaning attention pattern, so a systematic misreading recurs across all of them. That is what MathVerse's text-only paradox suggests. MATH-Vision's finding that chain-of-thought can amplify hallucinated visual claims points the same way. Evidence that voting is not settled already exists in Thought Rollback's comparison against forward-only search on text math. Still, we will settle the question directly with token-matched self-consistency curves. If voting matches us on vision-only MathVerse at equal tokens, the amplified re-perception step adds nothing, and we would report that.

Why not just apply text Thought Rollback to the multimodal model? Uniform rollback cannot tell a misreading from a bad deduction. When it retries, it rethinks from a context in which the misread fact is still implicitly trusted. The ported-Thought-Rollback baseline and the entropy ablation measure exactly this. The prediction is that our advantage concentrates on the recognition-error class and on Chart101 elementary questions, and is roughly zero on reasoning-error items.

Why not just boost image attention throughout, or tell the model to "look again"? Global amplification risks the perturbation fragility documented in Plot Twist, and it ignores evidence that diagrams sometimes act as noise. A prompt to look again leaves the attention pattern unchanged. We run a global-$\gamma$ variant at its best tuned value as an additional baseline. If it matches targeted re-checking, then the ledger and rollback machinery are unnecessary, and the contribution reduces to a simple attention intervention.

Proposed Experiments

The main evaluation covers MathVerse across all six versions, using testmini for tuning and the full set for reporting, together with MATH-Vision broken down by discipline and MathVista. Chart101's synthetic and real plots serve for perceptual localization, and the dataset's 5% tolerance defines when a re-checked claim counts as wrong.

The first decisive check is the MathVerse modality gap. Our method should raise vision-only and vision-dominant accuracy more than text-dominant accuracy, which would narrow the paradox in which models do better without the diagram. A uniform gain across versions would mean we are only adding compute.

The second decisive check is localization on Chart101. The first ledger entry that fails re-checking should coincide with the ground-truth misread fact well above the rate expected from picking an entry at random. If localization is no better than chance, then the accuracy gains come from something other than the stated mechanism.

The third decisive check is token-matched accuracy against self-consistency and AGS. Here we report accuracy-versus-token curves rather than single points.

Secondary analyses report how often rollbacks land on perceptual versus reasoning steps, broken down by MATH-Vision discipline. GPT-4o is near 50% in Arithmetic and about 18% in Transformation Geometry, so we expect perceptual rollbacks to dominate in the geometry disciplines.