Vision-language models (systems that read an image and a question and then write an answer token by token) often fail on visual math and chart questions even when they have read the picture correctly. A second, separately studied failure in language models is degeneration: the model falls into repeating loops because each repeated phrase makes the next repetition more likely. This note proposes that in long step-by-step multimodal reasoning these two failures are partly the same event. As the written reasoning grows, the model attends less and less to the image tokens at the start of its input. With the picture effectively forgotten, it starts feeding on its own output and loops. If that is right, the fix is not to penalize repeated words. It is to detect the moment the image drops out of attention and make the image close again, positionally and in attention, before the loop locks in.
Concretely, we would build a training-free decoding intervention for autoregressive vision-language models. It watches a cheap signal, the share of attention that newly generated tokens place on image tokens, alongside standard loop signals. When a collapse fires, it trims the repetitive tail of the cache and re-rotates the cached image keys so they sit at a short positional distance from the current token. The test is whether this recovers accuracy on MathVista and MATH-Vision, not just whether it reduces repetition. It should also explain why step-by-step prompting sometimes hurts these models.
Core Claim
We claim that a measurable share of reasoning errors in autoregressive multimodal models on visual math and chart questions occurs in traces that degenerate. We further claim that these traces are preceded by a drop in attention to image tokens. Restoring that attention at the right moment reduces the errors more than generic loop-breaking does at matched compute. Let $a_t$ be the fraction of attention mass that the token at step $t$ places on image tokens, averaged over heads in the middle and late layers. Our hypothesis is that $a_t$ falls below a model-specific floor several dozen tokens before repetition metrics fire. Interventions keyed to that fall should beat interventions keyed to repetition alone.
The evidence for this mechanism is scattered across several papers but has not been joined. Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset finds that 42.2% of GPT-4V errors are reasoning errors made after correct perception, and that step-by-step prompting sometimes lowers accuracy. The authors attribute this to lost visual-textual consistency across steps, which motivates our claim. Chain-of-Thought Hijacking shows that over long reasoning chains, attention drifts from the prompt toward generated tokens. That paper studies refusal safety, and we reuse its observation for visual grounding. Mitigating Object Hallucination via Concentric Causal Attention shows that rotary position embeddings (a scheme that encodes token order by rotating query and key vectors, which weakens attention across long distances) cause early image tokens to be neglected. Its fix is a permanent change to positions, applied always. Ours is applied only when needed and during decoding. From Self-Attention to Markov Models explains why loops self-reinforce once context is dominated by generated tokens: transition probabilities track token frequencies in the context.
Research Program
The work has three stages, and each stage is allowed to kill the project.
The first stage is diagnosis. We run open autoregressive models, including Qwen3-VL-8B (the model used in PatternEval) and a LLaVA-family model, with long step-by-step reasoning on MathVista, MATH-Vision, ChartQA and Chart101. We log $a_t$ along with the loop signals used by LoopGuard: type-token ratio, compression ratio, confidence collapse and tail repetition. We also track structural repetition in any code rationales, following Rethinking Repetition Problems of LLMs in Code Generation. From these logs we estimate two quantities. One is the fraction of wrong answers that come from degenerate traces. The other is whether the drop in $a_t$ comes before the loop, as opposed to after it or not at all. A control condition replaces the image with a blank image of identical token count. This separates "the image fell out of attention" from "attention to any early tokens fell."
The second stage is the intervention. We reuse LoopGuard's debounced multi-signal trigger and add the visual-attention-share signal. When the trigger fires, we prune the repetitive tail as LoopGuard does. We then re-rotate the cached image keys. Rotary embeddings are rotations, so cached keys can be moved to new positions exactly, without re-encoding the image. That lets the image sit just before the current token at the cost of one rotation per image key. This is the key ingredient. It differs from LoopGuard, which is text-only and leaves the instruction anchors at their original far positions. It differs from Context Tokens are Anchors, which strengthens attention to context tokens in diffusion multimodal models whose repetition arises from cache disruption rather than causal context growth. It also differs from Concentric Causal Attention, because we intervene only on trigger and need no retraining.
The third stage is analysis of whether the fix touches reasoning or only cosmetics. We split outcomes by the error categories of MATH-Vision. We check whether the harm that step-by-step prompting causes relative to direct answering shrinks under re-anchoring. We also measure PatternEval failure triggers to see whether the intervention reduces repetition and leaked thinking markers without RL.
Baselines
The naive reference point is standard greedy and nucleus decoding with step-by-step prompting on the same models. We will also report direct answering with no chain of thought, because MATH-Vision finds that chain of thought sometimes underperforms it. If direct answering beats every chain-of-thought variant, including ours, that is itself a finding against the value of long multimodal traces. Published references for the absolute scale are GPT-4V at 49.9% accuracy on MathVista, reported in MathVista, and GPT-4o at 30.39% on MATH-Vision. These come from different, stronger models and serve as context, not as targets.
The strongest existing decoding-time competitor is LoopGuard applied unchanged to the vision-language model's text stream, using the same trigger thresholds and tuning budget as our method. On LoopBench-DC, text baselines loop at 100% while LoopGuard reaches rates as low as 1.3%, as reported in the LoopGuard paper. No reported result found on Lacuna for LoopGuard on any multimodal benchmark, so we establish that number ourselves. Beating it means matching its loop reduction while gaining accuracy on MathVista and MATH-Vision, at equal or fewer generated tokens, across three seeds.
The strongest training-time competitor is PatternRL from Beyond Correctness. It reduced the non-thinking failure-trigger rate on Qwen3-VL-8B by 14.35 percentage points over correctness-only RL, with less than 1% change in aggregate accuracy. We compare on PatternEval with the same base model. A training-free method that approaches that trigger reduction while improving accuracy would be a clear win. Falling short on triggers but gaining accuracy would still be informative.
For loop handling we also include Context Tokens are Anchors' attention-enhancement component, ported to causal attention. No per-dataset MathVista or ChartQA number is reported for it in the file. To isolate the key ingredient, we run four ablations: re-anchoring without tail pruning; pruning without re-anchoring (which is LoopGuard); re-anchoring the text instruction only, not the image; and always-on re-anchoring at every step. Every variant uses the same tuning budget for its thresholds on a held-out MathVista split.
Why not just…?
Why not just use repetition, presence or frequency penalties? These act on token recurrence. Structural repetition, where the model loops over the same syntax while renaming variables, is roughly nine times more common than literal repetition in code, according to Rethinking Repetition Problems. Chart reasoning also legitimately repeats axis values and labels, which such penalties would suppress. We include a tuned penalty baseline and Locally Typical Sampling. We expect them to cut repetition metrics without restoring accuracy, and Chart101 point-reading errors will show whether they harm numeric copying.
Why not just run LoopGuard on the vision-language model? This is the real threat. If loops in multimodal reasoning are ordinary text loops, LoopGuard alone will capture nearly all the gain. The "pruning without re-anchoring" ablation settles it directly. If re-anchoring adds no accuracy over LoopGuard at matched tokens, we drop the visual claim and report that LoopGuard transfers, which is a useful negative result.
Why not just resample when a loop is detected, or vote over several samples? Resampling is simple and may recover many answers. It costs whole regenerations, though, and it does not address forgetting that recurs on every long trace. We compare against detect-and-restart and against majority voting at matched total generated tokens. If either matches us at equal compute, the practical case for re-anchoring collapses, and we will say so.
Why not just scale the model or train with PatternRL? Beyond Correctness reports that pattern failures persist from 4B to 397B parameters. PatternRL helps, but it costs accuracy at 4B and leaves residual failures. Our method is training-free and complementary, so we also test it on top of a PatternRL-style checkpoint if one can be reproduced.
Proposed Experiments
The first decisive check is whether the target exists at all. If degenerate traces account for only a small share of wrong answers on MathVista and MATH-Vision, the accuracy upside is capped and the proposal shrinks to a loop-reduction study. The second check is about timing: does $a_t$ fall before repetition signals fire, more than in the blank-image control? If the drop follows the loop rather than leading it, attention loss is a symptom and re-anchoring should not help. The third check is the head-to-head against LoopGuard at matched tokens on MathVista, MATH-Vision and ChartQA. Here we report accuracy, loop incidence, compression ratio, four-gram repetition and tokens used. The fourth check asks whether the chain-of-thought-minus-direct accuracy gap on MATH-Vision narrows under re-anchoring, which would tie degeneration to the known failure of multimodal chain of thought. We also run POPE to confirm that re-anchoring does not increase object hallucination, and PatternEval for comparison with PatternRL. The outcome counts as a success only if accuracy gains concentrate in previously degenerate traces and in "correct perception, wrong reasoning" errors. If gains appear only in perception errors, the mechanism is something other than what we claim.
References
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Chain-of-Thought Hijacking
- Mitigating Object Hallucination via Concentric Causal Attention
- From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers
- LoopGuard: Breaking Self-Reinforcing Attention Loops via Dynamic KV Cache Intervention
- Rethinking Repetition Problems of LLMs in Code Generation
- Context Tokens are Anchors: Understanding the Repetition Curse in dMLLMs from an Information Flow Perspective
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
- Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs
- Locally Typical Sampling