Vision-language models can read many images and very long inputs. When asked to write long reports about those images, though, they slowly drift away from the image. In the first few hundred words, their statements about the image are usually correct. By word three thousand, they are inventing numbers and details that fit the story but not the picture. This proposal asks a narrow mechanical question about that decay and then builds the fix the answer points to.
The question rests on one observation about how these models decode. The keys and values a model computes for the image are produced once, when it first reads the prompt, and then sit frozen in its working memory (the key-value cache) for the rest of generation. So the image's stored representation cannot rot as the output grows. Whatever goes wrong must happen on the reading side: newly generated tokens get worse at retrieving from fixed image entries. We propose to split that retrieval failure into three causes that each have a clean intervention. The first is growing positional distance between the current token and the image. The second is dilution by thousands of competing text tokens. The third is drift in what the model is "asking" for. We then test whether the dominant cause can be removed at inference time.
Core Claim
Grounding errors that appear late in long multimodal outputs can be split into three parts. The first is a distance component, caused by rotary position embeddings (the scheme that encodes token position by rotating keys and queries, which weakens attention between far-apart tokens). The second is a dilution component, caused by the attention softmax spreading over an ever-larger text history. The third is a query-drift component, caused by the model's text-dominated internal state no longer forming queries that match image keys.
We hypothesize that the distance component is large and cheap to remove. Rotary embeddings act as an exact rotation, so cached image keys can be re-rotated so that they appear a fixed short distance behind the current token. This requires no recomputation and no retraining. If the hypothesis holds, a decoding-time "visual position refresh" should flatten the curve of grounding accuracy against output position. If it fails, the decomposition still tells the field which kind of fix (attention reweighting, re-encoding, or training) is worth pursuing.
This differs from the closest work in specific ways. More Thinking, Less Seeing? documents "attention drift": attention weight on visual tokens decays toward zero as reasoning chains lengthen. It does so observationally, within reasoning budgets far shorter than long-form writing, and proposes no positional intervention. Visual Representations inside the Language Model shows that visual information is present in image value tokens even when answers are wrong, for up to 33.3% of errors on a BLINK task. That is evidence for a retrieval failure rather than a storage failure, but the paper studies only short single-turn answers. PIC compresses position IDs in the middle of a text sequence to extend generation length. It is text-only and targets length and coherence. It does not target the distance between text and a specific block of visual tokens. LongWriter-V attacks the multimodal output ceiling with synthetic long-form data and segment-wise preference optimization. It treats the model as a black box and never measures where in the output grounding fails. A tempting framing in the research file asks whether positional drift "corrupts early visual KV tokens." Under standard caching that question is ill-posed, and recasting it as a retrieval question is the first contribution.
Research Program
Measuring grounding by output position. The first piece is a measurement instrument that existing benchmarks do not provide. MMLongBench-Write scores whole outputs with a judge, so it cannot localize where grounding fails. We will generate charts, tables-as-images and simple geometric diagrams from templates with known underlying values, following the template-and-tool data generation idea in Synthesize Step-by-Step. We will then ask models for 2,000 to 6,000-word analytical reports covering one to several such images. Every numeric or relational claim in the output can be extracted and checked against ground truth. That yields a grounding curve: claim accuracy as a function of output position, binned by every 250 tokens. We will also include a matched set of claims that need no image, such as arithmetic on values already stated earlier in the text. This control separates image-retrieval decay from general late-generation decay of the kind HelloBench and LongGenBench describe.
Measuring how much image information reaches each token. Attention weight is a weak proxy for information flow. For each generated token, we will therefore compute the norm of the summed attention-head outputs contributed by image positions, per layer. We will also train linear probes, in the style of Language Models Encode the Value of Numbers Linearly, to decode the true chart value from the hidden state just before the model emits a number. That paper found linear decodability above 0.95 R² for numbers in text. A drop in probe accuracy with output position, while the image value tokens stay unchanged, is the direct signature of a retrieval failure.
Decomposing the failure. The core analysis uses three interventions on the same generated prefixes. In re-rotation, at a late position $t$ we rotate all cached image keys so that their relative offset to the current query equals the offset they had at the start of generation. This removes the distance effect while leaving every other token untouched. In dilution compensation, we add a per-head logit bias to image keys equal to the log-ratio of attended text tokens at position $t$ versus at the start. This restores the image's share of attention mass without changing positions. In the teleport condition, we copy the current sentence and a short summary of the preceding text into a fresh context immediately after the image, and regenerate the claim. Teleport removes distance and dilution together. Write $g(t)$ for claim accuracy at position $t$, with subscripts for each condition. The gap $g_{\text{teleport}}(t) - g(t)$ bounds the total recoverable loss. We then measure the share recovered by re-rotation alone, by dilution compensation alone, and by both together. The share that teleport recovers but the two mechanical fixes do not is attributed to query drift.
We will also check depth. Visual Thoughts reports that attention to raw image tokens is near zero after layer 12 even in short chains. Some decay is therefore layer-driven rather than length-driven, and we will report the decomposition per layer group so the two are not confused.
Turning the diagnosis into a method. If distance dominates, the method is a visual position refresh applied at fixed intervals or whenever the image-contribution norm falls below a threshold. It costs one rotation of the image keys per refresh. If dilution dominates, the method is the calibrated image-key bias. If query drift dominates, inference-time fixes are insufficient. The right follow-up would be training signals that reward late-position grounding, which segment-wise preference optimization of the LongWriter-V kind could supply. We commit to reporting whichever outcome occurs.
Models are Qwen2.5-VL-7B, LongWriter-V-7B and LLaVA-OneVision. The first two share a backbone, which lets us ask whether long-output training already shrinks one of the three components.
Baselines
Unmodified decoding. The naive reference point is greedy and nucleus decoding with the unmodified model on the same prompts. It defines the grounding curve to be flattened.
Strongest trained competitor. The strongest existing method is LongWriter-V-7B, trained on LongWriter-V-22k with iterative preference optimization. LongWriter-V reports 84.6% on MMLongBench-Write for this model, against 62.7% for GPT-4o. It is the strongest competitor because it is the only open method built for long multimodal outputs. Our refresh is applied on top of it with identical prompts, decoding settings and length targets. Beating it means higher claim accuracy in the last third of outputs, not lower MMLongBench-Write. We will also report MMLongBench-Write to show that the refresh does not hurt judged quality. No reported result exists on Lacuna for position-resolved claim accuracy on any model, so the baseline curve is ours to measure.
Text-side position compression. We will apply Dynamic PIC from PIC unchanged, compressing the middle positions, to the vision-language model. This tests whether generic position compression already fixes grounding as a side effect.
Attention reweighting toward image tokens. We include the logit-bias arm, tuned on a held-out split with the same budget as the refresh interval. It also serves as the ablation that isolates positional distance from attention share. We additionally borrow the key-blocking intervention from Visual Representations inside the Language Model, which raised POPE F1 on LLaVA-OneVision from 78.1 to 81.2, to see whether input-agnostic image keys matter more late in generation.
Re-inserting the image. The last baseline re-appends the image mid-generation at the same interval as the refresh. This is the strongest naive competitor.
Why not just…?
Why not just re-insert the image every section, plan-and-write style? This is a real threat, because it plausibly removes distance, dilution and some query drift at once. We run it as a baseline at matched intervals. It costs a full re-encoding of hundreds of image tokens each time and grows the context, which brings the wall closer. LongWriter-V shows that frontier models driven by a plan-and-write agent still fall short of trained models on MMLongBench-Write. If re-insertion matches the refresh at comparable cost, we will say so. The decomposition still explains why it works, which a pipeline alone cannot.
Why not just boost attention to image tokens? That treats only dilution. Our dilution-compensation arm is exactly this intervention. If it recovers as much as re-rotation, the positional story is wrong and we report the simpler fix. The observational attention drift result in More Thinking, Less Seeing? cannot distinguish the two causes.
Why not just train on more long multimodal data? LongWriter-V-7B is that approach and is our main baseline. Two pieces of evidence give reason to doubt that training removes grounding decay. More Thinking, Less Seeing? finds that SFT-then-RL models hallucinate more than pure-RL ones; the top RH-AUC of 0.63 went to Ocean-R1-7B. LongWriter traces length failures to training data skew rather than capacity. Comparing the decomposition on Qwen2.5-VL-7B against LongWriter-V-7B directly measures which component training reduces.
Why not just apply PIC? PIC compresses middle text positions to keep the model in-distribution, but the image sits in the "initial context" that PIC deliberately leaves uncompressed. Text-image distance therefore still grows, more slowly. The PIC baseline arm settles this empirically.
Decisive Checks
The first check concerns the instrument. The probe for chart values must decode well near the start of generation, comparable to the text-only probe results of Language Models Encode the Value of Numbers Linearly. Without that, falling probe accuracy cannot be read as retrieval loss.
The main result is the decomposition itself. We report the fraction of the teleport-recoverable gap closed by re-rotation, by dilution compensation, and by both, at output positions of 1k, 2k, 4k and 6k tokens. The image-free control claims must stay flat under all interventions. Otherwise the interventions are acting on general generation quality rather than on image retrieval.
The positional hypothesis is falsified under either of two outcomes. One is that re-rotation recovers less than a quarter of the gap at 4k tokens while dilution compensation recovers more. The other is that re-rotation degrades text coherence, measured by MMLongBench-Write and repetition rate, by more than it gains in grounding.
As a transfer test, we apply the selected refresh to the reasoning half of RH-Bench at long reasoning budgets. If it raises RH-AUC above the untreated model on the same backbone, that is evidence that the mechanism behind long-report drift also drives long-reasoning hallucination.
References
- More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models
- Visual Representations inside the Language Model
- PIC: Unlocking Long-Form Text Generation Capabilities of Large Language Models via Position ID Compression
- LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in Vision-Language Models
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- Language Models Encode the Value of Numbers Linearly
- Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought
- LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs