Vision-language models (systems that read an image and write text about it) make more factual mistakes the longer they write. Across many models, the rate of invented objects rises almost linearly with caption length. Prior work explains this growth in two incompatible ways. One explanation is language drift: as the text grows, the model attends less to the image and more to its own previous words. The other is resolution starvation: long, detailed descriptions eventually reach small details that the image encoder has already blurred away by shrinking the picture, so the model fills the gap with what it "knows" usually appears in such scenes. The two explanations call for different fixes. Drift calls for re-anchoring attention to the image. Starvation calls for showing the model more pixels. Current methods pick one fix and apply it everywhere.
We propose to separate the two causes token by token, using a resolution counterfactual. For each object word a model writes from a low-resolution image, we replay the exact same text prefix with a higher-resolution view and check whether the hallucination disappears. Hallucinations that vanish are starved; those that survive are drifting. These labels let us do two things no prior work has done. First, we can split the length-hallucination slope into a part that resolution can remove and a part it cannot. Second, we can train a lightweight linear probe on the model's internal activations that predicts, while generating, when the model is about to substitute prior knowledge for missing pixels. That probe then triggers a targeted high-resolution zoom only at those moments.
Core Claim
The slope of hallucination against length is not one number. It is the sum of a resolution-independent drift term and a starvation term that shrinks as resolution grows. Write $\beta(r)$ for the extra hallucinated objects per additional word at input resolution $r$, the growth-rate slope introduced by LeHaCE. Our working hypothesis is
$$\beta(r) \approx \beta_{\text{drift}} + \beta_{\text{starve}}(r), \qquad \beta_{\text{starve}}(r) \to 0 \text{ as } r \text{ grows},$$
where $\beta_{\text{drift}}$ is the slope that remains once every queried detail is resolvable. The hypothesis has two testable consequences. Raising resolution should lower the slope but level off at a nonzero floor; it should not merely delay the onset of errors. The floor should be the part that attention-resetting methods address. If starvation can be detected from hidden states before the offending word is emitted, a model can spend extra visual tokens only where they help. It would then approach the hallucination rate of always running at maximum resolution, at a fraction of the cost.
Where This Sits
Three bodies of work motivate the split. LeHaCE shows a length-hallucination correlation above 0.9 across many models and blames language priors and transition words such as "also" and "in addition." It evaluates only at fixed image size, so it cannot say whether resolution changes the slope. Summary-Guided Decoding treats the problem as pure drift. It summarizes the generated history to reset attention, applied only to image-related word types. It assumes the visual tokens themselves are adequate. HallE-Control is the closest prior work and the strongest motivation. It shows that detailed captions hallucinate when they ask for finer detail than the encoder resolves, and that raising resolution helps more than enlarging the language model. Its remedy is a static switch between grounded and imaginative output. It neither measures how the slope moves with resolution nor separates which individual errors resolution would fix.
Two lines supply our method. Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning shows that conflicts between visual evidence and prior knowledge are linearly separable in mid-to-late layers (15–22 in 7B models) and can be steered. It labels conflicts on reasoning traces at fixed resolution, using constructed contradictions. We reuse its probing recipe but replace the label source with a resolution counterfactual, which ties the probe to a fix that can actually be carried out. Analysing the Residual Stream of Language Models Under Knowledge Conflicts shows that conflict is detectable several layers before the model commits to an answer. That finding is what makes a pre-emission trigger plausible.
Two systems constrain the design. CARES picks one resolution per query before generation and cuts visual tokens by 70–80% with little accuracy loss. It is single-turn and cannot react as a description moves into finer detail. CogCoM trains a 17B model to issue explicit crop-and-zoom actions. That requires a dedicated training corpus, and the decision to zoom is a learned plan rather than a reading of the model's internal state. Finally, evidential-conflict detection shows that internal feature conflict beats softmax entropy by 4–10 AUROC points at flagging hallucinations. We take this as a warning: an output-confidence trigger is likely to underperform, and we test that directly.
Research Program
The first component is counterfactual labeling. We use two open models with different visual front ends. LLaVA-1.5 7B uses a fixed 336-pixel encoder, so higher resolution means re-encoding crops. Qwen2.5-VL 7B accepts native resolution, so resolution is a continuous setting.
For each image, we generate captions at a low resolution $r_0$ under the 25 instruction templates used by LeHaCE, which span short to very detailed outputs. Each generated object word is checked against MSCOCO annotations using CHAIR's synonym lists. CHAIR measures the share of hallucinated sentences and object mentions. For each hallucinated object at position $t$, we teacher-force the identical prefix with the image at a ladder of higher resolutions. We then record whether the hallucinated word's probability collapses and whether a grounded alternative takes over. Tokens that flip at some resolution are labeled starved, along with the lowest resolution that fixes them. Tokens that persist at the highest resolution are labeled drift.
We apply the same procedure to correctly grounded object words. This gives a control set and guards against higher resolution simply making the model more cautious everywhere.
The second component is the slope decomposition. Using LeHaCE-style curve fitting at uniform lengths of 40, 60, and 80 words, we fit $\beta(r)$ at five or more resolutions per model. We report the starved and drift fractions as a function of position in the caption. We expect starvation to grow with length, because late sentences describe smaller items. We also expect $\beta(r)$ to level off at a drift floor. That floor should coincide with the slope remaining after Summary-Guided Decoding is applied at maximum resolution.
The third component is the starvation probe and the zoom it triggers. We train linear probes on residual-stream activations at each layer at the step before an object word is emitted. Each probe makes a three-way call: grounded, starved, or drift. At inference, when the starved score crosses a threshold, we take the image region receiving the most attention over the preceding few tokens. We re-encode that region at the resolution the labels say is usually sufficient, append the new visual tokens, and continue generating. When the drift score fires, we optionally apply a summary-guided reset instead. The method needs no fine-tuning of the base model, only the probe.
Baselines
The naive reference point is greedy decoding at the model's default resolution. This is the operating point from which LeHaCE measured its above-0.9 correlations. Every method is compared against its CHAIR scores and its fitted slope.
The strongest drift-side competitor is Summary-Guided Decoding. On LLaVA-1.5 7B its summary variant improved $CHAIR_S$ by 16.5% and $CHAIR_I$ by 19% relative to greedy decoding, with GPT-4-judged text quality of 3.75–4.84 (SumGD). We rerun it with the same models, images, prompts, and length targets.
The strongest resolution-side competitor is always running at maximum resolution: native resolution on Qwen2.5-VL, and full multi-crop tiling on LLaVA-1.5. This sets the accuracy ceiling for "more pixels" at the cost of maximum tokens.
For grounded detailed captioning, HallE-Control reports a 44% reduction in object existence hallucination versus LLaVA-7B on CCEval, a GPT-4-assisted evaluator for detailed captions (HallE-Control). We compare against that reported figure on CCEval. Where its checkpoints are available, we also run it on our CHAIR and length-curve protocol.
We include a CARES-style static selector that picks one resolution per caption request (CARES), run on the same images. CogCoM's trained zoom (CogCoM) reports 87.8 on POPE, a yes/no object-existence benchmark. We use that figure for a POPE sanity comparison. Its different base model means we treat it as a reference rather than a matched baseline.
Three ablations isolate the key ingredient, the counterfactually trained trigger. Each keeps the same zoom mechanism and the same average extra visual-token budget as our method. The first triggers zoom at random positions. The second triggers on high output entropy. The third triggers on the evidential-conflict score of Wu-style evidential detection. A fourth ablation trains the probe on the conflict labels of the long-chain reasoning study rather than our resolution-flip labels. All probe thresholds are tuned on a held-out split with the same search budget for every trigger.
Our method beats the field if it matches or exceeds maximum-resolution CHAIR while using at most half the extra visual tokens, and if it beats every equal-budget trigger. Combined with a drift reset, it should also beat Summary-Guided Decoding's reported relative gains on LLaVA-1.5 7B. No per-model $\beta$ values are reported on Lacuna for LeHaCE, so slope comparisons will use our own refits of the baselines.
Why not just…?
Why not just always use maximum resolution? This is the real threat. If $\beta(r)$ falls to near zero at high resolution, the drift term is negligible and our routing has nothing to add beyond saving compute. The decomposition experiment settles this directly. A flat floor near zero would mean we reframe the contribution as an efficiency method competing with CARES. Two pieces of evidence suggest a floor exists. Native high resolution inflates visual tokens to 97–99% of the total (CARES), so it is costly. And CrossCheck-Bench finds failures persist even after perception succeeds.
Why not just use Summary-Guided Decoding everywhere? It assumes every late error is drift. Our counterfactual labels measure what fraction of late errors are fixable by resolution. Any nonzero starved fraction is an error class that attention resets cannot recover, because the information was never encoded. The comparison against Summary-Guided Decoding at matched length will show whether that fraction matters in practice.
Why not just write shorter captions? Shorter captions reduce hallucination by giving up content, which is exactly the trade LeHaCE's uniform-length evaluation exists to expose. We report all methods at fixed lengths of 40, 60, and 80 words, together with object coverage from CCEval, so a method cannot win by saying less.
Why not trigger zoom on output uncertainty, or train a zoom policy like CogCoM? Softmax confidence is known to be near 99% on fabrications, and internal conflict beats entropy by 4–10 AUROC points (evidential conflict). The entropy-triggered ablation tests whether this carries over to zoom decisions. A trained policy is a legitimate alternative, but it needs new training data and a model retrained for each base model. Our probe is a few linear layers on a frozen model. If CogCoM-style zoom at equal token budget matches our method, the probe's value narrows to its low cost, and we will say so.
Proposed Experiments and Decisive Checks
The main data are MSCOCO validation images under the 25 LeHaCE templates, scored with CHAIR and LeHaCE length curves, plus CCEval for detailed captions. VisualOverload's 4K dense paintings serve as a stress test. They are where starvation should dominate: the best model reaches only 41.7% counting accuracy, and o3 reaches 19.6% on the hardest split (VisualOverload). POPE serves as a check that zooming does not distort short-answer behavior.
Three checks decide the project. First, the fitted $\beta(r)$ must show a nonzero floor that resolution cannot remove. A slope that only shifts in onset, or that falls to zero, falsifies the two-term picture. Second, the starvation probe must predict resolution-flip labels on held-out images substantially above an entropy predictor. Probe accuracy far below the 93–99% AUC reported for fixed-resolution conflict probes would suggest starvation is not linearly readable. Third, probe-triggered zoom must beat random and entropy triggers at equal token budget and approach maximum-resolution CHAIR at lower cost. If it fails, the useful contribution reduces to the measurement study and the counterfactual-labeled dataset, which we would release either way.
References
- Toward a Stable, Fair, and Comprehensive Evaluation of Object Hallucination in Large Vision-Language Models
- Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding
- HallE-Control: Controlling Object Hallucination in Large Multimodal Models
- Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning
- Analysing the Residual Stream of Language Models Under Knowledge Conflicts
- CARES: Context-Aware Resolution Selector for VLMs
- CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoning
- Visual hallucination detection in large vision-language models via evidential conflict
- CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict Resolution
- VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes