Vision-language models, systems that read an image through a vision encoder and then write text about it with a language model, often describe objects that are not there. One of the most effective fixes so far works inside the vision encoder. It finds image patches whose internal features are "uncertain," meaning a tiny adversarial nudge to the pixels changes them a lot, and it hides those patches from attention. The language model then has less noise to explain away and fewer objects to invent. The problem is that a real object only a few pixels wide is also exactly the kind of patch that a tiny nudge changes a lot. The fix that removes invented objects may therefore be quietly deleting real small ones.
This note proposes to measure that collateral damage and then remove it with a test borrowed from small-object detection. Detection researchers learned that a few pixels of displacement wreck overlap scores for tiny boxes, and they turned that fragility into a design principle. We turn it into a diagnostic. If the image is shifted by a few pixels, less than one patch width, the features of a real small object should move with it and keep meaning the same thing once mapped into the language model's input space. Unstructured noise should not. We propose to mask a patch only when it is both uncertain and fails this shift-consistency test, and to show that this keeps most of the hallucination reduction while recovering small-object recall.
Core Claim
Uncertainty masking, as introduced in On Epistemic Uncertainty of Visual Tokens for Object Hallucinations, estimates how uncertain each patch feature (a "visual token") is by measuring its sensitivity to small projected-gradient-descent (PGD) perturbations of the input image. It then masks high-uncertainty tokens in the middle layers (13–17) of LLaVA-1.5-7B's 24-layer encoder. This cuts the sentence-level caption hallucination rate $C_s$ from 47.4% to 29.2%. The authors themselves note that aggressive masking can discard fine detail.
Our first claim is empirical and falsifiable. On COCO, masked tokens overlap ground-truth small-object boxes far more often than chance. As a result, masking lowers the rate at which captions mention small objects that really are present.
Our second claim concerns the method. Shift-consistency separates real small-object evidence from noise, so filtering the mask with it moves the trade-off between hallucination and small-object recall outward rather than just sliding along it.
To define the test, let $z_i(x)$ be the feature of patch $i$ for image $x$. Let $P$ be the model's projector into the language model's input space. Let $T_\delta$ translate the image by a sub-patch offset $\delta$. Let $\tilde z_i(T_\delta x)$ be the shifted image's feature map, bilinearly resampled back by $\delta$ so that it lines up with patch $i$. The shift-consistency of patch $i$ is
$$c_i = \frac{1}{|\Delta|}\sum_{\delta \in \Delta} \cos\big(P z_i(x),\; P\tilde z_i(T_\delta x)\big),$$
where $\Delta$ is a small set of offsets of 2–7 pixels for a 14-pixel patch. The mask removes patch $i$ only if its uncertainty $u_i$ exceeds a threshold $\tau_u$ and $c_i$ falls below a threshold $\tau_c$.
Measuring consistency after the projector $P$ is deliberate, and it is where the cross-modal gap enters. A small object can have encoder features that are stable but that the connector maps to nowhere meaningful. Such a token is useless to the language model, and the test treats it accordingly.
Relation to Prior Work
Two lines of work motivate the proposal. Understanding Visual Detail Hallucinations builds a COCO-derived benchmark that separates small from regular objects for existence, color, and position. Across 11 models it finds systematic drops on small objects. It does not study what hallucination fixes do to small objects, and that question is ours.
Its own numbers also constrain us. For LLaVA-1.5-7B, existence F1 is 73.2% on regular objects and 73.7% on small ones, so the baseline small-object deficit for this particular model is negligible. We therefore frame the risk as damage introduced by masking, not as a pre-existing weakness. We also test a second model family where that paper reports steeper drops.
The uncertainty-masking paper supplies the mechanism we modify. Our contribution is a filter on its mask, not a replacement for its uncertainty estimate.
Detection work supplies the lens. A Normalized Gaussian Wasserstein Distance for Tiny Object Detection shows how sharply overlap scores react to a few pixels of displacement. For example, a 1-pixel shift drops the IoU of a 6×6 box from 0.53 to 0.06. The paper replaces IoU with a distance between Gaussian boxes and raises AI-TOD AP from 11.1% to 17.8% on Faster R-CNN. We reuse its insight twice. First, a pixel shift is the natural probe of small-scale behavior. Second, we use its Gaussian distance, not IoU, to score the positions that models generate for tiny objects, so that evaluation noise does not swamp the comparison. Noise-Robust Tiny Object Localization with Flows adds a related warning: tiny-object ground truth is itself noisy, which is another reason to avoid hard overlap thresholds in scoring.
Several methods are competitors rather than ingredients. ICT steers attention heads that respond to clear rather than blurred inputs. It improves POPE F1 by 7.09% on LLaVA-v1.5 at no added latency, and it locates object-level heads in deep layers. It never tests objects below 16×16 pixels, and blurring may erase tiny objects from its own calibration signal. MG-LLaVA fixes small-object blindness by adding high-resolution and detector-cropped object streams. That is an architectural answer requiring retraining, whereas ours is training-free and applies to an existing model. Visual contrastive decoding also contrasts clean and corrupted images, but it does so on output distributions at every decoding step, at 2.1–3.6× latency according to ICT. Our shifted images only decide which encoder tokens survive, and we use translations that preserve content rather than corruptions.
Research Program
Phase one: diagnosis. We run uncertainty masking on LLaVA-1.5-7B over COCO val2017 at several masking rates. For each image we compute the fraction of masked tokens whose patches overlap COCO small-object boxes (area below 32²) and compare it to the area-matched chance rate. We then compute small-object caption recall: the fraction of present small objects named in the caption, matched with the CHAIR synonym lists. We report this recall with and without masking, stratified by object area.
If masked tokens are not enriched on small objects, and small-object recall does not fall, the proposal stops here. That outcome would itself be a useful negative result.
Phase two: the shift-consistency filter. We compute $c_i$ with four to eight sub-patch offsets. This costs that many extra vision-encoder forward passes and no language-model passes, which is cheaper than the PGD step the base method already runs. We tune $\tau_c$ on a 1,500-image COCO split disjoint from evaluation, matching ICT's calibration budget.
We also test whether shift-instability alone, without PGD, is a good enough uncertainty estimate. If it is, the method becomes backpropagation-free.
Phase three: generality. We repeat both phases on a second, arbitrary-resolution model. The goal is to check that the effect does not depend on LLaVA's fixed 336-pixel input.
Baselines
The naive reference point is the unmodified LLaVA-1.5-7B. Its reported $C_s$ is 47.4% in the uncertainty-masking paper. This anchors both hallucination and the best achievable small-object recall.
The strongest direct competitor is uncertainty masking itself, run with the authors' layer choice (13–17) and PGD settings. Its reported result is $C_s$ dropping from 47.4% to 29.2% on LLaVA-1.5-7B (On Epistemic Uncertainty of Visual Tokens). We sweep its masking threshold to trace a full curve of hallucination against small-object recall, so that we are not comparing against a single operating point. Our method counts as beating it if, at a matched $C_s$ within one point of 29.2%, it recovers at least half of the small-object recall that masking removes, with paired-bootstrap significance across images. Equivalently, its curve should dominate the swept baseline curve.
The strongest zero-latency competitor is ICT, re-calibrated on the same 1,500 images. ICT reports a POPE F1 gain of 7.09% on LLaVA-v1.5 (ICT). We compare against it on the small-object split, where no reported result exists on Lacuna.
Two ablations isolate the key ingredient. The first protects tokens using a random mask of equal size. The second measures consistency before the projector rather than after it. If the pre-projector version works as well, the cross-modal part of the claim fails and we report that.
The detector-protection baseline described next is included at equal tuning budget.
Why not just…?
Why not just protect tokens inside an off-the-shelf detector's boxes? This is the most serious threat. Protection only needs recall, not precise localization. The counterargument is that detectors themselves miss tiny objects badly: on AI-TOD, AP is 11.1% for Faster R-CNN and 20.8% even with DetectoRS plus NWD (NWD). The detector approach also adds a separate model, as MG-LLaVA's offline pipeline does. Rather than argue, we run it. We use an open-vocabulary detector at a low score threshold to protect overlapping tokens, under the same compute budget. If it matches our curve, we will report that the simpler fix suffices and that shift-consistency is only useful when no detector is available.
Why not just lower the masking rate? A lower rate trades hallucination reduction for recall along the same curve. The threshold sweep in our baseline shows this directly. A filter is only worthwhile if it moves off that curve.
Why not just upscale the image so small objects become large? Understanding Visual Detail Hallucinations found that 2× resizing helps arbitrary-resolution models but hurts fixed-resolution LLaVA. MG-LLaVA notes the resulting growth in token count. Upscaling is also complementary, not competing: masking would still delete real tokens at the new scale. We include 2× resizing, alone and combined with our filter, on the arbitrary-resolution model.
Why not just use ICT, which never deletes anything? Steering does not delete tokens, but it calibrates on blurred images. Blurring removes tiny objects first, so the heads it selects may simply ignore them. Comparing ICT on the small-object split settles whether this concern is real.
Proposed Experiments
The evaluation uses four sources: CHAIR captions on COCO, with $C_s$ and $C_i$ and the new area-stratified small-object recall; POPE and the exhaustive POPE-C from THRONE, which better exposes "yes" bias; the small/regular existence, color, and position splits of the visual-detail benchmark; and THRONE free-form precision. THRONE matters because it shows that hallucination in direct questions and in long descriptions can move in opposite directions, and Why LVLMs Are More Prone to Hallucinations in Longer Responses shows that drift grows with length. We therefore report caption metrics bucketed by caption length.
For THRONE-style object enumeration with coordinates, which raised precision from 69.9% to 87.2% on LLaVA-v1.5, we score the positions of small objects by normalized Gaussian Wasserstein distance as well as IoU. This shows how much of the apparent positional error is just IoU breaking down at small scale.
As a stress test far outside COCO, we pose POPE-style "is there a bird" questions on SOD4SB frames (MVA2023), where 99.89% of objects are small. No reported LVLM result was found on Lacuna for this setting.
Three checks are decisive. First, phase one must show that masked tokens are enriched on small objects and that small-object recall drops. Second, the filtered mask must dominate the swept masking curve on both models. Third, the post-projector consistency test must outperform both the pre-projector variant and detector protection. If the third check fails, the honest conclusion is that a simple detector, not a cross-modal test, is the right guard.
References
- On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Models
- Understanding Visual Detail Hallucinations of Large Vision-Language Models
- A Normalized Gaussian Wasserstein Distance for Tiny Object Detection
- Noise-Robust Tiny Object Localization with Flows
- ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models
- MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning
- THRONE: An Object-Based Hallucination Benchmark for the Free-Form Generations of Large Vision-Language Models
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- MVA2023 Small Object Detection Challenge for Spotting Birds: Dataset, Methods, and Results