Authenticating Visual Claims with Secret Image Transforms

When a vision-language model says "there is a refrigerator next to the sink," there are two very different ways that sentence can come about. The model may have seen a refrigerator. Or its language habits, learned from text in which kitchens usually contain refrigerators, may have supplied one. Reading the sentence cannot tell these cases apart, and checking whether the answer stays the same under image changes does not help much either, because a guess driven by the model's language habits is often more stable than a real observation. We propose to treat each object claim like a message whose sender must be authenticated. Before re-querying the model, we apply a secret random spatial transform to the image, such as a flip, a rotation, or a quadrant swap, and ask the model where the claimed object is. An object the model actually sees must move exactly as the secret transform dictates. An object supplied by the language habits cannot follow a transform the model never observed, so its reported location will not track the key.

The idea comes from stochastic encoding in adversarial communication. There, a transmitter uses private randomness so that an adversary who knows the whole codebook still cannot forge a valid transmission. Authentication Against a Myopic Adversary proves that deterministic encoders lose all authentication capacity against a sufficiently informed adversary, while stochastic encoders recover it. Our analogue treats the model's language habits (we call these its "prior") as the forger. The secret transform plays the role of the private randomness. The model's spatial answer plays the role of the tag that must match the key. The result is a hallucination check that runs entirely inside the model being checked, needs no external detector, and makes a testable prediction: the chance that a claim passes by luck falls geometrically with the number of independent challenges, until one specific failure mode described below sets a floor.

Core Claim

Existing uses of randomness in hallucination mitigation all treat the random view as something to average over or subtract. VCD adds Gaussian noise to build a prior-heavy reference and subtracts its logits. The same paper documents that the noise pushes predictions toward stereotypes, which is why the noisy view is useful only as a negative. RITUAL ensembles predictions from the original image and one randomly transformed image, raising LLaVA-1.5 POPE accuracy from 84.13% to 88.87%. Both use randomness whose value is thrown away.

Our claim is that the value of the randomness is the useful signal. Because the transform is drawn privately and is invertible, we know exactly where a truly seen object must appear in the transformed image. Agreement with that prediction authenticates a claim, and disagreement flags it. This check separates grounded content from prior-driven content in a way that invariance checks cannot, because the prior is stable but blind to the key.

Concretely, let $o$ be an object the model mentions. Let $\ell_0(o)$ be the grid cell (out of $C$ cells) where the model places $o$ in the original image. Let $g_1,\dots,g_R$ be secret transforms drawn from a key space, and let $\ell_r(o)$ be the cell the model reports for $o$ in the image transformed by $g_r$. The authentication score is

$$s(o) = \frac{1}{R}\sum_{r=1}^{R} \mathbf{1}\left[\, g_r(\ell_0(o)) = \ell_r(o) \,\right],$$

where $g_r(\cdot)$ maps a cell to its position after the transform. The null model is a forger that cannot see the key and guesses cells without regard to it. Under that null, each term is a coin with success probability near $1/C$, so a claim reaching $s(o)=1$ by luck has probability roughly $C^{-R}$; this follows from treating the $R$ challenges as independent. The central empirical question is how far real prior-driven hallucinations depart from this blind-forger ideal.

The Main Threat: Anchored Forgeries

The analogy also predicts where the method breaks. Beemer et al. identify "overwritability": an adversary who observes the transmission clearly enough can forge a substitute that mimics legitimate noise, and authentication then fails. Our version of this is a hallucinated object anchored to a real one. If the model places an imaginary refrigerator "next to the sink," the sink moves with the transform, and the imaginary refrigerator may move with it, passing the check.

This is exactly the co-occurrence failure that POPE's adversarial split probes (Evaluating Object Hallucination in Large Vision-Language Models, where MiniGPT-4's F1 drops from 78.86 on random to 71.37 on adversarial). Rigid transforms such as flips and rotations preserve anchor-relative layout. They are therefore the "weak key" regime. Quadrant permutations break relative layout across quadrant boundaries while keeping most object content intact, so they should make anchored forgery harder. The research program measures this trade-off directly: stronger scrambling defeats anchored forgeries but damages real objects that span boundaries, lowering the pass rate of true claims. The best key space is the one that maximizes the gap between the pass rates of true and hallucinated objects, and finding it is part of the contribution.

Research Program

We will build the check on LLaVA-1.5 and LLaVA-Next, the two models with the most reported numbers in the file. Location is elicited as a multiple-choice question over a 3×3 grid ("In which cell is the refrigerator?"). This works for any instruction-tuned model and avoids depending on box-coordinate training. Where a model can emit box coordinates, we additionally score intersection-over-union after mapping the box through the key. The key space mixes the eight flip and rotation symmetries with the 24 permutations of image quadrants, and we study each family separately.

The check is used in two ways. For yes/no probing, the model answers "yes" only if the queried object's score exceeds a threshold. For free-form captions, we run the check only on object nouns, identified by part-of-speech tagging as in HALC and Summary-Guided Decoding, and we remove unauthenticated objects. Cost scales with $R$ times the number of object claims, using short location prompts, not with caption length. This matters because the open problems in the file emphasize the per-token overhead of VCD (two passes per token) and HALC (2.4× slowdown).

The analysis half of the program has three parts. First, we measure pass rates for ground-truth versus hallucinated objects as a function of $R$ and key family, and compare the observed decay against the $C^{-R}$ blind-forger prediction. Second, we group failures by whether the hallucinated object co-occurs with a present object, which tests the anchored-forgery account. Third, we relate authentication failure to response length. Why LVLMs Are More Prone to Hallucinations in Longer Responses shows that attention drifts from visual tokens toward generated text as captions grow. We therefore expect objects introduced late in long captions to fail authentication more often, and to fail as blind forgeries rather than anchored ones.

Baselines

The naive reference point is the unmodified model with greedy decoding. RITUAL reports it at 84.13% POPE accuracy for LLaVA-1.5, and SHARP reports 79.5% on POPE adversarial for LLaVA-Next. We will recompute both numbers under our prompts so that every method shares one evaluation harness.

The most direct competitor is RITUAL, which uses random transforms and ensembles output distributions. Its strongest reported result is 88.87% POPE accuracy on LLaVA-1.5. It is the strongest competitor because it uses the same raw ingredient as our method, randomly transformed images, in a non-authenticating way. We will run it with exactly our transform families and the same number of views $R$, so that the only difference is how the views are used. Beating it means higher POPE accuracy and F1 on the adversarial split at matched view count, not only on the random split.

The key ablation isolates the core ingredient. It uses the same transformed views and the same model, but scores each claim by whether the yes/no answer stays the same across views (invariance) rather than whether the location follows the key (equivariance). If invariance scoring matches equivariance scoring, the secret key contributes nothing and the proposal fails.

SHARP is the strongest single-pass internal method. It injects steering vectors at layer 10 and reports 82.8% POPE adversarial accuracy on LLaVA-Next, an MME total of 613.3 on LLaVA-v1.5, and CHAIRs reduced from 52.8 to 34.8. We will use its released vectors if available, or re-extract them on the same data the paper describes, and compare at wall-clock parity as well as at raw accuracy.

VCD is run with its default contrast weight and plausibility constraint. It reports a gain of up to +7.4 F1 on POPE random and an MME perception gain from 565 to 604 on LLaVA-1.5.

For captioning, the reference methods are HALC, which reduced CHAIRs on MiniGPT-4 from 30.87% to 17.80%, and VASparse, which reduced instance-level CHAIR on LLaVA-1.5 from 7.22% to 5.82%. Beating them means lower CHAIR at comparable recall of true objects. Removing objects trivially lowers CHAIR, so we report object recall against COCO annotations alongside it.

Finally, we compare against an external-detector verifier that accepts a claim only if Grounding DINO finds the object. This is the check HALC builds on internally.

Why not just…?

Why not just ensemble more random views, as RITUAL does? Averaging over views helps when the error is view-specific. Prior-driven hallucinations are not view-specific: the kitchen prior produces "refrigerator" in every view. More views should therefore mostly improve calibration on real objects and do little against confident prior hallucinations on POPE's popular and adversarial splits. The matched-$R$ comparison with RITUAL, broken out by split, tests this directly. If RITUAL with many views closes the adversarial-split gap, the authentication idea adds nothing beyond compute.

Why not just use self-consistency or answer invariance under transforms? This is the ablation described above. A stable answer is exactly what a strong prior produces, so invariance should flag real-but-uncertain objects more than hallucinated-but-confident ones. If the ablation performs as well as equivariance scoring, we drop the claim.

Why not just verify each claim with an external open-vocabulary detector? This is a real threat. For COCO objects, a strong detector may simply win on POPE, and we say so in advance. The case for our method is that it needs no second model and no fallback when the detector misses; the file notes HALC reverts to random sampling in that case. Our check also tests the generating model's own evidence rather than a third party's. We will report both methods and their combination. If the detector dominates everywhere, including on MME position questions, the contribution narrows to a diagnostic of why a model hallucinated (blind versus anchored forgery) rather than a better filter.

Proposed Experiments

The decisive first experiment runs on POPE's three splits with LLaVA-1.5. It plots the pass rate of true objects and of hallucinated objects against $R \in \{1,2,4,8\}$ for each key family. The method is justified if hallucinated-object pass rates fall roughly geometrically while true-object pass rates stay high, and if equivariance scoring clearly beats the matched invariance ablation on the adversarial split. If hallucinated objects pass at rates far above $C^{-R}$ on the adversarial split under rigid keys, but not under quadrant permutations, that confirms the anchored-forgery account and fixes the recommended key space.

The second experiment is CHAIR and object recall on COCO captions, with comparisons to HALC, VASparse, SHARP, and VCD, and with wall-clock time per image reported for every method.

The third uses the MME existence, count, and position subsets to test whether authentication extends past existence to spatial claims. Position is the natural target for an equivariance check. Color is not, and we expect no gain there. The file notes that POPE ignores attributes, and we do not claim to fix that.

The last experiment tests the length hypothesis. It checks whether objects introduced late in long captions fail authentication disproportionately, and whether those failures look like blind forgeries, which would connect the method to the context-drift mechanism reported in the HalTrapper paper.