Training Away Prompt-Fragile Hallucinations in Vision-Language Models

Vision-language models describe images in fluent text. They also hallucinate: they claim objects that are not in the picture, usually because the language model inside them "knows" that kitchens have refrigerators and tables have knives. A second, less discussed problem compounds the first. Ask the same yes/no question with a slightly reworded instruction and the answer often flips. Text-only work has shown that these two failures are tangled together. Many apparent hallucinations are really prompt-sensitive coin flips that occur where the model's next-token distribution is flat, rather than stable false beliefs. The fixes that work best for multimodal hallucination today run at test time. They decode twice with a perturbed image, contrast against an image-free pass, or inject steering vectors into hidden states, and so they add latency to every query.

We propose to move that work into training and then delete it. During fine-tuning, each image–question pair is shown under several rephrased instructions, plus one cheap pass with the image removed. These extra passes build a soft training target with two properties. It reflects what the model believes about the image across phrasings, and it is deliberately discounted wherever the image-free language prior is doing the talking. The model learns to put that distribution into a single ordinary forward pass. At deployment nothing extra runs. The hypothesis is that one training signal can reduce both co-occurrence hallucination and instruction brittleness, at zero inference cost, and do better than simply augmenting the data with paraphrased prompts.

Core Claim

Standard label smoothing mixes a small uniform mass into the one-hot target. It is the wrong regularizer for this problem. ATLaS shows that uniform smoothing hurts vision-language prompt tuning because it pulls the model away from sharp pretrained representations. LORAS shows it damages calibration on near-deterministic tokens. The usual repair spreads smoothing mass toward semantically similar classes, and for hallucination that repair backfires. "Knife" is semantically close to "fork," and smoothing toward it teaches exactly the co-occurrence the model should unlearn.

Our claim is that the smoothing distribution for a vision-language model should come from two sources computed only during training. The first is consensus across instruction variants, which addresses the flat, low-confidence regions that ProSA identifies as the source of prompt sensitivity. The second is a penalty against the image-ablated prior, which addresses the co-occurrence bias that POPE exposes through its adversarial split. Concretely, for an answer token position with gold token $y$, the training target is

$$q = (1-\alpha)\,e_y + \alpha\,\mathrm{norm}\!\left(\frac{\prod_{k=1}^{K} p_\theta(\cdot \mid v, x_k)^{(1+\beta)/K}}{p_\theta(\cdot \mid \varnothing, x_1)^{\beta}}\right).$$

The symbols are as follows. $e_y$ is the one-hot vector for the gold token. $p_\theta(\cdot\mid v, x_k)$ is the model's own stop-gradient next-token distribution given image $v$ and the $k$-th of $K$ rephrased instructions $x_k$. $p_\theta(\cdot\mid\varnothing, x_1)$ is the same distribution with the image tokens removed. $\beta \ge 0$ controls how strongly the language prior is discounted, $\alpha$ is the smoothing weight, and $\mathrm{norm}$ renormalizes to a probability vector.

The numerator is a geometric-mean consensus over phrasings. The ratio has the same form as visual contrastive decoding, but here it shapes a training target instead of being recomputed on every test query. All extra passes are discarded after training. Each instruction variant is also trained on with this target, so the scaffolding doubles as data.

Research Program

The method borrows from three prior lines and differs from each. LABO supplies the method's core: it proves that the optimal instance-level label regularizer is a temperature-scaled copy of the model's own predictions, and it costs 1.0–1.1× base training. We keep the self-derived target but change what it is derived from, aggregating across prompts and discounting the image-free prior, which LABO never considers.

BatchFormer and DAC-DETR show that relational computation used only during training can leave invariances in a backbone that survive its removal. Our cross-prompt grouping is the language-generation analogue. It is lighter, because it uses no added module, only grouped passes.

IC-VCO is the closest training-time hallucination method. It distills a multi-image contrastive preference signal into single-image inference. It needs image-editing pipelines and a preference (DPO) objective, and it does not address instruction brittleness. Our method needs no edited images and works on the token-level fine-tuning loss.

Two papers motivate the joint treatment. Rethinking Hallucinations shows that hallucination detectors track prompt consistency more than correctness, and it reports roughly 60% ambiguity for Llama2-13B-Chat on Med-HALT. That study is text-only. SHARP separates prior-driven hallucination from hallucination induced by misleading prompts inside vision-language models, but it intervenes only at test time.

The program has three stages.

Stage 1: diagnose. We extend prompt-multiplicity metrics to vision-language models, building POPE and AMBER question sets under the ProSA instruction families (plain, role-play, emotional framing, output-format requirements). For each base model we sort every instance into one of three groups: answered correctly regardless of prompt, wrong regardless of prompt, or prompt-sensitive. If prior-driven errors cluster in the "wrong regardless of prompt" group on POPE's adversarial split, and flat-confidence errors cluster in the prompt-sensitive group, the two-term target is justified.

Stage 2: train. We LoRA-fine-tune LLaVA-1.5-7B and LLaVA-NeXT-7B on their standard instruction mixture with the target above. We use $K=4$ phrasings drawn from a template pool, split by template family so that whole families are held out for evaluation. The image-free pass is cheap because it drops all visual tokens.

Stage 3: analyze. We test whether the learned model's image-free prior has shifted and whether the effect concentrates on co-occurring objects, by correlating per-object hallucination reduction with COCO co-occurrence statistics. ATLaS warns that soft targets can disrupt pretrained features, so we also test its alternating schedule (hard labels for $K-1$ epochs, soft on the $K$-th) as a fallback in case continuous smoothing hurts MME capability scores.

Baselines

The naive reference point is the base model fine-tuned on the same instruction data with ordinary hard-label cross-entropy for the same number of steps. Its POPE accuracy on LLaVA-1.5 is reported as 84.13% in RITUAL, and its CHAIR_s (the fraction of captions containing a hallucinated object) is reported as 52.8 in SHARP. We rerun this baseline ourselves so that every comparison shares a pipeline. The same run with uniform label smoothing, tuned over the same $\alpha$ grid, is the second reference, since it is the one-line alternative every reviewer will ask about.

The strongest training-free competitor is RITUAL, which decodes the image together with a randomly transformed copy and combines the two prediction streams. It reports 88.87% POPE accuracy on LLaVA-1.5 against 84.13% for the base model. We run the released method on our base checkpoints with its default settings and report wall-clock per sample. SHARP is the second test-time competitor. It reports 82.8% versus 79.5% POPE adversarial accuracy on LLaVA-NeXT, MME Perception rising from 510.0 to 613.3, and CHAIR_s falling from 52.8 to 34.8 on LLaVA-v1.5, at 7.04 s per sample against 13.97 s for visual contrastive decoding. Beating either one means matching or exceeding its accuracy at the base model's inference cost. Matching RITUAL's POPE gain at single-pass latency is the bar we would count as a win.

The strongest training-time competitor is IC-VCO. The file reports only relative gains for it: 1.19–1.24 points over its strongest baseline on LLaVA-NeXT-Interleave. No absolute POPE or CHAIR number for IC-VCO was found on Lacuna. We therefore compare on AMBER, where IC-VCO reports leading existence and attribute scores, and we rerun it if code permits.

LABO, applied to the vision-language loss with its closed-form self-target, isolates whether our gains come merely from self-distillation. We tune every method with the same budget of sweeps, the same LoRA rank, and the same token count. Prompt-augmentation baselines see exactly the same paraphrased instructions our method sees.

Two ablations isolate the key ingredient. Setting $\beta=0$ keeps cross-prompt consensus but removes prior discounting. Setting $K=1$ keeps prior discounting but removes consensus.

Why not just…?

Why not just augment the training data with paraphrased instructions and keep hard labels? This is the most serious threat, and it may well capture most of the brittleness gain. Hard labels teach the model which answer to give under each phrasing. They do not tell it how confident to be, and ProSA ties brittleness to flat confidence specifically. Hard labels also do nothing about the language prior, so we expect no gain on POPE's adversarial split beyond what more data provides. The matched-data augmentation baseline settles this directly. If it equals our method on held-out-template ambiguity and on adversarial POPE, the smoothing term is unnecessary and we would report that.

Why not just add a pairwise KL consistency loss between prompt variants? Consistency losses make the model agree with itself, and agreement includes consistently hallucinating. Rethinking Hallucinations found prompt-agnostic errors as a distinct category, and a consistency loss would leave those intact or harden them. The $\beta=0$ ablation is essentially this baseline in target form. We predict it improves self-consistency but leaves POPE adversarial accuracy and the "yes" rate, which POPE reports approaching 99% for some models, largely unchanged.

Why not just distill a test-time method such as RITUAL or contrastive decoding into the weights offline? This is a reasonable alternative and close to our prior-discounting term. It requires running the expensive decoder over the whole training set as a fixed teacher. That teacher is frozen to the base model's beliefs and to a single phrasing, whereas our target updates with the model and pools over phrasings. We include offline RITUAL distillation, using matched data, as an extra baseline. If it matches our method, the honest conclusion is that the cross-prompt component adds nothing for hallucination, and the contribution narrows to brittleness.

Why not just use uniform label smoothing? ATLaS directly shows that uniform smoothing harms vision-language prompt generalization, and LORAS shows it worsens calibration on structured tokens. More fundamentally, uniform mass on "knife" is the same as uniform mass on "giraffe." It cannot single out the co-occurring distractors that drive the adversarial-split drop, such as MiniGPT-4's F1 falling from 78.86 to 71.37 in POPE. Our uniform-smoothing run tests this empirically.

Proposed Experiments and Decisive Checks

We evaluate on POPE (random, popular, and adversarial splits; accuracy, F1, and yes-rate), on CHAIR for captions, on AMBER and HallusionBench, and on MME for capability retention. Every evaluation is run under held-out instruction families, and we report three instance-level quantities. Ambiguity is the fraction of instances whose answer changes across templates. Self-consistency is the probability that two random templates agree. The third is ProSA's PromptSensiScore. CHAIR's known instability is a confound: POPE reports a 3.22 template standard deviation for caption metrics against 0.78 for POPE. We therefore treat CHAIR results only as means over at least five caption prompts, with variance reported.

Three outcomes would decide the proposal. First, on held-out templates, our method should lower ambiguity relative to matched hard-label prompt augmentation while also improving adversarial POPE. A gain on only one of these would indicate that the two failures are separable after all. Second, the $\beta$ and $K$ ablations should show that prior discounting drives the adversarial-split and yes-rate improvement, while consensus drives the ambiguity improvement. If either term is inert, the joint target is not justified. Third, the deployed model must reach a POPE gain on LLaVA-1.5 comparable to RITUAL's reported 84.13% to 88.87% with identical single-pass latency to the base model, without lowering MME Perception. We will also report the training-time multiplier, expected to be roughly $K$ plus a small image-free overhead, so that the cost moved from inference into training is stated plainly.