Crediting the Image, Not the Distractor: Two-Probe Token Rewards for Multimodal Preference Training

Vision-language models describe things that are not in the picture, and one reliable trigger is the text around the question. If a prompt says "the sofa is green, but tell me what is on the table," the model is more likely to mention a sofa, or to bind "green" to the wrong object, even when the image contradicts both. Current fixes for hallucination train the model to rely on the image, but they decide which output words were supported by the image using only one test: blur the image and see which words become less likely. That test cannot say whether a word came from the picture or was planted by a distracting sentence in the prompt, because a planted word can still depend weakly on the image.

We propose a second, text-side test used during preference training. For each word in a training response, we measure how much its probability changes when the image is corrupted, and how much it changes when a known distractor span is deleted from the prompt. Words that the image supports and the distractor does not affect are rewarded. Words that exist mainly because of the distractor are penalized. Because we inject the distractors ourselves when building the training data, we know exactly which span to delete. The deployed model therefore needs no filter, no extra decoding passes, and no knowledge of where the noise is. The central question is empirical and falsifiable: does the distractor-deletion signal carry information about hallucinated words that image corruption alone misses?

Core Claim

Token-level preference optimization, meaning preference training where each output token receives its own weight instead of the whole response sharing one score, has already been applied to multimodal hallucination. Its credit signal, however, has only ever been conditioned on the image channel. TPO scores tokens by their log-probability drop under a corrupted image and reweights the preference loss with that score. TARS adversarially perturbs output tokens that look unrelated to the image. V-DPO contrasts image-conditioned and text-only likelihoods, but over the whole sequence. RLHF-V weights human-corrected segments. All four assume the text prompt is clean.

We claim that a token's causal source should be estimated on two axes, image and prompt distractor. Training on the joint signal should reduce hallucinations caused by distractors more than image-only token credit does when both are given the same distractor-augmented data. It should also leave clean-prompt hallucination rates and fluency no worse.

The two probes are defined as follows. Let $\pi_\theta$ be the policy, $v$ the image, $v_c$ a corrupted copy of the image, $x$ the prompt containing an injected distractor span $d$, $x_{\setminus d}$ the same prompt with $d$ deleted, and $y_{<t}$ the response prefix before token $y_t$. The two per-token probe scores, both computed by teacher forcing on the same response, are

$$s^{v}_t = \log \pi_\theta(y_t \mid v, x, y_{<t}) - \log \pi_\theta(y_t \mid v_c, x, y_{<t}), \qquad s^{d}_t = \log \pi_\theta(y_t \mid v, x, y_{<t}) - \log \pi_\theta(y_t \mid v, x_{\setminus d}, y_{<t}).$$

Here $s^v_t$ measures how much the image supports the token and $s^d_t$ measures how much the distractor produced it. The two scores are combined into a per-token weight, with a stop-gradient so the weight itself is not trained:

$$w_t = 1 + \alpha\,\sigma(s^{v}_t) \cdot [\text{chosen}] + \beta\,\sigma(s^{d}_t - s^{v}_t) \cdot [\text{rejected}].$$

In this expression, $\sigma$ is the logistic function, $\alpha$ and $\beta$ are scalar strengths, and the bracketed indicators select whether the token lies in the chosen or the rejected response. The weights multiply each token's log-ratio inside a token-decomposed DPO loss of the kind formalized by TDPO.

Function words have both scores near zero, so they keep a weight near 1. This matters for the concern raised against uniform sequence-level guidance in V-DPO: penalizing reliance on language priors everywhere also distorts grammar. Here the penalty lands only on the few tokens that are distractor-driven and weakly image-supported.

Research Program

The work has three parts.

Data. We build a distractor-augmented preference set on top of the RLHF-V images and responses (RLHF-V), restricted to images that have COCO object annotations. Object annotations let us verify absence automatically. For each image we inject one distractor sentence of one of three types:

  • Absent co-occurring object, sampled the way POPE's adversarial split samples negatives (POPE).
  • Present but irrelevant object, modeled on the "sofa" setting that cut OpenVLA's LIBERO-Goal success from 77.5% to 18.9% (Bring the Apple, Not the Sofa).
  • Attribute-conflicting mention, for example calling a blue cup red.

We then sample responses from the base model under the distracted prompt. Rejected responses are those that name an absent distractor object or a wrong attribute, which we check with AMBER-style noun and attribute extraction against annotations. Chosen responses are either grounded on-policy samples or the RLHF-V corrected response.

Diagnostic, run before any training. On a held-out set of a few hundred responses with token-level labels (grounded, distractor-induced hallucination, other hallucination, function word), we measure how well each score separates distractor-induced hallucinated tokens from grounded content tokens, using AUROC (area under the ROC curve). We compare $s^v$ alone, $s^d$ alone, and the pair. We also include the forward attention-influence score from Attention Illuminates LLM Reasoning as a comparison.

That paper rescales RL advantages by how much later tokens attend back to a token. The research file notes that this measures structural influence rather than truth. HalTrapper shows that hallucinated objects attract self-reinforcing attention, so we expect the attention score to rank some hallucinated tokens highly. This diagnostic decides whether the project proceeds. If the pair does not beat $s^v$ alone by a clear margin (we set 0.05 AUROC in advance), then image-only credit with augmented data is sufficient, and we report that instead.

Training and evaluation. We train LLaVA-1.5, the backbone used by TPO, TARS and V-DPO, with the two-probe weighted loss. We then evaluate on clean and distracted versions of the standard hallucination benchmarks. The extra cost is two teacher-forced forward passes per training pair, which is the same order as TPO's single image probe plus one more. Inference cost is unchanged.

Baselines

Unmodified base model. The naive reference point is LLaVA-1.5 with no additional training, evaluated with the plain prompt. It fixes how much distractors hurt before any intervention.

Negation instruction. A second naive point adds an instruction to the prompt: "answer only from the image; ignore objects mentioned in the text if they are not visible." This is in the spirit of the escape instruction in DAS, which reported a 64% average relative Exact Match gain in text-only open-domain QA. We add no training and give it the same evaluation prompts.

TPO trained on our distractor-augmented data. This is the strongest competitor, because it isolates exactly our key ingredient. It is the same kind of token-weighted DPO, but its weights come only from the image-corruption probe. We run it on identical preference pairs with the same backbone, the same number of epochs, and an equal hyperparameter sweep (eight configurations each over the weight strength and learning rate), selected on the same validation split.

For reference, TPO trained on standard RLHF-V data reports a 20.4% absolute F1 improvement on AMBER over base LLaVA-1.5 (TPO). TARS reports an AMBER hallucination rate reduced from 26.4% to 13.2% (TARS). No reported result was found on Lacuna for any method on distractor-injected versions of AMBER or POPE, so the distracted comparisons rest entirely on our reruns.

Beating TPO means a lower distractor-induced hallucination rate on distracted AMBER and distracted POPE-adversarial, with non-overlapping 95% bootstrap intervals over three seeds, and clean-prompt AMBER F1 within one point of augmented TPO.

Other trained and inference-time comparisons. We include:

  • Sequence-level DPO on the same augmented pairs.
  • V-DPO on the same pairs; it reports a 2.2-point CHAIR improvement on AMBER and a 9.01% qAcc gain on HallusionBench (V-DPO).
  • A text-only 3B LLM pre-filter in front of the base model, following Bring the Apple, Not the Sofa, where it restored VLA performance to about 98.5% of clean.
  • HalTrapper's contrastive contextual decoding at inference (HalTrapper).

Ablations. These use our loss with four alternative weightings: $s^d$ only, uniform weights, the attention-influence weighting, and a random-span deletion in place of the true distractor. The random-span control checks that the benefit comes from knowing which span is the distractor rather than from generic prompt perturbation.

Why not just…?

Why not just run TPO on distractor-augmented data? This is the main threat, and it may well suffice. A distractor-primed word such as "sofa" in a living-room image can still lose probability when the image is blurred, because the image makes a sofa plausible. If that happens, image-only credit rewards the word in chosen responses, or fails to single it out in rejected ones. Whether this confusion is frequent is exactly what the pre-training AUROC diagnostic measures, and the matched TPO rerun settles it at the level of benchmark outcomes. If augmented TPO matches us, the honest conclusion is that the data, not the credit signal, is what matters, and we will report it that way.

Why not just filter the prompt with a small language model? The pre-filter works when noise is recognizable from text alone. That is how Bring the Apple, Not the Sofa recovered 98.5% of clean performance. It cannot tell whether a mentioned object is present or relevant without seeing the image. It also adds a model call and can itself hallucinate. We will split the distracted test set into cases a text filter can resolve (pure fluff) and cases that need the image (a present-versus-absent object, or a conflicting attribute). We expect the filter to be competitive on the first split and weak on the second, and we report both splits.

Why not just use sequence-level DPO on hard negatives? With whole-response credit, the grounded parts of a rejected response are pushed down along with the planted word. Prior work reports the resulting failure modes: TDPO reports entropy falling to 3.196 under DPO versus 4.915 under TDPO2 on Anthropic HH, and TARS reports overfitting to surface patterns. The matched sequence-level DPO run, with response length and diversity reported alongside hallucination rates, tests whether this matters in our setting.

Why not just use contrastive decoding at test time? Contrastive decoding needs a second conditioning pass at every generated token. At deployment it also does not know which prompt span is the distractor, so it can only contrast against the full text or no text. That is the same limitation V-DPO raises about inference-time guidance cost. Our method pays the counterfactual cost once, during training, when the span is known. HalTrapper's decoding is included as a baseline, and we report its wall-clock decoding overhead next to its hallucination numbers.

Proposed Experiments

Benchmarks. The primary evaluation uses AMBER (AMBER) generative and discriminative splits and POPE random, popular and adversarial splits. Each is run twice: once as released and once with a distractor sentence injected into every prompt from a held-out pool whose objects and templates never appear in training. The headline metric is the distractor-induced hallucination rate, defined as the fraction of responses that name an absent object mentioned in the distractor or assign the distractor's attribute. We also report CHAIR, AMBER F1, and POPE's "yes" rate, since the "yes" bias can reach nearly 99% (POPE).

HallusionBench (HallusionBench) tests whether gains transfer beyond object mentions. MMBench and SEED Bench check that general capability does not regress.

Self-generated context. HalTrapper argues that the model's own earlier text acts as a distractor. We therefore report hallucination rate against response length on long-description prompts, to see whether training against external distractors also delays self-induced hallucinations.

Decisive checks. The first is the token-level AUROC diagnostic, which gates the project. The second is the head-to-head against augmented TPO on the image-dependent split of the distracted benchmarks, where the two-probe signal should matter most. The third is the random-span ablation, which should lose most of the gain if the true-distractor counterfactual is what does the work.

Exploratory transfer. An optional check fine-tunes an action model with the same loss and evaluates on LIBERO-Goal under the "sofa" distractors. It is explicitly exploratory and does not bear on the main claim.