Grounding Captions by Editing Them: Distantly Supervised Edit Tags Fused with Image Tokens

Vision-language models describe images fluently, but they often mention objects, colors, or spatial relations that are not in the picture. This failure is called multimodal hallucination. Most current fixes adjust the model while it writes: they shift its internal activations or reweight its next-word probabilities. We propose a different division of labor. A standard model writes a draft. A second, small pass then reads the image together with the draft and assigns each draft word one of a few explicit edit operations: keep it, delete it, or replace it with a word from a short, image-derived candidate list. The draft words and their candidate edit markers enter the network at the very first layer, alongside the image patches. Every word is therefore checked against the picture from the start, not after the language model has committed to it. The training labels for these edits come from distant supervision, meaning labels built automatically by aligning drafts with independent, noisy evidence such as agreement among several captioners and an object detector, with no human annotation.

The approach rests on two observations. First, code-change research shows that models reason much better about a change when it is presented as aligned operations rather than as raw before/after text. MergeBERT also showed that restricting a model's output to a small set of edit classes stops it from inventing content (Program merge conflict resolution via neural transformers). Second, a model that can only keep, delete, or choose from a constrained list can remove hallucinations but has very limited room to create new ones. If this works, it yields a cheap, auditable corrector whose every change can be inspected. It also gives a direct test of a puzzle in the literature: whether hallucination is only detectable in a model's middle layers, or whether that is an artifact of where generators happen to be probed.

Core Claim

Hallucination correction framed as per-token edit classification over a draft, with edit markers fused early with image tokens and trained on automatically generated edit labels, should reduce caption hallucination about as much as the best latent-steering methods. It should do so while introducing near-zero new hallucinations, adding one parallel forward pass rather than a second autoregressive decode, and improving most on attributes and relations, where existing methods are weakest.

Three findings in the literature motivate this. Hallucinated and faithful states inside the generator are linearly separable, with separation peaking in layers 10–19 (SHARP; HIRE; AFTER). Attention drifts away from the image as responses grow longer (Why LVLMs Are More Prone to Hallucinations in Longer Responses). Relation and attribute errors remain harder than existence errors (AMBER).

The edit pass sidesteps the drift problem because it reads the whole draft at once with the image present. It is not a continuation of the draft. The middle-layer finding is a claim about generators. We will test whether a verifier that receives explicit edit markers at the input separates grounded from ungrounded words earlier.

Relation to Prior Work

The closest work edits internal states rather than text. HIRE trains a small autoencoder to move intermediate hidden features along a "hallucination-reduction" direction, with a first-layer router choosing which tokens to edit. AFTER computes a steering vector from the difference between activations on an image and on factual text derived from COCO annotations. SHARP extracts cause-specific steering vectors from layer 10.

All three treat an "edit" as a continuous vector shift that no one can read, and all three rely on human-verified annotations or contrastive and preference pairs. Our edits are discrete, readable operations on output words, and our labels are built automatically from unannotated images.

A second group intervenes at decoding or after it. Summary-Guided Decoding re-summarizes the context for image-related nouns and adjectives (SGD). RITUAL averages over random image transformations at test time (RITUAL). SimProb re-ranks beam-search candidates using detector evidence (SimProb). SimProb is the nearest structural cousin, since it also corrects after the fact. However, it can only choose among whole candidate captions, fails when every candidate in the beam is wrong, and fuses visual evidence only at the final scoring step. We correct individual words inside a single draft and fuse the image from layer one.

The method borrows its representation from code-change work, which supplies the technique. Explicit keep/delete/insert tagging (Lu et al.; Lin) and token-level edit-aware embeddings (MergeBERT; CCRep) outperform raw diffs. Language models reading raw changes zero-shot are "change-blind" (Exploring the Capabilities of LLMs for Code Change Related Tasks). All of this work is text-only.

Distant supervision supplies the data recipe. WikiAsp and WikiCite turn Wikipedia structure into hundreds of thousands of training pairs (WikiAsp; Summary Cloze). MergeBERT mines git history. WikiAsp also warns that such labels are noisy and can themselves induce hallucination. That constraint shapes our labeling design: we label only when independent sources agree, and we measure label noise directly.

Fusion work explains why the edit markers enter early rather than through late concatenation. In mDBN, naive concatenation of modalities did worse than a single modality, while deeper joint layers helped (Multimodal DBN). Chameleon shows fully early-fused token streams are trainable at scale (Chameleon).

Research Program

Edit representation. A base model (LLaVA-1.5, matching the baselines' reported setting) writes a draft caption. We identify "content slots" in the draft: nouns, adjectives, and relation phrases, found with a part-of-speech tagger, the same grammatical categories SGD targets. Each slot receives a candidate set made of the slot's own word plus a few alternatives. The alternatives come from the image side, namely detector labels and attribute and relation terms for detected regions.

The editor reads three inputs in one sequence: the image patch tokens, the draft tokens, and a learned edit-role embedding added to each draft token. The edit-role embedding marks whether a token is a content slot and of which kind. It is added at the input layer, which is what makes this early fusion. For each slot the editor outputs KEEP, DELETE, or REPLACE(candidate). Non-slot tokens are always kept, and a light grammatical cleanup step repairs articles and conjunctions after deletions.

Free-form insertion is deliberately excluded. This limits how much the editor can add back, so object recall must be reported alongside hallucination rates. The editor is the base model's own backbone with LoRA adapters (low-rank weight updates trained while the original weights stay frozen) and a classification head over slots. It runs as a single non-autoregressive forward pass.

Distantly supervised labels. We take an unannotated image pool that excludes all COCO validation images, since CHAIR and POPE are built on those images. For each image we generate drafts from the base model at several sampling temperatures, which produces realistic hallucinations rather than injected ones. We then collect evidence from independent sources: captions from two other captioning models and an open-vocabulary detector.

Following the alignment hypothesis of distant supervision, a slot is labeled KEEP if at least two sources confirm it. It is labeled DELETE if no source mentions it or any synonym and the detector's score is below threshold. It is labeled REPLACE when sources agree on a different term at the same aligned position. Slots with ambiguous evidence are left unlabeled rather than guessed.

We measure label precision against COCO annotations on a held-out audit set of COCO training images. This number bounds what the editor can learn. As a secondary, exploratory source, we will mine naturally occurring caption revision histories, such as encyclopedia image-caption edits, and align them into the same edit tuples. This tests whether human revisions carry grounding signal or mostly stylistic change.

Analysis. Using linear probes, we compare where in the editor's layer stack grounded and ungrounded slots become separable, against the same probe on the generator. We repeat the comparison with edit-role embeddings injected at layer 0, at layer 10, or only at the output head. This directly confronts the middle-layer finding of HIRE, SHARP, and AFTER.

Baselines

The naive reference is the unmodified LLaVA-1.5 decode. The file reports CHAIRs of 52.8 for LLaVA-v1.5 (SHARP) and POPE accuracy of 84.13% (RITUAL). A trivial lower bound is "delete every object the detector does not see," which tests whether the learned editor adds anything beyond its own evidence source.

The strongest existing competitors are the latent-steering methods, because they target the same failure at low cost in a single pass. SHARP reports reducing CHAIRs on LLaVA-v1.5 from 52.8 to 34.8, and POPE adversarial accuracy on LLaVA-Next of 82.8% versus 79.5%. HIRE reports roughly 40% fewer sentence-level and 50% fewer instance-level CHAIR hallucinations on LLaVA-1.5 at 11.81 versus 10.23 TFLOPs. HIRE also claims top AMBER scores, but the file gives no numeric value, so no reported result was found on Lacuna for HIRE's AMBER score. AFTER reports up to a 16.3% hallucination reduction on AMBER. RITUAL reports POPE accuracy of 88.87% on LLaVA-1.5.

We will rerun SHARP and HIRE from their released settings on the same base checkpoint, the same evaluation images, and the same caption prompt and length cap. Where reruns are impossible, we compare to the reported numbers and flag the comparison as unmatched.

On captions, beating the field means CHAIRs below 34.8 on LLaVA-v1.5 at object recall no worse than SHARP's. Lower hallucination achieved simply by deleting content does not count.

SimProb-style re-ranking is run over the same top-20 beam with the same detector our labeler uses. This isolates "choose a whole caption" from "edit individual words."

Two ablations isolate the key ingredients. The first is a free-form rewriter: the same backbone, LoRA budget, and distant labels, but trained to regenerate the corrected caption as text instead of emitting edit classes. The second is a late-fusion editor, where draft and image representations are pooled separately and concatenated before the slot head. All learned variants get the same number of training steps and the same hyperparameter search budget.

Why not just…?

Why not just fine-tune the generator on the corrected captions? This is the most serious alternative. Training the generator with LoRA, or with DPO (direct preference optimization, the preference training HIRE uses), on our distantly labeled corrections might yield most of the gain without a second pass. Two things work against it. A generator fine-tuned this way still writes long responses autoregressively and remains exposed to attention drift (Why LVLMs Are More Prone to Hallucinations in Longer Responses). It also gives no inspectable record of what was changed. We will train this arm on identical labels and compare. If it matches the editor on CHAIR and AMBER at equal recall, the case for the separate editor rests only on auditability, and we will say so.

Why not just inject synthetic hallucinations into clean captions? Inserting plausible but absent objects into reference captions is cheaper than mining drafts and consensus evidence. Its errors, however, do not follow the base model's own co-occurrence biases, which are exactly what POPE's adversarial split exposes (POPE). We run this as a data-source arm with matched training size. Our prediction is a gap that is largest on POPE-adversarial and AMBER relations. If no gap appears, the distant-supervision pipeline is unnecessary and the simpler data should be used.

Why not just prompt a strong model to fix the caption, or use contrastive decoding? Prompted rewriting inherits change-blindness, and in-context gains plateau around 16 demonstrations (Commit Message Generation via ICL). Evaluators built from vision-language models themselves hallucinate (Chen et al.). Contrastive decoding roughly doubles compute: SHARP reports 13.97 s per sample for VCD versus 7.04 s for SHARP on LLaVA-Next. SGD also reports that it degrades fluency. We include a prompted-rewrite arm on the same backbone and report FLOPs for every method, so the cost argument is measured rather than assumed.

Proposed Experiments

The main evaluation uses CHAIRs and CHAIR1 together with object recall on COCO captions, and AMBER's generative existence, attribute, and relation scores. Both are computed deterministically, with no model acting as judge.

For POPE we use the editor as a presence probe. The input is a fixed template draft, "There is a {object} in the image," and the editor's KEEP probability serves as the yes/no score. This shows whether the editor also corrects the systematic "yes" bias. Because this is a different use of the model, we report it separately from generator-based POPE numbers.

Fluency is checked on MME perception (SHARP reports 510.0 to 613.3) and with a grammaticality check on edited captions. Cost is reported in TFLOPs against HIRE's 11.81.

Four checks decide the outcome. First, a hallucination-introduction rate: the fraction of edited captions that contain an ungrounded object absent from the draft. We expect this to be near zero for the edit-class editor and clearly nonzero for the free-form rewriter on identical data. If it is not, the structural restriction is not doing its job.

Second, the early-versus-late fusion comparison should show gains concentrated on AMBER attributes and relations. If late fusion matches early fusion, the fusion claim fails.

Third, the layer-probe study should show the editor separating grounded from ungrounded slots before layer 10 when edit-role markers are injected at the input. If separation still waits until the middle layers, the middle-layer finding of HIRE, SHARP, and AFTER is confirmed as a general property of these networks rather than an artifact of probing generators.

Fourth, a label-noise sweep, made by corrupting a controlled fraction of distant labels, will show how editor quality degrades. Together with the measured precision on the audit set, this tells us whether the WikiAsp-style noise problem is the binding constraint.