Pointing at Patches: A Frozen-Backbone Copy Head for Grounded Vision-Language Decoding

Vision-language models describe images fluently but often name things that are not there, or attach the wrong color to the right object. One cause is well documented. These models are built on text-trained language models, and their strong habits about which words co-occur ("bathroom" suggests "sink") can override weak evidence from the image. A second finding sets up this proposal. The image regions the model reads (patches) often already "know" the right word in the model's middle layers, and the decoder simply fails to say it. We propose to give the decoder an explicit way to copy that word. At each step, the decoder looks up which image patches it is currently attending to. It reads off the words those patches encode, and mixes that "copy" distribution with its ordinary next-word distribution through a small learned switch. This is the pointer-generator idea from text summarization, a decoder that blends generating from its vocabulary with copying from its input. Here it is extended so the decoder can point at image regions instead of source words.

The second ingredient is modular parameter isolation, meaning new capability lives in a separate small module while the original network stays untouched. It acts as a design constraint, not as decoration. The entire vision-language model is frozen. The copy switch and its readout form a module of roughly ten thousand parameters that sits after the output layer. It is initialized so the system reproduces the base model exactly. Whatever the module learns, it cannot rewrite the language model's grammar; it can only redirect probability mass on the steps where it opens. Because the copy distribution comes from the specific region being attended rather than from a bag of objects for the whole image, the method targets the failure that current fixes leave open: binding attributes to the right object.

Core Claim

A learned, attention-localized copy distribution over image patches will reduce object hallucination about as much as the best training-free inventory method. It will also reduce attribute hallucination substantially more, because it localizes. Keeping the copy head in an isolated module rather than spreading the same parameter budget into the backbone will preserve caption fluency and recall better than an equally sized fine-tune.

Concretely, at decoding step $t$ the output distribution is

$$P_t(w) = g_t\, P^{\text{vocab}}_t(w) + (1-g_t) \sum_{j \in \text{patches}} \tilde a_{t,j}\, q_j(w),$$

where $P^{\text{vocab}}_t$ is the frozen model's ordinary next-token distribution. Here $\tilde a_{t,j}$ is the attention from the current token to image patch $j$, renormalized over patches only and averaged over a fixed set of heads. The term $q_j(w)$ is the vocabulary distribution obtained by projecting patch $j$'s hidden state at an intermediate layer through the model's own output head (the "logit lens"), sharpened by a learned temperature. Finally, $g_t \in [0,1]$ is a sigmoid gate computed from the decoder state and the attended visual context.

In a decoder-only model, patch hidden states do not change after the image is first processed, because each position attends only to earlier positions. So every $q_j$ is computed once per image, and the method needs a single forward pass per token.

Relation to Prior Work

This proposal draws on four strands of prior work, and each plays a different role.

Supplying the method. The mixture and the gate come from Get To The Point, which blends vocabulary generation with copying from source text through a learned switch. The idea of building the copy head almost for free, by reusing attention the model already computes, comes from FiD-PGN in A Copy-Augmented Generative Model for Open-Domain Question Answering. FiD-PGN adds 1,537 parameters to a 220M text model. Both methods copy only discrete text tokens whose surface form is already a word. Image patches have no surface form, and the logit-lens readout $q_j$ is what gives them one.

Closest prior work. LargePiG shows that frozen LLMs and LLaVA models contain an implicit pointer-generator. It copies from attention to input text tokens, using a heuristic gate based on the divergence between layers. It never points at visual tokens, and its gate is not learned. PatchGate provides the motivating evidence that patches encode correct object words in later decoder layers. However, it collapses this evidence into a single image-level inventory of single-token WordNet nouns, then adds fixed bonuses and penalties to matching tokens. Our method differs from PatchGate in three ways. It is step-wise and attention-weighted rather than image-global, so "blue" is copied from the patch the model is currently attending to rather than from anywhere in the image. It uses the full vocabulary rather than a noun list. And it replaces fixed bonuses with a learned soft gate.

Constraining the design. Summary-Guided Decoding shows that interventions should touch only image-related words and leave function words to the language prior, or fluency suffers. Our gate is meant to learn this behavior rather than take it from a part-of-speech tagger, and we test that directly. Why LVLMs Are More Prone to Hallucinations in Longer Responses and AMBER document that attention drifts away from the image as captions grow. This is a direct threat to any method that reads visual attention, and our renormalization over patches is the specific response we test.

Motivating isolation. Residual adapters and L2R motivate isolating the new capability. Residual adapters initialize to an exact identity and beat full fine-tuning on Visual Decathlon (2643 vs 2500). L2R trains modules in isolation to avoid interference. We adopt their identity-at-initialization, frozen-backbone discipline. We deliberately do not adopt routing across many modules, because the hallucination benchmarks in hand are almost all natural-image benchmarks with no domain structure to route over.

Research Program

We build the copy head for LLaVA-v1.5-7B, the model every strong baseline in the file reports on, and port it to LLaVA-Next to check transfer.

The first stage is diagnostic and needs no training. We measure whether the attention-weighted lens distribution $\sum_j \tilde a_{t,j} q_j$ ranks the correct word above the hallucinated word at the steps where the frozen model errs. We do this for objects and, separately, for attributes, using AMBER's attribute annotations. We also choose the readout layer and attention heads on a held-out split. If localized readout does not beat PatchGate's global inventory on attribute words in this offline test, the attribute claim is dead before any training, and we will report that.

The second stage trains only the gate, the lens temperature, and one scalar layer-mixing weight, keeping every backbone weight frozen. Training uses reference captions from COCO train. The loss is the likelihood of the reference caption under the mixture. We add an unlikelihood term on object tokens in the model's own sampled captions that COCO segmentation labels mark as absent. These labels come from the train split, which is disjoint from CHAIR's evaluation images.

The unlikelihood term matters. Pure caption likelihood may teach the gate to stay closed, because the frozen vocabulary distribution already fits references well. Because both terms use COCO's 80-class space, we treat AMBER, with its 337 objects and Unsplash images, as the real test of whether the gate generalizes beyond the training classes. The gate's bias is initialized so that $g_t \approx 1$ and the system starts as the unmodified model.

The third stage is analysis. We record where the gate opens, broken down by part of speech and by position in the caption. This tells us whether the learned gate rediscovers the noun-and-adjective selectivity that SGD hand-codes, and whether copying still helps late in long captions, where visual attention has drifted.

Baselines

The naive reference point is the frozen LLaVA-v1.5-7B with greedy decoding under our exact prompts and caption-length settings. Every other method is reported as a change from it.

The strongest existing competitor is PatchGate. It uses the same internal evidence we use (patch states read through the logit lens) and it is training-free, so any gain we show must come from localization and learning, not from the evidence source. PatchGate reports, for LLaVA-v1.5-7B, a POPE F1 of 89.6 and a CHAIR reduction from 7.5 to 6.6. It also reports an AMBER object-coverage increase from 49.4 to 56.0 (PatchGate). We will rerun PatchGate in our evaluation harness and tune its bonus and penalty on the same validation split we use to choose our layer and heads. Beating it means matching its POPE F1 and CHAIR within noise across three seeds of caption sampling, while improving AMBER attribute and relation scores with non-overlapping confidence intervals.

SHARP (SHARP) is the strongest internal-steering competitor. It reports sentence-level CHAIR on LLaVA-v1.5 falling from 52.8 to 34.8, and adversarial POPE accuracy on LLaVA-Next of 82.8 against 79.5 for sampling. Its CHAIR baseline differs from PatchGate's by a factor of seven, which can only reflect different caption lengths or protocols. This is exactly why we will rerun all methods under one protocol rather than compare published numbers.

We also run Summary-Guided Decoding, which reports a 16.5% CHAIRs improvement for LLaVA 1.5 (SGD), and RITUAL (RITUAL) as the multi-pass representative, timing both. LargePiG is evaluated by applying its divergence-based gate to our patch copy distribution. For LargePiG there is no reported result found on Lacuna on POPE, CHAIR, or AMBER; it reports only truthfulness gains over DoLa and about 6% latency on LLaMA2-7B.

Three ablations isolate the proposal's ingredients. The first replaces attention-weighted pointing with a uniform average over patches, which makes our method a learned-gate version of PatchGate's global inventory and tests localization. The second replaces the learned gate with LargePiG's divergence gate and, separately, with SGD's part-of-speech rule, which tests learning. The third puts the same parameter budget and the same training data into LoRA on the backbone's top layers instead of the isolated head, with an identical tuning budget, which tests isolation.

Why not just…?

Why not just use PatchGate and tune its bonuses? For object existence this may be nearly as good, and we expect to roughly tie on POPE and CHAIR. The threat is real there. But PatchGate's inventory has no notion of which region a word belongs to, so it cannot tell "red cup, blue plate" from the reverse. The uniform-pointing ablation settles the question. If it matches full pointing on AMBER attributes, localization adds nothing and the method reduces to a learned PatchGate.

Why not just LoRA-fine-tune the model on the same anti-hallucination data? This is the most serious alternative, because a small fine-tune might suppress the "Yes" bias that POPE documents just as well. Two pieces of evidence point the other way. The residual-adapter result shows a frozen backbone regularizes better than full fine-tuning. POPE shows that fine-tuned models still degrade from random to adversarial sampling (MiniGPT-4 F1 drops from 78.86 to 71.37). The equal-budget LoRA ablation decides the matter on three measures: hallucination, object recall, and caption perplexity under a separate reference language model. If LoRA wins on all three, we will report that isolation was unnecessary.

Why not just gate by part of speech, as SGD does? A tagger is free and already encodes the fluency lesson. But it fires on every noun, including nouns in hedges and discourse, and it cannot open the gate partway. The gate-swap ablation measures whether learning buys anything beyond the rule. If it does not, the simpler rule should be preferred.

Why not just use contrastive or multi-view decoding? Contrastive decoding costs about twice the time per sample of single-pass intervention (13.97 s vs 7.04 s for VCD on LLaVA-Next, per SHARP), and RITUAL doubles or triples the number of passes. Our head adds one precomputed matrix product per step. We report wall-clock time alongside accuracy so the trade-off is explicit, not assumed.

Proposed Experiments and Decisive Checks

All methods run on LLaVA-v1.5-7B, with LLaVA-Next as the transfer model, using identical prompts, a fixed maximum caption length, and the same 500-image CHAIR sample. We report CHAIR at both sentence and instance level, POPE F1 and accuracy in the random, popular, and adversarial splits, and AMBER generative and discriminative scores broken down into existence, attribute, and relation. We also report the MME perception subtotal and LLaVA-Bench for general capability. Fluency is measured by perplexity under a separate reference language model, and cost by latency per token and memory.

The first decisive check is the AMBER attribute and relation breakdown against PatchGate and the uniform-pointing ablation. This is where the core claim lives. A null result there falsifies the localization hypothesis regardless of how the object metrics look.

The second check plots hallucination rate against token position in long captions, with and without the head. It tests whether renormalized patch attention still carries usable evidence after visual attention has drifted, or whether the copy distribution flattens and the gate learns to stay closed late in the caption.

The third check is the isolation test. With the gate forced to one, outputs must match the base model token for token, and MME and LLaVA-Bench scores must not fall below base when the head is active. The equal-budget LoRA ablation is expected to show a measurable drop on these benchmarks if the isolation argument is right.