News and encyclopedia captions name people, places and organizations that cannot be read off the pixels. They have to come from the surrounding article. Models built for this task usually include a copy mechanism, a decoder component that can lift a word straight out of the source text instead of generating it from its own vocabulary. The weakness is that the decision to copy a name is made by reading text alone. The model will happily write "Chancellor Merkel speaks to reporters" because Merkel dominates the article, even when the photo shows an empty podium or three unidentified men.
We propose to make every copied name pass through the image first. The captioner writes a visual description phrase ("the woman at the lectern") and then emits a segmentation token, a special vocabulary item whose hidden state is decoded into a pixel mask. The mask, its quality score, and the visual features pooled inside it are fed back into the decoder. Only after that feedback does a span-level pointer choose which article entity, if any, to attach to that region. The model can also decline to name anyone and keep the generic phrase.
Core Claim
Grounding cannot verify identity. A mask decoder does not know what Angela Merkel looks like. What grounding can verify is anchoring: whether a region of the right type, count and salience exists for a name to attach to. Our claim is that conditioning the copy decision on the mask the model just produced cuts "named but not shown" errors, the copying shortcut documented by Show, Interpret and Tell, without losing fluency or entity recall on ordinary captions. We also claim this works better than either training on counterfactual data alone or filtering names after the fact.
The novel part is the direction of information flow. In pixel-grounding models such as LISA and GLaMM, the segmentation token triggers a mask and nothing flows back into the text. In copy-based captioners, the decision to copy never sees space.
Relation to Prior Work
Three papers supply the motivation, and each stops short of spatial verification.
The Visual News captioner filters article paragraphs by visual relevance and copies entities with a multi-head pointer-generator, but the pointer attends only over the 1D article sequence and produces no grounding. Good News, Everyone! fills type placeholders from the article post hoc, using a global image vector. Show, Interpret and Tell diagnoses the shortcut and fights it with masked-entity pretraining. That method changes what the model learns but gives it no explicit check at decoding time. We keep their diagnosis and replace the implicit fix with an architectural one, and we test the two head to head.
The method comes from two other lines of work. GLaMM provides phrase-then-mask interleaved generation and a SAM-based decoder, but it takes no external document and its vocabulary is entirely parametric. The copy formulation of Get To The Point provides the gated mixture of generating and copying, whose gate sees only the decoder state, attention context and input token. We change both the gate's inputs and the unit being copied.
Two papers act as constraints. LargePiG shows that LLMs already copy implicitly through attention, so an explicit copy gate is not new in itself. Any gain must come from the spatial conditioning, and our ablations isolate exactly that. MMR shows that a single segmentation token collapses multiple instances. We therefore emit one token per named entity, so "Biden and Sunak" yields two masks and two pointer decisions.
The nearest structural precursor is Language-Driven Region Pointer Advancement. There, a NEXT token moves attention along pre-extracted boxes, without document copying or dense masks. Its ablation showing that token-synchronized region features beat average pooling is useful evidence for region-conditioned decoding.
Mechanism
We start from a GLaMM-style model. The article is reduced to its top-$k$ paragraphs by image–text similarity, in the spirit of Visual News' visual selection, and placed in the prompt. Named-entity spans in those paragraphs are pre-extracted. Each span $j$ gets one embedding $u_j$, the mean of its subword states. This avoids the pointer attention fragmenting across byte-pair pieces of a single name.
When the decoder emits a segmentation token for region $r$, the mask decoder returns three things: a mask, a predicted-IoU quality score $q_r$, and a feature $z_r$ pooled from the image features inside the mask. These are projected into one feedback token that the decoder reads before its next step. At the next step, $s_t$ is the decoder state. The model then picks a candidate $c$ from the article spans plus an abstain option $\varnothing$:
$$P(c \mid r) = \mathrm{softmax}_c\left(f(s_t, z_r, q_r)^\top u_c\right),\quad c \in \{u_1,\dots,u_M,u_\varnothing\}$$
Here $f$ is a small MLP and $M$ is the number of candidate spans. A gate $p_{\text{copy}} = \sigma(w^\top [s_t, z_r, q_r])$ mixes this span distribution with ordinary vocabulary generation, as in See et al.
Supervision
News images have no gold masks, so mask supervision is pseudo-labelled. We strip names from reference captions, replacing them with their entity type ("a person at the lectern"), and ground those phrases with off-the-shelf GLaMM. We keep only high-confidence masks.
Copy targets are free: the name strings in the reference caption match article spans.
Abstention targets come from counterfactual images. We crop or inpaint away the pseudo-masked region of a named person and relabel the caption with the generic phrase. We also pair images with same-topic articles from other stories. This counterfactual data is a strong ingredient on its own, so a key ablation trains a model without masks on the same data.
Baselines
The naive reference point is the same backbone given the selected article paragraphs in its prompt. It has no segmentation tokens and no feedback, and it copies only implicitly through attention, which LargePiG suggests is already substantial. It is fine-tuned on the same Visual News and GoodNews training data with the same compute and tuning budget. This tells us whether any structure is needed at all.
The strongest existing captioner is the Visual News captioner, which reports CIDEr 50.5 on Visual News with 93M parameters. We compare against that number on the same split and, if code permits, retrain it on our paragraph selection.
For GoodNews, the original Good News system reports about 12.7 CIDEr and 9.19% entity exact match. These are weak reference points, not targets.
For the grounding side, GLaMM reports 43.5 CIDEr and 65.6 mIoU on grounded conversation generation. We run it zero-shot on our annotated subset to measure how well an unmodified pixel-grounding model handles news imagery. The file has no reported entity-precision or grounding numbers for any news benchmark: no reported result found on Lacuna for entity grounding on Visual News or GoodNews.
Four controls isolate the key ingredient, all sharing one budget:
- Pipeline control. An MLLM captions with the article, then off-the-shelf GLaMM grounds each named phrase's type and drops names whose phrase gets no confident mask.
- No-feedback ablation. Segmentation tokens are emitted, but the pointer never sees $z_r$ or $q_r$.
- Counterfactual-only ablation. The same abstention data is used without any masks.
- Masked-entity pretraining. The method of Show, Interpret and Tell, applied to the naive model.
Beating these means two things. First, a statistically significant drop in unanchored-name rate on the probe sets, relative to the counterfactual-only and no-feedback ablations. Second, entity precision no worse than, and CIDEr within one point of, the naive model on the standard test sets.
Why not just…?
Why not just caption with a strong MLLM and then check names with a referring segmenter? A segmenter cannot ground "Angela Merkel" by name. The pipeline must ground a type phrase and sees none of the reasoning that produced the caption. LISA reports a decoupled LLM plus Grounded-SAM pipeline at about 26 gIoU against 65.0 for its end-to-end token design on ReasonSeg. That is a different task, so it is suggestive rather than decisive. We run the pipeline control directly. If it matches our unanchored-name rate, the integration claim fails and we will report that.
Why not just rely on a large MLLM's implicit copying? This is a real threat. LargePiG shows that LLMs behave like pointer generators already, so we do not claim that copying itself is the gain. Mao et al. found a small specialized controller outscoring LLaVA-1.5 on highlight relevance (1.73 vs 1.27) in Ctrl-CIC, which hints that scale alone does not solve context selection. The naive baseline and the no-feedback ablation test this directly. If they close the gap, the explicit span pointer is unnecessary.
Why not just train on counterfactual images without masks? This may well capture most of the gain, because the model could learn "no visible face, no name" from data alone. The counterfactual-only ablation exists for this reason. We expect masks to matter most for multi-entity captions and for partial occlusion, where the count and position of persons decide how many names can be attached. We will report results stratified by number of named people.
Why not just filter names by a person-detector count? This is cheap and catches over-naming. However, it cannot decide which of several article names belongs to the visible person, and it ignores places and organizations. We include it as a post-hoc filter on the naive model.
Proposed Experiments
Training and standard evaluation use Visual News and GoodNews. For transfer, we use WIT, reporting its Easy and Hard splits separately, since Show, Interpret and Tell identifies the Hard split as where shortcuts dominate. Metrics are CIDEr, entity precision and recall against reference-caption entities, and per-type breakdowns.
We will build two new evaluation resources. The first is a human-annotated subset of roughly 1,500 Visual News test images with entity-to-mask links, for measuring grounding mIoU and whether each name is attached to the correct region. The second is a set of probe sets: test images with the named person cropped or inpainted out, and images paired with mismatched same-topic articles. On these probes the primary metric is the unanchored-name rate, meaning names emitted with no corresponding visible referent.
The decisive check is the comparison against the counterfactual-only and no-feedback ablations on the probes, stratified by the number of named people. If the feedback model lowers the unanchored-name rate mainly in multi-person and occluded cases, and keeps entity recall on unmodified images, the claim holds. If the counterfactual-only model matches it everywhere, the segmentation pathway is not earning its compute, which GLaMM-style decoders make substantial. In that case the correct conclusion is that data, not architecture, fixes the copying shortcut.
References
- Show, Interpret and Tell: Entity-Aware Contextualised Image Captioning in Wikipedia
- LISA: Reasoning Segmentation via Large Language Model
- GLaMM: Pixel Grounding Large Multimodal Model
- Visual News: Benchmark and Challenges in News Image Captioning
- Good News, Everyone! Context driven entity-aware captioning for news images
- Get To The Point: Summarization with Pointer-Generator Networks
- LargePiG: Your Large Language Model is Secretly a Pointer Generator
- MMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentation
- Language-Driven Region Pointer Advancement for Controllable Image Captioning
- Controllable Contextualized Image Captioning: Directing the Visual Narrative Through User-Defined Highlights