Sibling Photos as a Test and a Cure for Name-Copying in News Captioning

A news caption names people, places and events. Most of those names cannot be read off the pixels, so modern captioners get them from text: the accompanying article, sentences retrieved from it, or an external knowledge base. That is retrieval-augmented generation, meaning the model looks up text at generation time and conditions on it. It also opens an easy failure. The model can name whoever the article talks about most, whether or not that person is in the photo. Such a caption is fluent, scores well on overlap metrics, and is factually wrong in a specific way: a real name attached to the wrong image. We propose to measure and train against this failure with a signal the news datasets already contain and nobody uses. Many articles carry several photographs, and each has its own caption. Two photos from the same article share the same text, so whatever differs between their correct captions must come from the image.

The core idea fits in two sentences. Hold the article and the retrieved context fixed, swap in a sibling photo from the same article, and check whether the captioner's names change the way the reference captions do. A model that copies from text will not change. We use that sibling swap first as a hallucination benchmark and then as a contrastive training loss. The loss rewards a caption for being more likely under its own photo than under its sibling.

Why this gap is real

Prior work motivates the problem from three sides. In generic captioning, Understanding Retrieval Robustness for Retrieval-augmented Image Captioning shows with attribution analysis that retrieval-augmented captioners learn to copy tokens from retrieved captions. Those tokens overlap the target 88–99% of the time in training, and with irrelevant retrievals CIDEr drops by up to 50%. Retrieval-augmented Image Captioning reports that its decoder shifts attention from the image to retrieved text in later layers. Blacking out the input image only lowers COCO CIDEr from 121.2 to 102.1, which says much of the caption survives without the photo. In contextual captioning, Show, Interpret and Tell documents models copying names from metadata while ignoring visual features. Enhancing Journalism with AI finds that full-article context drives METEOR down to 0.164 and induces hallucination. Its clean-entity setting reaches 0.344, but those entities are taken from the ground-truth captions, so it is an oracle.

The news captioning systems themselves supply the retrieval methods we build on. None of them tests whether names depend on the image. EAMA fine-tunes InstructBLIP to pick relevant article sentences and entities before captioning. VACNIC retrieves article sentences by CLIP similarity to the image, aligns detected faces with names, and adds a margin loss against a text-only language model. MERGE retrieves from an external entity knowledge base using CLIP matching and face recognition. All three are scored mainly on CIDEr and named-entity precision/recall against the reference caption. A copied name of the article's protagonist is often in the reference too, so these metrics cannot separate "named because seen" from "named because prominent."

Two-input conflict methods constrain the design. KCM reweights attention and prunes neurons so that a vision-language model trusts retrieved text over its internal memory in question answering. That pushes in exactly the wrong direction for our failure, where the text is over-trusted relative to the image.

Core Claim

We claim two things. First, a large share of person names in current news captions is invariant to which photo from the article is shown, and this invariance is invisible to CIDEr and reference-based entity F1. Second, a training loss built on sibling photos removes much of that invariance without losing the entity recall that context provides.

The sibling contrast differs from each neighbouring technique. Sample-$k$ training varies the retrieved context while holding the image fixed. VACNIC's margin loss contrasts against a model that has no image at all. That penalizes generic, uninformative captions, but it does not penalize a caption that names the article's main figure for every photo. Blank-image contrastive decoding removes the image entirely, which also strips the legitimate textual evidence for the correct, depicted protagonist. A sibling photo is a realistic counterfactual. It comes from the same story, the same visual style, and often shows overlapping people, so only the image-specific part of the caption is pushed apart.

Research Program

The first piece is the benchmark, which we call the sibling-swap entity probe. Multi-image articles are common: Visual News pairs over a million images with about 600,000 articles. We will count the exact sibling subsets of GoodNews, NYTimes800k and Visual News. For each pair of sibling photos $(x_i, x_j)$ sharing article $c$, we define the discriminative entities of $x_i$ as names in its reference caption that are absent from the reference caption of $x_j$.

We report two quantities. Sibling-discriminative recall asks whether the model names those entities for $x_i$. Sibling leakage asks whether it also names them for $x_j$. We also report an unsupported-depiction rate in the style of CHAIR from Object Hallucination in Image Captioning: the fraction of generated person names that are mentioned in the article but not in that photo's reference caption. Reference captions do not always name every visible person. We will therefore human-audit about a thousand flagged cases to estimate how often a "leak" is actually correct.

Evaluating Object Hallucination in Large Vision-Language Models found that free-form caption hallucination metrics swing across prompts (standard deviation 3.22 versus 0.78 for yes/no probing). Following its design, we add yes/no probes of the form "Is [name] shown in this photograph?". The negatives are drawn adversarially from names in the same article that appear only in sibling captions, the news analogue of POPE's co-occurrence sampling.

The second piece is the training signal. Let $y_i$ be the reference caption for photo $x_i$, $c$ the retrieved context (article sentences, or knowledge-base entries in a MERGE-style setup), $E_i$ the token positions of sibling-discriminative entities in $y_i$, and $\log p(\cdot)$ the captioner's summed token log-likelihood over $E_i$. We add a hinge term to the usual likelihood loss:

$$ \mathcal{L} = \mathcal{L}_{\text{MLE}} + \beta \max\big(0,\; m - [\log p(y_i^{E_i} \mid x_i, c) - \log p(y_i^{E_i} \mid x_j, c)]\big), $$

where $m$ is a margin and $\beta$ a weight. Restricting the hinge to $E_i$ matters. Names, places and events shared by both siblings are legitimately text-derived and are left alone. Only the names that should depend on the photo must depend on it. The loss can be added to any of the captioners above. We implement it on an InstructBLIP captioner with EAMA-style sentence selection and on a LLaVA-style model with retrieved context, so that results are not tied to one architecture.

The third piece is analysis across retrieval sources. We run the probe with three kinds of context: full-article context, intra-article retrieval (CLIP sentence selection as in VACNIC, or learned selection as in EAMA), and external entity retrieval as in MERGE. The question is whether better retrieval reduces sibling leakage, or whether it simply delivers the prominent name more efficiently. We suspect the latter for CLIP retrieval: siblings that look alike will pull up the same sentences.

Baselines

The naive reference point is a vision-language model prompted with the full article, as in the LLaVA and GPT-4v conditions of Enhancing Journalism with AI. That paper reports METEOR 0.164 for BLIP-2 + GPT-3 with full-article context on GoodNews. We rerun this condition on our sibling split with the same base model as our method. A blank-image run of the same model, where the photo is replaced by a gray image, gives the pure text-copying floor for sibling leakage.

The strongest existing method on reported numbers is EAMA, which reports 87.00 CIDEr and 32.24% entity recall on NYTimes800k against a previous best of 83.72 CIDEr. MERGE reports only relative gains in the file (+6.84 CIDEr and +4.14 NER F1 on GoodNews; +1.16 CIDEr on NYTimes800k). We found no absolute MERGE score reported on Lacuna. We also found no reported result on Lacuna for any sibling-swap or depiction-precision metric, so those baseline numbers will come from our own reruns.

For fairness, we retrain EAMA and a MERGE-style external-retrieval variant on the same base model and the same training images. Each gets the same number of optimizer steps and the same hyperparameter search budget (a fixed number of trials over learning rate and, for our method, $\beta$ and $m$). Our method counts as beating them if it lowers sibling leakage and unsupported-depiction rate by a margin larger than the seed-to-seed spread. At the same time, sibling-discriminative recall must hold, overall entity recall must stay within one point of the baseline, and CIDEr must be no worse than one point below.

Three ablations isolate the key ingredient by swapping the counterfactual and keeping everything else fixed. The first is Sample-$k$ training applied to news context: sampling $k$ sentences from a wider top-$N$ pool, as in Understanding Retrieval Robustness, which reports CIDEr 98.9 versus 73.2 for the standard model under random-noise captions on COCO. The second is a VACNIC-style text-only contrast. The third is the same hinge with a random photo from another article, or a blank image, in place of the sibling. If the sibling is the ingredient that matters, it should beat all three on leakage.

We also include a post-hoc face-recognition filter. It drops generated person names whose retrieved reference faces do not match any detected face, using the same kind of face matching as MERGE and VACNIC.

Why not just…?

Why not just apply Sample-$k$ training to news articles? Sample-$k$ breaks the habit of copying from any one retrieved passage by randomizing which passages appear. In a single article, though, the prominent name appears in many sentences, so every sample still contains it, and the image never has to discriminate. We expect it to cut leakage from noisy external retrievals, but not leakage of the article's main figure. The context-varying ablation tests this directly. If it matches the sibling loss on leakage, the sibling loss adds nothing and we would report that.

Why not just verify names with face recognition? This is the most serious threat for people who are celebrities and visible. MERGE and VACNIC already use face matching, and VACNIC reports a 5% absolute gain in person recall on NYTimes800k from its face-naming module. Face matching needs a reference gallery for each name and a frontal, detectable face. It does not help for little-known people, back-of-head or crowd shots, or non-person depicted entities such as buildings, vehicles and sports teams. We will split the results by whether a gallery face exists. If the face filter matches our method on the gallery subset, we will say so. Our case then rests on the non-gallery subset and on combining the two.

Why not just contrast against a blank image at decoding time? Removing the photo also removes the evidence that the article's protagonist is likely to be shown. Such contrast should therefore penalize correct protagonist names as much as copied ones, buying lower leakage at the cost of recall. The blank-image ablation measures this trade-off. If it turns out favourable, the cheaper training-free method wins and we would adopt it.

Why not just select better sentences, as EAMA does? Sentence selection decides which text the model sees. Siblings that differ in content but look similar will often be given the same sentences, and the generator can still name whoever those sentences emphasize. Running EAMA on the sibling probe shows whether its selection already discriminates between siblings. If it does, the gap we claim is smaller than we think.

Decisive Checks

The first check comes before any training: the sibling-swap probe on existing captioners. If names already change across siblings at close to the reference rate, the thesis is wrong and the project becomes a negative result about name-copying. Our prediction is that blank-image, full-article and EAMA-style models produce nearly the same person names for siblings, at leakage rates far above the references.

The second check is whether the sibling loss lowers leakage and unsupported-depiction rate on GoodNews and NYTimes800k while keeping entity recall. It must also beat the random-photo and blank-image versions of the same loss, which separates "any image contrast" from "a matched counterfactual."

The third check is transfer. We train on GoodNews and evaluate zero-shot on Visual News, the setting where MERGE reports its largest gain (+20.17 CIDEr over the runner-up). The point is to see whether lower leakage carries over across news agencies. We also see whether the yes/no adversarial probes agree with the caption-level metrics. If they disagree, we treat the probe results as the more trustworthy, given the stability evidence from POPE.