Diversify the Evidence, Not the Words: Hop-Level Critics for Grounded Expressive Captioning

The best captions and stories about an image often depend on a fact that is not in the picture. A New Yorker cartoon joke turns on an idiom. A wry tweet caption relies on a cultural reference. A story about a graduation album needs a link between "diplomas" and "parents crying." Current systems produce variety by adding randomness to the words they write, through hotter sampling or adversarial rewards on the final sentence. That randomness tends to invent things. The words become more varied, but the scene they describe is no longer the one in the image. We propose to move the source of variety earlier, to the short chain of retrieved facts that connects the visible scene to the eventual caption. A model first proposes these intermediate "hops," meaning retrieval queries and the bridging statements they support. A learned critic rewards a hop only when it is both checkable against the image or a retrieved document and unlike the hops the model usually takes. A separate stage then turns a verified chain into an expressive caption.

The two-sentence version is this. Variety in expressive captions should come from choosing different true connections, not from saying the same connection in stranger words. We train a retrieval-and-reasoning policy whose intermediate steps are pushed apart by an adversarial "typicality" critic, and we gate that push with a per-step grounding check so that novelty cannot compound into hallucination.

Core Claim

The research file identifies a real tension between two lines of work. Adversarial diversification methods reward the long tail of language and pay for it in fluency and precision. DP-GAN replaces the binary discriminator with a language model and raises distinct trigrams on Yelp from 6.6K to 49.6K, but it notes semantic drift. Multi-hop retrieval, in turn, is fragile: one bad hop corrupts everything downstream. Combining the two naively, by attaching a diversity reward to the output of an interleaved retriever, should make hallucination cascades worse.

Our claim is that this tension disappears if two things hold. First, the diversity pressure must act on the discrete choice of which verified hop to take next, not on free text. Second, every hop must be verified before it can earn any reward. Under these conditions, more diverse captions should not cost more hallucinations. Concretely, across $N$ captions sampled for one image, the number of distinct verified bridge facts should rise while the rate of unsupported claims stays flat. Every baseline that injects diversity at the word level should show those two quantities moving together.

Relation to Prior Work

Three papers motivate the gap.

Knowledge-Enriched Visual Storytelling (KG-Story) is the closest prior work. It searches Visual Genome and OpenIE graphs for one- and two-hop paths between words detected in adjacent images. It scores those paths with a fixed language model and then writes the story. Its paths are static and chosen by likelihood, so it favors generic links, and the file notes it can inject content the images do not support. We differ in three ways: our hops are generated and retrieved dynamically, chosen to be atypical, and checked one at a time.

IRCoT supplies the retrieval mechanism we reuse. It interleaves single chain-of-thought sentences with retrieval, where each sentence becomes the next search query. This improves recall by 11–21 points and QA F1 by up to 15 points over one-step retrieval. IRCoT is text-only and deterministic, and nothing in it seeks variety.

Humor in AI gives the strongest motivation. Text-only models given human descriptions of a cartoon beat multimodal models given the raw image. Claude-3-Opus captions were preferred by a New Yorker editor only 1.6% of the time, against 35.4% for average crowd submissions. This suggests the bottleneck is finding the right non-visual connection, which is what our hop policy targets. It is also why the hop policy first rewrites the image as explicit entity descriptions before retrieving.

Four papers supply methods for critics and rewards. Show, Reward and Tell pairs a visual-relevance critic with a story-style critic on the final paragraph. AREL learns an adversarial reward from human stories, again on final text. Adversarial Semantic Alignment uses a co-attention image–text discriminator with self-critical sequence training, a policy gradient that uses the model's own greedy output as the baseline. This doubles vocabulary coverage, from 5.1% to 11.0%. All of them apply the critic after the caption is written. We keep their two-critic design and the self-critical baseline, but move the novelty critic onto intermediate hops and make the grounding critic a hard gate rather than an additive term.

Several stylized captioners constrain the design. MSCap uses back-translation to check that a stylized caption still describes the image. ADS-Cap samples diverse style phrases from a conditional variational autoencoder over a fixed 1,600-word object vocabulary. Both show that style can be added once content is fixed, which justifies our split between finding content and rendering style. Neither retrieves anything.

RONA prompts proprietary multimodal models with five discourse coherence relations to obtain pragmatically varied captions. We reuse its relations as a prompt for the rendering stage, but RONA itself has no retrieval and no learned critic.

Research Program

The system has three parts.

The hop policy is an open vision-language model fine-tuned to do two things. It first writes a short literal inventory of the image. This deliberately turns the image into text, given the finding in Humor in AI. It then alternates, IRCoT-style, between writing a bridging statement or query and retrieving from a Wikipedia passage index plus the Visual Genome and OpenIE graphs used by KG-Story. A chain stops after at most four hops.

The critics score each hop $h_t$ with a reward:

$$r(h_t) = g(h_t)\,\big[-\log q_\phi(h_t \mid h_{<t}, x)\big],$$

The symbols are as follows. $x$ is the literal image inventory. $g(h_t)\in[0,1]$ is a grounding score: an entailment model checks that the hop's claim is supported by the retrieved passage, and an image–text matcher checks that the chain's anchor entity is actually in the image. $q_\phi$ is a small language model that is continually re-trained on the policy's own recent hops. Hops the policy keeps producing therefore become predictable and earn little. This follows the DP-GAN idea of using a language model as a continuous discriminator, but here the model tracks the generator's habits rather than human text.

Because $g$ multiplies the novelty term, an unsupported hop earns zero no matter how surprising it is. This is the mechanism meant to stop novelty from cascading into hallucination. The policy is trained with self-critical policy gradient on the sum of hop rewards, after supervised warm-up on chains distilled from a strong model.

The renderer writes the final caption from a verified chain, conditioned on a target style or a RONA coherence relation. It is trained with DPO on the New Yorker ranking data, since Humor in AI found DPO helps there, and with supervised learning on FlickrStyle10K and VIST. The renderer is held fixed across all comparisons below, so any difference comes from the content chain.

Baselines

The naive reference point is the same backbone vision-language model captioning directly, with no retrieval. It is sampled at temperature 1.0 and top-p 0.9 to get $N=10$ captions per image, then passed through the same renderer. This measures how much variety and how much hallucination come for free.

The strongest prompt-based competitor is RONA-style prompting with five coherence relations, run on the same open backbone so that compute matches, plus a version on the proprietary models RONA used for reference. RONA reports wins in 7 of 8 settings on Tweet Subtitles and ANNA for Self-BLEURT and Div-2. No absolute Div-2 value for RONA was found on Lacuna, so we will re-run it ourselves. Beating it means higher verified-fact diversity at equal or better BLEURT and CLIPScore.

On VIST, the strongest reported systems are StoryLLaVA, at RoViST-C 0.833 against 0.759 for a LLaVA+GPT-4 pipeline (StoryLLaVA), and KG-Story, at CIDEr 0.096 against 0.049 for GLAC. KG-Story is the most direct competitor because it also uses multi-hop knowledge paths. We will re-implement its path enrichment over the same graphs and feed its chosen paths to our renderer, which isolates static, likelihood-chosen paths from our dynamic, critic-chosen ones. Beating it means matching StoryLLaVA's RoViST-C while lowering the human-annotated rate of unsupported story details relative to KG-Story.

On the New Yorker task, the reference number is Claude-3-Opus's 54.4% win rate against crowd top-10 captions under a GPT-4 judge, reported in Humor in AI. We use the same judging protocol and also collect a small expert-style human evaluation, since the 1.6% editor result shows the AI judge is unreliable.

Three ablations isolate the key ingredient. In the first, chains are sampled at high temperature and the critic is removed, so there is diversity but no learned pressure. In the second, the novelty term is kept but the grounding gate $g$ is dropped. This tests the tension directly: we expect diversity to rise and hallucination to rise with it. In the third, the adversarial $q_\phi$ is replaced by a fixed inverse-frequency penalty on retrieved documents, which tests whether an adversarial critic is needed at all.

Every variant gets the same retrieval index, the same renderer, the same number of samples per image, and the same hyperparameter search budget.

Why not just…?

Why not just sample many captions at high temperature and rerank them with a grounding scorer? This is the most serious threat, because best-of-N with a strong verifier can approximate our gate at inference time without any training. Our answer is that reranking can only choose among connections the model already tends to propose. If the policy always reaches for the same three idioms, reranking returns grounded but repetitive captions. The experiment that settles this is a comparison of best-of-$N$ reranking, scored with the same $g$, against our trained policy at equal total generation compute. If reranking matches us on distinct verified bridge facts per image, the training component is unnecessary, and we would report the method as an inference-time search procedure instead.

Why not just prompt a strong multimodal model with coherence relations and one round of retrieval? RONA shows prompting produces surface-level pragmatic variety. One-shot retrieval, however, cannot find a second-hop fact whose query depends on the first; IRCoT measured this gap at up to 15 F1 points in text QA. Humor in AI shows that prompted frontier models still fall to 1.6% under expert judgment. We include this configuration as a baseline rather than assume it fails.

Why not just apply DPO to final captions using the New Yorker preference data? We do use DPO, but only for the renderer. The concern is that preference tuning on final text pulls toward whatever raters reward on average. Humor in AI notes that creative rewards are "spiky" and hard for static reward models. Comparing DPO-only against DPO with critic-chosen chains, using the same renderer, shows whether choosing content contributes anything beyond style preferences.

Why not just use KG-Story's static paths? Static paths ranked by likelihood are exactly the generic links we expect to avoid, and they are not checked against the image. Feeding KG-Story's paths to our renderer, as described above, answers this question directly.

Proposed Experiments

We evaluate on four benchmarks: New Yorker contests, VIST, FlickrStyle10K and SentiCap for stylized single images, and Tweet Subtitles and ANNA for pragmatic captions. For each image we sample ten outputs and measure three things. The first is verified diversity: the number of distinct bridge facts across the ten outputs that human annotators confirm as true and relevant. The second is the unsupported-claim rate, the fraction of captions asserting something contradicted by or absent from the image and the retrieved evidence. The third is quality, measured by GPT-4-judge win rate on the New Yorker task, RoViST-C on VIST, style accuracy on FlickrStyle10K and SentiCap, and BLEURT and CLIPScore on the pragmatic sets.

The decisive check is a plot of verified diversity against unsupported-claim rate as the novelty weight is increased. The core claim predicts a flat hallucination curve for the gated critic and a rising one for the ungated and temperature ablations. If the gated curve rises too, the claim is wrong.

A second decisive check runs the hop policy, with its critic, on MuSiQue questions. MuSiQue is a multi-hop QA benchmark built so that single-paragraph shortcuts fail. If diversity pressure makes chains reach correct answers less often than plain IRCoT, the policy is buying novelty with broken reasoning. If F1 holds, the critic is choosing among valid paths, which is what expressive captioning needs.