When Does a Stylized Caption Lie? Measuring Hallucination by What the Reader Believes

Vision-language models now write captions that are funny, romantic, sarcastic or poetic, and they also hallucinate, meaning they describe objects that are not in the picture. The standard tests for hallucination check whether an object word in the caption matches an object annotated in the image. That test breaks on expressive language. "This sofa is a cloud" mentions a cloud that is not there. "Dessert on the patio" names a patio nobody photographed. "What glorious weather" may describe a storm. A word-matching checker flags all three as errors, and a word-matching corrector rewrites them back into flat inventory. We propose a different test. A caption hallucinates when a reader who has not seen the image would come away believing something false about it. Under this test, the cloud in a metaphor does not count, while a patio stated as fact does.

The proposal builds a reader-belief hallucination measure, checks it against human readers, and uses it to answer a question no one has measured: do the decoding-time fixes for hallucination keep their benefit, or destroy the requested style, when the model is asked to be expressive? It then tests one lightweight mitigation. The model generates several expressive candidates and keeps the one whose induced reader beliefs agree with the model's own careful yes/no answers about the image. Whether this beats simply writing a literal caption and restyling it with a text-only model is a central, openly contested question in the experiments.

Core Claim

Our claim is that object-mention metrics overcount hallucination in expressive captions, and that they do so unevenly across styles. Metaphor, hedged inference and irony get penalized heavily. Confident literal insertion of unseen objects gets penalized no more than it should be. Because of this distortion, current comparisons of hallucination fixes on expressive output are unreliable. A measure based on reader belief should agree with CHAIR on literal captions. On expressive captions it should diverge in predictable ways, and once it does, the ranking of mitigation methods should change.

The prior work splits into three roles. Some of it motivates the claim. RONA prompts GPT-4o and Claude with discourse coherence relations to get pragmatically diverse captions. One of those relations, Insertion, explicitly asks for "a salient object that isn't explicitly mentioned in the image," and RONA never checks these captions with an object-hallucination metric. Describe Images in a Boring Way builds sarcasm by reversing the image's emotional tone, which a survey taxonomy such as Siren's Song in the AI Ocean would file as input-conflicting hallucination. Born Pragmatic, Trained to Hallucinate? shows in text-only settings that alignment training pushes models toward implied readings, with an average Context Sensitivity Gap of -14.0%, so the literal/implied boundary is already known to be where models slip.

Other work supplies the measurement machinery we reuse. Object Hallucination in Image Captioning introduced CHAIR, which matches caption nouns against COCO segmentation labels. It also showed that optimizing CIDEr inflates hallucination. POPE replaced free-form scoring with yes/no object questions, which are far more stable across prompts (standard deviation 0.78 versus 3.22). Our reader probe is POPE asked of a caption instead of an image.

A third group constrains the mitigation design. VCD subtracts the predictions made from a noise-corrupted image to cancel language priors. OPERA penalizes attention collapse onto summary tokens during beam search. Woodpecker extracts noun phrases, verifies them with Grounding DINO and BLIP-2, and rewrites the text. All three define the target as literal inventory, so all three risk treating requested style as a prior to be removed.

Research Program

The first component is the measure itself. Each caption goes to a reader who cannot see the image. For each object in a POPE-style question set, the reader answers "would you believe there is an X in this picture?" The question set mixes objects that are present, objects that are absent, and absent objects that commonly co-occur with present ones (adversarial negatives). Reader-induced hallucination is the rate of "yes" on absent objects, with each caption compared against the same image's literal reference captions. An optional extension probes attributes such as colour and weather.

We collect human readers' answers on a calibration set. We then fit a text-only LLM proxy reader and report its agreement with those humans before using it at scale. This definition takes no position on whether a figure of speech is good. It only asks whether the caption misinforms, which is what matters when captions serve as alt text or reach people who never see the image. Irony is the hard case, because a reader who cannot see the image cannot detect it. We therefore report sarcastic captions separately and treat them as a stress test, not something the measure settles.

The second component is an audit. We generate captions with LLaVA-1.5 and InstructBLIP, the open models used by OPERA and VCD. Each model receives a literal prompt, positive and negative sentiment prompts, humorous and romantic prompts, and RONA's five coherence-relation prompts. Generation runs under plain decoding, VCD, OPERA and Woodpecker post-correction. For every cell we report CHAIR, reader-induced hallucination, style strength, and diversity.

The third component is the mitigation, belief-consistency reranking. The model samples N expressive candidates. The proxy reader lists the objects each candidate would make it believe are present. The same model then answers a binary probe for each listed object while looking at the image, using VCD-style decoding because yes/no probing is the regime where VCD's gains are documented. Each candidate is penalized by the number of believed-but-unsupported objects, combined with a style score. The method rewrites nothing, so a metaphor survives unless it plants a false belief.

Baselines

The do-nothing reference is plain sampling from LLaVA-1.5 and InstructBLIP under each style prompt. It shows how much hallucination expressiveness costs before any fix.

The strongest existing methods are VCD and OPERA, run on the same expressive prompts with the hyperparameters their papers recommend. OPERA reports roughly a 35% CHAIR improvement over DoLa, and VCD reports up to +7.4 POPE F1, both on literal tasks. For hallucination on stylized captions specifically, we found no reported result on Lacuna, so these methods are being tested outside the conditions they were measured under. Woodpecker is the strongest post-hoc competitor; it reports raising MiniGPT-4 POPE accuracy from 54.67% to 85.33%. We run it unchanged on the expressive outputs.

To isolate the key ingredient, we keep belief-consistency reranking fixed and swap the proxy reader's belief list for plain noun extraction. That makes it a Woodpecker-style selector that does no rewriting, so any gap between the two variants is attributable to reading for belief rather than for surface words. A second ranking baseline is Best-of-N selected by CLIPScore plus a style classifier, mirroring the EGRM ranker.

Every reranking method gets the same N, the same base model and the same verification budget, and all hyperparameters are tuned on one shared development split. A method beats VCD or OPERA if it lowers reader-induced hallucination at equal style strength. It must do so with style strength judged by humans within noise of plain sampling, while VCD or OPERA either lower style strength or fail to lower reader-induced hallucination.

Why not just…?

Why not just write a literal caption and have a text LLM restyle it? This is the most serious threat, and we run it as a full competitor. A grounded caption from VCD or OPERA is passed to a strong text LLM told not to add entities. It may well achieve the lowest reader-induced hallucination. The evidence against it concerns expressiveness, not truthfulness. In Humor in AI, text-only models given human descriptions were competitive, yet an expert editor preferred AI captions only 1.6% of the time. RONA reports richer pragmatic variation when models see the image directly. We will plot both methods on a hallucination-versus-human-rated-expressiveness frontier. If the restyling pipeline dominates that frontier, we report that the mitigation is unnecessary. The measurement contribution and the audit stand either way.

Why not just prompt the model to be expressive but not invent objects? PaCE shows that strict instructions can swing GPT-4o's gap from -10.4% to +26.1%, which suggests that overcorrecting is easy. Why LVLMs Are More Prone to Hallucinations in Longer Responses shows that attention drifts away from the image as text grows, and a prompt cannot fix that drift. We include constrained prompting as a baseline and expect it to trade style for literalness.

Why not just run Woodpecker or VCD as-is? Our audit tests exactly this. The prediction is that Woodpecker's mis-corrections concentrate on figurative mentions. Its reported overall rate is about 6.2%, but that figure comes from literal text. We also predict VCD lowers style scores because the stylistic association is part of what it subtracts. If either prediction fails, the existing tools are adequate and we will say so.

Why not just filter figurative words out of CHAIR with an LLM judge? That is close to our proxy reader. The difference is that we validate the proxy against what human readers actually believe, rather than against a rule list for figurative language. We report how far a simple figurative-word filter gets as its own baseline.

Proposed Experiments

The images come from 500 COCO Karpathy test images, following the CHAIR and OPERA protocol, together with the COCO images underlying UTStyleCap4K and the SentiCap rewrites. These come with segmentation labels for checking objects and human-written stylized captions as a human reference point. RONA's Tweet Subtitles and ANNA test images add abstractive cases. For these, and for FlickrStyle10K humorous and romantic prompts, we annotate absent-object ground truth by hand for a 300-image subset. Expressiveness is measured by human style ratings, a style classifier, and Self-BLEURT diversity as in RONA.

Four checks decide the project. First, on literal captions, reader-induced hallucination must correlate strongly with CHAIR. If it does not, the measure is broken. Second, on expressive captions, a substantial share of CHAIR-flagged mentions must not be believed by human readers. This share should be largest for metaphor and hedged Insertion and smallest for flat Insertion. If readers believe nearly everything CHAIR flags, the core claim fails. Third, the ranking of VCD, OPERA and Woodpecker by reader-induced hallucination on expressive prompts must differ from their ranking by CHAIR. Fourth, belief-consistency reranking must outperform its noun-extraction ablation and sit on or beyond the restyling pipeline's frontier at matched expressiveness. Losing that comparison is a publishable negative result about where hallucination fixes should live.