Anti-Hallucination Prompts That Survive the Move to Closed Models

Vision-language models describe things that are not in the image. They report a refrigerator in a kitchen photo because kitchens usually have one, and they answer "yes" to almost any "is there an X?" question. Many of the strongest fixes need access to the model's internals: subtracting logits, steering hidden states, or running extra forward passes. Commercial models such as GPT-4o, Gemini and Claude expose none of this, so for them the only lever is the input. One hand-written instruction already helps on these models. Text-only work shows that prompts optimized on a small open model can carry over to a large closed one. The obvious next step is to search for anti-hallucination prompts on open models and ship them to closed ones.

That step is not obviously safe. There is direct evidence that prompt-side fixes for hallucination can flip sign from one architecture to the next. The same rendered instruction helps one model and collapses another. The best visual marker differs between LLaVA and GPT-4o. Top-performing prompts barely overlap across model families. This proposal treats transfer as the thing to be engineered rather than assumed. We search for anti-hallucination instructions against a panel of open models with deliberately different vision encoders. We keep only prompts whose benefit is consistent across that panel and is not explained by simply making the model say "no" more often. We then test whether panel consistency predicts which prompts help closed models.

Core Claim

Anti-hallucination prompts optimized on a single open model mostly exploit that model's quirks. The commonest quirk is a shift in its yes/no answer bias, which looks like a hallucination reduction on balanced benchmarks such as POPE but involves no better looking at the image. Selecting prompts by the consistency of their gain across architecturally diverse proxies, under a constraint that holds the yes-rate fixed, should produce instructions that transfer to black-box models better than either single-proxy optimization or the best hand-written prompt. Under this claim, panel consistency is a cheap, measurable predictor of black-box benefit. The claim is falsifiable on two counts: the rank correlation between panel consistency and closed-model gain could be near zero, or the gains could vanish once yes-rate is controlled.

What Exists and What Is Missing

The closest work falls into three groups.

The first group is hand-written or input-side anti-hallucination prompting. Magnifier Prompt (MagPrompt) prepends a fixed "look at the image, and prefer it over common sense" wrapper. It reports 84.5 macro F1 on the POPE adversarial split for LLaVA-1.5-13B, against 81.9 for visual contrastive decoding (VCD), and it helps GPT-4o and Gemini Pro. It never optimizes the wording, and it fails on InstructBLIP. MagPrompt motivates this proposal: it shows text-side fixes reach closed models. It also sets the bar to beat.

Black-Box Visual Prompt Engineering (BBVPE) trains an image-only router over visual markers. It raises LLaVA-1.5 POPE F1 from 89.71 to 91.42, but finds that LLaVA prefers "reverse blur" while GPT-4o and Claude prefer cropping or arrows. Cure or Poison? renders instructions into the image. This improves Qwen2.5-VL from 80.2% to 84.3% POPE accuracy but drops LLaVA-1.5 from 84.0% to 55.0%, with a yes-ratio near 0.99. These two papers constrain the design. A prompt can be a cure on one vision encoder and a poison on another, so any method that tunes against one model is suspect.

The second group is prompt optimization that already transfers. Black-Box Prompt Optimization (BPO) trains a 7B query rewriter that improves GPT-4 on general dialogue. PromptBridge learns systematic cross-model prompt edits for code tasks. UniAPO runs multimodal automated prompt optimization: it clusters failure cases and lets an LLM rewrite the prompt. UniAPO moves GPT-4o-optimized prompts to Qwen2.5-VL-72B, but only for classification and keyword extraction. IPO shows that natural-language prompts transfer better than learned embedding prompts, but only for CLIP classifiers. These papers supply the search machinery we reuse. None targets hallucination, and none checks whether an apparent gain is grounding or answer bias.

The third group is the counter-evidence on transfer. PromISe finds almost no overlap between the top prompts of different model families. TP-Eval argues that a single shared template underrates multimodal models. In Their Own Words and Self-Renewal Prompt Optimizing show that prompts overfit the model they were tuned on. We take this group seriously. Our bet is not that optimal prompts coincide across models. It is that a smaller set of robustly good prompts exists and can be found by asking for agreement.

Research Program

The proxy panel spans the vision-encoder split that Cure or Poison? identified. It includes two CLIP-encoder LLaVA-family models (LLaVA-1.5-7B and LLaVA-Next), Qwen2.5-VL-7B, whose encoder was trained with heavy OCR (text-in-image) data, and InstructBLIP, which was instruction-tuned on short prompts. mPLUG-Owl and LLaVA-1.5-13B are held out as open transfer targets. GPT-4o, Gemini Pro and Claude 3.5 Sonnet are the black-box targets.

Candidate instructions come from an LLM optimizer in the style of UniAPO's failure-clustering loop. The search signal comes from POPE-style questions built on the COCO train split and from captions scored by CHAIR. POPE asks yes/no questions about object presence. CHAIR counts objects in free-form captions that are absent from the COCO annotations. A prompt $p$ is scored by

$$S(p) = \bar{\Delta}(p) - \lambda\,\sigma_{\Delta}(p) - \mu \max_m \left| y_m(p) - y_m(\varnothing) \right|.$$

Here $\Delta_m(p)$ is the hallucination improvement on proxy model $m$ relative to no added prompt, computed as POPE adversarial F1 gain plus CHAIR sentence-level reduction, with object recall held above a floor. $\bar{\Delta}$ and $\sigma_{\Delta}$ are the mean and spread of these improvements across the panel. $y_m(p)$ is the model's yes-rate on POPE with prompt $p$, and $y_m(\varnothing)$ is its yes-rate with no added prompt. The spread term rewards agreement across proxies. The yes-rate term penalizes prompts that "win" by changing the answer bias rather than by grounding the answer in the image. We also cap prompt length, because MagPrompt's InstructBLIP failure suggests long wrappers hurt models with short instruction data.

The second deliverable distills the selected prompts into a small text-only rewriter, trained BPO-style on preference pairs in which the preferred rewrite is the one with the higher $S$ score. The rewriter reshapes each user query, for example by adding instructions that tie attributes and counts to visible regions. It never sees the image, so it cannot introduce visual claims of its own. If the rewriter fails to beat the single best selected prompt, we will report that. The panel-selection result stands on its own.

The analysis has three parts. For every candidate prompt we log its panel statistics and its gain on each held-out and black-box target. We then measure how well panel mean, panel spread and single-proxy gain predict target gain. Finally, we classify the instructions that survive selection by content type: image priority, explicit uncertainty, enumeration before answering, or attention to the background.

Baselines

The naive reference is the target model with no added instruction, queried with the exact benchmark prompts. Every gain we report is measured against this, on the same images and question order, with greedy decoding wherever the API allows it.

The strongest existing black-box competitor is MagPrompt (Magnifier Prompt), the published wrapper used verbatim. It is the strongest because it is free, already transfers to GPT-4o and Gemini, and beats VCD on LLaVA-1.5-13B with 84.5 against 81.9 macro F1 on POPE adversarial. For GPT-4o the file gives no number: there is no reported MagPrompt POPE figure for GPT-4o on Lacuna beyond "comparable or superior". We will therefore re-run MagPrompt ourselves on every target. Beating it means a statistically significant gain in POPE adversarial F1 on at least two of three black-box targets, with no CHAIR regression and a yes-rate change no larger than MagPrompt's.

The second competitor is single-proxy optimization: the same optimizer, query budget and length cap, with all search on LLaVA-1.5-7B and the spread and yes-rate terms removed. This is the obvious version of the idea, and it is the ablation that isolates our key ingredient. Each panel-based search gets the same total number of proxy evaluations.

A further ablation removes only the yes-rate term. This shows how much of any apparent transfer is answer-bias shifting.

The third competitor is direct optimization on the target: the same optimizer run against GPT-4o with an API budget equal to the cost of our transfer evaluation. It serves as a reference ceiling for what proxies can buy.

For context on open models we report, without re-tuning, published numbers for methods that need internal access: RITUAL at 88.87% POPE accuracy on LLaVA-1.5 (RITUAL), and SHARP's CHAIR sentence-level rate of 34.8 down from 52.8 (SHARP). We will not claim to beat these. They show how far a prompt-only method sits from internal-access methods.

Why not just…?

Why not just use MagPrompt? It is a real threat. If one sentence already captures most of the transferable benefit, an optimized search adds little. Two things argue for search. MagPrompt fails on InstructBLIP, and PromISe and TP-Eval show a single template leaves performance on the table. Neither shows that an optimized prompt does better on closed models, though. The MagPrompt baseline settles this directly. If it comes within noise of our selected prompts, the paper's contribution narrows to the diagnostic finding about yes-rate and transfer prediction, and we will say so.

Why not just optimize directly on GPT-4o? With enough API budget this probably wins on GPT-4o itself. It must be repeated for every closed model and every model version, however, and PromptBridge documents that prompts drift as models change. The matched-budget comparison shows how close proxy transfer gets. A proxy-selected prompt that recovers most of the direct-search gain on Gemini and Claude, at zero target cost during search, is a useful result even if direct search stays ahead on its own target.

Why not just tell the model to answer "no" when unsure? On POPE, a bias toward "no" can raise F1 for models with a near-99% yes-rate (Evaluating Object Hallucination) without any better grounding. We include this prompt explicitly as a control. Its weakness should show on CHAIR and AMBER, where saying "no" is not available. Our selected prompts must beat it on those generative metrics at a matched yes-rate.

Why not just use visual prompts like BBVPE or RITUAL? These are complementary rather than competing. BBVPE's own finding is that the best marker differs between LLaVA and GPT-4o, which is exactly the transfer problem we study. We will run the text prompt together with RITUAL's image augmentation as a combination check, not as a replacement.

Decisive Experiments

The evaluation covers POPE (random, popular and adversarial splits, plus the A-OKVQA and GQA variants for distribution shift), CHAIR on held-out COCO val captions, AMBER, PhD in both neutral and misleading modes, HallusionBench, and the MME existence and count subsets. The search uses only COCO train images.

The first decisive check is a rank correlation, across all candidate prompts, between panel score and gain on the black-box targets, compared against the correlation for single-proxy gain. If panel consistency is not a better predictor, the core claim fails.

The second check asks whether panel-selected prompts beat both single-proxy prompts and MagPrompt on held-out open targets and closed targets, at matched yes-rate.

The third check is a leave-one-proxy-out test. We remove Qwen2.5-VL from the panel to see whether including an OCR-trained encoder is what protects against the Cure-or-Poison style collapse on other models.

The fourth check uses the matched-budget direct-optimization ceiling to show how much of the closed-model gain proxies can recover.

A negative result on the first check would still be informative. It would extend PromISe's finding of near-zero prompt overlap to hallucination control, and it would tell practitioners to stop expecting anti-hallucination prompts tuned on open models to carry over.