Contrary Probes: Deciding When a Multimodal Fact-Checker Should Search

Automated fact-checkers that search the web over several rounds are much more accurate than single-shot models, and much slower. Systems that plan a search, read the results, and search again (DEFAME) roughly double accuracy on claims made after the model's training cutoff. Each round, though, adds web queries and large-model calls, and debunking already lags behind viral spread. The obvious fix is to search only when the fast model is unsure. That fix depends on knowing when a vision-language model is unsure, and spatial and location claims are where its own confidence is least trustworthy. Vision-language models answer "is the cup on the table" and "is the cup under the table" with near-chance discrimination (What's "up" with vision-language models?), yet nothing in their output probabilities reveals that they are guessing.

The proposal is to ask the fast model a few cheap, paired questions before any search. Each pair states one assertion from the claim and its contrary: "taken in Kyiv" versus "taken in Lviv", or "the crowd is in front of the building" versus "behind it". Each pair is also asked in a few harmless rewordings. A model that says yes to both members of a pair, or that changes its answer when the wording changes, is not grounded on that assertion. Only those flagged assertions trigger search. The same check, rerun once evidence arrives, tells the loop when to stop. In two sentences: contradictions in a fast model's answers to paired spatial and location questions decide whether to search, what to search for, and when to stop. We test whether this matches always-search accuracy at a fraction of the latency, and whether it beats both ordinary confidence scores and paraphrase-only consistency.

Core Claim

SENSEL showed for text classifiers that answers which change under prompt rewording tend to be wrong. Its abstention rule, which measures how much predictions vary across perturbed prompts, beat maximum softmax probability by an average of 4.1 points of area under the F1-coverage curve on GPT-J-6B. ProSA traced this sensitivity to flat token distributions. Both look only at invariance: they ask whether the answer stays put when nothing meaningful changed.

The spatial failure documented by What'sUp is the mirror image of that. The model stays put when something meaningful did change, such as swapping "left" for "right". A purely paraphrase-based signal cannot see this, because a model that ignores prepositions is perfectly stable under rewording.

Our claim is that a two-sided signal predicts when retrieval will help much better than either confidence or paraphrase stability alone, especially on spatial and geolocation claims. The two sides are invariance to meaning-preserving rewordings and incompatibility of answers to mutually exclusive assertions. For an atomic assertion $a$ extracted from the claim, with contrary $\bar a$ and image $x$, define its incoherence as

$$ I(a) = \big| p(\text{yes} \mid x, a) + p(\text{yes} \mid x, \bar a) - 1 \big| + \lambda \,\mathrm{Var}_{t \in T}\, p_t(\text{yes} \mid x, a). $$

Here $p$ is the fast model's probability of answering yes, $T$ is a small set of instruction templates, $p_t$ is that probability under template $t$, and $\lambda$ weights the two terms. A grounded model gives complementary probabilities to the pair (first term near zero) and is stable across templates (second term near zero). The claim is escalated to retrieval if $\max_a I(a) > \tau$. Each assertion with high $I(a)$ becomes the target of one specific query.

Research Program

The system has three stages that share one backbone. Stage zero is the cheap path for recycled claims. It uses BM25 plus dense matching against past fact-checks, as in That is a Known Lie, which reports a mean reciprocal rank of 0.788 on Snopes. Claims with a confident match exit there.

Stage one is the probe step. An open vision-language model (Qwen2.5-VL-7B, so we can read token probabilities) breaks the claim into atomic assertions, following FIRE's atomic-claim decomposition (FIRE). Contraries come from templates. Spatial relations are swapped with their opposite. Locations are replaced with a plausible distractor from the same region; this is a hard negative, not a random city. Explicit left/right claims additionally get a horizontally flipped image. Dates and times of day get an adjacent alternative. All probes reuse the cached image encoding and are batched, so the added cost is a few short decoding passes rather than any network round trip.

Stage two is retrieval targeted to flagged assertions. A flagged location assertion triggers reverse image search and geolocation. A flagged entity or event assertion triggers text web search. A flagged spatial-relation assertion that has no external referent goes to a stronger verifier model rather than to search. After each round, the retrieved evidence is placed in context and $I(a)$ is recomputed. The loop stops when every assertion is coherent or a budget of $B$ searches is spent. This gives a calibrated stopping rule, which both the iterative-refinement literature and EarthWhere (Where on Earth?) identify as missing.

Alongside the method we contribute a measurement that Gap 4 of the source material identifies as absent: verdict flip rates of existing fact-checkers under twelve instruction templates, across their True / Miscaptioned / Out-of-context labels. We will report it for every system we run.

Relation to Prior Work

Three systems motivate the latency problem and supply the iterative loop we reuse. DEFAME runs a six-stage GPT-4o agent with web, image, reverse-image, and GeoCLIP tools, and loops whenever evidence is judged insufficient. MultiReflect adds multi-source retrieval with evidence ranking across several attributes. RAMA sends every query through a fixed ensemble of prompts and vision-language judges at about 3.5 seconds per query. In all three, the decision to search is either always on or made by the agent's own judgment. Our proposal replaces that decision with a measured signal and makes the signal choose the query. SNIFFER grounds verdicts in detected entities and reverse image search. Like the others, it checks who and what, not whether stated spatial and locational facts are consistent with the image.

Two papers constrain the design. EarthWhere found that adding web search lowered GPT-4o's country-level geolocation accuracy by 13.2% while helping street-level tasks by a relative 6.5%. So "search when unsure" must be judged by whether search helps, not merely by whether accuracy is high. What'sUp found that fine-tuning on spatial data produces shortcuts that fail on controlled counter-examples, which is why we detect spatial ungroundedness rather than try to train it away. Hidden Sensitivity in Spatial Reasoning Evaluation shows that spatial scores are confounded by answer distributions. We therefore balance contrary pairs so that "yes" is the correct answer exactly half the time.

SENSEL and ProSA supply the core idea that instability predicts error. We extend it from rewording invariance to contrary-pair incompatibility, from text to images, and from abstention to routing retrieval.

Baselines

The naive reference point is the fast backbone alone: a single pass with chain-of-thought and no search. This sets the latency floor and the accuracy we must improve on. The published analogue is vanilla GPT-4o at 35.2% on ClaimReview2024+, as reported by DEFAME.

The accuracy ceiling is always-search. Every claim goes through the full iterative loop with all tools and budget $B$, using the same backbone and the same frozen search snapshot. We re-implement DEFAME's plan–execute–judge loop on our backbone so that tools and models are identical. We also report DEFAME's published numbers as an external reference: 69.7% on ClaimReview2024+, and 70.5% on AVeriTeC. For VERITE, the strongest number in the file is MultiReflect's binary accuracy of 0.78 (GPT-4V), with multi-class accuracy 0.50. On the Multimedia Verification Challenge, RAMA reports 0.91 accuracy at about 3.5 seconds per query. On NewsCLIPpings, SNIFFER reports 88.4%. No reported result was found on Lacuna for DEFAME's per-claim wall-clock latency, so we will measure it ourselves. Beating always-search means staying within one accuracy point of it while escalating at most 40% of claims and cutting median latency by at least half.

The strongest gating competitors all decide which claims to escalate. One gates on maximum softmax probability of the verdict. One gates on the model's verbalized confidence. One is FIRE's self-judgment rule, under which the model decides at each step whether its evidence suffices (FIRE). One is SENSEL-style paraphrase variance alone, which is the ablation that removes contrary pairs; a matching ablation removes paraphrases and keeps contrary pairs only. Finally, random escalation at matched rate shows what any gate must exceed.

All gates are compared at matched escalation rates on the same claims. Thresholds are tuned on the same held-out development split, with the same number of tuning trials. Winning means a higher area under the accuracy-versus-escalation-rate curve, with bootstrap confidence intervals that exclude zero. Routing gets its own ablation: our gate paired with generic query generation, as FIRE does it, against our gate paired with assertion-targeted queries, both at the same search budget.

Why not just…?

Why not just gate on the model's output probability or stated confidence? It costs a single pass, so it is the real threat. SENSEL found that maximum probability and entropy trailed sensitivity-based abstention in text, and What'sUp's near-chance spatial results suggest confident wrong answers on exactly the claims we care about. That evidence comes from other models and tasks, however. If probability gating matches our area under the curve on the spatial subset, the contrary-pair term is not earning its cost, and we will report that and recommend the cheaper gate.

Why not just let the agent decide when to search, as FIRE does? FIRE's self-judgment cut search cost 16.5x on textual claims. But the decision is made by the same model whose grounding is in doubt, and EarthWhere's overthinking result shows that model-initiated search can hurt. Our design measures, per item, whether search actually changed correctness. That tells us directly whether self-judgment or incoherence better predicts claims where search helps rather than hurts.

Why not just paraphrase the prompt, as SENSEL does? A model that ignores spatial words is stable under rewording, so paraphrase variance should miss precisely the spatial failures. The contrary-only and paraphrase-only ablations test this. Our prediction is that they are similar on entity claims and diverge on spatial and geolocation claims. If they do not diverge, the spatial motivation is wrong.

Why not just fix spatial reasoning with spatial fine-tuning such as SpatialThinker? The two approaches are complementary, but What'sUp shows that spatial fine-tuning learns dataset shortcuts that break on controlled examples. Under fact-checking distribution shift, knowing when the model is ungrounded remains necessary. We will run our gate on top of a spatially fine-tuned backbone to check that it still adds value.

Proposed Experiments

The evaluation spans five benchmarks. VERITE covers true, miscaptioned, and out-of-context image-caption pairs. NewsCLIPpings provides scale. MOCHEG covers multimodal evidence. ClaimReview2024+ holds 300 claims made after the training cutoff, where retrieval is genuinely needed. The Multimedia Verification Challenge test set allows a latency comparison with RAMA. On VERITE and ClaimReview2024+ we will annotate a spatial and location subset, meaning claims whose truth depends on where something is or how objects are arranged. Every search result is frozen in a snapshot so that all systems see identical evidence.

Before building the gate, we validate the probe signal itself. On What'sUp and the EarthWhere country and street splits, we test whether contrary-pair incoherence predicts per-item error better than output probability. On EarthWhere specifically, we test whether it predicts which items search improves versus degrades.

Metrics are accuracy and macro-F1, median and 95th-percentile wall-clock latency, searches per claim, area under the accuracy-versus-escalation-rate curve, and verdict flip rate across templates.

Three checks decide whether the idea works. First, at matched escalation rates, the two-sided gate must beat probability, verbalized-confidence, FIRE-style, and paraphrase-only gates on the spatial subset, and must not lose elsewhere. Second, assertion-targeted queries must beat generic queries at equal search budget. Third, the gated system must reach always-search accuracy on ClaimReview2024+ within one point, at under half its median latency.