Out-of-context misinformation is a real, unedited photo paired with a false caption, for example a ten-year-old flood photo presented as last week's disaster. Because the pixels are genuine, detectors cannot look for editing traces. Recent detectors instead use a multimodal large language model (a model that reads both images and text) to write an explanation and then a verdict: "the image shows X, the web evidence says Y, so the caption is false." These explanations are the main reason fact-checkers would trust such systems. Nobody has checked whether the explanations are true. Vision-language models are known to produce confident statements about images that the image does not support. They are also known to give a correct top-level answer while making false claims in the supporting text. A detector can therefore reach the right verdict for invented reasons and still be scored as correct.
We propose an auditing method for these explanations that runs entirely on open-weight models. Each explanation is split into small atomic claims. Each claim is labeled by what it is about: what the image shows, what the caption says, or what the retrieved evidence says. Each claim is then checked only against that source. The audit has two uses. First, it measures how much of current detectors' measured accuracy rests on fabricated reasoning. Second, it acts as a filter: a verdict whose supporting claims fail verification is turned into an abstention. We expect this filter to help most in the two regimes where these detectors are known to fail, polluted web evidence and images with no web history.
Core Claim
Verdict accuracy on out-of-context benchmarks overstates detector reliability. A source-routed, claim-level checker built from open models can measure explanation faithfulness about as well as a proprietary judge. Withholding verdicts whose explanations contain unverified image or evidence claims should improve the accuracy of the verdicts that are issued, at a stated coverage, more than confidence thresholds or self-consistency can.
The new element is routing. A judge that receives the whole explanation together with the image, caption and evidence can be persuaded by fluent text. This is the failure documented for multimodal judges in Chen et al. and Kaiyi Huang. Routing narrows the judge's task. A visual claim is checked against the image alone. An evidence claim is checked against the retrieved text alone. The judge never sees the explanation it is auditing as a whole.
How This Relates to Prior Work
Several papers motivate the gap. The out-of-context detectors SNIFFER, CMIE, MAD-Sherlock and COVE work as follows:
- SNIFFER uses InstructBLIP with reverse image search.
- CMIE first generates how the image and text could coexist, then scores the evidence.
- MAD-Sherlock has several agents debate over Bing Visual Search summaries.
- COVE reconstructs seven context fields, such as date, location and people, before deciding.
All four present explanations as a selling point. All four evaluate verdicts, not whether the explanation's claims are true.
Hallucination research says those explanations should not be trusted by default:
- Yan et al. and Zenz et al. show correct answers paired with fabricated justifications.
- Why LVLMs Are More Prone to Hallucinations in Longer Responses shows that attention drifts away from the image as text grows. Multi-step explanations are exactly this kind of long output.
- POPE documents a strong tendency to agree with affirmative questions. This mirrors the bias toward predicting "real" that CMIE reports for out-of-context detection.
Other work constrains the design. CMIE and Mitigating GenAI-powered Evidence Pollution show that retrieved evidence is noisy and can be deliberately polluted. Evidence claims must therefore be checked against the retrieved text, and the auditor must not assume that text is true. SNIFFER notes that grounding collapses for images with no web footprint, so the audit must still work when evidence is absent. The multimodal-judge failures cited above rule out a single holistic judge.
Other work supplies the method:
- VALOR-EVAL provides two-stage decomposition: an LLM parses text into objects, attributes and relations, and a second stage matches them.
- HaloQuest's Auto-Eval shows that an LLM can extract semantic intent with Pearson $r = 0.99$ against humans. That result used Gemini, not an open model, and was on VQA rather than misinformation.
- RITUAL contributes multi-view image checking: the image is cropped and transformed, and a visual claim is accepted only if it holds across views.
- HerO and AGORA show that pipelines of specialized open models can match GPT-4o-class systems. They also warn that errors compound across stages and that heavy agent loops can hurt small models. This is why our pipeline has three fixed stages and no loop.
Research Program
The auditor has three stages. In stage one, an open text model (Llama-3.1-70B or Qwen2.5-72B) splits the detector's explanation into atomic claims. Each claim gets a source tag: image, caption, evidence, or world-knowledge inference. In stage two, each claim is routed to a verifier for its source:
- Image claims go to an open vision-language model (Qwen2.5-VL-7B and -72B, or Molmo2), asked a yes/no question about the original image and two RITUAL-style transformed views. To counter the agreement bias, the same claim is also asked in negated form.
- Caption claims are checked by textual entailment against the caption.
- Evidence claims are checked by entailment against the retrieved snippets the detector actually saw.
- Inference claims are marked unverifiable rather than judged.
In stage three, a claim graph is aggregated into two outputs. The first is a faithfulness score: the share of checkable claims that are supported. The second is a decisive-claim flag, set when a claim the verdict depends on fails. Dependence is determined by asking the decomposer which claims the conclusion cites.
We will build a human-annotated audit set. It consists of explanations from SNIFFER, CMIE, MAD-Sherlock and COVE-style pipelines, all re-run with matched open backbones, on NewsCLIPpings test pairs across its semantic, person-match and scene-match splits. Two annotators label every atomic claim. This set serves to validate the auditor and to measure how often detectors hallucinate.
Baselines
The naive reference point is verdict-only scoring. It accepts every detector verdict and reports accuracy, which is the current practice. On the full NewsCLIPpings setup we found no reported result on Lacuna for verdict accuracy of SNIFFER, CMIE, MAD-Sherlock or COVE. We will therefore reproduce all four under identical retrieval and backbones rather than quote numbers.
The strongest competitor for measuring faithfulness is a holistic proprietary judge. GPT-4o receives the image, caption, evidence and explanation and grades faithfulness, in the style of HaloQuest's Auto-Eval. The closest reported number is that paper's $r = 0.99$ human correlation for Gemini Auto-Eval on VQA intent extraction. We found no reported result on Lacuna for any judge on out-of-context explanation faithfulness. The comparison is fair because both judges see identical inputs and get the same prompt-tuning budget, measured as the number of prompt revisions on a held-out development split. Beating this baseline means one of two things. Either the open routed auditor matches the proprietary judge's claim-level agreement with humans (Cohen's kappa within its bootstrap interval), or it exceeds that agreement on visual claims, where we expect holistic judges to be fooled by fluent text.
The ablation that isolates the key ingredient is the same open models used holistically: no decomposition and no routing. A second ablation keeps decomposition but sends every claim to the vision-language model with all context visible, which isolates routing from decomposition.
For using the audit as a filter, the strongest competitors are the detector's own verdict confidence with a tuned threshold, and self-consistency (five sampled explanations, majority verdict, abstain on disagreement). All methods are compared at matched coverage. Beating them means higher accuracy on issued verdicts at 70%, 80% and 90% coverage.
Why not just…?
Why not just use GPT-4o as the judge? It may in fact be accurate on most claims. That is why it is the main baseline and not a strawman. Two problems remain. Fact-checking organizations often cannot send unpublished or sensitive media to a cloud API, and API versions drift, which undermines reproducibility. Multimodal judges are also documented to hallucinate visual errors (Chen et al.). If the routed open auditor merely ties GPT-4o, the contribution becomes an offline, reproducible equivalent plus the hallucination measurements. We will report it that way.
Why not just calibrate a confidence threshold? A threshold catches verdicts the model is unsure of. It cannot catch confidently right-for-wrong-reasons verdicts, or confidently wrong verdicts built on a hallucinated visual detail, which is exactly the pattern long-response hallucination predicts. This is a real threat if most detector errors happen to be low-confidence. The matched-coverage comparison settles it directly: if the threshold does as well as the audit, filtering adds nothing beyond measurement.
Why not just probe hidden states for hallucination, as SHARP does? SHARP finds that faithful and hallucinated activations are linearly separable in middle layers. That is cheaper than a three-stage pipeline. However, Latent Fact-Checking reports that such probes fail on external, unmemorized facts, and evidence claims about breaking news are exactly that kind of fact. A probe also cannot say which claim failed, so it cannot produce an auditable explanation. We will train a probe on the same annotated data and compare it with the auditor separately for each claim type. We predict the probe is competitive on image claims and weak on evidence claims.
Proposed Experiments
The first decisive check is the measurement itself. For each detector we report the rate of unsupported claims among correct verdicts, split by source tag. The core claim survives only if this rate is substantial. We would consider it substantial at about one in five correct verdicts or more, and if it is not, we will say so.
The second check is auditor validity. We report claim-level kappa against humans for the routed open auditor, its two ablations, the holistic open judge and GPT-4o.
The third check tests filtering under stress. We compare accuracy at matched coverage under three conditions: clean evidence, evidence polluted following the evidence-pollution setup, and evidence removed to simulate images with no web footprint. In the no-evidence condition only image and caption claims remain checkable. The prediction is that the auditor's advantage over thresholding is largest under pollution, because polluted evidence produces confident but unsupported evidence claims.
We also report the cost of running the auditor. VCD-style contrastive decoding takes 13.97s per sample, as reported in the file's discussion alongside VALOR-EVAL and RITUAL. Our auditor needs one decomposition call plus one short query per claim, and we will report the measured cost per explanation rather than assume it is cheaper.
References
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
- T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation
- Sniffer: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection
- CMIE: Combining MLLM Insights with External Evidence for Explainable Out-of-Context Misinformation Detection
- MAD-Sherlock: Multi-Agent Debate for Visual Misinformation Detection
- COVE: COntext and VEracity prediction for out-of-context images
- Evaluating the Quality of Hallucination Benchmarks for Large Vision-Language Models
- Can Vision-Language Models be a Good Guesser? Exploring VLMs for Times and Location Reasoning
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- Evaluating Object Hallucination in Large Vision-Language Models
- Mitigating GenAI-powered Evidence Pollution for Out-of-Context Multimodal Misinformation Detection
- VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models
- HaloQuest: A Visual Hallucination Dataset for Advancing Multimodal Reasoning
- RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in LVLMs
- HerO at AVeriTeC: The Herd of Open Large Language Models for Verifying Real-World Claims
- Unifying Language Agent Algorithms with Graph-based Orchestration Engine for Reproducible Agent Research
- NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media
- SHARP: Steering Hallucination in LVLMs via Representation Engineering
- Latent Fact-Checking: Detecting Misinformation through Activation Engineering