TL;DR
Current multimodal misinformation benchmarks suffer from data contamination (models simply memorize old events) or artificial synthesis. The authors introduce XFACTA, a contemporary, real-world benchmark of 2,400 post-2023 social media claims, and conduct a systematic study dissecting evidence retrieval versus reasoning in Multimodal Large Language Models (MLLMs). They find that combining reverse-image textual retrieval with multi-step reasoning yields state-of-the-art detection (88.8% accuracy on GPT-4o) and introduce a continuous, detection-in-the-loop update pipeline.
What Problem Does This Paper Address?
While MLLMs (like GPT-4o and Gemini) show immense promise for fact-checking multimodal social media posts (text paired with images), current evaluations face two major issues:
- Evaluation Bias: Existing benchmarks either contain historical events already baked into the pre-training data of modern LLMs (allowing models to succeed via memorization rather than actual verification) or consist of synthetic, AI-generated fakes that fail to reflect the subtle tactics used by real-world bad actors.
- Unclear Bottlenecks: It is poorly understood whether model failures stem from poor evidence retrieval (finding relevant web context) or flawed reasoning (synthesizing evidence to spot inconsistencies like out-of-context images).
Methods
The authors systematically unpack and optimize the two core stages of MLLM-based fact-checking:
- Evidence Retrieval Strategies: They evaluate eight search formulations—including unimodal text search ($T \to E_t$), cross-modal image-to-text ($I \to E_t$) and text-to-image ($T \to E_i$), LLM-generated search queries, and search engines (Google vs. DuckDuckGo). They also evaluate evidence post-processing via Domain Filtering (removing low-credibility sources) and LLM-Based Evidence Extraction (filtering noise before reasoning).
- Reasoning Architectures: They benchmark standard Chain-of-Thought (CoT), Prompt Ensembles, Self-Consistency, and Multi-Step Reasoning. Multi-step reasoning explicitly disentangles the verification pipeline:
- Verify caption authenticity using textual evidence ($T \to E_t$).
- Verify image misuse/context using image-retrieved text ($I \to E_t$).
- Synthesize a final judgment.
- Detection-in-the-Loop Expansion: To prevent benchmark obsolescence, their top-performing pipeline monitors live accounts, flags suspicious content with generated rationales, and routes candidates to human fact-checkers for ongoing dataset expansion.
Datasets
- XFACTA (Introduced): 2,400 posts from X (Twitter) published between January 2024 and April 2025 (safely after the training cutoffs of models like GPT-4o). It contains 1,200 real posts (from outlets like BBC, CNN, and Reuters) and 1,200 fake posts (flagged by journalists and X Community Notes). Real posts are balanced against fakes via topic alignment and Optimal Transport matching over SigLIP visual embeddings. Fake posts are annotated across three categories: Deepfakes, Image Out-of-Context (OOC), and Text Misleading.
- Evaluation on Other Benchmarks: The authors compare performance against existing datasets, including VERITE, Snopes+Reuters, and NewsCLIPpings, as well as a newer out-of-distribution test set of 1,200 Snopes articles from July 2024 to July 2025.
Key Insights and Ideas
- Prior benchmarks rewarded memorization, not verification: On older datasets (VERITE, Snopes+Reuters), evidence-free GPT-4o achieved 80–92% accuracy purely through parametric memory. On XFACTA, zero-shot evidence-free accuracy dropped to 70.5% (and a dismal 51% on real posts), demonstrating that contemporary benchmarks are strictly required to assess true fact-checking ability.
- $I \to E_t$ is the silver bullet for Out-of-Context fakes: Querying the web using the post's image to retrieve text ($I \to E_t$) dramatically outperforms searching caption text for matching images ($T \to E_i$). Image search pinpoints original reporting and context, whereas text-to-image search returns superficial visual matches that fail to catch relocated or recycled photos.
- The "Confirmation Trap" of Text Search: Searching only the post caption ($T \to E_t$) improves accuracy on real posts, but often hurts accuracy on out-of-context fakes. The retrieved evidence confirms that the event described in text occurred, misleading the LLM into overlooking that the accompanying image depicts a completely different event.
- Multi-step reasoning enforces skepticism: Without structured steps, models tend to lump evidence together and ignore subtle visual mismatches. Explicitly breaking verification into "Caption Authenticity" followed by "Image Misuse Check" prevents caption evidence from drowning out visual discrepancies.
Key Results
- Benchmark Superiority: On the XFACTA test set, the proposed pipeline (GPT-4o + Multi-Step Reasoning + $T \to E_t$ & $I \to E_t$) achieves 88.8% overall accuracy (87.2% on real posts, 90.4% on fakes).
- Comparison to Baselines: It significantly outperforms fine-tuned specialist baselines (e.g., SENs at 52.6%, HAMMER at 57.2%, Sniffer at 56.1%) and prior LVLM frameworks (MMFakeBench at 68.2%, LEMMA at 77.3%), both of which struggled to generalize to real-world social media data.
- Out-of-Distribution Generalization: Evaluated on 1,200 recent, out-of-distribution Snopes fact-checks (mid-2024 to mid-2025), the detector maintained 89.2% accuracy.
Limitations and Future Work
- Overly Strict Misuse Detection: Multi-step reasoning occasionally flags benign posts from reputable outlets as fake if the publisher used generic illustrative stock photos rather than on-the-scene documentation.
- Inference Latency & Cost: Performing multiple rounds of search (Google API), evidence extraction, and multi-step LLM prompting introduces significant computational and monetary overhead per post.
- Search Dependency: The framework relies heavily on external search engines (Google substantially outperformed DuckDuckGo); API rate limits, search ranking drift, and anti-scraping policies remain operational bottlenecks.
Summary updated