Mitigating GenAI-powered Evidence Pollution for Out-of-Context Multimodal Misinformation Detection

Zehong Yan, Peng Qi, Wynne Hsu, Mong-Li Lee

Abstract: While large generative artificial intelligence (GenAI) models have achieved significant success, they also raise growing concerns about online information security due to their potential misuse for generating deceptive content. Out-of-context (OOC) multimodal misinformation detection, which often retrieves Web evidence to identify the repurposing of images in false contexts, faces the issue of reasoning over GenAI-polluted evidence to derive accurate predictions. Existing works simulate GenAI-powered pollution at the claim level with stylistic rewriting to conceal linguistic cues, and ignore evidence-level pollution for such information-seeking applications. In this work, we investigate how polluted evidence affects the performance of existing OOC detectors, revealing a performance degradation of more than 9 percentage points. We propose two strategies, cross-modal evidence reranking and cross-modal claim-evidence reasoning, to address the challenges posed by polluted evidence. Extensive experiments on two benchmark datasets show that these strategies can effectively enhance the robustness of existing out-of-context detectors amidst polluted evidence.

The Challenge of Evidence Pollution

Out-of-context (OOC) misinformation involves pairing authentic images with deceptive captions—for instance, using a decade-old protest photo to describe current events. Modern detection systems combat this by retrieving web evidence to verify claims. However, the rise of Generative AI (GenAI) has introduced a critical vulnerability: evidence-level pollution.

Malicious actors can now use Large Language Models (LLMs) and Diffusion Models to flood the internet with synthetic articles and images that appear to "corroborate" false claims. If a detector retrieves this polluted evidence, it may mistakenly validate a lie as truth. Research indicates that state-of-the-art detectors suffer a performance drop of over 9 percentage points when faced with such contaminated data.

Illustration of how GenAI-powered evidence pollution can mislead misinformation detectors by providing fake corroborating content.

Proposed Defense Strategies

To counter this threat, the authors introduce two "plug-and-play" strategies designed to help detectors filter out AI-generated noise and focus on factual signals:

  1. Cross-modal Evidence Reranking: Standard search engines often rank results based on text-to-text relevance. This strategy uses CLIP (a model that understands both images and text) to re-sort evidence. It prioritizes textual evidence that matches the claim's image and visual evidence that matches the claim's caption. This cross-check makes it significantly harder for hallucinated or irrelevant AI content to influence the final decision.
  2. Cross-modal Claim-Evidence Reasoning: This adds a layer of semantic analysis focusing on the "main entity" of a claim. It looks for fine-grained inconsistencies between the claim’s image and the most relevant retrieved text. This allows the model to spot when evidence is merely echoing a false narrative rather than providing genuine factual context.

Experimental Setup and Datasets

The study utilized two primary benchmarks to evaluate these defenses:

  • NewsCLIPpings: A large synthetic dataset for OOC detection created by mismatching images and captions from news archives.
  • VERITE: A real-world benchmark consisting of actual out-of-context pairs collected from fact-checking websites.

To simulate a contaminated web environment, the authors augmented these datasets with over 1.5 million pieces of "polluted" evidence generated via GPT-4 (for text) and Depth-Conditional Stable Diffusion (for images).

Key Insights: The Impact of Pollution

  • Textual Vulnerability: The researchers found that textual pollution is far more damaging to detectors than visual pollution. Current models rely heavily on the rich semantic relationships (who, what, where) found in text, which LLMs can now hallucinate with high convincingness.
  • Human Impact: In a human evaluation study, participants correctly identified AI-generated evidence only 49% of the time. Furthermore, reading polluted evidence caused 41.8% of participants to reverse their initially correct judgments, highlighting the danger this poses to human information consumption.
  • Performance Recovery: By implementing the reranking and reasoning strategies, the authors boosted the accuracy of the SNIFFER model from 76.42% to 88.82% on the NewsCLIPpings dataset—surpassing the model's original performance on "clean" data.

Limitations and Future Directions

While effective, the reranking strategy depends on the quality of the cross-modal encoder (CLIP). If pollution becomes sophisticated enough to fool CLIP's semantic matching, the defense may weaken. Additionally, reranking large volumes of evidence in real-time introduces computational overhead. Future research should explore more complex "reasoning-heavy" pollution, where AI creates logically consistent but entirely false narratives, requiring even deeper semantic verification.

Summary updated

Abstract

While large generative artificial intelligence (GenAI) models have achieved significant success, they also raise growing concerns about online information security due to their potential misuse for generating deceptive content. Out-of-context (OOC) multimodal misinformation detection, which often retrieves Web evidence to identify the repurposing of images in false contexts, faces the issue of reasoning over GenAI-polluted evidence to derive accurate predictions. Existing works simulate GenAI-powered pollution at the claim level with stylistic rewriting to conceal linguistic cues, and ignore evidence-level pollution for such information-seeking applications. In this work, we investigate how polluted evidence affects the performance of existing OOC detectors, revealing a performance degradation of more than 9 percentage points. We propose two strategies, cross-modal evidence reranking and cross-modal claim-evidence reasoning, to address the challenges posed by polluted evidence. Extensive experiments on two benchmark datasets show that these strategies can effectively enhance the robustness of existing out-of-context detectors amidst polluted evidence.
BibTeX
@misc{yan2025mitigating,
  author = {Zehong Yan and Peng Qi and Wynne Hsu and Mong-Li Lee},
  title = {{Mitigating GenAI-powered Evidence Pollution for Out-of-Context Multimodal Misinformation Detection}},
  year = {2025},
  eprint = {2501.14728},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.14728},
}

Loading related content...

Figure