Abstract
Fact-checking claims is a highly laborious task that involves understanding how each factual assertion within a claim relates to a set of trusted source materials. Existing approaches typically make sample-level predictions but fail to identify the specific aspects of the claim that are problematic or the specific evidence relied upon. In this paper, we introduce M3D, a method and new benchmark for this challenging task. Our method predicts the fine-grained logical relationship of each aspect of a claim from a set of multimodal documents, including text, images, videos, and audio. We also introduce a new benchmark (M3DC) of claims requiring multimodal multidocument reasoning, constructed using a novel claim synthesis technique. Experiments show that our approach outperforms existing models on this task while providing finer-grained predictions, explanations, and evidence.
Introduction: The Challenge of Fine-Grained Fact-Checking
Automated fact-checking is a high-stakes task that currently suffers from three major flaws: coarse granularity, single-source bias, and a "neutral" blind spot. Most systems flag an entire claim as "false" without explaining which specific detail (e.g., the person, action, or location) is incorrect. Furthermore, they often rely on a single document, whereas real-world verification requires synthesizing information from multiple media types. Finally, many models struggle to identify when evidence is simply missing, often mislabeling "unknown" information as "contradicted."
To address these gaps, the M3D framework treats fact-checking as a fine-grained entailment problem across a multimodal, multi-document landscape. By breaking claims down into semantic graphs, M3D can pinpoint exactly which part of a claim is inconsistent and identify the specific evidence that refutes it.
Methodology
1. Data Synthesis and the M3DC Benchmark
To train a model for fine-grained detection, the authors developed a sophisticated data synthesis pipeline to create the M3DC dataset. This benchmark contains approximately 400,000 claims across 10,000 topics. The pipeline involves:
- Knowledge Graph (KG) Construction: Extracting information from news clusters (text, images, video, and audio) and linking them via co-reference resolution.
- Claim Generation: Using graph traversal to find related "Knowledge Elements" and converting them into natural language sentences using AMRBART.
- Claim Manipulation: Utilizing a Llama-2 model fine-tuned with Reinforcement Learning (PPO) to intentionally alter claims into "Neutral" or "Contradicted" versions while maintaining syntactic similarity to the original "Entailed" claim.
2. The M3D Model Architecture
The M3D model dissects claims using Abstract Meaning Representation (AMR) graphs, where nodes represent entities or events and edges represent relations. The architecture employs LongT5 for text encoding and ImageBind for visual and audio context. A two-layer Graph Convolutional Network (GCN) then propagates information through the claim's AMR graph to predict a label (Entailed, Neutral, or Contradicted) for every single node and edge, as well as a global label for the entire claim.
Key Insights and Results
- AMR as the "Skeleton" of Truth: By using AMR graphs, the model performs "semantic bookkeeping." If a "location" node is contradicted, the system knows exactly which part of the sentence is incorrect.
- Superior Fine-Grained Accuracy: On the M3DC benchmark, M3D significantly outperformed baselines. In the "Contradicted" category for fine-grained labels, M3D achieved an F1-score of 0.41, compared to the baseline FGVE's 0.09.
- Handling the "Neutral" Case: While state-of-the-art Large Vision-Language Models (LVLMs) like LLaVA-1.5 achieved 0% F1-scores on neutral cases, M3D successfully identified claims where evidence was missing, reducing the risk of automated hallucinations.
- Cross-Modal Reasoning: The research demonstrates that adding visual and audio context (such as background sirens in a video contradicting a claim of a "peaceful" event) provides a necessary layer of verification that text alone often misses.
Conclusion
M3D represents a significant step forward in automated fact-checking by providing granular, explainable results across multiple documents and modalities. While the current model relies on generated captions for visual data and assumes source materials are trustworthy, future work will explore raw pixel processing and methods for handling conflicting information within trusted sources.
Summary updated