Abstract
Large Vision-Language Models (LVLMs) often struggle with Multimodal Deepfake Detection (MDD) due to the subtlety of cross-modal inconsistencies. While fine-tuning offers a solution, it is computationally expensive and lacks generalization. This paper introduces GASP-ICL (Guided Adaptive Scorer and Propagation In-Context Learning), a training-free framework that treats deepfake detection as a sophisticated retrieval and in-context learning task. By utilizing a novel graph-based scoring mechanism (GSTAS), GASP-ICL identifies the most relevant "real vs. fake" examples to guide off-the-shelf LVLMs, significantly improving detection accuracy across various models without gradient updates.
Introduction
The rise of sophisticated generative AI has made Multimodal Deepfake Detection (MDD) a critical challenge. MDD requires identifying manipulations in images, text, or the semantic alignment between them. While state-of-the-art Large Vision-Language Models (LVLMs) like GPT-4o possess broad reasoning capabilities, they often overlook the minute artifacts characteristic of deepfakes.
The proposed GASP-ICL framework addresses these limitations by enhancing LVLMs through in-context learning (ICL). Instead of traditional fine-tuning, it leverages a smart "librarian" to provide the model with highly relevant reference cases, enabling frozen models to catch forgeries effectively.
The Challenge of Multimodal Deepfake Detection
Current approaches to MDD face three primary hurdles:
- Resource Intensity: Fine-tuning massive models for every emerging forgery type is computationally unsustainable.
- Subtle Cues: Deepfakes often contain logical "glitches" or tiny visual artifacts that general-purpose models are not inherently biased to detect.
- The Retrieval Gap: Standard ICL relies on similarity-based retrieval. However, simple semantic similarity often fails to distinguish between content-related samples and those that share structural forgery patterns.
Methodology: The GASP-ICL Framework
The core of GASP-ICL is a "knowledge bridge" built between a query sample and a library of known examples using graph theory.
1. Multimodal Retrieval and Clustering
The process begins with a CLIP-based encoder, specifically fine-tuned for deepfake features, to extract candidates from a sample library. The system evaluates similarity across three domains: image-to-image, text-to-text, and joint image-text embeddings.
2. Graph Construction and Fusion
Rather than selecting the nearest neighbors in isolation, the framework constructs a Query-Centric Fused Graph. The query sample serves as the central node, connected to candidates based on modality-specific similarities. This structure captures the inter-relationships between samples, providing a richer context for reasoning.
3. GSTAS: Graph-Structured Taylor Adaptive Scorer
The technical centerpiece is the Graph-Structured Taylor Adaptive Scorer (GSTAS). This mechanism propagates activation signals from the query node across the graph. Using a Taylor-expanded gating mechanism, it weights these signals to "up-weight" samples sharing consistent manipulation patterns while suppressing noise. This allows for a tunable "receptive field" on the graph, enabling the model to learn from higher-order relationships.
4. In-Context Inference
The top-$k$ samples identified by GSTAS are formatted into a prompt and presented to a frozen LVLM. By observing these relevant examples, the model performs zero-shot classification with significantly higher precision.
Experimental Evaluation
The framework was evaluated on the DGM4 dataset, covering four manipulation categories: Text Swap, Face Swap, Text Attribute, and Face Attribute.
Key Results
Testing across seven LVLMs (including Qwen2.5-VL and LLaVA-v1.6) revealed:
- Consistent Performance Gains: Qwen2.5-VL's accuracy improved from 49.71% (zero-shot) to 53.05% using GASP-ICL.
- F1 Score Improvements: F1 scores frequently increased by over 10 percentage points, indicating a better balance between precision and recall.
- Retrieval Efficacy: Using an adapted CLIP model for retrieval significantly outperformed standard frozen versions, highlighting the importance of forgery-aware feature extraction.
Conclusion
GASP-ICL demonstrates that sophisticated retrieval and graph-based reasoning can substitute for expensive model training in the domain of deepfake detection. While the method depends on the diversity of its sample library and introduces some inference latency, it provides a scalable and interpretable path forward for multimodal security. Future work will explore extending this graph reasoning to temporal inconsistencies in video deepfakes.
Summary updated