TL;DR
This paper introduces a novel multi-modal deepfake detection method that leverages pre-trained foundation models (like CLIP and Whisper) and adapts them using learnable "prompts." By employing a multi-task learning approach and a new frame-level cross-modal feature matching loss, the method achieves state-of-the-art performance and superior generalization capabilities, even with limited training data.
What Problem Does This Paper Address?
The proliferation of sophisticated deepfake videos, which combine manipulated audio and visual elements, poses a significant threat to trust and information integrity. Existing deepfake detection methods often struggle with these multi-modal threats. Many approaches either focus on a single modality (audio or video), tune all parameters on limited datasets leading to poor generalization, or rely on coarse-grained consistency supervision. This paper aims to address these limitations by developing a method that effectively learns from both audio and visual cues simultaneously, leveraging the power of pre-trained models to improve robustness and generalization.
Methods
The core of the proposed method is a multi-task audio-visual prompt learning framework. It utilizes a two-stream architecture, with each stream employing a frozen foundation model: CLIP for the visual branch and Whisper for the audio branch.
- Prompt Learning: Instead of fine-tuning the entire foundation model, the authors introduce sequential visual prompts and short-time audio prompts. These learnable prompts are injected into specific layers of the respective foundation models. This parameter-efficient approach allows the models to adapt to the deepfake detection task while retaining their strong pre-trained capabilities.
- Multi-task Learning: The framework is designed for multi-task learning, meaning it aims to predict both the overall video authenticity and potentially the authenticity of individual modalities. This encourages the model to learn more robust and generalizable features.
- Frame-level Cross-Modal Feature Matching (CMFM) Loss: A key innovation is the CMFM loss function. This loss encourages fine-grained alignment between audio and visual features at the frame level. It works by minimizing the distance between corresponding audio-visual features in real videos and maximizing it for fake videos, effectively capturing subtle audio-visual inconsistencies that are crucial for detecting sophisticated deepfakes.
Datasets
The primary dataset used for training and evaluation is FakeAVCeleb (Khalid et al. 2021b), which contains real videos and various types of deepfake videos with different ethnic groups. The authors also evaluate their method on a subset of KoDF (Kwon et al. 2021) for cross-dataset generalization testing.
Key Insights and Ideas
- Foundation Models are Powerful: Leveraging large, pre-trained foundation models like CLIP and Whisper significantly boosts generalization capabilities, especially when training data is scarce.
- Prompts are Efficient: Parameter-efficient prompt tuning is a viable and effective strategy for adapting powerful foundation models to specific downstream tasks like deepfake detection, avoiding the need for extensive fine-tuning.
- Fine-grained Consistency Matters: Capturing subtle, frame-level audio-visual inconsistencies is crucial for detecting advanced multi-modal deepfakes, and the proposed CMFM loss effectively achieves this.
- Multi-task Learning Enhances Generalization: Training for multiple related tasks (e.g., overall video authenticity and modality-specific authenticity) can lead to more robust and generalizable representations.
Key Results
The proposed method achieves state-of-the-art performance on the FakeAVCeleb dataset, with an accuracy of 99.84% and an AUC of 99.98%. It also demonstrates superior generalization capabilities on unseen manipulation categories and cross-dataset evaluation (KoDF), outperforming existing methods. Notably, the method achieves this with a significantly lower number of learnable parameters (4.4M) compared to other state-of-the-art approaches.
Limitations and Future Work
The authors acknowledge that their method is trained on specific datasets and that further exploration is needed to apply the prompt learning approach to other foundation models. They also suggest future work could focus on solving more complex problems like deep forgery localization. The current work primarily focuses on detecting the authenticity of entire videos, and extending this to pinpointing the exact manipulated regions or segments could be a valuable next step.
Summary updated