Abstract
Video synthesis methods rapidly improved in recent years, allowing easy creation of synthetic humans. This poses a problem, especially in the era of social media, as synthetic videos of speaking humans can be used to spread misinformation in a convincing manner. Thus, there is a pressing need for accurate and robust deepfake detection methods, that can detect forgery techniques not seen during training. In this work, we explore whether this can be done by leveraging a multi-modal, out-of-domain backbone trained in a self-supervised manner, adapted to the video deepfake domain. We propose FakeOut; a novel approach that relies on multi-modal data throughout both the pre-training phase and the adaption phase. We demonstrate the efficacy and robustness of FakeOut in detecting various types of deepfakes, especially manipulations which were not seen during training. Our method achieves state-of-the-art results in cross-dataset generalization on audio-visual datasets. This study shows that, perhaps surprisingly, training on out-of-domain videos (i.e., not especially featuring speaking humans), can lead to better deepfake detection systems.
The Challenge of Generalization
Deepfake technology is evolving at breakneck speed, making it increasingly difficult to distinguish between real and synthetic humans. While many detection models perform well on "in-distribution" data (forgeries similar to those seen during training), their performance often plummets when faced with new manipulation techniques or different datasets.
The core challenge is generalization. Most existing methods rely on supervised learning with massive annotated datasets or specific facial pre-processing (like lip-region extraction). This creates a "coupling" between the detector and specific facial attributes or dataset-specific artifacts. Furthermore, many detectors ignore the auditory modality entirely, missing out on crucial "mismatch" cues between what we see and what we hear.
The FakeOut Methodology
The authors propose FakeOut, a multi-modal approach that breaks the reliance on domain-specific pre-training by leveraging general-purpose video representations.
1. Out-of-Domain Backbone
Instead of pre-training on faces, FakeOut utilizes a Multi-modal Versatile Network (MMV) backbone. This backbone is trained in a self-supervised manner on massive, general-purpose datasets like HowTo100M (instructional clips) and AudioSet. The central hypothesis is that learning task-invariant representations from general videos helps the model generalize better to the specific task of deepfake detection than training on talking heads alone.
2. Multi-modal Fusion and Pipeline
FakeOut processes both video and audio streams to identify inconsistencies. The video stream uses a Temporal Shift Module (TSM-50), while the audio stream utilizes a ResNet-50 architecture to process log-MEL spectrograms. These representations are concatenated into a multi-modal representation and fed into a classification head.
To ensure robustness, the system includes an end-to-end face tracking and alignment pipeline. This pipeline is designed to handle multi-person scenarios and filter out "distractors" (such as dog-mask filters) that often cause false positives in other detectors.
3. Data Enrichment
Many popular deepfake datasets are silent, which limits multi-modal research. The authors developed an "enrichment process" to map manipulated videos back to their original YouTube sources, re-inserting the correct audio to enable multi-modal training on standard benchmarks like FaceForensics++ (FF++).
Key Insights
- The Power of "Irrelevant" Data: Surprisingly, training on videos of people playing guitar or cooking leads to a more robust deepfake detector than training only on talking heads. This suggests the model learns fundamental spatio-temporal and audio-visual correlations that are harder to "fool" with specific forgery artifacts.
- Audio as a Critical Signal: The interaction between audio and video is a powerful indicator of forgery. Even when the visual forgery is near-perfect, subtle misalignments with the audio can reveal the manipulation.
- Fine-tuning is Essential: While the pre-trained backbone provides a strong starting point, the authors found that fine-tuning the entire network (rather than just the final layer) is necessary to achieve top-tier generalization.
Results and Performance
FakeOut achieves state-of-the-art results in cross-dataset generalization, particularly on the most challenging benchmarks:
- DFDC (Filtered): FakeOut achieved a 78.8% ROC-AUC, outperforming major competitors like RealForensics (75.9%) and LipForensics (73.5%).
- Cross-Dataset Generalization: When fine-tuned on enriched FF++ and tested on FaceShifter and DeeperForensics, FakeOut reached near-perfect scores of 99.8% and 99.9% ROC-AUC, respectively.
- Confidence: The model demonstrated a high degree of confidence, with prediction scores clearly clustering near 0 (real) or 1 (fake), whereas simpler "linear probing" methods remained uncertain.
Conclusion
FakeOut demonstrates that out-of-domain self-supervision is a potent tool for deepfake detection. By moving away from face-specific pre-training and embracing multi-modal signals, the model achieves superior generalization across diverse forgery techniques. Future work may explore attention-based mechanisms for more sophisticated cross-modal fusion and audio-specific augmentations to defend against voice cloning and other auditory forgeries.
Summary updated