TL;DR
This paper introduces a novel method for deepfake source detection by analyzing subtle motion artifacts amplified through a dual motion magnification technique. By combining traditional and deep learning-based motion magnification, the authors achieve state-of-the-art results in identifying not only if a video is fake but also which generative model created it.
What Problem Does This Paper Address?
The proliferation of deep generative models has led to increasingly realistic deepfakes, making it difficult to distinguish them from authentic videos. While existing deepfake detection methods often focus on identifying general artifacts, this paper tackles a more challenging problem: source detection. This involves identifying the specific generative model used to create a deepfake. The authors argue that subtle, sub-muscular motion patterns in human faces, which are difficult for current generative models to perfectly replicate, serve as a unique "watermark" of authenticity. They aim to exploit discrepancies in how these subtle motions are rendered in real versus fake videos, and how these discrepancies vary across different deepfake generators.
Methods
The core of the proposed method lies in motion magnification, a technique used to amplify subtle movements in videos. The authors employ a dual approach:
- Deep Motion Magnification: This leverages a CNN-based approach to learn and amplify motion patterns, making it robust to complex motions.
- Phase-Based Motion Magnification: This traditional method, based on Eulerian approaches and steerable pyramids, is particularly effective at capturing small, local motions by analyzing phase variations.
By combining these two techniques, the authors create a richer representation of motion that captures both general movement and fine-grained details. This dual representation is then fed into a 3D Convolutional Neural Network (3D CNN). The 3D CNN is chosen for its ability to process spatio-temporal data effectively. The system involves several steps:
- Frame Sampling: Fixed-window samples are extracted from videos to ensure consistent input.
- Face Detection and Alignment: Faces are detected and aligned to standardize the input for motion analysis.
- Motion Magnification: Both deep and phase-based magnification are applied to the aligned faces.
- Input Aggregation: The outputs from both magnification methods are combined into a tensor.
- 3D CNN Training: The 3D CNN is trained to classify videos into different source categories (real or specific generators).
- Prediction Aggregation: Predictions from individual samples are aggregated to provide a final video-level classification.
Datasets
The paper utilizes two multi-source datasets for training and evaluation:
- FaceForensics++ (FF): This dataset contains videos from 5 generators.
- FakeAVCeleb (FAVC): This dataset includes videos from 3 generators.
Additionally, they use an "in-the-wild" dataset from [20] for cross-model evaluation to assess generalization capabilities.
Key Insights and Ideas
- Motion as a "Human Watermark": The paper posits that subtle sub-muscular facial movements are a fundamental characteristic of authentic human motion that current deepfake generators struggle to perfectly replicate.
- Dual Motion Magnification Synergy: Combining deep and phase-based motion magnification provides a more comprehensive representation of motion artifacts, revealing differences that single methods might miss. Deep magnification captures general motion, while phase-based magnification excels at highlighting subtle, localized movements.
- Generative Residue Amplification: Deepfake generators introduce "noise" or artifacts. When motion is magnified in deepfakes, this generative noise gets amplified, leading to deformations and inconsistencies that differ from the amplified natural motion in real videos.
- Source-Specific Motion Signatures: The amplified motion artifacts exhibit distinct patterns for different generative models, enabling source detection.
Key Results
The proposed approach achieves impressive results:
- FaceForensics++ (FF): 97.77% video source detection accuracy.
- FakeAVCeleb (FAVC): 94.03% video source detection accuracy.
- The method outperforms the best prior deepfake source detector by 4.08%.
- The model demonstrates good generalization, achieving 92.64% real class accuracy on an "in-the-wild" dataset with unknown generators.
- Ablation studies confirm the importance of both motion magnification components and the chosen 3D CNN architecture.
Limitations and Future Work
- Computational Cost: Motion magnification, especially the phase-based method, is computationally intensive, though it's an offline process.
- Demographics Analysis: While the paper analyzes performance across genders and skin tones, it notes discrepancies, particularly for African males and Asian women, suggesting that deepfake generators may not create these demographics with the same fidelity. This indicates a potential area for bias that needs further investigation and mitigation.
- Future Work: The authors suggest further analysis of motion in conjunction with other signals (audio, gaze, gesture) in a multi-modal setting. They also aim to explore open-set scenarios with entirely new generators and investigate how to retrain models for them.
Summary updated