Abstract
Deepfake detection has increasingly relied on multi-modal analysis to identify synchronization mismatches between audio and visual tracks. However, existing methods often fail to capture the underlying statistical differences between real and fake media distributions. This paper presents the Statistics-aware Audio-visual Deepfake Detector (SADD), a lightweight model that leverages a novel statistics-aware loss function. By analyzing feature distribution means and standard deviations, SADD achieves state-of-the-art performance (96.69% AUC) with reduced computational overhead compared to traditional deep models.
Introduction
As deepfake technology advances, the need for robust detection becomes critical. Current multi-modal detectors typically use contrastive learning to measure the distance between isolated audio and visual feature pairs. A significant limitation of this approach is that while individual pairs might be pushed apart, the overall feature distributions for real and fake media often remain indistinguishable. This makes robust classification difficult and sensitive to thresholding.
SADD addresses these challenges by moving beyond point-to-point distances to distribution-level statistics. The core insight is that forcing the means of audio-visual feature distributions apart for fakes—and together for reals—creates a more reliable decision boundary.
Methodology
The SADD architecture, illustrated in Figure 1, introduces several optimizations to the standard detection pipeline to improve both speed and accuracy.
Architectural Innovations
- Shallower Feature Extraction: SADD employs shallower convolutional networks than standard forensic models. This preserves low-level signal artifacts often lost in deeper layers while reducing the parameter count from 122.78M to 107.10M.
- Raw Audio Processing: Instead of converting audio into 2D Mel-spectrograms, SADD ingests raw 1D audio waveforms. This prevents artifact loss during conversion and eliminates the need for manual hyperparameter tuning.
- Statistics-aware Loss ($\mathcal{L}^s$): This novel loss function calculates the distance between the means (first-order statistics) of audio and visual features, forcing them apart for fakes and together for reals.
- Adaptive Margin: SADD dynamically sets the contrastive loss margin using the standard deviations of the features. This ensures that wider distributions are pushed further apart to maintain distinctness.
Experimental Results
The model was evaluated primarily on the DeepFake Detection Challenge (DFDC) dataset and tested for zero-shot generalization on the FakeAVCeleb dataset.
Performance Comparison
SADD achieves state-of-the-art results among audio-visual (AV) detectors. As shown in Figure 2, SADD outperforms previous benchmarks like MDS and AvoiD-DF.
Key findings include:
- Accuracy: SADD reached 96.69% AUC on DFDC, a significant improvement over the 90.66% achieved by the MDS baseline.
- Efficiency: By utilizing a shallower network, the model simultaneously increased accuracy while reducing computational complexity.
- Data Efficiency: The statistics-aware loss provided a 2% AUC boost in data-constrained training scenarios, demonstrating its robustness.
Discussion and Future Work
While SADD significantly improves detection accuracy and efficiency, cross-dataset generalization remains a challenge. On the FakeAVCeleb dataset, SADD improved baseline performance but still trails behind some heavier, transformer-based models. Future research will explore integrating transformer architectures and more complex distribution metrics, such as Kullback–Leibler (KL) divergence, to further enhance robustness.
Summary updated