Statistics-Aware Audio-Visual Deepfake Detector

Marcella Astrid, Enjie Ghorbel, Djamila Aouada

Abstract

Deepfake detection has increasingly relied on multi-modal analysis to identify synchronization mismatches between audio and visual tracks. However, existing methods often fail to capture the underlying statistical differences between real and fake media distributions. This paper presents the Statistics-aware Audio-visual Deepfake Detector (SADD), a lightweight model that leverages a novel statistics-aware loss function. By analyzing feature distribution means and standard deviations, SADD achieves state-of-the-art performance (96.69% AUC) with reduced computational overhead compared to traditional deep models.

Introduction

As deepfake technology advances, the need for robust detection becomes critical. Current multi-modal detectors typically use contrastive learning to measure the distance between isolated audio and visual feature pairs. A significant limitation of this approach is that while individual pairs might be pushed apart, the overall feature distributions for real and fake media often remain indistinguishable. This makes robust classification difficult and sensitive to thresholding.

SADD addresses these challenges by moving beyond point-to-point distances to distribution-level statistics. The core insight is that forcing the means of audio-visual feature distributions apart for fakes—and together for reals—creates a more reliable decision boundary.

Methodology

The SADD architecture, illustrated in Figure 1, introduces several optimizations to the standard detection pipeline to improve both speed and accuracy.

The SADD architecture featuring visual and audio extractors, distance loss, and the novel statistics-aware loss function.

Architectural Innovations

  1. Shallower Feature Extraction: SADD employs shallower convolutional networks than standard forensic models. This preserves low-level signal artifacts often lost in deeper layers while reducing the parameter count from 122.78M to 107.10M.
  2. Raw Audio Processing: Instead of converting audio into 2D Mel-spectrograms, SADD ingests raw 1D audio waveforms. This prevents artifact loss during conversion and eliminates the need for manual hyperparameter tuning.
  3. Statistics-aware Loss ($\mathcal{L}^s$): This novel loss function calculates the distance between the means (first-order statistics) of audio and visual features, forcing them apart for fakes and together for reals.
  4. Adaptive Margin: SADD dynamically sets the contrastive loss margin using the standard deviations of the features. This ensures that wider distributions are pushed further apart to maintain distinctness.

Experimental Results

The model was evaluated primarily on the DeepFake Detection Challenge (DFDC) dataset and tested for zero-shot generalization on the FakeAVCeleb dataset.

Performance Comparison

SADD achieves state-of-the-art results among audio-visual (AV) detectors. As shown in Figure 2, SADD outperforms previous benchmarks like MDS and AvoiD-DF.

Comparison of SADD against state-of-the-art unimodal and multi-modal deepfake detection methods.

Key findings include:

  • Accuracy: SADD reached 96.69% AUC on DFDC, a significant improvement over the 90.66% achieved by the MDS baseline.
  • Efficiency: By utilizing a shallower network, the model simultaneously increased accuracy while reducing computational complexity.
  • Data Efficiency: The statistics-aware loss provided a 2% AUC boost in data-constrained training scenarios, demonstrating its robustness.

Discussion and Future Work

While SADD significantly improves detection accuracy and efficiency, cross-dataset generalization remains a challenge. On the FakeAVCeleb dataset, SADD improved baseline performance but still trails behind some heavier, transformer-based models. Future research will explore integrating transformer architectures and more complex distribution metrics, such as Kullback–Leibler (KL) divergence, to further enhance robustness.

Summary updated

BibTeX
@inproceedings{astrid2024statistics,
  author = {Marcella Astrid and Enjie Ghorbel and Djamila Aouada},
  title = {{Statistics-Aware Audio-Visual Deepfake Detector}},
  booktitle = {ICIP},
  year = {2024},
  url = {https://openreview.net/forum?id=OChiH1rTv8},
}

Loading related content...

Figure