TL;DR
DF-MoE is a novel audio-visual deepfake detection framework that achieves exceptional generalization by aggregating high-level semantic cues (like gaze, head pose, and heart rate) from frozen, pre-trained models using a sparse Mixture-of-Experts (MoE) transformer. Combined with a new anchor-based contrastive loss, it consistently outperforms state-of-the-art detectors across multiple challenging open-set and cross-domain benchmarks.
What Problem Does This Paper Address?
As generative AI continues to advance, creating photorealistic deepfakes has become remarkably easy. While existing deepfake detectors boast up to 99% accuracy on standard benchmarks, they suffer from a fatal flaw: they fail to generalize. Most detectors overfit to the low-level pixel artifacts or specific blending patterns of the generative models they were trained on. When faced with a deepfake generated by a newer, unseen method, their performance plummets.
The authors of this paper tackle the challenge of out-of-domain generalization. Instead of training a model to spot specific pixel-level anomalies, they propose looking at high-level, multi-modal behavioral inconsistencies—such as unnatural head movements, irregular gaze patterns, missing physiological signals, or audio-visual mismatches—which persist across various deepfake generation methods.
Methods
The proposed architecture, DF-MoE (Deepfake Detection via Mixture-of-Experts), splits the detection task into two distinct phases: feature extraction and temporal/cross-modal consistency modeling.
- Frozen High-Level Feature Extraction: To prevent the network from overfitting to dataset-specific artifacts, the authors assemble a suite of frozen, pre-trained models to extract diverse semantic cues:
- Visual Cues: Head pose (HopeNet), eye gaze tracking (ResNet-34), face segmentation (BiSeNet), remote photoplethysmography (rPPG for heart rate tracking), and facial expressions (EfficientNet-B0).
- Audio Cues: Speech embeddings (Whisper-Tiny) and audio emotion recognition.
- Multimodal Cues: Lip-sync consistency (via a custom AV transformer) and deep audio-visual features from a self-supervised baseline (AVFF).
- Specialized Adapters & Sparse MoE: These diverse features are projected into a shared embedding space using specialized adapters. The resulting tokens are fed into a lightweight, trainable Sparse Mixture-of-Experts (MoE) transformer. A gating network routes each token to the top-$k$ ($k=2$) most relevant experts out of $n=6$ available experts, allowing different parts of the network to specialize in specific types of behavioral anomalies.
- Contractive-Repulsive Objective (CRO) Loss: To further boost generalization, the authors introduce a novel contrastive loss function. Instead of performing computationally expensive pairwise hard-sample mining, CRO utilizes learnable class anchors (one for 'Real', one for 'Fake'). It minimizes the distance between a sample's feature vector and its corresponding class anchor (intra-class contraction) while enforcing a strict margin between the two anchors (inter-class repulsion).
Datasets
The framework was rigorously evaluated across five distinct deepfake benchmarks:
- AVLips: Focused on lip-sync forgery detection.
- MAVOS-DD: A massive multilingual benchmark used for both closed-set and open-set (unseen languages and generation methods) evaluations.
- PolyGlotFake: A multilingual, multimodal dataset featuring text-to-speech and voice conversion.
- BioDeepAV: A dataset containing advanced generative techniques like NeRF-based synthesis, Gaussian Splatting, and diffusion models.
- FakeAVCeleb: A highly imbalanced dataset, which the authors balanced by adding 19,000 real videos from VoxCeleb2 (referred to as Vox+FakeAVCeleb).
Key Insights and Ideas
- Behavior Over Pixels: By forcing the model to rely on high-level semantic features (e.g., whether a person's heart rate signal matches their facial micro-movements), the system becomes agnostic to the underlying generative architecture.
- The Power of "Frozen" Weights: Keeping the upstream feature extractors strictly frozen acts as a powerful regularizer. The model simply cannot learn to detect low-level textures, forcing the trainable MoE layers to focus entirely on high-level orchestration and consistency.
- Inherent Interpretability: Because the MoE uses a gating mechanism with attention weights, the model provides a degree of transparency. Researchers can inspect the attention scores to see why a video was flagged as a fake (e.g., detecting that a video was deemed fake primarily due to irregular gaze or poor lip-syncing).
Key Results
DF-MoE achieved state-of-the-art performance across all evaluated datasets, showing a massive leap in generalization capability:
- Open-Set Robustness: On the MAVOS-DD "open-set full" benchmark (unseen methods and languages), DF-MoE achieved 0.98 AUC and 92.95% accuracy, significantly outperforming the closest competitor, AVFF (0.93 AUC, 84.85% accuracy).
- Cross-Dataset Generalization: When trained on MAVOS-DD and tested on the highly challenging Vox+FakeAVCeleb dataset, DF-MoE achieved an AUC of 0.88, compared to just 0.70 for AVFF and 0.57 for TALL.
- Efficiency: Despite integrating multiple models, the lightweight MoE backbone ensures near real-time inference speeds on a single modern GPU (e.g., Nvidia RTX 5090).
Limitations and Future Work
The authors identify two primary limitations in their current framework:
- Feature Dominance: In certain scenarios, highly performant multimodal features (like AVFF or the AV Transformer) can overshadow subtler cues (like gaze or rPPG), leading to misclassifications if the dominant features fail.
- Upstream Sensitivity: Because the feature extractors are frozen, the final classification head is entirely dependent on the accuracy of these upstream models. For instance, an inaccurate emotion recognition step can propagate errors downstream and cause false positives.
Future Directions: The authors suggest exploring additional pre-trained models to capture even more complementary signals and adapting this MoE architecture to other complex multi-modal tasks, such as video planning.
Summary updated