Abstract
Deepfake detection has become increasingly complex as forgeries now encompass both visual and auditory modalities. Traditional multi-stream architectures often fail to capture the fine-grained correlations between these modalities and suffer from high computational costs. This paper introduces SS-AVD, an ultra-lightweight single-stream framework for audio-visual deepfake detection. By employing a Collaborative Audio-Visual Learning (CAVL) block for iterative fusion, SS-AVD achieves state-of-the-art performance with only 0.48 million parameters. Experimental results on benchmark datasets like FakeAVCeleb and DFDC demonstrate its superior accuracy and efficiency, making it highly suitable for resource-constrained environments.
Introduction
The rapid advancement of generative AI has led to the creation of highly realistic deepfakes that manipulate both facial features and speech. Detecting these multi-modal forgeries is critical for maintaining digital trust. Most current detectors rely on multi-stream architectures where video and audio are processed by separate, heavy encoders, with fusion occurring only at the final classification stage. This "late fusion" approach often misses the subtle synchronization between visual cues (like lip movements) and auditory signals. Furthermore, the redundancy of dual encoders leads to massive parameter counts, hindering real-time deployment on mobile devices.
Proposed Method: SS-AVD
The proposed Single-Stream Audio-Visual Detection (SS-AVD) framework addresses these limitations by integrating audio and visual features throughout the network. The core of the architecture is the Collaborative Audio-Visual Learning (CAVL) block, which is stacked throughout the network to facilitate iterative fusion across multiple stages.
Key innovations within the framework include:
- Visual Preprocessing Module (VPM): This module utilizes spatial attention and depth-wise convolutions to extract long-range spatial dependencies within video frames efficiently with minimal parameters.
- Self-Attention based Audio-Visual Module (SAAVM): This module tokenizes both visual and audio features, applying a self-attention mechanism to learn joint spatial-temporal correlations. A "channel shrinking" bottleneck is used to minimize the parameter footprint.
- Multi-Modal Style-Shuffle Augmentation (MMSSA): To ensure the model focuses on semantic content rather than dataset-specific artifacts (e.g., compression noise), styles are randomly shuffled between samples during training.
- Latent-Shuffle Augmentation (LSA): This adversarial strategy intentionally mismatches audio and video pairs to improve robustness against sophisticated deepfakes where only one modality might be manipulated.
Experiments and Results
SS-AVD was evaluated on three major benchmarks: DF-TIMIT, FakeAVCeleb, and the Deepfake Detection Challenge (DFDC). The results highlight a remarkable balance between efficiency and performance.
As shown in the table above, SS-AVD achieves a 98.31% AUC on the FakeAVCeleb dataset while utilizing only 0.48M parameters. This is significantly more efficient than competing models like VFD (122.7M parameters) or 2+1 Stream (64.5M parameters). On the DF-TIMIT benchmark, the model achieved near-perfect scores. Cross-dataset evaluations further confirmed the model's ability to generalize beyond specific training distributions, focusing on fundamental forgery cues rather than memorizing artifacts.
Conclusion
SS-AVD represents a significant step forward in making multi-modal deepfake detection practical for real-world applications. By shifting from a multi-stream to a single-stream architecture with iterative fusion, the framework achieves high accuracy with a fraction of the parameters required by previous methods. Future research will focus on optimizing hyperparameter tuning and testing the model's resilience against extremely low-quality or heavily filtered "in-the-wild" content.
Summary updated