Lightweight Joint Audio-Visual Deepfake Detection via Single-Stream Multi-Modal Learning Framework

Kuiyuan Zhang, Wenjie Pei, Rushi Lan, Yifang Guo, Zhongyun HuaarXiv 2025 · DOI · arXiv

Abstract

Deepfake detection has become increasingly complex as forgeries now encompass both visual and auditory modalities. Traditional multi-stream architectures often fail to capture the fine-grained correlations between these modalities and suffer from high computational costs. This paper introduces SS-AVD, an ultra-lightweight single-stream framework for audio-visual deepfake detection. By employing a Collaborative Audio-Visual Learning (CAVL) block for iterative fusion, SS-AVD achieves state-of-the-art performance with only 0.48 million parameters. Experimental results on benchmark datasets like FakeAVCeleb and DFDC demonstrate its superior accuracy and efficiency, making it highly suitable for resource-constrained environments.

Introduction

The rapid advancement of generative AI has led to the creation of highly realistic deepfakes that manipulate both facial features and speech. Detecting these multi-modal forgeries is critical for maintaining digital trust. Most current detectors rely on multi-stream architectures where video and audio are processed by separate, heavy encoders, with fusion occurring only at the final classification stage. This "late fusion" approach often misses the subtle synchronization between visual cues (like lip movements) and auditory signals. Furthermore, the redundancy of dual encoders leads to massive parameter counts, hindering real-time deployment on mobile devices.

Comparison of Multi-Stream (a) vs. Single-Stream (b) Multi-Modal Learning architectures.

Proposed Method: SS-AVD

The proposed Single-Stream Audio-Visual Detection (SS-AVD) framework addresses these limitations by integrating audio and visual features throughout the network. The core of the architecture is the Collaborative Audio-Visual Learning (CAVL) block, which is stacked throughout the network to facilitate iterative fusion across multiple stages.

Key innovations within the framework include:

  • Visual Preprocessing Module (VPM): This module utilizes spatial attention and depth-wise convolutions to extract long-range spatial dependencies within video frames efficiently with minimal parameters.
  • Self-Attention based Audio-Visual Module (SAAVM): This module tokenizes both visual and audio features, applying a self-attention mechanism to learn joint spatial-temporal correlations. A "channel shrinking" bottleneck is used to minimize the parameter footprint.
  • Multi-Modal Style-Shuffle Augmentation (MMSSA): To ensure the model focuses on semantic content rather than dataset-specific artifacts (e.g., compression noise), styles are randomly shuffled between samples during training.
  • Latent-Shuffle Augmentation (LSA): This adversarial strategy intentionally mismatches audio and video pairs to improve robustness against sophisticated deepfakes where only one modality might be manipulated.

Experiments and Results

SS-AVD was evaluated on three major benchmarks: DF-TIMIT, FakeAVCeleb, and the Deepfake Detection Challenge (DFDC). The results highlight a remarkable balance between efficiency and performance.

Performance comparison of SS-AVD against state-of-the-art methods across multiple datasets, highlighting its extreme parameter efficiency.

As shown in the table above, SS-AVD achieves a 98.31% AUC on the FakeAVCeleb dataset while utilizing only 0.48M parameters. This is significantly more efficient than competing models like VFD (122.7M parameters) or 2+1 Stream (64.5M parameters). On the DF-TIMIT benchmark, the model achieved near-perfect scores. Cross-dataset evaluations further confirmed the model's ability to generalize beyond specific training distributions, focusing on fundamental forgery cues rather than memorizing artifacts.

Conclusion

SS-AVD represents a significant step forward in making multi-modal deepfake detection practical for real-world applications. By shifting from a multi-stream to a single-stream architecture with iterative fusion, the framework achieves high accuracy with a fraction of the parameters required by previous methods. Future research will focus on optimizing hyperparameter tuning and testing the model's resilience against extremely low-quality or heavily filtered "in-the-wild" content.

Summary updated

Abstract

Deepfakes are AI-synthesized multimedia data that may be abused for spreading misinformation. Deepfake generation involves both visual and audio manipulation. To detect audio-visual deepfakes, previous studies commonly employ two relatively independent sub-models to learn audio and visual features, respectively, and fuse them subsequently for deepfake detection. However, this may underutilize the inherent correlations between audio and visual features. Moreover, utilizing two isolated feature learning sub-models can result in redundant neural layers, making the overall model inefficient and impractical for resource-constrained environments. In this work, we design a lightweight network for audio-visual deepfake detection via a single-stream multi-modal learning framework. Specifically, we introduce a collaborative audio-visual learning block to efficiently integrate multi-modal information while learning the visual and audio features. By iteratively employing this block, our single-stream network achieves a continuous fusion of multi-modal features across its layers. Thus, our network efficiently captures visual and audio features without the need for excessive block stacking, resulting in a lightweight network design. Furthermore, we propose a multi-modal classification module that can boost the dependence of the visual and audio classifiers on modality content. It also enhances the whole resistance of the video classifier against the mismatches between audio and visual modalities. We conduct experiments on the DF-TIMIT, FakeAVCeleb, and DFDC benchmark datasets. Compared to state-of-the-art audio-visual joint detection methods, our method is significantly lightweight with only 0.48M parameters, yet it achieves superiority in both uni-modal and multi-modal deepfakes, as well as in unseen types of deepfakes.
BibTeX
@misc{zhang2025lightweight,
  author = {Kuiyuan Zhang and Wenjie Pei and Rushi Lan and Yifang Guo and Zhongyun Hua},
  title = {{Lightweight Joint Audio-Visual Deepfake Detection via Single-Stream Multi-Modal Learning Framework}},
  year = {2025},
  eprint = {2506.07358},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.07358},
}
Figure