A Spatial-Frequency Aware Multi-Scale Fusion Network for Real-Time Deepfake Detection

Libo Lv, Tianyi Wang, Mengxiao Huang, Ruixia Liu, Yinglong Wang

Abstract

With the rapid advancement of real-time deepfake generation techniques, forged content is becoming increasingly realistic and widespread across applications like video conferencing and social media. Although state-of-the-art detectors achieve high accuracy on standard benchmarks, their heavy computational cost hinders real-time deployment in practical applications. To address this, we propose the Spatial-Frequency Aware Multi-Scale Fusion Network (SFMFNet), a lightweight yet effective architecture for real-time deepfake detection. We design a spatial-frequency hybrid aware module that jointly leverages spatial textures and frequency artifacts through a gated mechanism, enhancing sensitivity to subtle manipulations. A token-selective cross attention mechanism enables efficient multi-level feature interaction, while a residual-enhanced blur pooling structure helps retain key semantic cues during downsampling. Experiments on several benchmark datasets show that SFMFNet achieves a favorable balance between accuracy and efficiency, with strong generalization and practical value for real-time applications.

Introduction

As deepfake generation becomes more sophisticated, forged content is moving beyond static images into real-time streams like social media live-feeds and video calls. While current state-of-the-art (SOTA) detectors are highly accurate, they are often computationally heavy, relying on massive transformer architectures or deep CNNs that struggle to run at high frame rates on consumer hardware.

Furthermore, many existing models treat spatial clues (like unnatural skin textures) and frequency clues (mathematical artifacts left by generative AI) as separate entities. Neglecting the joint interaction between these two domains leads to a loss of sensitivity to subtle manipulations, especially when attempting to maintain a lightweight model profile.

Methodology

The Spatial-Frequency Aware Multi-Scale Fusion Network (SFMFNet) addresses these challenges through three core innovations designed for efficiency and sensitivity.

Overview of the SFMFNet architecture, highlighting the SFHA, DBP, and TSCA modules.

Spatial-Frequency Hybrid Aware (SFHA) Module

This module processes features in two parallel branches to capture a holistic view of the input. One branch utilizes the Haar Wavelet Transform to extract frequency-domain artifacts, while the other employs Coordinate Attention to capture spatial textures. These features are fused via a dynamic gating mechanism that highlights the most suspicious regions of an image, allowing the model to "decide" which domain provides the strongest evidence of forgery for a specific patch.

Token-Selective Cross Attention (TSCA)

Standard cross-attention mechanisms are computationally expensive due to quadratic complexity relative to pixel count. TSCA mitigates this by sampling and selecting only the most salient "tokens" (feature patches) before performing attention. This allows high-level semantic information to guide low-level detail extraction efficiently, providing the global reach of a Transformer with the speed of a lightweight CNN.

Downsample with BlurPool (DBP)

Standard downsampling operations like MaxPool can cause aliasing, which often destroys the subtle high-frequency artifacts critical for deepfake detection. The DBP module uses blur pooling and a residual path to retain structural and edge details during the feature extraction process, preserving the "fingerprints" left by generative models.

Experimental Setup

The researchers evaluated SFMFNet on several major deepfake benchmarks to test both raw accuracy and generalization:

  • FaceForensics++ (FF++): The primary benchmark containing four different forgery techniques (Deepfakes, Face2Face, FaceSwap, NeuralTextures).
  • Generalization Sets: To assess performance on "unseen" forgeries, the model was tested on Celeb-DF v2, DeepFake Detection (DFD), DFDCP, DFDC, and UADFV.

Results and Discussion

SFMFNet strikes an impressive balance between accuracy and computational efficiency.

Performance comparison (AUC) of SFMFNet against state-of-the-art models on the FaceForensics++ (FF++) dataset.

Performance and Efficiency

As shown in the experimental results, SFMFNet achieved an average AUC of 0.9867 on the FF++ dataset, outperforming heavier models like Xception and RegNet. Crucially, the model requires only 1.27 GFLOPs and 6.64M parameters. This represents an 87.9% reduction in computation compared to the UCF model and 78.9% less than FFD, while maintaining superior performance.

Generalization

On the challenging Celeb-DF v2 dataset, SFMFNet achieved an AUC of 0.8111, demonstrating a robust ability to detect forgeries it was not specifically trained on, surpassing many existing lightweight competitors.

Conclusion

While SFMFNet demonstrates excellent results, real-world deployment faces ongoing challenges. The rapid emergence of new deepfake generation techniques requires constant adaptation. Additionally, extreme video compression in low-bandwidth scenarios can "smear" frequency artifacts, potentially impacting detection. Future work will focus on further optimizing the token selection process to enable even leaner models suitable for mobile or edge-device deployment.

Summary updated

Abstract

With the rapid advancement of real-time deepfake generation techniques, forged content is becoming increasingly realistic and widespread across applications like video conferencing and social media. Although state-of-the-art detectors achieve high accuracy on standard benchmarks, their heavy computational cost hinders real-time deployment in practical applications. To address this, we propose the Spatial-Frequency Aware Multi-Scale Fusion Network (SFMFNet), a lightweight yet effective architecture for real-time deepfake detection. We design a spatial-frequency hybrid aware module that jointly leverages spatial textures and frequency artifacts through a gated mechanism, enhancing sensitivity to subtle manipulations. A token-selective cross attention mechanism enables efficient multi-level feature interaction, while a residual-enhanced blur pooling structure helps retain key semantic cues during downsampling. Experiments on several benchmark datasets show that SFMFNet achieves a favorable balance between accuracy and efficiency, with strong generalization and practical value for real-time applications.
BibTeX
@inproceedings{lv2025spatial,
  author = {Libo Lv and Tianyi Wang and Mengxiao Huang and Ruixia Liu and Yinglong Wang},
  title = {{A Spatial-Frequency Aware Multi-Scale Fusion Network for Real-Time Deepfake Detection}},
  booktitle = {PRCV},
  year = {2025},
  doi = {10.1007/978-981-95-5676-2\_36},
}
Figure