Beyond Frame-Level Accuracy: Analyzing Video-Level Stability in Transformer-Based Deepfake Forensics

Syed Azmul Hasan, Mowmita Pervin Hera, Afroza Lona Mim, M M Jihad Hossain, Hazrat Ali, Mohammad Farhad BulbularXiv 2026 · DOI · arXiv

TL;DR

This paper provides a rigorous, controlled comparison between Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) for deepfake detection. The authors find that while CNNs perform well on individual frames, Transformers offer significantly better "video-level stability"—producing more consistent and reliable results when predictions are aggregated across an entire video—though cross-dataset generalization remains a significant challenge for all architectures.

What Problem Does This Paper Address?

The field of deepfake forensics is currently a "Wild West" of evaluation metrics. Most papers report high performance, but they use heterogeneous preprocessing pipelines, inconsistent data splits, and model-specific training protocols. This makes it nearly impossible to tell if a performance boost comes from a superior architecture or just a better data augmentation strategy.

Furthermore, most research focuses on frame-level accuracy. In the real world, we care about the video-level verdict. CNNs often capture local texture artifacts effectively but can be "jittery," producing wildly different scores for adjacent frames. This paper investigates whether the global receptive field of Transformers provides more stable, reliable video-level forensics.

Methods

The authors implement a unified, face-centric detection framework to ensure a "level playing field" for three backbones: ResNet-18, EfficientNet-B0, and ViT-B/16.

Fig. 1: Overall workflow of the proposed deepfake detection pipeline.

Key methodological choices include:

  • Unified Preprocessing: All models use MTCNN for face detection, 224x224 cropping, and identical data augmentations.
  • Video Aggregation: The final verdict is reached by simple mean pooling of frame-level probabilities.
  • Statistical Rigor: Unlike many papers that report a single "best" run, this work uses three independent random seeds and reports bootstrapped 95% confidence intervals (CI).
  • Architecture Focus: The ViT-B/16 model uses 12 transformer encoder layers with multi-head self-attention to capture long-range spatial dependencies across the face.
Fig. 2: Architecture of ViT-B/16 used for deepfake detection.

Datasets

The study utilizes two primary benchmarks, both of which are class-imbalanced (skewed toward "fake" samples):

  1. FaceForensics++ (FF++): A standard dataset for controlled manipulation studies.
  2. Celeb-DF v2: A more challenging, high-quality dataset with fewer visual artifacts and more sophisticated blending.
Dataset split statistics and class balance

Key Insights and Ideas

  • Global Attention = Stability: The "aha moment" is that global self-attention allows the Transformer to integrate evidence across multiple facial regions (eyes, mouth, nose) simultaneously. This makes it less sensitive to isolated texture artifacts that might trip up a CNN on a single frame, leading to more stable predictions across a video sequence.
  • The Generalization Gap is Asymmetric: Models trained on the diverse manipulations of FF++ generalize better to Celeb-DF than vice versa. This suggests that training on a broader "palette" of forgery types is more important than the raw quality of the fakes.
  • Calibration Failure vs. Signal Loss: When testing on unseen datasets, accuracy often plummets while the Area Under the Curve (AUC) remains moderately high. This indicates a calibration failure: the model still "knows" which videos are fakes relative to others, but its probability scores have shifted such that the standard 0.5 decision threshold no longer works.

Key Results

The ViT-B/16 outperformed the CNN baselines in video-level stability within its training domain:

  • Celeb-DF v2: Achieved 92.8% video-level accuracy and an AUC of 0.950 (3-seed bootstrapped).
  • FaceForensics++: Achieved 77.4% accuracy and an AUC of 0.840.
  • Efficiency: Despite the complexity of Transformers, the ViT-B/16 achieved 73.7–84.75 FPS on a single Tesla T4 GPU, making it viable for practical deployment.
Fig. 4: ROC curve of ViT-B/16 on Celeb-DF v2.

Qualitative analysis via attention maps showed that the Transformer consistently focuses on semantically meaningful areas like blending boundaries around the mouth and nose, rather than background noise.

Limitations and Future Work

  • Cross-Domain Generalization: Even with global attention, the "generalization gap" remains large. A model trained on Celeb-DF v2 collapsed to near-chance accuracy when tested on FF++.
  • Temporal Modeling: The authors tested a Gated Recurrent Unit (GRU) for temporal aggregation, but it failed to outperform simple mean pooling. They suggest that more sophisticated spatiotemporal Transformers are needed.
  • Scope: The study was limited to GAN and autoencoder-based fakes. Future work is needed to address newer threats like diffusion-based synthesis and adversarial perturbations.

Summary updated

Abstract

Deepfake videos generated by advanced generative models pose serious threats to digital security, misinformation control, and media forensics. Although numerous detection methods have been proposed, their reported performances are often based on heterogeneous preprocessing pipelines, inconsistent data splits, and model-specific training protocols, hindering fair architectural comparison. This work presents a controlled and reproducible comparative study of convolutional neural networks (ResNet-18 and EfficientNet-B0) and a transformer-based architecture (ViT-B/16) within a unified, face-centric deepfake detection framework. All models are trained using identical preprocessing, augmentation, optimization settings, and data splits on Celeb-DF v2 and FaceForensics++, and results are validated across three independent random seeds with bootstrapped confidence intervals. Experimental results show that while CNN-based models perform competitively at the frame level, their performance degrades after video-level aggregation. In contrast, ViT-B/16 demonstrates more stable video-level performance, achieving 92.8\% [95\% CI: 91.7--94.0] video-level accuracy and an AUC of 0.950 [95\% CI: 0.934--0.964] on Celeb-DF v2 (3-seed bootstrapped). Cross-dataset evaluation across the same three random seeds reveals a large, asymmetric generalization gap (AUC 0.79 FF++$\rightarrow$Celeb-DF vs. 0.64 Celeb-DF$\rightarrow$FF++), indicating that in-domain video-level stability does not transfer to unseen manipulation domains. These findings indicate that global self-attention improves robustness and video-level stability for deepfake detection within a given data domain, while highlighting the need for further work on cross-domain generalization.
BibTeX
@misc{hasan2026beyond,
  author = {Syed Azmul Hasan and Mowmita Pervin Hera and Afroza Lona Mim and M M Jihad Hossain and Hazrat Ali and Mohammad Farhad Bulbul},
  title = {{Beyond Frame-Level Accuracy: Analyzing Video-Level Stability in Transformer-Based Deepfake Forensics}},
  year = {2026},
  eprint = {2611.10249},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10249},
}
Figure