FInfer: Frame Inference-Based Deepfake Detection for High-Visual-Quality Videos

Juan Hu, Xin Liao, Jinwen Liang, Wenbo Zhou, Zheng QinAAAI 2022 · OpenReview

As generative AI advances, the "uncanny valley" of Deepfakes is rapidly closing. Early Deepfakes were easy to spot—they blinked weirdly, had blurry chin lines, or suffered from strange color flickering. However, modern high-visual-quality Deepfakes have largely eliminated these obvious artifacts, making them nearly indistinguishable from real footage to the naked eye.

In a paper presented at AAAI-22, researchers Juan Hu and colleagues introduced FInfer (Frame Inference), a framework that stops looking for what a fake face looks like and starts looking at how a face behaves over time.

The Challenge of High-Quality Fakes

Most Deepfake detectors are "artifact-hungry." They look for microscopic inconsistencies in pixels, frequency distributions, or biological signals. But as generative models like NeuralTextures improve, these spatial artifacts vanish.

Comparison of low-visual-quality vs. high-visual-quality deepfakes, showing the disappearance of obvious artifacts.

The authors identify three major hurdles in the current landscape:

  1. Vanishing Traces: High-quality fakes have almost no perceptible distortions or color mismatches.
  2. Overfitting: Models trained on one dataset’s specific artifacts often fail when tested on a different dataset.
  3. Lack of Interpretability: Many "black-box" detectors offer no theoretical explanation for why they flag a video as fake.

The FInfer Methodology

FInfer shifts the focus from spatial artifacts to temporal predictability. The framework operates on the insight that real human expressions are temporally consistent and "predictable," whereas synthetic faces—despite looking perfect in a single frame—lack the natural coherence required to sustain a believable sequence over time. The process consists of four main stages:

  1. Faces Preprocessing: The system extracts faces and applies a Gaussian-Laplace pyramid to amplify boundary details and subtle textures that might be tampered with.
  2. Faces Representative Learning: A CNN encoder maps these high-dimensional face images into a compact 128-dimensional latent space, focusing on core facial features.
  3. Faces Predictive Learning: An autoregressive model (specifically a Gated Recurrent Unit or GRU) takes a sequence of "source" frames and attempts to predict the latent representation of future frames.
  4. Correlation-Based Learning: The system compares the predicted future representation with the actual future representation. In real videos, these match closely. In fakes, the unnatural transitions in the generative process cause a mismatch, resulting in a lower correlation score.

Information Theory Foundation

The authors use entropy and mutual information to provide a mathematical basis for their approach. Their analysis indicates that fake videos have higher joint entropy (more uncertainty/randomness) between frames. This proves that the correlation between predicted and referenced representations is significantly higher in real videos than in high-visual-quality Deepfakes.

Key Results and Performance

FInfer was evaluated against industry-standard benchmarks including FaceForensics++, Celeb-DF, WildDeepfake, and DFDC-preview.

  • State-of-the-Art Accuracy: FInfer achieved 90.47% accuracy on the challenging Celeb-DF dataset, outperforming established models like Xception (90.34%) and Meso4 (67.53%).
  • Superior Generalization: In cross-dataset testing (trained on FF++ and tested on Celeb-DF), FInfer maintained a high AUC of 70.60%, suggesting it detects fundamental temporal flaws rather than dataset-specific quirks.
  • Efficiency: FInfer is remarkably lightweight, requiring only 96.75 million Multi-Adds. This makes it nearly 10x more efficient than Xception, making it a viable candidate for real-time detection.

Conclusion

While FInfer is powerful, it currently relies on successful face detection (via dlib) and requires facial variation to be effective. However, by moving the detection frontier from static pixel analysis to temporal inference, FInfer provides a robust defense against the next generation of high-visual-quality synthetic media.

Summary updated

Abstract

Deepfake has ignited hot research interests in both academia and industry due to its potential security threats. Many countermeasures have been proposed to mitigate such risks. Current Deepfake detection methods achieve superior performances in dealing with low-visual-quality Deepfake media which can be distinguished by the obvious visual artifacts. However, with the development of deep generative models, the realism of Deepfake media has been significantly improved and becomes tough challenging to current detection models. In this paper, we propose a frame inference-based detection framework (FInfer) to solve the problem of high-visual-quality Deepfake detection. Specifically, we first learn the referenced representations of the current and future frames’ faces. Then, the current frames’ facial representations are utilized to predict the future frames’ facial representations by using an autoregressive model. Finally, a representation-prediction loss is devised to maximize the discriminability of real videos and fake videos. We demonstrate the effectiveness of our FInfer framework through information theory analyses. The entropy and mutual information analyses indicate the correlation between the predicted representations and referenced representations in real videos is higher than that of high-visual-quality Deepfake videos. Extensive experiments demonstrate the performance of our method is promising in terms of in-dataset detection performance, detection efficiency, and cross-dataset detection performance in high-visual-quality Deepfake videos.
BibTeX
@inproceedings{hu2022finfer,
  author = {Juan Hu and Xin Liao and Jinwen Liang and Wenbo Zhou and Zheng Qin},
  title = {{FInfer: Frame Inference-Based Deepfake Detection for High-Visual-Quality Videos}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=KdBFCghwRP},
}
Figure