Semantic Contextualization of Face Forgery: A New Definition, Dataset, and Detection Method

Mian Zou, Baosheng Yu, Yibing Zhan, Siwei Lyu, Kede Ma

Abstract

Current deepfake detection methods often struggle with generalization because they focus on low-level digital artifacts rather than human-perceivable semantic changes. This paper introduces a novel perspective by redefining face forgery based on psychophysical thresholds of human detection. We present the Face Forgery in the Semantic Context (FFSC) dataset, which categorizes manipulations into a hierarchical structure of global and local attributes. Furthermore, we propose a Semantics-Oriented (SO) detection framework that leverages this hierarchy through bi-level optimization. Experimental results demonstrate that our approach significantly improves generalization across diverse manipulation methods and unseen semantic attributes.

Introduction

The rapid advancement of deepfake technology has created a "generalization gap" in detection systems. Models trained on specific datasets often fail when encountering new manipulation techniques because they overfit to signal-level artifacts, such as blending boundaries or frequency anomalies, rather than understanding the facial content. Moreover, the field lacks a rigorous definition of what constitutes a "forgery." Is every pixel-level edit malicious?

Example of a semantic attribute change (adding glasses).

This research argues that a face is only "fake" when computational manipulation alters semantic attributes—such as identity, expression, or age—beyond the threshold of human perception. By shifting the focus from microscopic artifacts to human-perceivable semantic changes, we can develop more robust detectors that align with real-world impact.

Methods

The proposed framework consists of three core components: a psychophysical definition of forgery, a new large-scale dataset, and a hierarchical detection model.

Psychophysical Definition and FFSC Dataset

The authors established a forgery threshold using the "staircase method" from psychophysics, identifying the point where 75% of human observers can distinguish a manipulated image. Based on this, they developed the Face Forgery in the Semantic Context (FFSC) dataset. FFSC contains 63,344 real images and 83,840 fake images generated via 12 manipulation methods, each annotated with a rich hierarchy of semantic labels.

Hierarchical Graph Modeling and SO-Detection

To capture the relationships between different types of manipulations, facial attributes are organized into a tree structure. The root "Face" (Real/Fake) branches into five global attributes: Age, Expression, Gender, Identity, and Pose. These are further decomposed into local regions such as the eyes, mouth, and nose.

Face parsing into semantic regions for hierarchical detection.

The SO-detection method performs multi-task learning across this hierarchy. It utilizes the Auto-$\lambda$ algorithm for bi-level optimization, ensuring the model prioritizes the primary real/fake classification while effectively using semantic attributes as auxiliary clues. This structure ensures that detection of a local anomaly (e.g., in the mouth region) correctly propagates to the global "Face" label.

Results and Discussion

The SO-detection method was evaluated against several state-of-the-art benchmarks, including FF++, DFDC, and Celeb-DF.

Superior Generalization

Models trained on the FFSC dataset exhibited significantly better cross-dataset performance. For instance, a standard detector retrained on FFSC achieved a 92.99% AUC on the FF++ dataset, outperforming versions trained on other large-scale sets like DFDC. This suggests that learning semantic contexts helps the model avoid "cheap shortcuts" like background artifacts.

Robustness to Unseen Attributes

In "Protocol-2" testing—where the model must detect a fake attribute (e.g., Pose) it never encountered during training—the SO-detection method consistently outperformed traditional binary and multi-class classifiers. Visualizations using Grad-CAM confirmed that the SO-detector focuses on meaningful facial regions like the eyes and lips, whereas baseline models often rely on irrelevant background signals.

Conclusion

By contextualizing face forgery within a semantic and psychophysical framework, this research provides a more robust path forward for deepfake detection. While currently focused on visual facial attributes, future work aims to incorporate temporal and audio forgeries, such as videos where speech is manipulated through frame-rate adjustments. The next steps include expanding the attribute set to include subjective social attributes and integrating this semantic approach with multimodal analysis to detect more complex, multi-layered forgeries.

Reference: Zou, M., Yu, B., Zhan, Y., Lyu, S., & Ma, K. (2024). Semantic Contextualization of Face Forgery: A New Definition, Dataset, and Detection Method. IEEE Transactions on Information Forensics and Security.

Summary updated

BibTeX
@article{zou2026semantic,
  author = {Mian Zou and Baosheng Yu and Yibing Zhan and Siwei Lyu and Kede Ma},
  title = {{Semantic Contextualization of Face Forgery: A New Definition, Dataset, and Detection Method}},
  journal = {IEEE Transactions on Information Forensics and Security},
  year = {2026},
  url = {https://openreview.net/forum?id=hLz6q8lLFb},
}
Figure