Face deepfake detectors are often trained on two kinds of fakes: real deepfakes, and cheap "pseudo-fakes" made by blending a face back onto itself or onto another person. One would expect the mixture to help. It usually hurts. A recent fix orders the four kinds of training images (real, self-blended, cross-blended, deepfake) along a hand-coded ladder of three-bit vectors and pushes the detector's features to follow that ladder. This proposal replaces those arbitrary bit vectors with the text embeddings of plain-language descriptions of how each image was made, taken from a vision-language model's text encoder. Examples are "the skin at the face border was blended from a different photo," "the person's identity was replaced," and "the face was produced by a neural generator." The detector is trained to place each image near the embedding of its own description.
The bet is that language supplies two things a bit vector cannot. First, distances between forgery types come from a space the image encoder already partly shares, so the targets should be easier to fit and the training more stable. Second, a forgery attribute never seen in training can be added at test time by writing a sentence, with no retraining. Both claims can be falsified. The central control is a detector trained on the same compositional structure with random vectors in place of the text embeddings. If that control does as well, semantics adds nothing, and the proposal says so.
Core Claim
The claim is specific. Suppose the targets used to organize the forgery spectrum in Can We Leave Deepfake Data Behind in Training Deepfake Detector? are replaced by compositional sentence embeddings from a frozen text encoder, with the backbone, data and pseudo-fake recipe held fixed. The claim is that this matches or beats the hand-coded targets on cross-dataset face benchmarks. It should also lower variance across random seeds in the low-data regime, and let the detector rank images by an artifact attribute that was held out of training entirely.
The second and third parts are what make this more than relabelling. The paper above showed that organizing the feature space fixes the "1+1 < 2" effect, where adding blended pseudo-fakes to real deepfakes lowers accuracy. It did not ask whether the organizing coordinates could carry meaning. It used a counter from [0,0,0] to [1,1,1], and a counter cannot be extended to a fourth attribute without inventing a new bit and new data.
Relation to Prior Work
Two vision-language detectors motivate the idea and bound it.
C2P-CLIP decoded CLIP detection features into words. It found that they cluster around a shared concept of "fakeness." It then trained CLIP with the single-word prompts "Camera" versus "Deepfake" and discarded the text branch at inference. The resulting 93.8% mean accuracy on UniversalFakeDetect is evidence that language targets help. Its prompts carry no information about how an image was forged, however, and it cannot be queried with a new description.
Knowledge-Guided Prompt Learning builds prompts from descriptive words that GPT-4 associates with deepfakes, such as "blurred" and "inconsistent," and adapts them at test time using pseudo-labels. Those descriptions are about how fakes look in general. Ours are about the specific construction of each training image. Ours are also checkable, because the pseudo-fake generator knows exactly which operations it applied.
Three methods show that richer structured supervision can work in explainable forensics, but they are built on multimodal language models that generate text. PRPO optimizes free-form forensic paragraphs. Fake-in-Facext uses a 112-concept facial ontology as discrete token targets and segmentation targets. We need neither generation nor an autoregressive decoder. The label embeddings are computed once and cached, so inference is a single image forward pass followed by a dot product.
The method's ingredients come from text classification. Description-Enhanced Label Embedding aligns inputs with embeddings of multi-aspect label definitions. It also reports that raw definitions are noisy unless filtered. Label-Description Training shows that descriptions alone support zero-shot classification of unseen labels. We take their central lesson, that a label should be a point in a meaning space rather than an index. We handle description noise by averaging several paraphrases per attribute rather than adding cross-attention, which keeps the method cheap.
One paper pulls the other way. PlainProbe reports that auxiliary objectives make training unstable when data are scarce. Its seed standard deviation on Celeb-DF v2 was 0.0558 for a reference method with auxiliary objectives, against 0.0141 for plain binary cross-entropy on normalized frozen CLIP features. Our loss is an auxiliary objective of exactly this kind, so we take this result as the bar the method must clear, not as a problem to explain away.
Research Program
Attributes and labels. Every training image is described by which of a small set of construction attributes it carries. The pseudo-fake generator controls these exactly. They are: a blending boundary; a color or illumination mismatch at the boundary; identity displacement, meaning that a different person's face was pasted in; and generative synthesis, meaning that the face texture was produced by a neural network.
For each attribute $a$ we write about ten paraphrased descriptions. We encode them with the frozen CLIP text encoder, average them, and normalize the result to get an attribute vector $e_a$. We also encode a "real camera photo of a face" description to get a real-image anchor $e_0$. An image $x$ with attribute set $S(x)$ receives the target
$$t(x) = \mathrm{normalize}\Big(e_0 + \sum_{a \in S(x)} e_a\Big).$$
The normalization step, $\mathrm{normalize}(v) = v/\lVert v\rVert$, puts every target on the unit sphere. This reproduces the nesting of the earlier ladder, where each forgery type adds one attribute on top of the previous one. The distances between rungs, however, now come from language rather than being set by hand.
Training. The image encoder is CLIP ViT-L/14 adapted with PlainProbe's LayerNorm-only tuning, so the number of trainable parameters stays tiny. It produces a normalized feature $f(x)$. We train it with binary cross-entropy on real versus fake plus $\lambda$ times a contrastive loss. In that loss, $f(x)$ must be closer to its own target $t(x)$ than to the targets of the other attribute combinations present in the batch. The weight $\lambda$ is tuned on FF++ validation only. At inference the detection score is the binary head. The attribute vectors are used only for attribute ranking.
Zero-shot test. In a leave-one-attribute-out protocol, one attribute (for example, color mismatch) is never switched on during training. At test time we write its description, encode it, and ask whether cosine similarity to it separates held-out images that carry the attribute from those that do not.
Baselines
The naive reference is a binary cross-entropy detector on the same backbone. It is trained on FF++ deepfakes plus self- and cross-blended images with no structure imposed. This should reproduce the 1+1 < 2 degradation and confirms that our pipeline sees the problem the prior paper described.
The strongest simple competitor is PlainProbe itself, meaning frozen normalized CLIP features with LayerNorm tuning and binary cross-entropy. The file reports 0.7287 AUROC on Celeb-DF v2 and 0.8410 AUROC on WILDD from 960 FF++ training frames (PlainProbe). We rerun it with its recipe on our splits and seeds. Beating it means a higher mean AUROC with a seed standard deviation no worse than its own, which the paper reports as 0.0141 on Celeb-DF v2.
The strongest existing method on the specific question is the oriented progressive spectrum with bit-vector targets from Can We Leave Deepfake Data Behind. The file gives only relative gains for it: a 5.26% average AUC gain across four test sets and 9.42% on Celeb-DF-v1. No absolute AUC is reported on Lacuna for this method. We therefore reimplement its bit-vector targets inside our backbone and loss. Only the target vectors differ, and each arm gets the same data, the same pseudo-fake generator, the same epochs and a 20-trial tuning budget. Beating it means higher average cross-dataset AUC plus the zero-shot capability it lacks.
Coarse prompting in the style of C2P-CLIP, using "Camera" and "Deepfake" as the only two targets, is run in the same harness. Its 93.8% mean accuracy on UniversalFakeDetect (C2P-CLIP) comes from whole-image synthesis, not face swaps. No reported result on face benchmarks was found on Lacuna for C2P-CLIP.
The ablation that isolates our key ingredient is the random-anchor control. It uses the identical compositional formula with the $e_a$ replaced by random unit vectors. We match their pairwise angles to the text embeddings in one variant and leave them orthogonal in another. A secondary ablation uses single-word descriptions instead of paraphrase sets.
Why not just…?
Why not just use PlainProbe? In the low-data regime it may genuinely win, because its authors show that extra losses add seed variance. That is why stability is a primary metric here and not an afterthought. Our one argument for expecting otherwise is that text targets lie in the space the frozen image features already inhabit, so they should conflict less with the binary loss than arbitrary targets do. If the 960-frame experiment shows higher variance or lower mean AUROC than PlainProbe, we will report that semantic anchoring helps only with full data, if at all.
Why not just keep the hand-coded bit-vector spectrum? This is the real threat. If the structure alone fixes 1+1 < 2, text adds nothing to in-distribution accuracy. The random-anchor control settles it. If random vectors with matched angles equal text anchors on cross-dataset AUC, the accuracy claim is dropped. The remaining contribution then rests solely on the leave-one-attribute-out test, which bit vectors cannot attempt at all because a held-out bit has no coordinates.
Why not just use coarse prompts like C2P-CLIP? C2P-CLIP's gains are on full-image generators, where one "fakeness" direction may suffice. Face swaps mix several distinct artifacts, and that mixture is exactly what causes the 1+1 < 2 conflict. A two-point target cannot order them. The coarse-prompt arm in our harness tests directly whether the richer descriptions matter.
Why not just train on real images only, as RealNet does? RealNet generalizes across generators by modelling real-image statistics and perturbing them. Its reported gains are on general AI-generated imagery, and the file gives no face-swap numbers for it. It also produces no attribute-level output. We include it as a reference on Celeb-DF where it can be run. We do not claim to beat it on generator-level detection.
Proposed Experiments
Training uses FF++ HQ, with the LQ split used for a compression check (FaceForensics++). Testing is cross-dataset on Celeb-DF v1 and v2, whose smoothed swaps defeat splicing shortcuts (Celeb-DF), and on WILDD. We run both the full-data regime and PlainProbe's 960-frame regime, with five seeds each.
We report AUROC, seed standard deviation, and the true-positive rate at 0.1% false-positive rate. We include the last because PlainProbe's authors concede that it is where more complex models still lead.
There are three decisive checks. The first is whether text anchors beat random anchors with matched angles on average cross-dataset AUROC by more than the seed spread. The second is whether they match or exceed the bit-vector spectrum while keeping seed variance at or below PlainProbe's. The third concerns a held-out attribute's text description, with the pass mark fixed before running. It must rank held-out Celeb-DF and FF++ images that carry that attribute above those that do not with AUROC of at least 0.65. It must also clearly beat the same query made with a random vector, which should sit near 0.5.
A positive result on the first and third checks would show that forgery labels benefit from meaning. A negative result on the first check but a positive one on the third would narrow the contribution to open-set attribute querying, and we would state it that way.
References
- Can We Leave Deepfake Data Behind in Training Deepfake Detector?
- C2P-CLIP: Injecting Category Common Prompt in CLIP to Enhance Generalization in Deepfake Detection
- Knowledge-Guided Prompt Learning for Deepfake Facial Image Detection
- PRPO: Paragraph-level Policy Optimization for Vision-Language Deepfake Detection
- Fake-in-Facext: Towards Fine-Grained Explainable DeepFake Analysis
- Description-Enhanced Label Embedding Contrastive Learning for Text Classification
- The Benefits of Label-Description Training for Zero-Shot Text Classification
- PlainProbe: A Stable Cross-Entropy Baseline for Data-Scarce Deepfake Detection
- RealNet: Efficient and Unsupervised Detection of AI-Generated Images via Real-Only Representation Learning
- FaceForensics++: Learning to Detect Manipulated Facial Images
- Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics