Deepfake detectors look nearly perfect on the dataset they were trained on and then fail in two ways once deployed. They collapse on forgeries from newer generators: methods scoring above 80% AUC on FaceForensics++ average 56.9% AUC on Celeb-DF (Celeb-DF). They also collapse under small adversarial perturbations, meaning tiny pixel changes crafted to flip the decision: a RealForensics detector drops to about 1.0% AUC under a 10-step projected-gradient attack (AFSL). The only forensic defense in this literature, AFSL, repairs the second failure with full adversarial training, which runs a multi-step attack inside every training step. Its authors name that cost as the open problem.
This note proposes that both failures share a cause, and that one cheap training change can address both. When a small detector is trained by imitating a larger "teacher" model (knowledge distillation), the training images the small model cannot robustly fit are, we hypothesize, mostly the ones whose labels are only recoverable through generator-specific shortcut artifacts. A teacher that is confident on those images forces the student to memorize the shortcuts. A teacher that is deliberately uncertain on exactly those images tells the student to stop trying. The prediction is that the student becomes harder to attack and also transfers better to unseen generators, with a teacher that can be made robust in a few hundred optimization steps instead of through full adversarial retraining.
Core Claim
Toward Understanding Adversarial Distillation: Why Robust Teachers Fail studies adversarial distillation, where the student is trained on attacked inputs to match a robust teacher. It defines the student's robustly unlearnable set: training samples the student lacks the capacity to classify correctly under attack. On CIFAR and Tiny-ImageNet, the teacher's entropy on that set, not the teacher's own robust accuracy, predicted the student's final robustness (Pearson r > 0.9). Switching to a high-entropy teacher cut the student's late-training robustness loss from 7.23% to 0.53%. That work motivates ours, but it is confined to balanced multi-class object recognition, and its theory is proven only for two-layer networks.
Our claim is specific to forensics and is falsifiable. On FaceForensics++, a compact detector's robustly unlearnable set is enriched for samples that carry generator-specific, high-frequency evidence. Teacher entropy on this set therefore predicts the student's cross-dataset AUC on Celeb-DF, DeeperForensics and DFDC, as well as its robust AUC. It does so better than the teacher's clean or robust AUC. Shaping teacher outputs toward maximum uncertainty on this set only, while keeping them sharp everywhere else, should produce a student that matches AFSL's robustness at a fraction of its training cost and generalizes better across datasets.
The binary setting is where the claim could break. In real-versus-fake detection, a maximum-entropy target is a flat 0.5, which literally says "no opinion." It is not obvious that this regularizes rather than blurs the decision boundary between authentic faces and subtle forgeries. Testing this is part of the contribution.
Research Program
The work has three parts.
Part 1: a cheap way to find the unlearnable set. Lee et al. identify the set by intersecting failures across several independently adversarially trained models, which is too expensive for video forensics. We will use a proxy instead. We train two seeds of the compact student for a short schedule with a cheap single-step attack, and mark the training frames that both seeds misclassify under a stronger attack at the end. On a FaceForensics++ subset, we validate the proxy against the full multi-model intersection by measuring overlap and checking that it reproduces the entropy–robustness correlation. If the proxy fails this check, the method falls back to the gold-standard set on a reduced training pool, and we report the cost.
Part 2: characterizing what the unlearnable set contains. We break it down by manipulation method (DeepFakes, FaceSwap, Face2Face, NeuralTextures), by compression level (raw, c23, c40), and by how much of each sample's forgery signal lies in high-frequency DCT bands. We also measure how much a frozen frequency-only probe, in the spirit of the two-stream detectors (Two-Stream CNN), relies on these samples compared with the rest. The shortcut hypothesis predicts strong enrichment for low-level-artifact samples. If the set instead looks random or is dominated by heavy c40 compression, the cross-dataset half of the claim is in trouble, and we say so.
Part 3: building the distilled detector. We take a teacher pool and apply an entropy-shaping rule: on samples in the unlearnable set, the teacher's soft target is interpolated toward 0.5 with a strength chosen on held-out FaceForensics++ data. Everywhere else the teacher's output is left unchanged.
The student is trained by adversarial distillation. To avoid the cost of running the teacher inside the attack loop, it uses the first-order Taylor approximation of teacher feedback from SAAD, which reported roughly a 4× speedup over full teacher-in-the-loop distillation.
The teacher pool spans four kinds of teacher:
- a clean Xception;
- a PGD-adversarially-trained Xception;
- an AFSL-trained Xception;
- a frozen DINOv2 with a linear real/fake head, hardened with Robustness Tokens. That method appends about ten learnable input tokens to a frozen vision transformer and trains only those tokens, in about 200 steps, to map attacked features back to clean ones.
The fourth teacher is where the overhead reduction comes from: the expensive robust model never undergoes full adversarial training. Two constraints shape the design. Cross-architecture distillation (transformer to CNN) is known to transfer invariances worse than CNN to CNN (What Knowledge Gets Distilled), so the homogeneous Xception teachers stay in the pool as a control. And whether Robustness Tokens suppress adversarial noise without also erasing the residual traces that reveal forgeries is untested, so we measure the token-hardened teacher's clean forensic AUC before using it.
Proposed Experiments
We train on FaceForensics++ c23 (FaceForensics++) and evaluate on four held-out sets: FaceForensics++ c40, Celeb-DF, DeeperForensics-1.0 and DFDC. The student is EfficientNet-B3, a backbone already used in this literature by AdvOU.
The attacks are PGD-10 under the protocol AFSL reports, plus a binary adaptation of AutoAttack: its gradient-based APGD-CE component and the query-based Square attack. Square needs no gradients, which guards against defenses that only mask gradients.
The metrics are:
- clean AUC;
- robust AUC on each dataset, including attacked Celeb-DF;
- total training GPU-hours, with teacher preparation included.
To our knowledge, nobody reports robust cross-dataset AUC for an adversarially trained deepfake detector. We found no reported result on Lacuna for adversarially robust Celeb-DF detection, so the experiment produces the first such numbers rather than comparing to them.
The first study is correlational. Across the teacher pool, crossed with several entropy-shaping strengths, we regress student cross-dataset AUC and robust AUC on four teacher properties: entropy on the unlearnable set, entropy on a size-matched random set, clean AUC and robust AUC. This replicates the Lee et al. test in a new domain and extends it to a new outcome, cross-dataset transfer. The second study is the head-to-head comparison against the baselines below.
Baselines
Naive reference points. The first is a student trained with plain cross-entropy, which shows the size of the Celeb-DF drop and the collapse under attack. The second is clean distillation with feature matching in the style of FReTAL, which reached 86.97% accuracy on low-quality FaceForensics++ but was never attacked. It serves as the zero-overhead distillation reference and shows how much inherited vulnerability costs.
AFSL, the strongest existing competitor. AFSL trains the detector itself with PGD-10. Its loss pulls clean and attacked features of the same image together and pushes real and fake feature clusters apart. On FaceForensics++ it reports 80.8% AUC under PGD-10, 89.7% clean AUC (against about 73.5% clean for standard adversarial training) and 83.6% attacked AUC on unseen DeeperForensics. It is the only published defense on this exact task. We rerun it on our student backbone and splits, under the same attack and the same hyperparameter-search budget in number of configurations, and we log its GPU-hours. Beating it means matching or exceeding its robust and clean AUC on FaceForensics++ within one standard deviation across three seeds, exceeding it on clean and attacked Celeb-DF AUC, and using at most half its training compute with teacher preparation counted.
Standard PGD adversarial training of the student. This is the no-teacher reference for whether distillation helps at all.
SAAD applied unchanged. SAAD reweights training samples by teacher entropy under student attacks and reached 50.35% AutoAttack accuracy on CIFAR-10, against 40.57% for RSLAD. It isolates our difference: SAAD reweights the student's samples using whatever entropy the teacher happens to have, whereas we reshape the teacher's targets on a capacity-defined subset.
Ablations. These isolate the key ingredient by replacing targeted entropy shaping with three alternatives:
- uniform temperature or label smoothing on all teacher outputs;
- entropy shaping on a random subset of the same size;
- deleting the unlearnable set from training.
All three use the same student, data, attacks and tuning budget.
Why not just…?
Why not just run AFSL on the small detector and skip the teacher? This may well match our robustness on FaceForensics++. AFSL's own evidence covers in-distribution and DeeperForensics robustness only, not the Celeb-DF transfer gap, and its cost is the problem its authors flag. The head-to-head comparison settles it. If AFSL equals us on attacked Celeb-DF at similar cost, the proposal has no case.
Why not just smooth the teacher's outputs everywhere? This is the most serious threat, because uniform smoothing also raises entropy on the unlearnable set. Lee et al. found entropy on that set specifically to be predictive, and smoothing confident correct samples throws away useful supervision. That finding comes from CIFAR, however. The uniform-smoothing ablation decides the question. If it matches targeted shaping, we report the simpler method as the finding and drop the unlearnable-set machinery.
Why not just delete the hard samples? In a binary task, a 0.5 target and deletion might behave almost identically. The difference is that the 0.5 target still trains the student to be locally uncertain around those faces under attack, whereas deletion leaves that region of input space unconstrained. The deletion ablation, together with calibration measured on the held-out datasets, shows whether that difference matters.
Why not just use a frozen vision-language detector like AntifakePrompt? AntifakePrompt tunes 4,864 prompt parameters on InstructBLIP and reaches 92.11% average accuracy on unseen generators. However, it was never trained against attacks, it degraded on inpainting, and it carries billions of parameters at inference time. It could serve as a further teacher in our pool, but it is not a replacement for a compact robust detector.
Decisive Checks
Two outcomes would sink the idea. The first is that teacher entropy on the unlearnable set fails to beat teacher robust AUC as a predictor of student Celeb-DF AUC. The second is that the unlearnable set shows no enrichment for artifact-heavy samples. Either would mean the forensic analogue of the CIFAR finding does not hold.
One check guards against false success: an adaptive attack that has gradient access to the full student. Any gain that disappears under that attack, or under the query-based Square attack, does not count.
References
- Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics
- Adversarially Robust Deepfake Detection via Adversarial Feature Similarity Learning
- Toward Understanding Adversarial Distillation: Why Robust Teachers Fail
- Fake visual content detection using two-stream convolutional neural networks
- Sample-wise Adaptive Weighting for Transfer Consistency in Adversarial Distillation
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- What Knowledge Gets Distilled in Knowledge Distillation?
- FaceForensics++: Learning to Detect Manipulated Facial Images
- Open-Unfairness Adversarial Mitigation for Generalized Deepfake Detection
- FReTAL: Generalizing Deepfake Detection Using Knowledge Distillation and Representation Learning
- AntifakePrompt: Prompt-Tuned Vision-Language Models are Fake Image Detectors