Deepfake detectors score above 99% on the dataset they were trained on and then fall toward chance on a new one. The usual explanation is that they learn a shortcut: a seam, color mismatch, or compression signature that happens to mark most fakes in the training set. This proposal borrows a tool from the spurious-correlation literature. Record how each training example's loss changes over the epochs. The examples a detector resolves last are the ones its shortcut does not cover. We claim these late-learned examples are a label-free stand-in for the unseen test datasets. If so, they can be used twice: upweighted during training, and used on a held-out source split to choose which checkpoint to ship.
The second use may matter most. Deepfake practitioners pick checkpoints by accuracy on the source validation split. That split shares the source dataset's shortcuts, so it rewards the checkpoint that relies on them most. A cheap, falsifiable test follows. Across checkpoints of one training run, accuracy on the hardest discovered validation cluster should track cross-dataset area under the ROC curve (AUC) on Celeb-DF and DFDC much better than overall validation accuracy does. If that correlation is absent, the whole idea fails early and cheaply.
Core Claim
Cluster each class's per-epoch loss histories into two groups. The smaller, slowly learned group of fakes consists of forgeries that lack the dominant source artifact. The slowly learned group of real faces consists of genuine images the detector nearly flags as fake. We claim that treating these clusters as groups, with no manipulation or domain labels, does two things. It improves cross-dataset AUC when those examples are upsampled. It also yields a model-selection rule, worst-cluster performance on source validation, that predicts target-domain performance. What is new is the unit of analysis. This is not another augmentation or architecture. It is a label-free partition of forensic training data that is used for both training and checkpoint choice. The claim must hold where no manipulation labels exist, namely training on single-method Celeb-DF v2.
Where This Sits
The mechanism comes from four label-free group-discovery methods. Efficient Bias Mitigation Without Privileged Information (TAB) records every sample's loss at each epoch and runs two-cluster k-means per class. It shows that a single end-of-training loss snapshot is corrupted by memorization, and we reuse its clustering directly. Exploiting What Trained Models Learn for Making Them Robust to Spurious Correlations without Group Annotations (EVaLS) retrains only the last layer on balanced high-loss and low-loss samples. It also selects checkpoints by worst accuracy over inferred environments. We adapt that selection idea, but our validation groups come from loss histories rather than a separate environment-inference method. Bias Amplification Enhances Minority Group Performance (BAM) deliberately makes a helper model more biased so that its error set is cleaner, and it reports that naive two-stage error selection breaks if stage one runs too long. That result constrains us: we must show loss-history clusters are stable across stage-one lengths. Correct-N-Contrast (CNC) argues that reweighting fails when the representation has already collapsed onto the shortcut. Its answer is contrastive alignment, which raised CelebA worst-group accuracy from 81.1% to 88.8%. This is the strongest argument against our simpler upsampling design, so we test a CNC-style contrastive stage as an ablation. All four papers were evaluated only on Waterbirds, CelebA, CivilComments, and similar benchmarks, never on forensic data. There the "group" is an unannotated artifact type, not a background or attribute.
On the forensics side, three papers motivate the proposal. Celeb-DF documents the collapse: detectors above 80% AUC on FaceForensics++ average 56.9% on Celeb-DF. FaceForensics++ shows that compression alone moves XceptionNet from 99.26% to 81.00% accuracy. That means compression level is a plausible hidden group our clusters might recover. What Makes Fake Images Detectable? shows that small-patch classifiers generalize better because they cannot use global content.
Three methods remove shortcuts but need privileged information, and this contrast motivates the label-free design. Improving Cross-dataset Deepfake Detection with Deep Information Decomposition (CrossDF) requires manipulation-technique labels. Towards Generalizable Deepfake Detection with Locality-aware AutoEncoder (LAE) requires human pixel masks on under 3% of the data, chosen by active learning. BPL pairs each fake with its real source or with a nearest neighbor found by CLIP retrieval, and tunes only the bias terms of CLIP ViT-L/14. BPL is the closest existing work and the strongest competitor. It finds hard comparisons through an external image pool. We find hard examples inside the detector's own training history. The two are not exclusive, so we test ours on top of BPL.
Detecting Deepfakes with Self-Blended Images (SBI) builds fakes by blending a perturbed copy of a real face back onto itself. It shows its own limit: 69.11% AUC on fully synthesized StyleGAN faces, where no blending seam exists. That is exactly the kind of "fake without the dominant artifact" our late-learned cluster is meant to capture.
Research Program
Stage one: record loss histories. A detector is trained normally for a fixed, short budget. At every epoch we record the loss of every training video and every held-out source-validation video. For validation videos we compute the loss but never take a gradient step on them. Losses are averaged over each video's frames before clustering. Frames from one video are near-duplicates and would otherwise form trivial clusters.
Clustering. Within each class (real, fake) we run two-cluster k-means on the loss-history vectors, as TAB does. The smaller, slower-converging cluster is labeled "late-learned."
Stage two: retrain. A fresh detector is trained with the late-learned training clusters upsampled to parity. Two ablations replace this step. The first is last-layer retraining, as in EVaLS. The second is a CNC-style contrastive loss. Its anchors are early-learned examples, its hard positives are late-learned examples of the same class, and its hard negatives are early-learned examples of the other class.
Selection rule. For model selection we compute four AUCs on source validation: late-learned fakes against all reals, early-learned fakes against all reals, and the two real clusters against all fakes. We take the minimum and select the checkpoint that maximizes it. This worst-cluster AUC is the rule under test.
Analysis. A separate analysis asks what the clusters actually contain. On FF++ we compare the late-learned fake cluster's makeup with the hidden labels: manipulation method (the four techniques), compression level (c23 vs. c40), and face attributes. We also check stability by rerunning the clustering with different stage-one lengths and seeds and measuring agreement. This speaks directly to BAM's warning about long stage-one training.
Baselines
Naive reference. The do-nothing reference is standard binary training, selected by overall source-validation AUC. It is run on both backbones we use: Xception, the FaceForensics++ standard, and CLIP ViT-L/14 with bias-only tuning, as in BPL. Every other method must beat it on target AUC, not source AUC.
Oracle labels. This is the most dangerous baseline. On FF++, manipulation method labels are free. So we train with per-method upsampling and select by worst-method validation AUC, using the true labels. This is the "oracle groups" reference, and on FF++ our method may at best match it. It is run with the same backbone, epochs, and selection budget.
SBI. SBI is trained on the same real videos with its published augmentation. On DFDC, BPL reports 65.5% AUC for SBI under the Celeb-DF v2 training protocol.
Strongest existing method. The strongest competitor is BPL. The file gives three reported figures for it: 83.8% AUC when trained on Celeb-DF v2 and tested on DFDC; an 88.86% average AUC over DF40, under a training protocol the file does not specify; and CrossDF's 0.779 AUC on Celeb-DF v2 when trained on FF++ HQ. We rerun BPL from its description with the same CLIP backbone, bias-only tuning, and data. Our method is then added as a sampling and selection layer on BPL's objective. Beating BPL means a gain in DFDC AUC over our reproduced BPL that exceeds seed-to-seed spread across at least three seeds. We also require no loss on DF40. If our BPL reproduction falls short of 83.8%, we report both numbers.
Single-snapshot ablation. The ablation that isolates the key ingredient replaces loss histories with final-epoch loss, taking the top-k highest-loss videos. Focal loss serves as a second, weaker variant. If this matches the history-based clusters, the temporal component is unnecessary and we will say so.
Fairness. Every method gets the same number of total training epochs, including stage one, and the same hyperparameter search size. Every method is selected without target-domain data.
Why not just…?
Why not just use the FF++ manipulation labels as groups? This is a real threat, and on FF++ it may win. Our answer is that labels exist only for curated benchmarks. When training on Celeb-DF v2, which uses one synthesis method, there are no method labels, yet subgroups plainly exist in lighting, quality, and blending. That protocol is where the method must earn its keep. The cluster-composition analysis also tests whether clusters simply rediscover method labels. If they instead cut across method, for example by compression level or by particular identities, that is evidence they capture something labels miss. If the method loses to oracle labels on FF++ and gives no gain on Celeb-DF training, the idea is dead.
Why not just use SBI or other synthetic fakes? SBI covers blended forgeries. However, its own paper reports 69.11% AUC on StyleGAN faces, and BPL reports it at 65.5% on DFDC. Our method is not an alternative source of fakes. It decides which examples to emphasize and which checkpoint to keep, so it can be applied on top of SBI training. One experiment runs trajectory clustering on SBI's self-blended samples to see whether the late-learned blends are the subtle ones.
Why not just fine-tune a foundation model? BPL itself reports that full fine-tuning and LoRA overfit CLIP to specific forgery cues, and that only bias-only tuning preserved generalization. We therefore adopt BPL's bias-only CLIP setup as a backbone rather than competing against it. The question becomes whether our sampling and selection add anything to the strongest recipe.
Why not just mine the hardest examples at the last epoch? TAB shows that end-of-training losses are flattened by memorization, and BAM shows that simple error-set selection breaks as stage one lengthens. Forensic data is small at the video level (Celeb-DF has 590 real videos), and EVaLS notes that loss sampling weakens on small datasets. So memorization is a serious risk here. The single-snapshot ablation and the stability analysis across stage-one lengths settle this directly.
Proposed Experiments
Protocol A. Train on FF++ c23 and test at frame and video level on Celeb-DF v2, DFDC, DFDC Preview, and DF40. The metric is AUC.
Protocol B. Train on Celeb-DF v2 and test on DFDC, matching BPL's reported 83.8% protocol. Here no group labels exist.
First decisive check: does the selection rule predict transfer? For every run we save all checkpoints. We compute the rank correlation between each candidate selection signal and target AUC: overall validation AUC, worst-cluster validation AUC, and, on FF++, worst-method AUC. We predict that worst-cluster AUC tracks target AUC substantially better than overall AUC. If it does not, selection is not the mechanism.
Second decisive check: which of the two uses helps? We run a two-by-two design crossing training (standard vs. cluster upsampling) with selection (overall vs. worst-cluster). This separates how much gain comes from training and how much from selection.
Third decisive check: is the late-learned cluster really "missing the shortcut"? On FF++ we measure how many late-learned fakes are NeuralTextures or c40 videos. On the targets, we train a simple probe on source cluster membership and ask whether it predicts which target fakes the baseline detector misses. If late-learned source fakes resemble the target fakes the baseline fails on, the core claim has direct support. If they look random, the claim does not hold even if AUC improves.
References
- Efficient Bias Mitigation Without Privileged Information
- Exploiting What Trained Models Learn for Making Them Robust to Spurious Correlations without Group Annotations
- Bias Amplification Enhances Minority Group Performance
- Correct-N-Contrast: a Contrastive Approach for Improving Robustness to Spurious Correlations
- Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics
- FaceForensics++: Learning to Detect Manipulated Facial Images
- What Makes Fake Images Detectable? Understanding Properties that Generalize
- Improving Cross-dataset Deepfake Detection with Deep Information Decomposition
- Towards Generalizable Deepfake Detection with Locality-aware AutoEncoder
- BPL: Generalizable Deepfake Detection via Bias-only Pair-aware Learning
- Detecting Deepfakes with Self-Blended Images