Is the Deepfake Detector's Failure in the Features or in the Boundary?

Deepfake detectors score near-perfectly on the dataset they were trained on and then collapse on a new one. Nine detectors evaluated on Celeb-DF averaged 56.9% AUC (area under the ROC curve, where 50% is chance) (Celeb-DF). The field almost always explains this as a feature problem: the network learned generator-specific shortcuts. The fixes follow from that diagnosis. Some restrict the network to local patches, some regularize attention with forgery masks, and some re-shape feature statistics. Work on class imbalance offers a competing explanation that nobody has tested on forgeries. There, the features are often adequate and the final linear layer is what goes wrong (Adjusting Decision Boundary for Class Imbalanced Learning).

This note proposes to settle which explanation holds for deepfake detection, and then to act on the answer. We first measure, with oracle linear heads trained on frozen features, how much of the cross-dataset gap a better linear boundary could recover on its own. If that share is large, we train only the final linear head, the "decision head," on a frozen backbone. The head is anchored on the real-image distribution. It is chosen to separate real faces from every training manipulation at once rather than from their average. Held-out manipulations, not target data, select its hyperparameters. The core claim is falsifiable. For strong backbones, a substantial part of the FF++→Celeb-DF/DFDC gap is a boundary-direction error that can be corrected without touching the features and without any target labels. If the oracle diagnostic shows otherwise, the method is abandoned and the diagnostic is the contribution.

Why the obvious combination is the wrong one

The obvious bridge is to take the post-hoc weight re-scaling of Kim et al. and apply it to a deepfake detector. That method multiplies each class's final-layer weight vector by $(n_1/n_i)^\gamma$ to undo frequency-driven norm growth, where $n_i$ is the number of training samples in class $i$ and $n_1$ is the count for the largest class. In a two-class detector this changes little that AUC can see. The logit difference moves mostly by a scalar offset, and AUC does not change when the threshold shifts. The method would help accuracy at a fixed threshold and would leave the headline cross-dataset metric largely untouched.

The candidate-gap framing in the research file treats "boundary encroachment" as the likely culprit. The honest version of that question is sharper. It separates three possible failures:

  • Ranking failure: frozen features cannot rank target fakes above target reals under any linear head. This is a representation problem.
  • Direction failure: a linear head could rank them, but the head learned on the source dataset points the wrong way.
  • Offset failure: the ranking is fine but the threshold is miscalibrated.

Only the second is a boundary problem that AUC can see. It is the one no surveyed paper isolates.

Core Claim

Let $z$ be the frozen features. Let $w_S$ be the source-trained head direction and $w_T^\star$ a target-oracle linear probe fit on labeled target features. The gap between within-source AUC and $\mathrm{AUC}_T(w_S)$ splits into two parts:

  • a representation part, within-source AUC minus $\mathrm{AUC}_T(w_T^\star)$;
  • a direction part, $\mathrm{AUC}_T(w_T^\star) - \mathrm{AUC}_T(w_S)$.

Offset errors appear only in accuracy and equal error rate (EER, the operating point where false-accept and false-reject rates are equal).

We hypothesize the following for backbones with rich pretrained features, CLIP ViT-L/14 in particular. The direction part is large. It arises because cross-entropy on the source dataset tilts $w$ toward whichever of the four FaceForensics++ manipulations is easiest to separate. That tilt mirrors how majority classes dominate the boundary under imbalance (Kim et al.; Tian et al.). Here the "majority" is the manipulation with the strongest shortcut, not the most frequent one.

Research Program

Stage one: the diagnostic. We run it on three frozen feature extractors:

  • an Xception trained end-to-end on FF++ c23;
  • CLIP ViT-L/14 with no forensic training;
  • CLIP ViT-L/14 after BPL-style bias-only tuning (BPL).

For each target set we fit the oracle probe on half the target data and score it on the other half. That gives the representation and direction parts per backbone and per target. We also measure the test-time variance asymmetry that Kim et al. describe for minority classes. Concretely, we compare the spread of target fake and target real features along $w_S$ to their spread on the source. The aim is to show which class spills over the boundary.

Stage two: the decoupled head, if the direction part is large. The design borrows two ideas from the literature on unequal class geometry. From anchored-prototype methods it takes the idea of fixing geometry rather than learning it (PAL). From real-emphasis forensics it takes the idea of treating real media as the compact, privileged class (REFD). It departs from PAL's symmetric equiangular prototypes, which assume a closed set of classes. It departs from REFD, which trains the backbone end to end and uses RegMixup to disperse fakes. We freeze the backbone, estimate the real-feature mean $\mu$ and covariance $\Sigma$ on source reals, and whiten features by the real distribution. The head direction is then chosen to maximize the worst-case margin over the source manipulation types $m \in \{\text{Deepfakes}, \text{Face2Face}, \text{FaceSwap}, \text{NeuralTextures}\}$:

$$\min_{\|w\|=1,\,b}\ \max_{m}\ \mathbb{E}_{(z,y)\sim \mathcal{D}_{\text{real}}\cup\mathcal{D}_m}\,\ell\big(y,\ w^\top \Sigma^{-1/2}(z-\mu)+b\big) \;+\; \lambda\,\|P_{\text{real}}\,w\|^2 .$$

In this objective, $\ell$ is the logistic loss and $y$ is the real/fake label. $\mathcal{D}_m$ holds the fakes produced by manipulation $m$. $P_{\text{real}}$ projects onto the top principal directions of the real features, and $\lambda$ weights that penalty. The first term is group distributionally robust optimization (minimizing the loss of the worst-served group) over manipulation types. It stops one easy manipulation from setting the direction. The penalty discourages boundaries that separate classes along directions in which real faces themselves vary, such as pose, identity or lighting, because target reals will vary along those directions too. The unit-norm constraint removes norm effects, following the spirit of weight vector normalization.

Hyperparameters, the principal-direction rank and $\lambda$, are chosen by leave-one-manipulation-out AUC within FF++. This addresses the known reliance of post-hoc boundary methods on in-distribution validation sets (Kim et al.).

This is cheap: a convex-ish problem on cached features, with no retrieval pool and no multi-GPU training. That contrasts with the overheads of BPL and $SD^2$.

Baselines

The naive reference is a standard Xception trained end to end with cross-entropy on FF++ c23 and tested on Celeb-DF. RDBC reports 56.75% AUC for it. We retrain it under our data pipeline so the number is reproduced rather than borrowed.

The strongest existing method on the FF++-trained protocol is RDBC. It models the real-image population by maximum likelihood and whitens features with matrices sampled from that model during end-to-end training. It reports 91.82% average cross-dataset AUC across its external test sets and 92.44% on Celeb-DF with an Xception backbone. RDBC is the most dangerous competitor for a specific reason. It already exploits the real distribution, which is our anchor, but it does so inside the feature extractor rather than at the head. For a fair comparison we use its published protocol: FF++ c23 training, the same external test sets, frame-level AUC. We then apply our head on top of the frozen RDBC-trained Xception. Beating RDBC means one of two things. Either our head raises the average AUC above 91.82% when stacked on RDBC features, or it matches RDBC when stacked on a plain Xception with no feature retraining. The second outcome would show that the boundary alone accounts for RDBC-sized gains.

For the Celeb-DF-trained protocol the reference is BPL, which tunes only the bias parameters of CLIP ViT-L/14 with a pair-aware discrepancy loss. It reports 88.86% average AUC on DF40 and 83.8% on DFDC. We replace BPL's head with ours on the same frozen bias-tuned backbone and give both heads the same budget of hyperparameter trials. Improving DF40 average AUC beyond 88.86% counts as beating it.

The test-time augmentation agent of Nadimpalli & Rattani reports 0.669 AUC on Celeb-DF. It serves as a reference for input-side correction rather than as a contender.

Three ablations isolate the key ingredient:

  • Plain probe: a logistic-regression head on the same frozen features, with no anchor, no worst-group term and no penalty.
  • Real anchor only: a real-only Mahalanobis score in the spirit of the strict one-class detectors (Learning Tight Rejection Boundaries without Negatives).
  • No worst-group term: the full head with the group objective replaced by average loss.

All heads use identical cached features, identical training data and the same leave-one-manipulation-out validation.

Why not just…?

Why not just re-scale or re-threshold the existing head? For a two-class detector, re-scaling in the style of Kim et al. acts mostly as an offset, and AUC does not respond to offsets. We will report it anyway. We expect it to improve accuracy and EER while leaving AUC flat. That result is itself evidence that the useful boundary correction is a change of direction, not a shift.

Why not just fit a linear probe on frozen CLIP features? This is the real threat. If a plain logistic probe already recovers most of the oracle direction part, our worst-group and real-subspace terms add nothing. The first ablation settles this directly. If the plain probe comes within one AUC point of our head on average, we will say that frozen features plus any linear head suffice, and the diagnostic becomes the paper's finding.

Why not just model the real distribution alone, one-class style? Real-only scoring ignores everything the source fakes teach. Its known weakness is high-quality fakes, like Celeb-DF's, that lie close to the real distribution. REFD's gains came from combining a compact real class with explicit handling of the fakes, not from the real class alone (REFD). The real-anchor-only ablation measures how much the fake-aware direction adds beyond the anchor.

Why not just fine-tune more of the network, or train on more data? BPL reports that LoRA, which adds low-rank weight updates, generalized worse than bias-only tuning (BPL). $SD^2$ reports that standard detectors plateau or degrade as training data grows. Both results suggest that more feature adaptation does not reliably close the gap. Our diagnostic also tests this directly. If the representation part dominates, feature-side methods are the right tool and we will say so.

Proposed Experiments

The decisive first check is the gap decomposition. We train on FF++ c23, test on Celeb-DF v2, DFDC, DFDCp and UADFV, and repeat for each of the three backbones.

The kill criterion is this: if the oracle probe on frozen features gains less than five AUC points over the source head on Celeb-DF and DFDC, the boundary hypothesis is wrong for this problem and stage two is dropped. The five-point bar is an operating choice. It is roughly the size of the gains separating recent methods in the file.

If the criterion passes, we run the head comparison under both the RDBC and BPL protocols, including DF40 for the latter. We report AUC, plus accuracy and EER at the source-chosen threshold, so that direction and offset effects appear separately.

A leave-one-manipulation-out study within FF++ tests whether validation on held-out manipulations predicts true cross-dataset AUC. We report the rank correlation between the two across hyperparameter settings. Without a positive correlation, the target-label-free tuning claim fails.

Finally, as a secondary probe of whether a well-aligned direction also has more margin, we attack the frozen-backbone heads with small-budget projected gradient descent at $\epsilon = 8/255$, following RAID's protocol. We report robust F1 against the plain probe. We state no expectation beyond "not worse." Adversarial robustness is not a claim of this proposal.