Choosing Which Layers to Trust for Deepfake Detection That Transfers

Deepfake detectors built on large pretrained image encoders usually read only the encoder's last layer. They often score near-perfectly on the dataset they were trained on and then drop sharply on deepfakes made by other methods or collected from the internet. The information needed to generalize may already be present in the frozen encoder's intermediate layers. The hard part is deciding which layers, and which features within them, to read. Today that choice is made by hand, or by sweeping every layer and keeping the one that scores best on held-out data from the training distribution. This proposal argues that the selection criterion is the real problem. A layer that looks best on in-distribution validation data is often best because it carries the training generator's fingerprints, and those fingerprints are exactly what fail to transfer.

We propose to select intermediate features of a frozen CLIP encoder by how well they survive shifts we can simulate inside the training set, rather than by in-distribution accuracy. FaceForensics++ already contains four manipulation methods and several compression levels. Treating each method-by-compression combination as a separate "environment," we train a sparse multi-layer probe. Its objective rewards features whose real-versus-fake signal holds in every environment, and it removes whole feature groups that do not help. The claim is testable and falsifiable. If layer rankings under leave-one-manipulation-out validation predict layer rankings on Celeb-DF and WildDeepfake better than in-distribution rankings do, then transfer-aware selection is possible without any target labels. If they do not, the intersection of these directions is weaker than it looks, and we will have shown that cleanly.

What is already known, and the gap

Several papers establish that layer depth matters for forgery detection. Aggregating Layers for Deepfake Detection attaches small feature extractors to every block of a CNN and combines them with a linear head. It finds that different manipulations peak at different depths: mid layers for NeuralTextures, later layers for identity swaps. Keeping only three blocks costs less than 0.25% in performance, but that trimming was chosen by hand after training. What Makes Fake Images Detectable? goes the other way. It truncates CNNs at Block 2 so the classifier sees only small patches, which improves transfer across generators but makes the model blind to semantic forgeries. DeCLIP sweeps CLIP layers manually for forgery localization and finds that ViT-L/14 transfers best from higher layers while ResNet-50 transfers best from lower ones. These three papers motivate the proposal: they show the right depth varies and matters. None of them selects depth automatically, and none selects with transfer as the criterion.

Two papers set the regime and constrain the design. PlainProbe shows that with a frozen CLIP ViT-L/14, tuning only LayerNorm on the final layer, adding L2 normalization and training with plain cross-entropy is more stable and transfers better than elaborate contrastive objectives. This warns us against adding complex auxiliary losses. Any extra term we add must earn its place against seed variance. A Rich Knowledge Space for Scalable Deepfake Detection fuses all CLIP layers through dense cross-layer attention, trained on 3.6M images with text supervision. It shows multi-layer access helps at scale but selects nothing, and it is not a fair comparison in a FaceForensics++-only regime.

The method comes from Head2Toe. Head2Toe pools and normalizes features from all layers of a frozen backbone, ranks them with Group Lasso (a penalty that zeros out whole groups of weights at once), and keeps a sparse subset. It also reports a correlation of −0.75 between how close a task is to the pretraining data and how much intermediate layers help. Forensics sits far from CLIP's pretraining, so this is a direct reason to expect gains. Head2Toe, however, selects features using in-distribution loss on VTAB classification tasks. It was never tested under a train/test distribution shift of the kind forensics faces. Your representations are in the network documents "layer affinity," meaning different tasks peak at different depths, but still requires a per-task sweep. Its result is also a natural in-domain selection baseline. The broader probing literature makes the same point. Beyond Linear Probes states outright that its probes still need a layer search, and Adaptive Safety Probing for VLA models picked layer 8 by manual validation.

The contrast with all of this work is narrow and specific. No paper here chooses among a frozen detector's internal features using a criterion built to predict cross-dataset transfer. On Improving Cross-dataset Generalization of Deepfake Detectors comes closest in spirit. It adapts to the test sample at test time, but it does so by choosing input augmentations and treats the backbone as a black box.

Core Claim

In-distribution accuracy is the wrong signal for choosing which internal features a forgery detector should read. Robustness across simulated environments inside the training data is a better signal. It can be turned into a sparse, cheap probe over a frozen encoder's layers, and it needs no labels from the target domain.

Research Program

The backbone is a frozen CLIP ViT-L/14, matching PlainProbe, DeCLIP and SD². We add EfficientNet-V2-Small to allow direct comparison with Aggregating Layers. From each of the 24 transformer layers we extract two things: the class token and the mean of the patch tokens. Each is L2-normalized, following both PlainProbe and Head2Toe. Each (layer, token type) pair forms one feature group $g$. Activations are cached once, so every selection method below is trained on identical inputs at negligible cost.

Environments $e$ are the FaceForensics++ manipulation methods (Deepfakes, FaceSwap, Face2Face, NeuralTextures), crossed with c23 and c40 compression. That gives eight environments, each sharing the same real videos. The probe is a linear head with weights $w$ on the concatenated features. It is trained to minimize

$$ \frac{1}{|E|}\sum_{e} R_e(w) \;+\; \beta\,\mathrm{Var}_e\big[R_e(w)\big] \;+\; \lambda \sum_{g} \lVert w_g \rVert_2 , $$

where $R_e$ is binary cross-entropy on environment $e$, $E$ is the set of environments, $\mathrm{Var}_e$ is the variance of those losses across environments, $w_g$ is the slice of weights for group $g$, and $\beta,\lambda$ are penalty strengths. The first term is ordinary training. The variance term penalizes features that help on some manipulations but not others; in the invariance literature this is the risk-extrapolation idea, and here it is applied to feature selection. The Group Lasso term, taken from Head2Toe, drives whole layers out of the model.

After this selection pass, we retrain a plain cross-entropy linear head on the surviving groups, as Head2Toe does. This keeps the final detector as simple as PlainProbe. Hyperparameters $\beta$ and $\lambda$ are tuned by leave-one-manipulation-out AUC within FaceForensics++. Target datasets are never touched until the final evaluation.

The analysis half of the program matters as much as the method. For every single layer, we compute three AUCs: in-distribution, leave-one-manipulation-out, and on each target dataset. We then measure rank correlation between the within-training rankings and the target rankings. This tests directly whether FaceForensics++ contains enough diversity to predict transfer. If the selected layers turn out to be shallow, the forward pass can stop at the deepest selected layer. That yields a real compute saving, which we report, although DeCLIP's finding that ViTs prefer high layers makes a large saving unlikely.

Baselines

The naive reference is a linear probe on the final-layer class token of the frozen CLIP ViT-L/14. It uses the same cached features, the same training data and a tuning budget matched to our method. Every other method must clear it.

The strongest published method in the matching regime is PlainProbe. It reports AUROC 0.7287 on Celeb-DF v2 and 0.8410 on WildDeepfake when trained on 960 FaceForensics++ frames. It is the strongest competitor because it shares our backbone, our data regime and our emphasis on simplicity, and it deliberately reads only the last layer. We will rerun it with the paper's recipe across at least five seeds in both the 960-frame setting and the full FaceForensics++ setting. Beating it means a higher mean AUROC on both targets, with a gap larger than the seed standard deviation PlainProbe reports (0.0141 on Celeb-DF v2). The test-time augmentation method in On Improving Cross-dataset Generalization reports AUC 0.669 on Celeb-DF, but it uses a different backbone. We cite it as context and do not treat it as a head-to-head result. SD² reports 87.79% mAUC on WildDeepfake after training on 3.6M images. Because the training data differ by orders of magnitude, we will not claim to beat it. If compute allows, we will rerun a CLAM-style dense all-layer attention head on FaceForensics++ only.

The manual-selection baseline is a single-layer probe at the layer with the best in-distribution validation AUC. This is the DeCLIP- and InCA-style sweep, run with the same features. A second multi-layer baseline concatenates all layers under L2 regularization, which is the spirit of Aggregating Layers. We also run the Aggregating Layers architecture itself on EfficientNet-V2-Small, for comparison with its own backbone. A third multi-layer baseline is vanilla Head2Toe, meaning Group Lasso with $\beta=0$.

Two ablations isolate the key ingredient. One sets $\beta=0$ but tunes on leave-one-manipulation-out validation. The other is a single-layer sweep that picks the layer by leave-one-manipulation-out AUC. Together they separate how much gain comes from the robustness-aware criterion and how much from multi-layer feature selection.

Why not just…?

Why not just sweep layers and keep the best one on validation data? This is what current practice does. Its failure mode is exactly our hypothesis: FaceForensics++ validation rewards layers that encode FaceForensics++ fingerprints. The rank-correlation analysis tests this directly. If in-distribution layer rankings already track Celeb-DF and WildDeepfake rankings as well as leave-one-manipulation-out rankings do, the hypothesis is false and we will report that.

Why not just sweep layers but validate on a held-out manipulation? This is the real threat. It is far simpler than our objective and may capture most of the benefit. We include it as an explicit ablation. If it matches the full method within seed noise, the honest conclusion is that the contribution is the selection criterion, not the variance penalty. We will present the finding that way and not argue for the more complex model.

Why not just concatenate every layer and let the linear head sort it out? Aggregating Layers and SD² suggest dense multi-layer access helps. A regularized linear head, however, will still put weight on whatever separates the training data best, fingerprints included. It has no reason to prefer features that transfer. The all-layer L2 baseline measures this directly. If it wins, sparsity and selection are unnecessary, though the selected model would still be cheaper to run.

Why not just fine-tune the backbone? PlainProbe shows that even small auxiliary objectives destabilize training on limited forensic data. Full fine-tuning adds far more capacity for fitting the training generator. Head2Toe also argues that fine-tuning updates mostly approximate linear combinations of existing internal features, which our probe reads directly. We therefore treat full fine-tuning as out of scope rather than as a main baseline.

Proposed Experiments

All methods train on FaceForensics++ in two budgets: PlainProbe's 960-frame budget and the full training split. Testing is on Celeb-DF v2, WildDeepfake, DFDC and DeeperForensics-1.0. We report frame-level and video-level AUC, the standard deviation over five seeds, and the number of layers retained. For cross-generator image experiments we also train on PGAN and test on StyleGAN, StyleGAN2, Glow and GMM, reporting AP. Aggregating Layers reports 88.54% mean AP across generators in this setting, which gives a reference point. The decisive checks come in three parts. First, leave-one-manipulation-out layer rankings must correlate with target-dataset layer rankings more strongly than in-distribution rankings do. Second, the full method must beat PlainProbe on both Celeb-DF v2 and WildDeepfake by more than seed noise. Third, the variance term must add gain beyond the leave-one-manipulation-out single-layer sweep. A final diagnostic checks whether the selected groups differ across the 960-frame and full-data regimes and across the two backbones. If they are stable, the selection reflects forensic structure in the encoder. If they are not, it reflects noise.