Many Independent Cues: Decorrelated Evidence Blocks for Deepfake Detectors That Survive New Manipulations

Deepfake detectors work well on the forgery methods they were trained on and fail on new ones. A detector trained on face-swap videos may score above 95% area under the ROC curve (AUC) on familiar data and drop toward chance on talking-face or reenactment fakes. The usual explanation is that the detector latches onto one cheap artifact, such as the blending seam a face-swap leaves, and that artifact simply does not exist in the new fakes. This proposal treats the failure as a geometry problem. A binary real-versus-fake loss is satisfied once a single direction in feature space separates the classes, so nothing pushes the network to learn a second, third, or fourth way of recognizing a fake. When the one cue it relies on disappears, the detector has no fallback.

We propose to train detectors whose evidence is redundant while their features are not. The final feature vector is split into several blocks. Each block has its own small classifier and must detect fakes on its own. A decorrelation penalty, adapted from self-supervised learning, forbids the blocks from encoding the same information. The result works like a built-in ensemble of detectors that are forced to look at different things. A shift in manipulation method then has to defeat every cue at once, rather than only the dominant one. We also introduce a diagnostic for existing detectors: count how many mutually orthogonal directions in their feature space separate real from fake. We predict that this count, more than within-domain accuracy, tracks how well a detector transfers across datasets.

Core Claim

The claim has two parts, and both can be tested. The first is diagnostic. Detectors that generalize poorly have a small "discriminative rank." We measure it by repeatedly fitting a linear real/fake probe on frozen features, projecting out the found direction, and refitting. The count is the number of rounds before held-out training-domain AUC falls below a fixed threshold. We expect this count to correlate with cross-dataset AUC across the detectors in DeepfakeBench.

The second part is interventional. Enforcing several individually sufficient, mutually decorrelated feature blocks raises discriminative rank and improves transfer to unseen manipulation scenarios. Simply decorrelating the whole feature vector does not, and independently trained ensembles do not either.

Concretely, let the head output $K$ blocks $z^{(1)},\dots,z^{(K)}$, each with its own linear classifier giving logit $s_k$. Let $\tilde z^{(k)}$ denote block features standardized within each class (real and fake separately), so that the real/fake mean difference is not itself penalized. The training loss is

$$ \mathcal{L} = \sum_{k=1}^{K} \mathrm{BCE}(s_k, y) \;+\; \lambda \sum_{c\in\{\text{real},\text{fake}\}} \sum_{k\neq l} \big\| C^{(c)}_{kl} \big\|_F^2 \;+\; \mu \sum_{k} \sum_{j} \max\!\big(0,\, 1-\mathrm{std}(\tilde z^{(k)}_j)\big), $$

where $y$ is the real/fake label, $C^{(c)}_{kl}$ is the cross-correlation matrix between blocks $k$ and $l$ computed over class-$c$ samples in the batch, and the last term is a variance hinge on each dimension $j$ that keeps blocks from going dead. At test time the detector averages the block logits. We also report the maximum over blocks, which implements the rule "any cue suffices."

Penalizing correlation within each class is the key design choice. A whole-batch penalty, as in DeCov or VICReg, would fight the label signal itself or spread irrelevant identity variation across dimensions. The within-class version only removes shared nuisance and shared artifact structure between the blocks.

Positioning Against Prior Work

The redundancy literature supplies the method. Barlow Twins drives the off-diagonal entries of a cross-correlation matrix to zero. VICReg separates this into variance, invariance, and covariance terms. We reuse the covariance penalty and the variance hinge, but apply them between supervised blocks rather than between augmented views.

DeCov penalizes activation covariance during supervised training. It shrinks generalization gaps on CIFAR and ImageNet, and on a biased MNIST task it resisted spurious correlations better than dropout. That evidence motivates this proposal. The same paper also constrains it: the benefits were muted when fine-tuning pretrained models. For that reason we place the trainable blocks in a fresh head on top of any pretrained backbone. Fly-CL reports strong multicollinearity in frozen pretrained embeddings, which suggests that a decorrelating head over a frozen model has room to help. Jing et al. show that strong augmentation and implicit weight alignment cause dimensional collapse. This is a warning that heavy forensic augmentation may make the low discriminative rank we predict worse.

On the forensics side, three papers motivate the gap. Celeb-DF++ shows that detectors trained on FaceForensics++ average only 73.8% video-level AUC across 22 face-swap, reenactment, and talking-face methods. It attributes the drop to generator variety and never examines the geometry of the detectors' features. Chai et al. obtain cross-generator transfer by truncating networks to small receptive fields, which removes global shortcuts by removing capacity. LAE keeps capacity but requires pixel masks to steer attention.

Our approach keeps full depth and needs no masks. It changes what the head is allowed to learn, not where the network looks.

The closest prior work is CrossDF. It splits features into forgery, domain, and irrelevant spaces and uses adversarial mutual-information minimization to separate them, reaching 0.779 AUC from FF++ to Celeb-DF v2. CrossDF decorrelates forgery features from nuisance. It leaves the forgery space itself free to collapse onto one artifact, and it needs domain labels and adversarial training. We do the reverse: we decorrelate forgery evidence from other forgery evidence, using closed-form correlation penalties and no domain labels.

BPL is the strongest detector in the file. It tunes only the bias terms of CLIP, the pretrained vision-language model, on retrieved real-fake pairs, and uses a heuristic threshold, "discrepancy inversion," to stop over-separation. It protects pretrained geometry by restricting updates, but it never measures or controls how many cues the detector uses. Our head can sit on top of BPL's backbone, so the two ideas can be combined as well as compared.

Research Program

The first phase is diagnostic. We run the iterative-nullspace probe on frozen penultimate features from the DeepfakeBench detectors, trained under its standard FF++ protocol, and correlate discriminative rank with each detector's cross-domain AUC. We also report the effective rank of the full feature covariance. This tests whether discriminative rank carries information that generic dimensional collapse does not. If discriminative rank shows no relation to transfer, the motivation for the method weakens, and we will report that.

The second phase builds the block head on two backbones. The first is Xception trained from scratch, where DeCov-style penalties have been shown to work. The second is CLIP ViT-L/14 with bias-only tuning, matching BPL's setup. We sweep $K$ in {1, 2, 4, 8} at fixed total width, along with the penalty weight $\lambda$. Here $K=1$ reduces to a plain classifier with a variance hinge.

One known weakness is that linear decorrelation can be satisfied by nonlinear copies of the same cue. We therefore add an ablation that replaces the correlation penalty with a kernel independence measure (HSIC). We also measure, on fake samples, how often blocks fail on the same inputs, which is the quantity that actually matters for fallback.

The third phase is mechanistic. Using Celeb-DF++'s scenario labels, we train only on face-swap and check whether different blocks become the ones that carry reenactment and talking-face detection. If they do, decorrelation preserved cues that a single-direction detector discards.

Baselines

The do-nothing reference is the DeepfakeBench Xception and EfficientNet-B4 recipe: plain binary cross-entropy with the benchmark's standard augmentation, run through its codebase with the identical face preprocessing used for every other method. DeepfakeBench shows that this naive baseline is competitive with specialized detectors, so any gain has to clear it first. On cross-domain transfer the strongest DeepfakeBench detector is SPSL, at 78.75% average cross-domain AUC. We retrain it in the same pipeline rather than quoting the number.

The strongest existing method is BPL. It reports 83.8% AUC when trained on Celeb-DF v2 and tested on DFDC, and 88.86% average AUC on DF40 cross-manipulation. We first reproduce BPL from its described recipe on the same backbone and training set. Our block head is then added on the same bias-only CLIP backbone with the same number of training steps and the same tuning budget, a fixed small grid per method selected on a held-out training-domain split, never on test domains. Beating BPL means a gain in DFDC and DF40 average AUC that is outside seed variance over at least three seeds. On those benchmarks we expect, at best, a modest increase. The more important evidence is a larger gain on the Celeb-DF++ scenario transfer, where BPL has no reported result in the file.

CrossDF is the closest decorrelation competitor (0.779 FF++ to Celeb-DF v2; 0.802 on diffusion-generated faces from DiFF). We compare against its reported numbers and, if the method can be reimplemented in DeepfakeBench, against a retrained version.

Two ablations isolate the key ingredient. The first applies a whole-vector VICReg or DeCov penalty at the same width and weight: the obvious idea, with no blocks and no per-block heads. The second uses blocks with per-block heads but no cross-block penalty. The proposal only works if the full method beats both.

Why not just…?

Why not just add DeCov or VICReg to the detector's features? This is the obvious combination. It decorrelates dimensions but places no requirement that more than one direction be predictive. A whitened feature vector still lets the classifier rely on a single seam direction, and whole-batch penalties partly fight the real/fake signal itself. Cogswell et al. also found that the benefit is muted in fine-tuning, which is how most detectors are trained. The whole-vector ablation tests this directly. If it matches the block method, our contribution reduces to "use DeCov on forensics," and we will say so.

Why not just ensemble several independently trained detectors? This is the most serious threat. Ensembles also give multiple votes. Our expectation is that independently trained members, seeing the same data and the same cheapest artifact, converge to correlated shortcuts and fail on the same unseen fakes. We test it with a $K$-member ensemble of the same total parameter count and compute. We measure the members' error correlation on held-out manipulations as well as AUC. If ensemble members turn out to be naturally diverse and match our method, the decorrelation penalty is unnecessary.

Why not just augment more aggressively, or truncate the receptive field? DeepfakeBench reports that blur-style augmentation trades high-quality accuracy for compression robustness. Jing et al. show that strong augmentation itself causes collapse, so heavy augmentation could lower discriminative rank. Patch truncation as in Chai et al. removes the global context needed for reenactment and talking-face fakes, whose local textures are clean. We include a truncated Xception and a heavy-augmentation Xception as baselines, and we report their discriminative rank, so the diagnostic shows whether these methods help for the reason we claim or for a different one.

Decisive Experiments

The headline test is scenario transfer on Celeb-DF++ under its GF-eval protocol: train on face-swap methods only, test separately on reenactment and talking-face, against the 73.8% FF++-trained reference. We also run the GFQ-eval compression protocol, to check that independent cues survive H.264 compression better than a single cue. Secondary tests are FF++ leave-one-manipulation-out, FF++ to Celeb-DF v2 (against CrossDF's 0.779), Celeb-DF v2 to DFDC (against BPL's 83.8%), and DF40 average AUC (against BPL's 88.86%). Metrics are video- and frame-level AUC, plus discriminative rank and pairwise block error correlation.

The proposal succeeds if three things hold together. Discriminative rank predicts cross-dataset AUC across existing detectors. The block method raises discriminative rank and beats both the whole-vector ablation and the matched ensemble on the reenactment and talking-face splits. And the blocks that remain predictive on unseen scenarios are different blocks from the ones that dominate on face-swap. If gains appear only on DFDC and DF40, and not on scenario transfer, the cue-fallback explanation is wrong even if the numbers improve.