Fair Supernets as Instruments for Predicting Unseen-Forgery Robustness

Deepfake detectors routinely score above 99% on the kinds of fakes they were trained on. They then fall toward chance on fakes made by a different method. Choosing a detector's architecture is therefore really a question of which designs will hold up on manipulations nobody has shown them yet, and the only test is to train each candidate separately and check it on a different dataset. That is expensive. A weight-sharing supernet could make the question cheap. A supernet is one large network that contains many candidate architectures as sub-paths sharing a single pool of weights, so any candidate can be scored without training it from scratch. The catch is that supernet scores are trustworthy only if every sub-path received a fair share of training. "Fair path sampling" methods enforce this, so that the supernet's ranking of candidates matches their true ranking.

This note proposes to treat a fairly trained supernet as a measurement instrument for cross-dataset robustness, and to test that instrument directly. The plan is to train a supernet over forensic design choices on three of the four FaceForensics++ manipulation methods. Candidates are ranked by how well they detect the held-out fourth method. The project then measures whether that ranking predicts each architecture's true stand-alone performance on Celeb-DF and Celeb-DF++, which are never used for selection. The central question is whether fairness in path sampling is what makes such a proxy trustworthy. The worry is that unfair sampling confuses two things: an architecture being under-trained, and an architecture generalizing poorly.

Core Claim

The claim has three parts.

  1. The usual proxy fails. Within-domain validation accuracy, the standard ranking signal in neural architecture search, does not predict cross-dataset robustness for forgery detectors.
  2. A held-out-manipulation proxy may work, but only under fair sampling. Ranking by held-out-manipulation accuracy inside the supernet can predict it, provided every candidate operation and every exit depth is trained equally.
  3. Unfair sampling biases the ranking in a predictable direction. Fast-converging, shortcut-prone paths would be favored, which is exactly the failure that kills cross-dataset transfer.

If the claims hold, the payoff is a cheap, trustworthy selection procedure, plus a map of which architectural features carry robustness. If they fail, we get a clean negative result: architecture rankings for forensics cannot be estimated from shared weights, and search in this field should stop.

Where This Sits

The forensics literature motivates the gap.

  • Celeb-DF built a harder face-swap benchmark and found detectors averaging 56.9% AUC on it (area under the ROC curve, where 50% is chance).
  • Celeb-DF++ extended this to 22 face-swap, reenactment and talking-face methods. It reports a 73.8% average video-level AUC when training on FF++, and a further 7.5% drop under heavy compression.
  • DeepfakeBench standardized 15 detectors. It found that plain Xception and EfficientNet-B4 match elaborate methods once trained with shared augmentations, and that depthwise separable convolutions help.

The last finding hints that architecture choices matter, but DeepfakeBench compares fixed published backbones rather than exploring a design space.

Two forensics papers supply hypotheses about which design dimensions matter.

We reuse both ideas as search dimensions instead of hand-fixing them.

Two prior attempts sit at the intersection.

Both select architectures by in-domain signals. Both let candidate operations compete jointly for gradients, which is the regime where fast-converging operations win. Neither asks whether its search signal predicts robustness on unseen data. The present proposal differs on all three points: it uses decoupled, fair weight training, it uses a held-out-manipulation selection signal, and it explicitly validates rank correlation against stand-alone cross-dataset results.

The NAS literature supplies the method and also constrains it.

  • Single Path One-Shot NAS decouples supernet training from search by sampling one path uniformly at random each step.
  • FairNAS goes further. It samples, without replacement, a set of paths per step so that every candidate operation in every layer is trained exactly once per step, and it accumulates their gradients into one update. FairNAS reports raising Kendall's tau, a rank-agreement score between −1 and 1, from about 0.4 to 0.9487 on ImageNet.

FairNAS's strict fairness requires the same number of choices in every layer. That constrains our search-space design.

Both are reasons to expect the instrument might fail, and the experiments are designed to detect that.

Research Program

Search space. The search space is a Xception-like backbone of five stages. Each layer offers the same four choices, so FairNAS strict fairness applies without padding:

  • a 3×3 depthwise-separable convolution;
  • a 5×5 depthwise-separable convolution;
  • a 3×3 dilated separable convolution;
  • a 3×3 standard convolution.

Two further dimensions handle the forensic hypotheses directly.

Input stem. The stem is a per-network choice between RGB and RGB plus block wavelet and Fourier channels in YCbCr color, following TwoStreamNet. Both stems are trained every step, again by sampling without replacement.

Exit depth. Every stage carries a lightweight patch-level classification head that scores each spatial location and averages the scores. This follows the truncated classifiers of Chai et al. All heads receive loss on every step. This makes the comparison across depths fair by construction, and it avoids the trap of shallow and deep paths converging at different speeds and being sampled unequally. An architecture is then a tuple of (stem, per-layer operations, exit stage).

Training. The supernet trains on FF++ at c23 compression using DeepfakeBench preprocessing and augmentations. The training split uses three manipulation methods, held out in rotation over four folds. Two sampling regimes are compared:

  • strict fairness, with exactly-once sampling and gradient accumulation;
  • uniform single-path sampling.

A third, deliberately unfair regime uses sampling weighted by current loss. It mimics the "rich get richer" dynamics of joint gradient search.

Ranking signal. Each candidate is scored inside the supernet by its AUC on the held-out FF++ manipulation, averaged over folds. This is the only selection signal. Celeb-DF, Celeb-DF++ and DFDC-P are touched only for final evaluation and for measuring ranking quality.

Ground truth. About 60 architectures are sampled to span the space, stratified by exit stage and stem. Each is trained from scratch under an identical recipe and evaluated on Celeb-DF v2 and on Celeb-DF++ under its generalized-forgery and generalized-cross-dataset protocols. The central quantities are three Kendall's tau values, each computed against stand-alone cross-dataset AUC:

  • supernet in-domain rank;
  • supernet held-out-manipulation rank;
  • stand-alone held-out-manipulation rank.

The third isolates whether failures come from the proxy itself or from weight sharing.

Failure analysis. If the unfair regime ranks badly, we test the specific mechanism. If loss-weighted sampling systematically over-ranks deep, high-capacity exits and under-ranks shallow patch exits relative to their stand-alone robustness, that confirms the confound between "under-trained" and "poorly generalizing."

Baselines

Naive reference: Xception under DeepfakeBench. The do-nothing reference is Xception trained on FF++ c23 with DeepfakeBench's unified augmentations. DeepfakeBench shows it to be as strong as most specialized detectors. Its reported cross-dataset weakness, 65.50% AUC from FF++ to Celeb-DF in ADD, sets the floor. EfficientNet-B4 is run alongside it.

Strongest published detector: SPSL. The strongest existing method on the same question is SPSL, the frequency-based detector that DeepfakeBench reports as having the highest cross-domain average AUC, 78.75%. We rerun it inside the same DeepfakeBench pipeline, on the same FF++ c23 training data, with the same tuning budget.

Context, not head-to-head: Face X-Ray. Face X-Ray reports 80.58% AUC on Celeb-DF. It is trained on synthetic blended images, so it uses different data. It is reported for context, and also combined with our selected architecture as a data-level add-on rather than treated as a direct competitor.

Prior NAS-for-forensics: ADD. ADD's 66.48% from FF++ to Celeb-DF is the prior search result. We reimplement its DARTS-style search over our space with the same compute, which separates the effect of our search space from the effect of our selection procedure.

Bar for success. The searched model should exceed SPSL's cross-domain average under identical evaluation. It should also exceed the truncation-sweep baseline below on Celeb-DF++ face-reenactment and talking-face subsets. These subsets matter because no candidate has seen anything like them.

Ablations isolating the key ingredient. Two ablations do this. The first swaps strict fairness for uniform and loss-weighted sampling with everything else fixed. The second swaps the held-out-manipulation signal for in-domain validation with everything else fixed.

Why not just…?

Why not just use Xception or EfficientNet-B4 with strong augmentation? DeepfakeBench makes this the honest default, and it may well be competitive. The evidence against it as an endpoint is the size of the cross-dataset drop it still suffers, documented by ADD, Face X-Ray and Celeb-DF++. Our ground-truth set includes Xception-like configurations, so the experiment directly shows whether any point in the space beats it under identical augmentation. If none does, that is the result.

Why not just truncate a standard backbone at different depths and pick the best? This is the most serious threat. Chai et al.'s finding suggests that receptive-field size may be most of what matters, and a five-point depth sweep is far cheaper than a supernet. We take it seriously by running the truncation sweep as a baseline with the same held-out-manipulation selection rule. We will also report how much of the stand-alone variance across our 60 architectures is explained by exit stage alone. If exit stage explains nearly all of it, we will report that the search is unnecessary and the instrument's value reduces to confirming the depth result at scale.

Why not just select stand-alone models by held-out-manipulation accuracy, with random search and no supernet? This removes weight sharing and its interference risks, and the stand-alone held-out-manipulation tau is computed precisely to measure it. If that tau is high and the supernet's is much lower, the honest conclusion is that random search plus this selection signal is the method. The supernet then only buys cost reduction, which we quantify in GPU hours.

Why not just use DARTS as ADD did? Joint gradient optimization lets candidate operations compete for updates. FairNAS links that kind of competition to roughly 0.4 rank correlation. Our reimplemented DARTS search and the loss-weighted sampling ablation test whether that bias specifically penalizes robust paths in forensics.

Proposed Experiments and Decisive Checks

Datasets. Training uses FF++ c23. Evaluation covers four settings:

  • Celeb-DF v2;
  • DFDC-P;
  • Celeb-DF++ GF-eval and GFD-eval, broken down by face-swap, reenactment and talking-face;
  • GFQ-eval for heavy compression.

Metrics. Frame-level and video-level AUC for detection, and Kendall's tau between proxy ranks and stand-alone cross-dataset AUC for the instrument. For the instrument there is no reported result found on Lacuna for supernet ranking correlation against cross-dataset forensic performance. The only reference tau is FairNAS's 0.9487, and that is in-domain on ImageNet.

First decisive check: does strict fairness improve the proxy's rank correlation? The test is whether strict-fairness tau on held-out-manipulation rank, measured against Celeb-DF++ AUC, is clearly higher than the uniform and loss-weighted taus, with bootstrap confidence intervals over the 60 architectures. If it is not, fairness is irrelevant to this problem.

Second decisive check: does the held-out-manipulation proxy beat in-domain validation? The test is whether the held-out-manipulation proxy beats in-domain validation as a predictor. If it does not, no supernet procedure can select for robustness from FF++ alone.

Third decisive check: is the gain more than depth? The test is whether the architecture selected under the fair, held-out regime beats both SPSL's 78.75% average under our rerun and the truncation sweep on Celeb-DF++ reenactment and talking-face subsets. Those subsets are where Face X-Ray-style blending cues are weakest and where any gain must come from the learned design rather than from seen artifacts.