Context-Anchored Normalization for Deepfake Detectors Under Domain and Label Shift

Deepfake detectors that look nearly perfect on their training data fail badly on new sources. A detector trained on FaceForensics++ falls to around 57% average AUC on Celeb-DF. Much of this failure comes from nuisance factors such as camera, lighting and video compression, not from the forgery itself. A cheap and well-known fix for this kind of shift is to re-estimate the network's batch normalization statistics on unlabeled test data. Batch normalization layers standardize each feature channel using a stored mean and variance; the fix swaps in the test data's own mean and variance, with no retraining and no gradients. In a forensic setting this fix has a hidden hazard. When the test data is mostly fake, the re-estimated statistics absorb the forgery signal itself, and normalization partly removes the evidence the detector needs. Real deployments rarely look like balanced benchmarks. A feed may be almost entirely authentic, a disinformation burst may be almost entirely fake, and a single video is by definition all one class.

We propose to estimate the test-time statistics only from the part of each frame that a face-swap or reenactment does not touch: the background, hair, neck and shoulders surrounding the face. That surrounding region shares the video's camera, codec and lighting, so it carries the domain shift. It is unmanipulated whether the video is real or fake, so it carries no information about the label. The detector therefore adapts to the domain without seeing the class. The method needs no backward pass, no synthetic training pairs and no labels, and it adds a single forward pass over frames the detector already processes.

Core Claim

Our central claim is that context-anchored normalization matches whole-image adaptive batch normalization when the test stream is balanced, and holds up where whole-image adaptation breaks: in streams with skewed fake prevalence and in per-video adaptation. It should also be competitive with gradient-based test-time adaptation at a small fraction of the latency. What is new is the source of the statistics, not the act of re-estimating them. Adaptation is keyed to a spatial region that is invariant to the label by construction, which turns the known spatial-locality weakness of batch normalization into a source of label-free domain information.

Where This Sits in Prior Work

The mechanism comes from the adaptive normalization literature. Revisiting Batch Normalization For Practical Domain Adaptation showed that swapping in target statistics at every batch normalization layer recovers much of a domain gap without touching weights. On satellite cloud segmentation, for example, it raised mIoU from 38.95% to 64.50%. Test-time Batch Statistics Calibration for Covariate Shift interpolates between source and target statistics (α-BN) to stabilize small batches. The Norm Must Go On updates the statistics online with decaying momentum and converges within a few hundred samples. We reuse these estimators and change what they are computed on.

Two papers constrain the design. Limitations of Post-Hoc Feature Alignment for Robustness shows three things: adaptive normalization collapses under label shift; spatially uniform statistics are corrupted by localized changes; and deep layers, which encode class semantics, should be excluded from re-estimation. Our method is built directly around these three findings. The second constraint is Un-Mixing Test-Time Normalization Statistics, which attacks label correlation in streams by fitting a mixture of normalization components. It handles correlated labels statistically. We try to remove the label from the statistics entirely, and we test UnMix-TNS head to head.

Three forensic papers motivate the design. DeepFake Detection Based on Discrepancies Between Faces and Their Context establishes that face-swaps leave the surrounding context intact, and exploits this by comparing face and context identity embeddings. Its encoders keep frozen statistics. We use the same observation for a different purpose: as a clean domain probe rather than a detection cue. What Makes Fake Images Detectable? and Finding Facial Forgery Artifacts With Parts-Based Detectors show that forgery cues are local, concentrated at blending boundaries, eyes and jawline. This is why a spatial split between manipulated and unmanipulated pixels is meaningful.

The closest work is test-time adaptation for deepfakes. Think Twice Before Adaptation (T2A) updates batch normalization parameters and gradient-aligned weights using an uncertainty-aware negative-learning loss. It reports that plain entropy minimization collapses on deepfakes, and it costs about 0.5 s per iteration. OST synthesizes a blended pseudo-fake for each test image and takes one meta-learned gradient step. It depends on landmark detection and on blending-type artifacts. Neither method isolates parameter-free statistics adaptation, and neither addresses what happens to normalization when the label distribution of the test stream is skewed.

Research Program

The method has two phases, one during source training and one at test time.

During source training on FaceForensics++, we record a second set of running statistics for each early batch normalization layer, computed only over spatial positions in a context mask. The mask is the outer ring of the face-detector crop, with the inner face ellipse removed and a margin band around the blending boundary also removed. It is derived from the detection box alone, so there is no landmark dependence of the kind that limits OST.

At test time, for a batch of target frames (a streaming window or a single video's frames), we compute the same context-only statistics. We then transfer the observed context shift to the full-map statistics used for normalization:

$$\mu_t = \mu_s + (\hat\mu^{c}_t - \mu^{c}_s), \qquad \sigma_t = \sigma_s \cdot \hat\sigma^{c}_t / \sigma^{c}_s$$

In this equation, $\mu_s,\sigma_s$ are the stored full-map source statistics of a channel, $\mu^{c}_s,\sigma^{c}_s$ are the stored source context-only statistics, and $\hat\mu^{c}_t,\hat\sigma^{c}_t$ are the context-only statistics measured on the target batch. The shift-and-scale form assumes that the domain acts as a roughly channel-wise affine change that is shared across face and context positions at shallow depth. That is the regime in which Burns & Steinhardt found adaptive normalization works. The target estimates are smoothed with α-style interpolation toward source.

Adaptation is restricted to shallow layers. There are two reasons. Deep layers carry class semantics, and deep features at context positions have receptive fields that reach into the face. We will measure this leakage by comparing context statistics of real and fake videos layer by layer, and choose the cutoff as the deepest layer at which those statistics are statistically indistinguishable on held-out FaceForensics++ data. No target labels are involved in this choice.

Two further analyses test the premise directly. First, we quantify how much of the real/fake difference lives in per-video channel means. We measure the separability of real and fake videos using only their early-layer statistics. If it is high, whole-image adaptation must be erasing signal. Second, we test whether context statistics track domain. We check that context-statistic shifts predict full-map shifts across compression levels and datasets on real videos, where no forgery is present to confound the comparison.

Baselines

The naive reference is the source-trained detector with frozen statistics: Xception and EfficientNet-B4 trained on FaceForensics++ c23. For calibration, the file reports Xception falling from 99.26% accuracy on raw FaceForensics++ to 81.00% on heavily compressed data (FaceForensics++). It also reports 69.70% video-level AUC for Xception from FaceForensics++ to Celeb-DF (Two-Branch Recurrent Network). We will retrain our own models and report our own reference numbers.

The most direct competitor is whole-image adaptive normalization in three variants. The first is plain AdaBN applied to all layers. The second is α-BN restricted to the same shallow layers we adapt. The third is DUA-style online momentum updates. These isolate our key ingredient, because the estimator, the layers and the smoothing are identical and only the spatial source of the statistics differs. No reported result was found on Lacuna for AdaBN or α-BN on any deepfake benchmark, so we will produce these numbers ourselves. UnMix-TNS, run with its published component count and momentum, is the strongest statistics-only method aimed at label correlation.

The strongest existing deepfake-specific method is T2A. The file reports it improving average AUC of source detectors by 2.93% under unknown post-processing, including +4.25% for CORE and +3.86% for EfficientNet-B4 (T2A). OST reports 0.854 average cross-dataset AUC and 0.748 on Celeb-DF (OST). We will run T2A on identical source checkpoints, the same six target sets and the same post-processing suite, using its published hyperparameters. Any tuning budget given to T2A will also be given to our method's single interpolation weight, and tuning will be done on FaceForensics++ validation only.

Three ablations isolate the context premise. The first is a random mask of equal area, which tests whether the benefit comes from which pixels are used or merely from using fewer pixels. The second is an inner-face-only mask, which should be the worst option. The third is an oracle that computes statistics from real target videos only, giving the upper bound on a label-free estimate.

We count as beating whole-image α-BN any result that ties it within noise at 50% prevalence while showing a significantly smaller drop in balanced accuracy and false-positive rate at 1% and 99% prevalence and under per-video adaptation. We count as beating T2A matching its average AUC gain at under a tenth of its latency, or exceeding it in the skewed regimes.

Why not just…?

Why not just apply standard AdaBN or α-BN to the whole face crop? This is the real threat, and on balanced benchmarks it may perform as well as our method. The case against it is mechanistic. Burns & Steinhardt show whole-batch statistics absorb the class distribution under label shift, and per-video adaptation is the extreme case, because every frame in the batch shares a label. The prevalence sweep settles the question. If whole-image α-BN restricted to shallow layers shows no degradation at skewed prevalence, the problem we target does not exist in practice, and we would report that as a useful negative result rather than argue around it.

Why not just train with heavy compression and blur augmentation? Augmentation reduces codec shift for anticipated corruptions. It cannot anticipate unknown pipelines, however, and T2A motivates test-time adaptation precisely by the failure of fixed detectors under unseen post-processing. Augmentation also creates the train/test statistics mismatch that Burns & Steinhardt call the augmentation gap. We will train an augmented source model and apply every method on top of it. If augmentation alone closes the gap on Celeb-DF, DFDCP and the held-out corruptions, the adaptation step is unnecessary.

Why not just use gradient-based test-time adaptation like T2A or OST? These methods carry costs and dependencies beyond speed: backward passes at about 0.5 s per iteration for T2A, and landmarks plus blending-specific pseudo-labels for OST. T2A also reports that the default entropy-minimization variant collapses on deepfakes. Their pseudo-labels are also driven by the model's own predictions, so they are exposed to the same label skew. We do not assume we win on accuracy. The head-to-head on identical checkpoints, together with a combined variant that initializes T2A from context-anchored statistics, will show whether the methods compete or complement each other.

Why not just estimate the fake prevalence and correct the statistics for it? Estimating prevalence requires trusting the detector's predictions on the shifted domain, which is the thing that is broken. The inner-face-only and oracle ablations bracket how much any label-aware correction could gain.

Proposed Experiments

Source models are Xception and EfficientNet-B4, trained on FaceForensics++ c23. Targets follow T2A's protocol: Celeb-DF v1 and v2, DFD, DFDCP, UADFV and FaceShifter. Each target is evaluated clean and under blur, noise and c40-style re-encoding. Beyond the standard shuffled, balanced evaluation, we build streams with controlled fake prevalence of 50%, 10%, 1%, 90% and 99%, as well as a strict per-video mode in which statistics are computed from one video's frames alone.

We report frame-level and video-level AUC. Because AUC ignores thresholds and hides calibration damage under skew, we also report balanced accuracy and false-positive rate at the threshold fixed on FaceForensics++ validation, expected calibration error, and wall-clock latency per frame.

The decisive checks are four. The first is whether whole-image adaptation measurably loses separability or calibration as prevalence departs from 50% while context-anchored adaptation does not. The second is whether the random-mask ablation fails to reproduce the benefit, which would confirm that the gain comes from label invariance and not from sample count. The third is whether early-layer context statistics of real and fake FaceForensics++ videos are in fact indistinguishable up to the chosen depth. The fourth is whether the method closes a meaningful fraction of T2A's reported gain without any backward pass.

We scope the claims to face-swap and reenactment video. Chai et al. find that fully synthesized faces carry cues in hair and background, so the context there is not clean. We include a small fully synthetic set only to document where the method's assumption breaks.