Balancing Test-Time Memory Without Trusting the Detector's Labels

Deepfake detectors trained on one benchmark lose much of their accuracy on others. Nine detectors averaged 56.9% AUC on Celeb-DF despite strong scores on FaceForensics++ (Celeb-DF). A growing response is test-time adaptation: the detector keeps updating itself on the unlabeled media it sees after deployment. Current deepfake methods do this either one image at a time or one batch at a time, and they keep no memory of the stream. Real deployment streams are badly suited to that. Most uploads are genuine, fakes arrive in bursts from a single tool, and consecutive frames come from the same video. In general vision, the standard fix is a class-balanced memory bank: a small buffer that holds equal numbers of each predicted class and replays them for adaptation. Deepfakes break this fix in a specific way. Under cross-dataset shift the detector's own real/fake predictions are close to chance, so a buffer balanced on predicted labels is balanced on noise.

We propose to balance the test-time memory on structure that does not depend on the detector's verdict. The buffer is split into quota-limited sub-buffers, one per cluster of a frozen feature space, with an extra split by source video when one is available. Within each sub-buffer, the oldest item is evicted first. Adaptation then replays from this buffer and is anchored by a tiny labeled memory of source exemplars, a few per manipulation type. The central question is empirical and has not been tested: when a forgery stream is skewed and temporally correlated, which partitioning key keeps adaptation from collapsing? The candidates are predicted label, label-free cluster, or none.

Core Claim

Our claim is that the right balancing key decides whether memory-based test-time adaptation helps or hurts deepfake detection. We expect three things. Memoryless adaptation in the style of T2A will degrade on real-heavy, bursty streams. A direct port of pseudo-label-balanced memory from RoTTA will recover only part of the loss, because its class quotas are filled by mislabeled items. Cluster-keyed per-partition queues, combined with a small labeled source anchor, will close most of the gap to an oracle buffer balanced on true labels.

The novelty is not the memory bank itself. It is the claim that, for a binary task with near-chance pseudo-labels, balancing must be decoupled from the prediction being adapted. It is also a testable account of when that matters.

Grounding in Prior Work

Three deepfake papers motivate the gap by adapting at test time without a memory of the stream.

  • OST takes one meta-learned gradient step per test image, using a synthesized blended fake, and then resets. It reports 0.854 average cross-dataset AUC. It carries nothing across images and depends on facial landmarks and blending.
  • T2A adapts batch by batch with uncertainty-aware negative learning, which trains on "not this class" signals that tolerate wrong pseudo-labels better than entropy minimization does. It reports a 2.93% average AUC gain on unknown post-processing. It was evaluated on target sets, not on skewed or ordered streams.
  • Continual deepfake work, namely CDDB, CoReD, and the exemplar-free Prompt2Guard, shows that memory matters across sequential generators. CDDB reports that shrinking the exemplar budget hurts every method. All of these use labeled task stages, not unlabeled online streams.

The method comes from the class-balanced memory literature. Queue-Based Resampling shows that per-class first-in-first-out queues make rare items survive longer in memory, a form of temporal oversampling. Tiny Episodic Memories shows that one to three exemplars per class, kept in class-partitioned ring buffers, beat reservoir sampling. We reuse the second result for the labeled source anchor.

Two papers constrain the design. RoTTA and CPL-NC balance their caches on predicted classes. The research file itself flags pseudo-label pollution of class sub-buffers as an open problem. In deepfakes this problem is at its worst: there are two classes and accuracy is near chance, so we change the key rather than inherit it.

What Makes Fake Images Detectable? finds that generalizable cues are local textures. This suggests clustering on patch-level rather than whole-face features, which we test as a secondary choice.

Research Program

The source detector is Xception trained on FaceForensics++ at c23 compression, matching T2A's setup. We also test EfficientNet-B4, which T2A also uses.

Clustering. A frozen copy of the source backbone embeds every incoming frame. Online k-means assigns each frame to one of $K$ clusters. When frames carry video IDs, the key becomes (cluster, video), and each video is capped at a few slots. This stops one long video from monopolizing a partition.

Buffer. Each partition has an equal quota and its own first-in-first-out queue. A burst of a single manipulation therefore fills only its own clusters. Earlier real and fake modes keep their slots.

Adaptation step. Every $s$ frames, the detector updates normalization statistics and affine parameters on a minibatch drawn uniformly across partitions, using T2A's negative-learning loss. We keep T2A's loss so that the only change is what gets replayed.

Labeled anchor. A supervised cross-entropy term is computed on a tiny labeled source memory: three exemplars per FaceForensics++ manipulation plus real. This term stops the adapted model from drifting into a solution where everything looks real. It deliberately relaxes strict source-free operation. That relaxation costs a few dozen stored images, and we ablate it.

Analysis. We also record how pure each partition is with respect to the hidden true labels. This tests directly whether clusters carry more label information than pseudo-labels do under shift.

Baselines

No adaptation. The naive reference is the unadapted FaceForensics++ Xception, evaluated on each stream. It sets the floor and shows how much any adaptation is worth on skewed streams.

T2A (strongest existing competitor). T2A is the strongest deepfake-specific online method in the file and the closest in spirit. It is the most direct threat to our claim. The file reports its 2.93% average AUC gain over the source model on unknown post-processing and wins on 5 of 6 targets. No reported absolute per-target AUC for T2A was found on Lacuna, and no reported result was found on Lacuna for T2A under skewed or correlated streams. We will rerun it on the identical source model, streams, batch size, and learning-rate grid. Beating it means higher stream AUC and balanced accuracy on the skewed and ordered streams, with no loss on the IID stream.

OST. OST is the strongest single-image method. It reports 0.854 average cross-dataset AUC and 0.748 on CelebDF from FaceForensics++ training. We run it per frame with its released protocol where possible and report its compute per frame alongside ours.

RoTTA ported to deepfakes. RoTTA keeps a memory balanced by predicted category, timeliness, and uncertainty. It reports a 5.9% error reduction over the best baseline on CIFAR-10-C. We give it the same buffer size and the same negative-learning loss so that only the balancing rule differs.

Ablations of the key ingredient. The ablations hold the buffer size fixed and vary the partition key. The keys are: none (reservoir sampling), predicted label, cluster, cluster plus video, and true label (oracle). We also run each key with and without the labeled anchor. Our claim survives only if the cluster key lands materially closer to the oracle than the predicted-label key does.

Why not just…?

Why not just correct the threshold for the class prior? If skew only shifted the fraction of fakes, estimating that fraction and moving the decision threshold would fix thresholded accuracy at no cost. This is a real threat for balanced accuracy at a fixed threshold. It cannot change AUC for a fixed model, though, because AUC does not depend on the prior. Skew also corrupts the normalization statistics and gradients used during adaptation. We will report a prior-corrected T2A baseline. If it matches our method on both AUC and balanced accuracy, the memory is unnecessary, and we will say so.

Why not just port RoTTA as is? Its quotas assume pseudo-labels are mostly right. Cross-dataset AUC near 57% (Celeb-DF) means they are not. The partition-purity analysis and the predicted-label ablation measure exactly how much this costs.

Why not just use a bigger pretrained detector and skip adaptation? SD² trains on 3.6 million images and still drops from 95.76% intra-domain to 87.79% mean AUC on WildDeepfake. Scale narrows the gap but does not close it. Adaptation can also be applied on top of a stronger backbone. We test this by applying our method to a CLIP-based detector where compute allows.

Why not just use larger batches or periodic resets with T2A? Larger batches dilute bursts but do not survive long runs of real videos. Resets throw away useful adaptation. We include T2A with a batch size matched to our buffer, and T2A with resets, as baselines.

Proposed Experiments

Data. We train on FaceForensics++ c23 (FaceForensics++). We test on CelebDF-v1, CelebDF-v2, DFD, DFDCP, UADFV, and FaceShifter, as in T2A, plus WildDeepfake for in-the-wild media.

Streams. We build three stream types from each target set:

  • an IID shuffle;
  • a real-heavy stream, at 90% and 98% real;
  • an ordered stream, where whole videos arrive contiguously and fakes come in manipulation-specific bursts.

As a secondary check, we run an online, label-free version of CDDB-Hard (CDDB) to cover fully synthetic generators where OST's blending cue is absent.

Metrics. We report AUC over the whole stream, balanced accuracy and fake recall at the source threshold, AUC on a held-out clean split measured after the stream (to capture forgetting), and wall-clock time per frame.

Decisive checks.

  • On the IID stream, our method should match T2A. A large gain there would mean the benefit comes from something other than balancing.
  • On the skewed and ordered streams, the cluster key should beat both the predicted-label key and T2A. Because the comparison is controlled, this gain is attributable to the key.
  • Cluster partitions should be measurably purer in true labels than predicted-label partitions. If they are not, the proposed mechanism is wrong even if the numbers improve.

If the prior-corrected baseline matches us, or if the cluster key is no better than the predicted-label key, the claim fails.