New image generators keep appearing, and a deepfake detector trained on last year's generators has to learn each new one without losing the old ones. Current continual detectors do this by updating network weights and replaying stored past images. We propose a different design. Keep the network completely frozen. Read features from a shallow layer, where each feature describes only a small image patch. Summarize each generator, and each source of real images, as a handful of stored patch "prototypes": average feature vectors that act as a lookup table. Adding a generator then means inserting new prototypes. No weight is ever overwritten, and no past image is ever stored.
Our core claim is that in this frozen regime forgetting takes a different form. It is no longer weights being overwritten. It becomes interference between prototypes in feature space, and the layer the prototypes are read from largely decides that interference. Shallow, patch-level prototypes should carry low-level forgery traces rather than image content, so they should both generalize to unseen generators and interfere less with one another. Because the design never trains the backbone, it should work comparably on convolutional networks and vision transformers. That would remove an architecture dependence that current continual detectors carry.
Why this, and what is already done
Four findings in the literature point to this design. The Continual Deepfake Detection Benchmark (CDDB) casts continual deepfake detection as class-incremental learning over 12 generator sources (A Continual Deepfake Detection Benchmark). It finds that every tested method (iCaRL, LUCIR, DyTox) degrades sharply when the exemplar buffer shrinks. Its best performer, DyTox, relies on transformer token expansion that does not port to CNNs. Separately, Replay can provably increase forgetting proves that replaying a small subset of past samples can turn vanishing forgetting into constant forgetting in over-parameterized models. Buffers are therefore not a safe foundation. What Makes Fake Images Detectable? shows that truncating a ResNet or Xception early, so that each output sees roughly a 30×30 patch, improves transfer to unseen generators because it blocks global content shortcuts. Conditioned Prompt-Optimization for Continual Deepfake Detection (Prompt2Guard) documents the same content-distraction problem in CLIP. It fixes the problem with text-conditioned prompts, which ties the method to vision-language models.
Two works come closest to our proposal. PUDD already uses prototypes on a frozen DINOv2 backbone, with a mean-minus-sigma rule that flags fakes as outliers from real prototypes. PUDD differs from our design in three ways. It is static, it models only real images, and it uses final-layer global embeddings. The paper reports that it breaks down on natural variation in real faces such as eye closure, pose, and lighting. Prompt2Guard is exemplar-free and continual, but it learns prompt tokens by backpropagation and uses K-means only to pick which task's prompt to apply.
Our proposal departs from both on three counts. It stores prototypes for both real and fake sources, per task. It reads them from shallow patch features rather than a global embedding. It treats the backbone family as an experimental variable instead of a fixed choice. Two further works play specific roles. Open-world recognition supplies the distance-based rejection rule we reuse for flagging unseen sources. Rethinking Architecture Design motivates a prediction: its finding that transformers forget far less than CNNs under sequential training should largely vanish once nothing is trained.
Research Program
Building the prototypes. For each backbone we pick a layer $l$ and extract patch features $z \in \mathbb{R}^D$: the spatial feature map for CNNs, the patch tokens for transformers. For each incoming task $t$ we run k-means separately on the real-image patches and the fake-image patches, keeping $K$ prototypes each. For each prototype we also store a diagonal variance, so that distances can be class-conditional Mahalanobis distances, following Lee et al.. Storage per task is $2K$ vectors plus variances. No images are kept.
Scoring an image. For each patch $i$ we compute a margin
$$s_i = d_{\text{real}}(z_i) - d_{\text{fake}}(z_i),$$
where $d_{\text{real}}$ and $d_{\text{fake}}$ are the Mahalanobis distances to the nearest real and nearest fake prototype, pooled over all tasks seen so far. The image score is the mean of the top-$q$ fraction of patch margins. We use top-$q$ pooling because forgery traces are often localized.
Flagging unseen sources. Following the Nearest Non-Outlier rule of open-world recognition, a patch far from every prototype is marked "unknown." If enough of an image's patches are unknown, the image is reported as coming from an unseen source. Once that source is labeled, new prototypes are inserted.
Choosing the layer. The layer is selected once, using only the first task. We do this by holding out one of that task's sources and picking the layer that best detects it. The layer is then frozen for the rest of the stream, so no information from later tasks leaks into the choice.
Analysis. The analytical part of the program measures prototype interference at every layer and on every backbone. Our interference measure is the fraction of past-task test patches whose nearest prototype changes label after a new task is inserted. We also measure how much of each layer's feature variance is explained by image content. Following Prompt2Guard's diagnosis, we probe content using zero-shot ImageNet labels as a content proxy. The central hypothesis is that retention loss, content leakage, and unseen-generator error all fall together as the layer gets shallower, up to the point where features become too weak to separate real from fake. We test this on ResNet-50, ConvNeXt, DINOv2 ViT-B, and the CLIP ViT-B image encoder. A second hypothesis is that at the chosen layer the CNN–transformer gap in forgetting is much smaller than the one Qu et al. report for trained models.
Baselines
Naive fine-tuning. The naive reference point is sequential fine-tuning of each backbone with a binary head and no retention mechanism. The literature already shows how badly this fails. On CFFD Protocol 1 it trails HDP by about 27% in previous-task retention. In CoReD, plain transfer learning falls to 62.25% F1, against 86.16% for CoReD. Fine-tuning gets the same data and a matched tuning budget. Its role is to calibrate how much forgetting there is to remove.
Prompt2Guard. The strongest exemplar-free competitor is Prompt2Guard, which reports 90.28% average accuracy and −0.71% average forgetting on CDDB-Hard, with 81.11% on the final SAN task (Conditioned Prompt-Optimization). It is the closest competitor because it is exemplar-free, continual, and explicitly designed around the content-distraction problem we target. For fairness we run our method on the same CLIP ViT-B image encoder, with its text tower unused. We use the same CDDB-Hard task order and train/test splits. Beating it means exceeding 90.28% average accuracy on that encoder, and matching or exceeding it on ResNet-50, where Prompt2Guard cannot run at all.
DyTox with exemplars. The strongest method that stores images is DyTox, reported at about 96% average accuracy on CDDB-EASY and 88% on CDDB-HARD (CDDB). We compare against these published numbers. We also rerun DyTox at reduced buffer sizes to map the accuracy-versus-memory curve, and report our method's memory in bytes against each buffer size.
HDP on CFFD. On the continual face forgery benchmark (CFFD), the comparison is HDP, run with its released detectors. We found no reported absolute CFFD numbers for HDP on Lacuna, so it must be rerun under all three protocols.
Ablations. Three ablations isolate the proposal's ingredients. The first is final-layer global-embedding prototypes on the same frozen backbone, which removes the shallow-patch ingredient. The second keeps real prototypes only, as in PUDD, which removes the fake prototypes. The third swaps top-$q$ pooling for mean pooling.
Why not just…?
Why not just keep a replay buffer and fine-tune? CDDB shows that every method depends heavily on buffer size, and Mahdaviyeh et al. prove that small replay can increase forgetting when tasks are correlated. Different GAN generators plausibly are correlated in this sense. Replay also stores face images, which is a privacy and legal liability. That said, replay with a generous buffer may still win on raw accuracy. The DyTox memory curve settles the question: it shows what each byte of stored images buys compared with each byte of prototypes.
Why not just take frozen foundation-model embeddings and a nearest-class-mean classifier? This is the most serious threat, because it is our method minus the one ingredient we claim matters. Prompt2Guard and X²-DFD both report that foundation-model features emphasize image content over forgery signal. That is suggestive evidence against the alternative, but not proof. The final-layer ablation tests it directly. If final-layer prototypes match shallow patch prototypes on SAN, WildDeepfake, and the held-out-generator tests, our claim collapses to "frozen features suffice," and we will report exactly that.
Why not just train a linear probe per task on frozen features and ensemble them? This alternative also avoids forgetting by construction. We run it on the same shallow features, with the same frozen layer. It needs a rule for choosing which task's head to trust, and Prompt2Guard shows that choosing the wrong task fails on unseen generators. It also offers no natural way to flag novel sources. If a probe ensemble matches prototypes on both accuracy and novel-source detection, the prototype choice is not essential, and the contribution narrows to the layer finding.
Why not just use maximum softmax probability to reject new generators? On ADD2023 Track 3, softmax-probability rejection reaches 39.45% F1 on the novel class and max-logit rejection 44.30%, against 58.10% for a structured metric space (Generalized Source Tracing). We rerun both rejection rules on CDDB leave-one-generator-out splits so the comparison is made on image data rather than audio.
Proposed Experiments
The main runs cover CDDB EASY, HARD, and LONG, and CFFD Protocols 1–3, on all four backbones. We report average accuracy, average forgetting, and final-task accuracy. To check robustness to task order, we repeat HARD under five random orders and report the spread, in the spirit of APD's order-disparity metric (APD). Generalization is measured by leave-one-generator-out AUC within CDDB. Open-world behavior is measured as F1 at flagging the held-out generator as unseen.
Four outcomes decide the proposal:
- Shallow patch prototypes must beat final-layer prototypes on SAN and WildDeepfake on at least three of the four backbones.
- The method on CLIP ViT-B must reach or exceed Prompt2Guard's 90.28% CDDB-Hard average accuracy.
- On ResNet-50, the method must come within a few points of that figure. This is the architecture-agnostic claim.
- The layer sweep must show the interference fraction and the content-leakage measure falling together with depth.
If the third outcome fails, with transformers winning by a wide margin even when nothing is trained, that result is informative in its own right. It would mean the transformer advantage lies in representation geometry rather than in resistance to weight updates, and we would report the finding in that form.
References
- A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentials
- Replay can provably increase forgetting
- What Makes Fake Images Detectable? Understanding Properties that Generalize
- Conditioned Prompt-Optimization for Continual Deepfake Detection
- PUDD: Towards Robust Multi-modal Prototype-based Deepfake Detection
- Towards Open World Recognition
- Rethinking architecture design for tackling data heterogeneity in federated learning
- A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
- Continual Face Forgery Detection via Historical Distribution Preserving
- CoReD: Generalizing Fake Media Detection with Continual Representation using Distillation
- X2-DFD: A framework for explainable and extendable Deepfake Detection
- Generalized Source Tracing: Detecting Novel Audio Deepfake Algorithm with Real Emphasis and Fake Dispersion Strategy
- Scalable and Order-robust Continual Learning with Additive Parameter Decomposition