Direct Discrepancy Replay: Distribution-Discrepancy Condensation and Manifold-Consistent Replay for Continual Face Forgery Detection

Tianshuo Zhang, Haoyuan Zhang, Siran Peng, Weisong Zhao, Xiangyu Zhu, Zhen LeiarXiv 2026 · arXiv

TL;DR

This paper introduces a novel replay framework for Continual Face Forgery Detection (CFFD) that avoids storing raw historical images or fragile adversarial perturbations. By condensing the statistical discrepancy between real and fake faces into a tiny bank of "distribution discrepancy maps," the method synthesizes highly diverse, privacy-preserving replay samples on the fly. This approach achieves state-of-the-art performance in preventing catastrophic forgetting under extremely tight memory budgets.

What Problem Does This Paper Address?

As deepfakes and generative AI models evolve, new face manipulation techniques emerge sequentially. To keep up, face forgery detectors must learn continuously. However, deep learning models suffer from catastrophic forgetting—as they learn to detect new types of forgeries, they lose the ability to recognize older ones.

Standard continual learning relies on replay methods, which typically fall into two camps:

  1. Selection-based replay: Storing a small subset of real and fake images from past tasks. Under strict memory constraints, this fails to capture the diversity of forgery cues and introduces severe privacy/identity leakage risks.
  2. Perturbation-based replay: Storing universal adversarial perturbations that mimic forgery traces. These are highly dependent on the model's current decision boundaries and break down when the model's backbone updates.

The authors argue that the fundamental goal of replay is not to show the model old images, but to reinstate the statistical distribution of previous forgery tasks.

Figure 1: Comparison of replay paradigms in CFFD. Selection- based replay uses sparse support samples and only coarsely approximates the previous fake distribution. Perturbation- based replay preserves boundary-related signals but ignores distribution alignment. Our method condenses the real-to- fake discrepancy and unfolds it through MCR in the next stage to produce replay samples that better match the previ- ous fake distribution. Radial charts show a CF-based distri- bution measure; closer overlap means better replay fidelity.

Methods

The authors propose a two-step framework: Distribution-Discrepancy Condensation (DDC) and Manifold-Consistent Replay (MCR).

1. Distribution-Discrepancy Condensation (DDC)

Instead of saving images, DDC models the mathematical difference between real and fake face distributions. It does this in the characteristic-function (CF) space (the Fourier transform of the probability distribution).

  • It factorizes the fake distribution as a combination of the real distribution and a surrogate discrepancy factor.
  • This factor is optimized into a tiny bank of spatial-domain Distribution Discrepancy Maps (DDMs) (only 50 maps per task).
  • A min-max adversarial training scheme ensures these maps capture the most informative frequency components where real and fake images differ.

2. Manifold-Consistent Replay (MCR)

During later training stages, the model needs to use these stored DDMs to generate replay samples. To ensure these synthetic samples look like natural images (staying on the correct "energy manifold"), the authors introduce a variance-preserving composition method.

By standardizing the DDMs and blending them with current-stage real faces using a mathematical formulation identical to the forward mixing process in Denoising Diffusion Probabilistic Models (DDPM), they inject historical forgery traces into new real faces without distorting natural image statistics.

The overall DDC and MCR pipeline.

Datasets

The framework was evaluated across two distinct incremental protocols using five prominent face forgery benchmarks:

  • FaceForensics++ (FF++): A hybrid-forgery dataset containing multiple manipulation types (Deepfakes, Face2Face, FaceSwap, NeuralTextures).
  • Deepfake Detection (DFD)
  • Deepfake Detection Challenge Preview (DFDC-P)
  • Celeb-DF v2 (CDF2)
  • DF40: A large pool used to source newly emerging forgery categories.

The protocols tested both Dataset Incremental scenarios (where both real and fake distributions change over time) and Forgery-Type Incremental scenarios (where the real face carriers remain consistent but the forgery types change).

Key Insights and Ideas

  • Replay the Discrepancy, Not the Data: Forgery detection relies on subtle traces rather than semantic image content. Shifting the replay target from the image space to the real-to-fake distribution discrepancy space solves both the memory bottleneck and the privacy issue.
  • Diffusion Math as a Blending Tool: Utilizing a DDPM-style variance-preserving constraint provides an elegant, training-free way to keep synthetic replay samples on the natural image manifold.
  • Privacy by Design: Because the DDMs only store abstract statistical discrepancies, they do not retain facial identity information. This makes the method highly resilient against identity linkage and membership inference attacks.

Key Results

The proposed method was tested in two variants: DDC-MIR★ (standalone replay) and DDC-MIR† (integrated into the prior state-of-the-art framework, SUR-LID).

  • Superior Performance: On the dataset-incremental benchmark (Protocol 1), the standalone method achieved 91.08% Average Accuracy (AA), outperforming all previous baselines despite using a fraction of the memory budget (50 maps vs. 500 raw images). When integrated with SUR-LID, it pushed the state-of-the-art to 92.16% AA.
  • Mitigating Forgetting: It significantly reduced Average Forgetting (AF) to just 2.85% on Protocol 1 and 2.10% on Protocol 2.
  • High Distribution Fidelity: Feature-space analysis proved that the generated samples closely align with historical fake distributions far better than perturbation or selection baselines.
Figure 3: Analysis of historical information encoded in replay samples under different replay budgets 𝐾∈{1, 5, 10, 20, 50}. (a) Cosine to Fake Centroid, (b) Fake-Real Margin, (c) Probe AUC, and (d) Probe AP.
  • Privacy Verification: Using a frozen ArcFace encoder, the identity linkability (ArcFace R1-Link at 0.1% FAR) dropped to 3.91% for this method, compared to over 24% for methods that store raw images, confirming a massive reduction in identity leakage risk.

Limitations and Future Work

The authors note that their framework relies on a key assumption: carrier-dependence. It assumes that real faces from the current training stage can serve as semantically compatible "carriers" for historical forgery cues. If future real face data distributions shift drastically from past real face distributions, the composition mechanism might face compatibility challenges. Investigating how to handle severe cross-stage real-face distribution drift remains an open direction for future research.

Summary updated

Abstract

Continual face forgery detection (CFFD) requires detectors to learn emerging forgery paradigms without forgetting previously seen manipulations. Existing CFFD methods commonly rely on replaying a small amount of past data to mitigate forgetting. Such replay is typically implemented either by storing a few historical samples or by synthesizing pseudo-forgeries from detector-dependent perturbations. Under strict memory budgets, the former cannot adequately cover diverse forgery cues and may expose facial identities, while the latter remains strongly tied to past decision boundaries. We argue that the core role of replay in CFFD is to reinstate the distributions of previous forgery tasks during subsequent training. To this end, we directly condense the discrepancy between real and fake distributions and leverage real faces from the current stage to perform distribution-level replay. Specifically, we introduce Distribution-Discrepancy Condensation (DDC), which models the real-to-fake discrepancy via a surrogate factorization in characteristic-function space and condenses it into a tiny bank of distribution discrepancy maps. We further propose Manifold-Consistent Replay (MCR), which synthesizes replay samples through variance-preserving composition of these maps with current-stage real faces, yielding samples that reflect previous-task forgery cues while remaining compatible with current real-face statistics. Operating under an extremely small memory budget and without directly storing raw historical face images, our framework consistently outperforms prior CFFD baselines and significantly mitigates catastrophic forgetting. Replay-level privacy analysis further suggests reduced identity leakage risk relative to selection-based replay.
BibTeX
@misc{zhang2026direct,
  author = {Tianshuo Zhang and Haoyuan Zhang and Siran Peng and Weisong Zhao and Xiangyu Zhu and Zhen Lei},
  title = {{Direct Discrepancy Replay: Distribution-Discrepancy Condensation and Manifold-Consistent Replay for Continual Face Forgery Detection}},
  year = {2026},
  eprint = {2604.12941},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.12941},
}

Loading related content...

Figure