TL;DR
This paper introduces a novel replay framework for Continual Face Forgery Detection (CFFD) that avoids storing raw historical images or fragile adversarial perturbations. By condensing the statistical discrepancy between real and fake faces into a tiny bank of "distribution discrepancy maps," the method synthesizes highly diverse, privacy-preserving replay samples on the fly. This approach achieves state-of-the-art performance in preventing catastrophic forgetting under extremely tight memory budgets.
What Problem Does This Paper Address?
As deepfakes and generative AI models evolve, new face manipulation techniques emerge sequentially. To keep up, face forgery detectors must learn continuously. However, deep learning models suffer from catastrophic forgetting—as they learn to detect new types of forgeries, they lose the ability to recognize older ones.
Standard continual learning relies on replay methods, which typically fall into two camps:
- Selection-based replay: Storing a small subset of real and fake images from past tasks. Under strict memory constraints, this fails to capture the diversity of forgery cues and introduces severe privacy/identity leakage risks.
- Perturbation-based replay: Storing universal adversarial perturbations that mimic forgery traces. These are highly dependent on the model's current decision boundaries and break down when the model's backbone updates.
The authors argue that the fundamental goal of replay is not to show the model old images, but to reinstate the statistical distribution of previous forgery tasks.
Methods
The authors propose a two-step framework: Distribution-Discrepancy Condensation (DDC) and Manifold-Consistent Replay (MCR).
1. Distribution-Discrepancy Condensation (DDC)
Instead of saving images, DDC models the mathematical difference between real and fake face distributions. It does this in the characteristic-function (CF) space (the Fourier transform of the probability distribution).
- It factorizes the fake distribution as a combination of the real distribution and a surrogate discrepancy factor.
- This factor is optimized into a tiny bank of spatial-domain Distribution Discrepancy Maps (DDMs) (only 50 maps per task).
- A min-max adversarial training scheme ensures these maps capture the most informative frequency components where real and fake images differ.
2. Manifold-Consistent Replay (MCR)
During later training stages, the model needs to use these stored DDMs to generate replay samples. To ensure these synthetic samples look like natural images (staying on the correct "energy manifold"), the authors introduce a variance-preserving composition method.
By standardizing the DDMs and blending them with current-stage real faces using a mathematical formulation identical to the forward mixing process in Denoising Diffusion Probabilistic Models (DDPM), they inject historical forgery traces into new real faces without distorting natural image statistics.
Datasets
The framework was evaluated across two distinct incremental protocols using five prominent face forgery benchmarks:
- FaceForensics++ (FF++): A hybrid-forgery dataset containing multiple manipulation types (Deepfakes, Face2Face, FaceSwap, NeuralTextures).
- Deepfake Detection (DFD)
- Deepfake Detection Challenge Preview (DFDC-P)
- Celeb-DF v2 (CDF2)
- DF40: A large pool used to source newly emerging forgery categories.
The protocols tested both Dataset Incremental scenarios (where both real and fake distributions change over time) and Forgery-Type Incremental scenarios (where the real face carriers remain consistent but the forgery types change).
Key Insights and Ideas
- Replay the Discrepancy, Not the Data: Forgery detection relies on subtle traces rather than semantic image content. Shifting the replay target from the image space to the real-to-fake distribution discrepancy space solves both the memory bottleneck and the privacy issue.
- Diffusion Math as a Blending Tool: Utilizing a DDPM-style variance-preserving constraint provides an elegant, training-free way to keep synthetic replay samples on the natural image manifold.
- Privacy by Design: Because the DDMs only store abstract statistical discrepancies, they do not retain facial identity information. This makes the method highly resilient against identity linkage and membership inference attacks.
Key Results
The proposed method was tested in two variants: DDC-MIR★ (standalone replay) and DDC-MIR† (integrated into the prior state-of-the-art framework, SUR-LID).
- Superior Performance: On the dataset-incremental benchmark (Protocol 1), the standalone method achieved 91.08% Average Accuracy (AA), outperforming all previous baselines despite using a fraction of the memory budget (50 maps vs. 500 raw images). When integrated with SUR-LID, it pushed the state-of-the-art to 92.16% AA.
- Mitigating Forgetting: It significantly reduced Average Forgetting (AF) to just 2.85% on Protocol 1 and 2.10% on Protocol 2.
- High Distribution Fidelity: Feature-space analysis proved that the generated samples closely align with historical fake distributions far better than perturbation or selection baselines.
- Privacy Verification: Using a frozen ArcFace encoder, the identity linkability (ArcFace R1-Link at 0.1% FAR) dropped to 3.91% for this method, compared to over 24% for methods that store raw images, confirming a massive reduction in identity leakage risk.
Limitations and Future Work
The authors note that their framework relies on a key assumption: carrier-dependence. It assumes that real faces from the current training stage can serve as semantically compatible "carriers" for historical forgery cues. If future real face data distributions shift drastically from past real face distributions, the composition mechanism might face compatibility challenges. Investigating how to handle severe cross-stage real-face distribution drift remains an open direction for future research.
Summary updated