One way to check whether a model has memorized its test set is to edit the test images, keep their labels, and re-test. A model that genuinely solves the task should hold its score. A model that memorized the originals should collapse. This works well for vision-language models that read the whole image. Most face-analysis systems are different: they are cascades. A face detector finds the face, a landmark model aligns and crops it, and only then does the downstream classifier (a deepfake detector, an action-unit recognizer) see anything. In a cascade, editing the image also changes what the detector returns. The crop shifts, the alignment moves, and the downstream model receives different input for reasons unrelated to memory. A score drop after editing can therefore mean either "this model memorized the benchmark" or "the face detector moved the box." Current audits cannot tell these apart.
We propose an audit protocol for cascaded face pipelines that splits each perturbation-induced score drop into two parts. One part comes from upstream crop drift. The other comes from downstream sensitivity to changed content, and only that part is evidence of memorization. The protocol runs every counterfactual through three arms: the detector re-run as deployed, the original crop transported onto the edited image, and a calibrated jitter control that applies only the measured box shift to the unedited image. We test it on deepfake detection, where contamination can be injected under control, and we ask a question with fairness stakes. Do naive audits falsely flag clean models more often on faces that detectors already handle badly?
Core Claim
Our claim is that perturbation-based contamination tests, carried over unchanged to cascaded face pipelines, confuse memorization with upstream localization fragility. The confusion is large enough to produce false contamination flags on clean models and to understate the drop on contaminated ones.
To fix notation, let $A$ be detection AUC on the original test videos. Let $B$ be AUC on counterfactuals with the detector re-run. Let $C$ be AUC on counterfactuals cropped with the original box and landmarks. Let $J$ be AUC on original videos whose crops are shifted by the box drift measured between original and counterfactual. The naive audit statistic is $A-B$. We propose two separate quantities instead. The memorization signal is $A-C$, and the upstream share is $C-B$, cross-checked against $A-J$. The new contribution is that this decomposition is validated against known ground-truth contamination, rather than assumed.
Positioning
The method comes from Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations. That paper edits benchmark images with a diffusion generator guided by edge maps and reads contamination from the sign of the score change. Clean models gain about 31.5 points on MMStar because the edited images render cleanly, while contaminated models fall as far as −45. The paper studies end-to-end models and never has to ask where a crop comes from. We keep its counterfactual logic and add the missing upstream accounting.
Three papers motivate the concern that upstream drift is large. MPIIGaze shows that imperfect alignment injects coordinate noise that bottlenecks gaze estimation. DeepfakeBench finds that earlier deepfake results were often irreproducible purely because of cropping choices. Hiding Faces in Plain Sight shows how brittle detectors are: imperceptible perturbations drive RetinaFace F1 on WIDER from 94.0% to 0.6%. We use their attack only as an extreme probe of drift, not as a defense.
Robustness Disparities in Face Detection shows that detectors degrade more under corruption for darker-skinned and dimly lit faces. That result predicts that naive audits will have demographically uneven false-flag rates, which is a hypothesis we test.
Implicit Identity Leakage is the closest prior work in spirit. It shows that deepfake backbones encode identity and therefore shortcut on identity correlations within the training split. Its identity probe detects label-correlated identity knowledge. It does not test whether a model has memorized specific test samples, and it does not separate that from crop effects. We use it as a competing audit.
LLM-side work such as DyVal and RADAR shows that static scores can hide memorization. Those methods operate on text and have no upstream stage, so they supply motivation rather than method.
Research Program
The testbed is the DeepfakeBench codebase, used for its standardized preprocessing and its 15 implemented detectors. For counterfactual generation we use the Celeb-DF++ generation code (OUC-VAS/Celeb-DF-PP), which covers 22 forgery methods.
Building counterfactuals. Counterfactuals must change content while keeping the real/fake label honest. For real videos, we relight the face and replace the background with a diffusion generator conditioned on the face landmarks and a face mask. Conditioning on landmarks rather than on edges keeps facial geometry in place while still changing face pixels. For fake videos, we cannot edit the output directly, because that would disturb the forgery artifacts. Instead, we apply the same edit to the source and target videos and re-run the original forgery method. The counterfactual is then a fresh fake made by the same generator.
Building the three arms. Each counterfactual passes through DLIB, RetinaFace, and MTCNN, the detectors whose disagreement DeepfakeBench documents. We log box IoU and landmark error relative to the original frame, which gives a per-sample drift value. This drift value defines the jitter control $J$ and lets us stratify all results by drift.
Ground-truth contamination. Following the injection recipe of the VLM contamination paper, we fine-tune detectors on a chosen fraction of test videos (0%, 10%, 50%, or 100% of test identities) for a fixed number of epochs. This gives labeled clean and contaminated models. We also audit, without ground truth, pretrained foundation backbones that may have seen celebrity frames during web-scale pretraining.
Baselines
Do nothing. The naive reference reports the raw within-dataset AUC, or the gap between within-dataset and cross-dataset AUC, as an informal contamination signal. DeepfakeBench reports a top average within-domain AUC of 95.27% for UCF and a top cross-domain average of 78.75% for SPSL. Celeb-DF++ reports a 73.8% average video-level AUC when training on FF++. These gaps mix domain shift with any memorization, so beating this baseline means producing a statistic that separates our injected-contamination models from clean ones at matched cross-dataset gaps.
Strongest existing method. The main competitor is the VLM semantic-perturbation delta $A-B$ applied unchanged to the cascade, with the detector re-run. It is the only published perturbation-based contamination test for vision. On LLaVA-v1.5-7B it moves clean MMStar accuracy from about 38% to about 69% under perturbation, while a contaminated model reaches about 54% on the originals and about 38% on the perturbed images. No reported result was found on Lacuna for this method applied to face cascades or deepfake detectors.
For a fair comparison, it uses the same counterfactual images, the same contaminated and clean models, and the same detectors. The decision threshold is tuned on the same held-out split for every method. We beat it if $A-C$ achieves higher video-level AUROC at separating contaminated from clean models. We also need a lower false-flag rate on clean models in the high-drift quantile, measured at matched true-positive rate.
Identity probe. As a competing audit, we fit the linear identity probe of Implicit Identity Leakage on frozen features, using the same models and identities.
Ablations. Two ablations isolate the ingredients. The crop-lock arm alone drops the jitter control. The edge-conditioned generator replaces landmark conditioning, which tests whether preserving geometry matters.
Why not just…?
Why not just reuse the original crops and skip the decomposition? This is the real threat, and it is one of our arms. If transported crops alone give the full separation, the honest conclusion is that a one-line protocol change suffices, and we will report that. Two things could make crop-locking insufficient. First, an edit or re-synthesis can move the face slightly, so the transported crop becomes misaligned with the new content. The jitter control $J$ and the drift stratification reveal whether this residual misalignment still produces false flags. Second, crop-locking cannot be used where the deployed pipeline must re-detect. In that setting, only the decomposition tells a practitioner how much of the observed drop to discount.
Why not just standardize on one detector, as DeepfakeBench does? Standardization fixes which detector is used. It does not fix how that detector reacts to an edited image. Robustness Disparities in Face Detection shows that every detector it tested degrades unevenly under corruption. Running the audit under each of the three detectors shows directly whether conclusions flip with the choice of detector.
Why not just compare against a fresh benchmark like Celeb-DF++? A fresh benchmark changes the generators, the identities, and the video quality all at once. The 73.8% cross-dataset AUC it reports cannot be attributed to memorization alone. Our injected-contamination grid holds all of these fixed and varies only exposure to the test set.
Why not just train downstream models with spatial jitter augmentation? An audit has to work on the model as released, not on a retrained one. DeepfakeBench also reports that blur augmentation degrades performance on high-quality data. We nonetheless include jitter-augmented clean models as a check. If their naive deltas become well-behaved, upstream drift was indeed the confound.
Decisive Checks
The first check concerns clean models: across drift quantiles, the naive delta $A-B$ should flag clean models at a rate that grows with drift, while $A-C$ stays flat. The second concerns contaminated models: $A-C$ should grow monotonically with the contaminated fraction. The third is internal consistency: $C-B$ should track $A-J$, since a mismatch would mean the jitter control misses some upstream effect such as alignment rotation. The fourth is the fairness check. Lighting metadata is unavailable for FF++, so we stratify videos by estimated face luminance and repeat the false-flag analysis. We test whether naive audits flag dim-lit clean videos disproportionately, as the detection-disparity results predict.
If none of these checks shows a gap between $A-B$ and $A-C$, then upstream drift is negligible for contamination auditing at realistic edit strengths. That negative result would itself justify applying VLM-style audits to cascades without modification.
References
- Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations
- MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation.
- DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection
- Hiding Faces in Plain Sight: Defending DeepFakes by Disrupting Face Detection
- Robustness Disparities in Face Detection
- Towards A Robust Deepfake Detector: Common Artifact Deepfake Detection Model
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- RADAR: Mechanistic Pathways for Detecting Data Contamination in LLM Evaluation