One-Sided Invisibility Training for Deepfake Detectors

A deepfake detector is a classifier that labels an image as real or machine-generated. It faces two attacks that turn out to be the same attack. At test time, someone can add a change no human would notice, such as tiny pixel noise, a smooth warp of the image geometry, or high-frequency color ripples, so that a fake is labeled real. At training time, someone can plant a backdoor: poisoned training images carry a hidden pattern (a trigger), so the finished detector calls any fake carrying that pattern real. The backdoor literature has already mapped which changes people cannot see. Smooth warps and optimized low-amplitude noise escape human inspection, while patches are caught 99% of the time (Hide and Seek). Current detectors lose to both attacks. White-box perturbations drive F1 to 0.0 on the RAID benchmark (RAID), and imperceptible poisoning reaches 100% attack success on FaceForensics++ (Real is not True). The usual fix, adversarial training (retraining on worst-case perturbed inputs found by many gradient steps), is expensive. It also works against the detector's purpose, because a forgery detector must stay sensitive to exactly the kind of faint signal that adversarial training teaches models to ignore.

This proposal trains a lightweight detector against a single set of threats defined by what humans cannot see, and applies that training in one direction only. Fakes must stay flagged as fake under any invisible edit. Real images carry no invariance requirement. The one-sided constraint is what should let the detector stay robust without losing its sensitivity to faint generator traces. A cheap learned adversary that searches the invisible-edit set inside the training loop should also find and erase backdoors planted in the training data, provided those backdoors use triggers from the same set.

Core Claim

We claim that, for a parameter-efficient detector built on a frozen vision-language backbone, one-sided adversarial training over a union of perceptually invisible edit families gives three things at modest compute. It gives robustness to additive perturbations comparable to standard adversarial training. It gives much better robustness to non-additive invisible edits, such as warps and chroma ripples, which standard adversarial training does not target. And it gives a smaller loss in cross-dataset accuracy than two-sided training. It should also reduce the success of invisible-trigger backdoors injected during adapter tuning, because the in-loop search over the edit set naturally finds the planted trigger direction and trains it away.

The objective, with label 1 meaning fake, is

$$\min_{\phi}\; \mathbb{E}_{x \sim \text{real}}\, \mathcal{L}(f_\phi(x), 0) \;+\; \mathbb{E}_{x \sim \text{fake}}\, \max_{t \in \mathcal{T}} \mathcal{L}(f_\phi(t(x)), 1).$$

Here $f_\phi$ is the frozen backbone plus trainable adapters $\phi$, $\mathcal{L}$ is cross-entropy, and $\mathcal{T}$ is the union of three edit families. The first is smooth elastic warps with a bounded displacement field, as in WaNet. The second is high-frequency wavelet perturbations in the chroma (UV) channels, as in Chroma Backdoor. The third is additive luminance noise under an $\ell_\infty$ bound. The inner maximization is not run by multi-step projected gradient descent (PGD). Instead, it is approximated by a small input-conditioned generator $g_\psi$, trained jointly as the adversary in the way LIRA trains its trigger generator. Each training step then costs one extra forward-backward pass instead of ten.

Research Program

The detector follows AdaptPrompt, which freezes CLIP, prunes the last transformer block so low-level forensic detail survives, and trains about 0.1% of the parameters. AdaptPrompt supplies the method and the clean-generalization reference. It never considers attacks. We add three pieces on top of it.

The first piece is the edit set $\mathcal{T}$ itself. Each family has a budget calibrated by a small referenceless human study that follows the protocol of Hide and Seek. That paper motivates the design in two ways: it shows $\ell_p$ norms track human perception poorly, and it identifies warps and continuous optimized noise as the attacks that evade inspection. We set each budget at the largest value where human detection stays near the clean false-alarm rate, which that paper reports reaching 23.7% in the referenceless setting. The threat is then defined by what people cannot see, not by an arbitrary radius.

The second piece is the amortized adversary $g_\psi$. It outputs a warp field, a chroma residual, and a luminance residual, each projected onto its budget. It is trained to push fakes toward "real". Chen et al. already train a synthesizer adversarially against a deepfake detector, but their synthesizer searches over blending configurations to create harder forgeries for generalization, and they never measure cost. Ours searches over evasion edits and is budgeted for compute. FrePGAN also trains a perturbation generator, but it uses frequency cloaking as data augmentation and does not treat it as a threat to defend against.

The third piece is the one-sided constraint. Perturbations are applied only to fakes. The prediction is that two-sided invariance forces the detector to ignore low-amplitude energy of any kind, including real generator traces, which is the tension raised by the tuning-versus-truncation contrast between AdaptPrompt and Robustness Tokens. One-sided training only forbids low-amplitude edits from reducing the fakeness score.

We then analyze whether the in-loop adversary also cleans backdoors. We poison the fake-class training data with triggers inside $\mathcal{T}$ (WaNet warps, chroma wavelets, DCT multi-channel steganography from Imperceptible and multi-channel backdoor attack) and with a trigger outside it (Bad-Deepfake high-frequency PGD triggers). Measuring attack success separately for triggers inside and outside the set tells us whether the protection is specific to the set or incidental.

Baselines

The naive reference is AdaptPrompt trained normally on the same source data, with no robustness mechanism. It sets the clean cross-dataset ceiling. The file reports 98.6% average precision across 25 test sets for AdaptPrompt (AdaptPrompt). We expect this baseline to collapse under attack, as RAID reports for standard detectors: white-box F1 near 0.0, and black-box AUROC dropping from 0.99 to 0.67 for the Corvi et al. detector (RAID).

The strongest competitor is two-sided PGD-10 adversarial training of the same adapters under an $\ell_\infty$ budget of 8/255, matching RAID's smallest noise level. This is the defense RAID itself recommends, restricted to adapters so it remains affordable. It gets the same data, the same adapters, and the same hyperparameter search budget. We report its GPU-hours alongside ours. The file contains no reported result for an adversarially trained detector on RAID. To beat it, our method must reach equal or higher robust AUROC on RAID's $\ell_\infty$ attacks at 8/255, higher robust AUROC against adaptive warp and chroma attacks, and a cross-dataset AP drop from the naive reference that is smaller by a margin clearly above seed variance, all at a quarter of the training compute or less. We also run a compute-matched version of this baseline: single-step adversarial training given the same GPU-hours as our method.

The second competitor is Robustness Tokens (Robustness Tokens) attached to the frozen detector backbone. This method appends ten learnable tokens to a frozen vision transformer and trains them so that features of attacked inputs match features of clean inputs. On ImageNet with DINOv2-Giant, it restored white-box accuracy from 0.3% to 63.1% in 200 steps. It is the cheapest credible defense available. We train it with the authors' two-part loss on the same data, and we evaluate it against white-box attacks that differentiate through the tokens, since its security otherwise rests on keeping the tokens secret.

For poisoning, the competitor is EPIC (Not All Poisons are Created Equal). EPIC drops training samples whose last-layer gradients are isolated from the rest, and it cut gradient-matching attack success on CIFAR-10 from 45% to 1%. We apply it to the adapter's last-layer gradients during tuning. The poisoning reference point is Bad-Deepfake's reported 100% attack success on FaceForensics++ (Real is not True).

Three ablations isolate the key ingredients. The first uses two-sided rather than one-sided training over the same edit set. The second uses the amortized generator restricted to $\ell_\infty$ edits only. The third replaces the generator with PGD over the edit set's parameters, which tells us what amortization costs in robustness.

Why not just…?

Why not just run PGD adversarial training on the adapters? This is the real threat, and it is the main baseline. Adversarial training on $\ell_\infty$ has no reason to protect against warps, which move pixel positions rather than adding value to them. WaNet shows that such warps survive even physical recapture (WaNet). Two-sided invariance may also erase the faint traces that AdaptPrompt had to prune layers to preserve. If PGD adversarial training matches us on warp and chroma robustness and on cross-dataset AP, the edit-set and one-sided ideas fail, and the only remaining contribution is cost. We will report that outcome as it stands.

Why not just use Robustness Tokens, which already cost almost nothing? Their loss pulls attacked features back to clean backbone features. That helps only if the backbone's features separate real from fake, and AdaptPrompt's finding that CLIP's final layers smooth away forensic detail suggests they may not. The tokens have also only been tested on additive noise, on ImageNet and ADE20K. Our white-box and non-additive evaluations settle whether the tokens transfer to forensics.

Why not just sanitize inputs with JPEG or blur? FreqBlender reports that forgery traces are spread across frequency bands, so filtering strips the evidence the detector needs. Warps are not high-frequency, so filtering does not remove them. We include a JPEG-plus-blur front end as a check, measuring both cross-dataset AP and robust AUROC.

Why not just filter poisons with EPIC and skip the in-loop adversary? EPIC was built against gradient-matching poisons on small classifiers, and its behavior on adapter gradients under dirty-label deepfake poisoning is unknown. It may well win on Bad-Deepfake, whose trigger lies outside our edit set. We therefore also report EPIC combined with our method, and we claim a backdoor benefit only for triggers inside the edit set.

Proposed Experiments

For general AI-generated images, we train on Diff-Gen and evaluate clean average precision on AdaptPrompt's 25-dataset suite. We evaluate robustness on RAID (aimagelab/RAID) at $\epsilon$ = 8, 16 and 32/255. We add adaptive white-box warp, chroma and combined attacks at the human-calibrated budgets, run by 100-step PGD over the edit parameters. For faces, we train on FaceForensics++ (c23) and report AUC on Celeb-DF, DFDC and DeeperForensics-1.0. Chen et al.'s 0.769 average AUC (Chen et al.) serves as the non-parameter-efficient reference. For poisoning, we inject each trigger type at 1%, 5% and 10% of the fake class in FaceForensics++ and measure attack success rate and clean AUC after adapter tuning. Compute is reported as GPU-hours and forward-backward passes per example.

Three results decide the claim. First, the one-sided versus two-sided ablation must show a statistically clear gap in cross-dataset AP with robustness held equal; if there is no gap, asymmetry is not the mechanism. Second, attack success for in-set triggers must fall well below that of the naive baseline while out-of-set Bad-Deepfake success stays high. That pattern would show the backdoor protection comes from searching the edit set and not from generic regularization. Third, the amortized generator must come within a few AUROC points of PGD over the same edit set when both face adaptive attacks. If it does not, the cost advantage is an illusion created by a weak adversary, and the proposal reduces to few-step PGD over the edit set, which we would then report with its true cost.