Some people protect their images from unauthorized model training by adding "unlearnable" noise. This is a tiny, carefully optimized perturbation that plants an easy shortcut, so a model trained on the images memorizes the noise instead of the content. Defenders who want to train on such data anyway have two options today. They can run adversarial training, which is expensive and costs clean accuracy. Or they can apply cheap filters such as JPEG compression, which newer poisons are built to survive. Both lines of evidence come from small convolutional networks trained from scratch. In practice, most people now train a small head or a few prompt tokens on top of a large frozen pre-trained vision transformer.
This proposal asks what happens to unlearnable data in that regime. Our claim has two parts. First, a learner restricted to frozen features learned from clean data can only latch onto shortcuts those features already expose, so standard poisons should largely fail against a plain linear probe. Second, because the backbone weights are public, an attacker can craft poisons directly against those features. For that adaptive case, we propose a defense that runs the usual adversarial-training game inside the backbone's feature space. Only the small trainable part is updated. The size of the allowed feature perturbation is calibrated from the backbone itself, so the defense provably covers what a pixel-bounded poison can do. Because the poisoned dataset is fixed, the frozen lower layers run once per image and their outputs are cached. The inner loop then touches only the top few blocks.
What the literature establishes and what it leaves open
Adversarial training, as formalized in Towards Deep Learning Models Resistant to Adversarial Attacks, solves a min-max problem: an inner attacker finds the worst perturbation, and the outer step trains on it. That paper also documents the method's two well-known costs. Each update needs many extra forward-backward passes, and clean CIFAR-10 accuracy drops from about 95% to about 87%. It supplies the core mechanism we reuse.
The poisoning literature motivates the problem.
- APBench and UEraser show that first-generation unlearnable examples are fragile shortcuts. JPEG compression, adversarial augmentations, or adversarial training restore accuracy to 80–92% (APBench) and 95.24% under the error-minimizing attack (UEraser), all on ResNet-18 trained from scratch.
- REM shows the fragility can be engineered away. Its noise is optimized against an adversarially training learner and holds that learner to 48.16% on CIFAR-10. It also states a "radius rule": the poison's budget must exceed the learner's perturbation radius.
- DUNE splits the poison into spatial and color components and keeps defended accuracy between 14.95% and 50.82%.
- EntF attacks robust semantic features and cuts an adversarially trained model from 84.88% to 71.57%.
These papers constrain the design. Simple filtering is not a sufficient answer, and any feature-space defense must be tested against attackers who craft poisons through the same features.
The parameter-efficient robustness work supplies the setting.
- Robustness Tokens adds ten learnable tokens to a frozen DINOv2 or OpenCLIP backbone and converges in 200 steps.
- Revisiting adapters with adversarial training reaches 83.76% clean and 49.87% robust top-1 on ImageNet with dual classification tokens adding 768 parameters.
- Reliably fast adversarial training via latent adversarial perturbation moves the inner attack into intermediate layers.
All three target test-time evasion attacks, not poisoned training data. None asks how large the latent perturbation must be to cover a given pixel-space threat.
Core Claim
Split a frozen backbone into lower blocks $z(\cdot)$ and upper blocks $g(\cdot)$. Let $h_\phi$ be the trainable part: a few prompt tokens inserted at the split plus a linear head, with parameters $\phi$. We train
$$\min_{\phi}\ \mathbb{E}_{(x,y)\in \tilde D}\ \max_{\|\eta\|_2\le \rho}\ \mathcal{L}\big(h_\phi(g(z(x)+\eta)),\,y\big),$$
where $\tilde D$ is the poisoned training set, $\eta$ is a perturbation of the cached intermediate features, and $\mathcal{L}$ is cross-entropy.
The radius $\rho$ is not a free hyperparameter. It is set from the backbone:
$$\rho = Q_{q}\Big(\|z(x+\delta^*)-z(x)\|_2\Big),\qquad \delta^* = \arg\max_{\|\delta\|_\infty\le\epsilon}\|z(x+\delta)-z(x)\|_2 .$$
Here $\epsilon$ is the pixel budget the poisoner is assumed to respect. $\delta^*$ is found by projected gradient ascent on a few hundred held-out images, and $Q_q$ is the $q$-th quantile of the resulting displacements. This is REM's radius rule, translated into feature space. If the learner's feature-space ball contains every displacement an $\epsilon$-bounded poison can cause at the split point, then no such poison can create a shortcut that survives the inner maximization.
The calibration is the new ingredient. It turns "latent adversarial training" from a heuristic into a defense with a stated coverage condition. The quantile $q$ is the single knob trading clean accuracy against coverage.
Caching changes the cost picture. $z(x)$ is computed once per image. Each inner step then costs only a pass through $g$ and $h_\phi$. When the split sits a few blocks from the top, this is a small fraction of a full backbone pass, against $(k+1)$ full passes for pixel-space adversarial training.
Research Program
The work has three stages, each of which can fail informatively.
Stage 1: are standard poisons already defeated by freezing? We train linear probes and prompt tokens with plain empirical risk minimization on frozen DINOv2, OpenCLIP and DeiT-III features. Training data is APBench's poisoned CIFAR-10, CIFAR-100 and ImageNet-100 sets, using the error-minimizing (EM), targeted adversarial poisoning (TAP), REM, DUNE and EntF attacks. One confound must be controlled: upsampling 32×32 poisons to 224×224 is itself a low-pass filter. We therefore also re-craft every poison at the backbone's input resolution, against a ResNet surrogate as the original papers did.
Stage 2: does an adaptive, backbone-aware attacker break the frozen learner? We build two such attackers, since the backbone weights are public:
- an error-minimizing poison optimized through the frozen backbone and a linear head;
- a REM-style min-min-max poison whose inner learner is our own latent defense at the calibrated $\rho$.
The second is the attacker that knows our defense, and it is the one that matters.
Stage 3: run the calibrated latent defense and measure everything a practitioner pays. We record poisoned-to-clean test accuracy, clean accuracy relative to a probe trained on unpoisoned data, and wall-clock and FLOPs per epoch. We sweep the split depth and $q$. The sweep tests a specific prediction from the coverage argument: robustness to a given poison should jump once $\rho$ crosses the measured displacement quantile of that poison, not rise smoothly.
Baselines
Naive reference points. A linear probe and a prompt-token learner trained with plain empirical risk minimization on poisoned data give the floor. The same learners trained on the clean dataset give the ceiling. Every defense is reported as the fraction of this gap it closes, at matched epochs and identical frozen backbones.
Strongest existing data-side defense. This is UEraser, which reports 95.24% CIFAR-10 test accuracy under the EM attack with ResNet-18 (UEraser). We run it two ways: as published, and transplanted onto the frozen-backbone learner with its adversarial augmentations. The transplanted version gets the same tuning budget (an equal number of configurations searched on a held-out clean split) and must forgo feature caching, since augmentations change inputs every epoch. We also include APBench's JPEG-based ISS defense and its reported 80–92% range (APBench).
Strongest training-side defense. This is pixel-space PGD adversarial training of the same tokens and head through the full frozen backbone at budget $\epsilon$. It provides REM's reference point of 48.16% under a 4/255 learner on CIFAR-10 (REM). It isolates whether moving the game into feature space loses protection.
Ablations. Two ablations isolate the key ingredients:
- random Gaussian feature noise of the same calibrated norm, which tests whether the maximization matters or noise alone suffices;
- latent adversarial training with $\rho$ tuned by grid search on clean validation accuracy rather than calibrated, which tests whether the calibration matters.
What counts as winning. The proposal beats the field if three things hold together under the adaptive Stage 2 attackers:
- it matches or exceeds pixel-space token adversarial training in poisoned-data accuracy;
- it stays within two points of the clean-data probe on clean accuracy;
- it costs a stated fraction of the pixel-space defense's compute.
Why not just…?
Why not just train a linear probe on the frozen backbone and stop? This is the most serious threat, and Stage 1 is designed to confirm it on non-adaptive poisons. If ordinary probes recover clean-level accuracy against EM, TAP and DUNE, that is a useful negative result for the poisoning literature. In that case the contribution narrows to the adaptive case. The backbone-aware poisons of Stage 2 are exactly what a probe cannot resist by construction: a poison optimized through the frozen features makes those features carry the shortcut. If even backbone-aware poisons fail against a plain probe, we will report that freezing alone suffices and drop the defense.
Why not just JPEG-compress or filter the data first? The evidence is against filtering as a general answer. IMPRESS states that JPEG, noise and resizing do not remove modern protections. DUNE's spatial-plus-color design specifically defeats single-domain filters. Poisons built with expectation over transformations, as in MetaCloak, survive blurring. Filtering also has to be applied before caching, so it can be combined with our learner. We test the combination and attribute gains accordingly.
Why not just apply UEraser-style adversarial augmentation to the frozen learner? It may work. It is a direct baseline above, not a strawman. Its cost is structural: augmented inputs must pass through the whole backbone every epoch. UEraser also reports needing many more epochs to converge. The decisive comparison is the accuracy-per-FLOP curve under the adaptive REM-style attacker. Augmentations are not optimized against a feature-space poison, so we expect them to degrade there. If they do not, the augmentation route is the better recommendation.
Why not just do full pixel-space adversarial fine-tuning of the backbone? It pays the $(k+1)$-pass cost on hundreds of millions of parameters. It also pays the clean-accuracy tax documented in Towards Deep Learning Models Resistant to Adversarial Attacks. And it overwrites the clean pre-trained features that are the source of the frozen learner's resistance in the first place. We run it on ImageNet-100 only, as a compute-unbounded reference.
Decisive Checks
The experiments are CIFAR-10, CIFAR-100 and ImageNet-100 with APBench's attack suite plus our two adaptive attackers, across three backbones. Results are reported as poisoned-data test accuracy, clean accuracy and GPU-hours.
Four outcomes would settle the claim:
- Calibrated threshold. Under the defense-aware REM-style attacker, the calibrated latent defense should hold accuracy well above REM's 48.16% reference, while plain probes collapse. Without that gap, the defense adds nothing over freezing.
- Maximization matters. The random-noise ablation at the same norm should fail where the adversarial version succeeds.
- Threshold, not slope. Robustness should change sharply near the displacement quantile, as the coverage argument predicts.
- Cost. Defense wall-clock should be reported against REM's own generation cost (over 50 hours versus 3.9 hours for plain EM on an ImageNet subset). This answers whether the defender's side of the race is now the cheaper one.
One threat to validity is pre-training overlap. DINOv2 and OpenCLIP may have seen images resembling these test sets. We guard against it with a DeiT-III backbone trained only on ImageNet-1K, evaluated on CIFAR, and we treat ImageNet-100 conclusions with corresponding caution.
References
- Towards Deep Learning Models Resistant to Adversarial Attacks
- APBench: A Unified Availability Poisoning Attack and Defenses Benchmark
- Learning the Unlearnable: Adversarial Augmentations Suppress Unlearnable Example Attacks
- Robust Unlearnable Examples: Protecting Data Privacy Against Adversarial Learning
- Dual-branch Robust Unlearnable Examples
- Is Adversarial Training Really a Silver Bullet for Mitigating Data Poisoning?
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Revisiting adapters with adversarial training
- Reliably fast adversarial training via latent adversarial perturbation
- IMPRESS: Evaluating the Resilience of Imperceptible Perturbations Against Unauthorized Data Usage in Diffusion-Based Generative AI
- MetaCloak: Preventing Unauthorized Subject-Driven Text-to-Image Diffusion-Based Synthesis via Meta-Learning