Many large vision systems now share one expensive image encoder that feeds many cheap task heads. Examples are the segmentation model SAM, the vision half of vision-language chatbots, and self-supervised backbones such as DINOv2. Attacks exploit this design. A small pixel perturbation that pushes the encoder's output features far from where they should be breaks every downstream prompt and task at once, and it needs no labels. Defending the encoder is the obvious fix. The standard defense, adversarial training (training on worst-case perturbed inputs found by many gradient steps), costs K extra backward passes through the largest component of the system for every training image.
We propose to replace that inner attack loop with a small learned attacker network. It takes an image and emits, in one forward pass, a bounded perturbation meant to scatter the encoder's features. It is trained online against a lightweight adapter on a frozen encoder. The attacker is in effect a generator of synthetic adversarial training data. Whether it pays off is a data-generation overhead question: does the one-off and per-step cost of training a generator beat regenerating each adversarial example with iterative search? It must do so without the attacker going soft and producing the false robustness that sinks single-step methods.
Core Claim
Our claim is that an image-conditioned, co-trained attacker targeting encoder feature distance can make a frozen foundation encoder robust at roughly the cost of single-step adversarial training. The robustness, measured under a strong adaptive white-box attack, should be close to what multi-step training reaches at several times the compute. Because the objective is label-free, the robustness should carry over to every downstream head, including SAM prompt types never seen in training.
Three things make this new relative to the file.
First, the attack objective comes from Segment (Almost) Nothing. That paper maximizes the ℓ2 distance between clean and perturbed SAM embeddings with Auto-PGD and drops ADE20K mIoU from 59.86% to 16.71% at ε=1/255 and 7.33% at ε=8/255. It only attacks; it proposes no defense.
Second, the amortization idea has a precedent in Deceptive Diffusion. It trains a diffusion model on a PGD-attacked MNIST set so that it "bakes in" adversarial examples, reaching 93.6% misclassification against LeNet. That generator is static, needs PGD on the whole dataset upfront, and targets one fixed classifier. Our attacker is co-trained against a moving defender, needs no precomputed attacks, and targets features rather than labels.
Third, the cheap-defense side is closest to Robustness Tokens and SLAT.
- Robustness Tokens trains 10 tokens on a frozen ViT in 200 steps, using iterative attacks inside its loss. It relies on the tokens staying secret.
- SLAT avoids the collapse of single-step training by perturbing hidden layers with gradients already computed during backpropagation, but only on small CNNs.
We keep their frozen-backbone economy. We ask for robustness when the attacker knows the adapter, and we cut the inner-loop cost rather than the parameter count.
Research Program
The defender is a frozen encoder $E_0$ plus a low-rank adapter with parameters $\theta$, giving $E_\theta$. We place the adapter on the encoder because CAT found that encoder-side LoRA undid protective perturbations on a diffusion model's VAE. There, encoder adapters restored Face Similarity from 0.160 to 0.572, while decoder-only adapters did almost nothing. CAT used this against defenses; we reuse the placement for defense.
The attacker $g_\phi$ is a small convolutional network with output squashed to the $\ell_\infty$ ball, $\delta = \epsilon \tanh(g_\phi(x))$. The two are trained in alternation:
$$\max_\phi \; \mathbb{E}_x \, \| E_\theta(x+\delta_\phi(x)) - E_\theta(x) \|^2, \qquad \min_\theta \; \mathbb{E}_x \Big[ \| E_\theta(x+\delta_\phi(x)) - E_0(x) \|^2 + \lambda \| E_\theta(x) - E_0(x) \|^2 \Big].$$
Here $\lambda$ weights how far the adapted encoder may drift from the original on clean images. The second term keeps every existing downstream head usable without retraining.
Per step, the cost is about two encoder forward-backward passes, which is comparable to single-step training. Multi-step training with K steps costs K+1.
The known failure mode is the one that kills single-step training. A weak attacker lets the defender learn a jagged feature map that only looks robust; plain FGSM reached 0.01% against PGD-50 on CIFAR-10 in the SLAT paper. We guard against it in three ways:
- A fraction $\rho$ of batches uses a short few-step search initialized at the attacker's output, so the attacker proposes and a cheap refinement corrects.
- Early stopping is keyed to the gap between the attacker's damage and that of a held-out many-step attack.
- The attacker is conditioned on the image rather than input-independent. Segment (Almost) Nothing shows input-independent universal perturbations are markedly weaker (11.24% vs 7.33% mIoU at 8/255), so a universal perturbation is too easy a training opponent.
A secondary arm tests the data-source side of the overhead question. Feeding the loop images from a one-step distilled generator, as in DAR-GDA, serves deployers who lack the encoder's pretraining data. DAR-GDA reports sampling over 100× cheaper than the multi-step teacher. We treat this arm as subordinate: if real unlabeled images are available, we expect them to be at least as good, and we will report whether they are.
Baselines
Do-nothing reference. The undefended encoder under the same attacks. For SAM on ADE20K the file gives 59.86% clean mIoU, falling to 16.71% (1/255) and 7.33% (8/255) under the prompt-agnostic embedding attack of Segment (Almost) Nothing. We will reproduce these before any comparison.
Strongest existing cheap defense. Robustness Tokens on DINOv2. The file reports the following for DINOv2-Giant:
- ImageNet-1K accuracy under white-box attack recovers from 0.3% to 63.1%, with 85.2% clean accuracy (85.6% unperturbed).
- ADE20K segmentation under attack rises from 4.7 to 23.3 mIoU.
We will run it on the same backbone, data and attack, in two settings: the paper's setting where the tokens are secret, and a fully adaptive setting where the attacker differentiates through the tokens. Beating it means higher ADE20K mIoU and ImageNet robust accuracy under the adaptive attack, at equal or lower GPU-hours, with clean accuracy within 1 point.
Strongest expensive reference. Multi-step feature-distance adversarial training (PGD-K with K in {3, 7, 10}) of the same adapter. This is the method we claim to approximate. There is no reported result on Lacuna for encoder-level PGD adversarial training of SAM or DINOv2 adapters, so we must produce it. On small models, the file's cost reference is SLAT's report that PGD-7 is about 3.7× slower than its roughly 104-minute CIFAR-10 run.
Single-step reference. Feature-distance FGSM training with random start, plus a SLAT-style hidden-layer perturbation variant. We use the same adapter, data and tuning budget.
Fairness. All methods get the same adapter rank, the same image stream, and an equal number of hyperparameter trials. They are compared on robustness-versus-GPU-hours curves, where attacker training time is charged to our method.
Ablations isolating the key ingredient. - The attacker replaced by a learned universal perturbation. - The attacker pretrained once and then frozen, which is the Deceptive Diffusion analog. - $\rho$ swept from 0 to 1, where $\rho=1$ recovers few-step training.
Why not just…?
Why not just use single-step feature-space training? It is the cheapest option and our method must beat it at equal cost, so it is in the baselines. The warning is SLAT's CIFAR-10 result: vanilla FGSM training collapses to 0.01% against PGD-50, compared with 47.06% for SLAT. We expect the same risk in feature space. The compute-matched curve and the gradient-masking checks below will show directly whether single-step training collapses here. If it does not, the attacker network is unnecessary, and we will say so.
Why not just run fewer PGD steps with a LoRA adapter? This is the real threat. On frozen-backbone adapters, PGD-2 or PGD-3 might already sit near the robustness-per-GPU-hour frontier. We do not argue around it. The K sweep puts these points on the same curve as our method. The claim survives only if the attacker at about two passes per step matches PGD-K robustness at a strictly lower total cost, with attacker training included.
Why not just use Robustness Tokens, which already converge in 200 steps? Their speed is measured with secret tokens, and even then ADE20K recovery reaches only 23.3 mIoU under attack. Robustness against an attacker who knows the defense needs far more training, and that is where inner-loop cost dominates. Running the tokens under the fully adaptive attack will show how much of their 63.1% survives.
Why not just purify inputs with a diffusion denoiser or prompt the model? Both alternatives fall short here:
- Purification adds multi-step generation to every inference call. CAT's comparison favors adapting the encoder over purifying each image, and DAR-GDA documents the hundreds-of-steps latency of multi-step sampling.
- Prompting works for chat models: an "adversarial possibility" prompt lifts LLaVA-7B from 12.54% to 91.46% under APGD on COCO in Improving Adversarial Robustness in VLMs. The same authors note an adaptive attacker can fold the prompt into the attack, and SAM or DINOv2 have no text interface to prompt.
Why not just precompute a large synthetic adversarial set once? A fixed set is tuned to the undefended encoder and goes stale as the adapter moves. The frozen-attacker ablation tests exactly this. Precomputation also carries the full upfront PGD cost that Deceptive Diffusion pays.
Proposed Experiments
The primary testbed is SAM on ADE20K with the prompt-agnostic protocol of Segment (Almost) Nothing. We report:
- mIoU over point, box and held-out prompt types, at ε of 1/255, 4/255 and 8/255;
- clean mIoU;
- clean-versus-attacked embedding cosine similarity;
- total GPU-hours.
The second testbed is DINOv2 on ImageNet-1K linear probing and ADE20K segmentation, for head-to-head comparison with Robustness Tokens.
A transfer check swaps an adapted CLIP encoder into a LLaVA-style model and measures COCO accuracy under APGD. This tests the finding in Improving Adversarial Robustness in VLMs that robustness is bottlenecked at the vision encoder: a robust encoder should help without touching the language model.
The decisive checks are three.
Adaptive-attack robustness. Every defended model is evaluated with Auto-PGD on the feature-distance objective, run through the adapter with at least 100 steps and multiple restarts. We also add a transfer attack crafted on the PGD-trained model. If transfer attacks beat white-box attacks on our model, it is masking gradients, and the result fails.
Compute-matched frontier. We compare robustness against GPU-hours for all methods. The core claim holds only if our curve lies above PGD-K and above single-step training across budgets.
Downstream generalization. Robustness must hold for SAM prompt types and ADE20K head configurations never used in adapter training. This is the payoff of a label-free encoder objective.
A negative result is also informative. If the attacker only matches few-step PGD at equal cost, then amortized adversarial data generation does not beat iterative search for foundation encoders, and cheap-inference results such as DAR-GDA's do not carry over to adversarial data.
References
- Segment (Almost) Nothing: Prompt-Agnostic Adversarial Attacks on Segmentation Models
- Deceptive Diffusion: Generating Synthetic Adversarial Examples
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Reliably fast adversarial training via latent adversarial perturbation
- CAT: Contrastive Adversarial Training for Evaluating the Robustness of Protective Perturbations in Latent Diffusion Models
- Generative Data Augmentation via Diffusion Distillation, Adversarial Alignment, and Importance Reweighting
- Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design