Most deployed vision models are now a public foundation model, such as DINOv2 or CLIP, with a small add-on trained for the task. The add-on is usually a parameter-efficient fine-tuning (PEFT) module: a few low-rank matrices, a handful of learned tokens, or rescaled normalization layers, while the rest of the network stays frozen. This creates a security problem that, as far as we can tell, nobody has measured. An attacker who never queries the private model already holds more than 95% of its weights, because those weights are the public backbone. Adversarial examples crafted on that backbone should carry over to the private model, an effect called transferability. They should carry over even better than they do to a fully fine-tuned model, which Two Sides of the Same Coin already found to be badly exposed to attacks crafted on its source model.
We propose to measure this threat across PEFT families and attack types, and to test a cheap defense against it. The defense trains the adapter on a bank of perturbations computed once on the public backbone and reused for every downstream task and every adapter type. Adversarial training is expensive because every update needs an inner loop of attack steps run through the whole network. A shared bank removes that per-task inner loop, so the defense costs roughly two clean training passes. The question is whether this targeted, amortized defense matches the transfer robustness of full PEFT adversarial training at a fraction of the compute, and how much white-box robustness it gives up.
Core Claim
We make two linked claims. The first is diagnostic: for models adapted with LoRA, normalization-augmented LoRA, prompt or prefix tokens, or bottleneck adapters, attacks crafted only on the public backbone recover most of the damage a full white-box attack does. In the vocabulary of Two Sides of the Same Coin, the gap $\gamma = (a_b - a_w)/a_w$ is close to zero, where $a_b$ is accuracy under black-box attack from the backbone and $a_w$ is accuracy under white-box attack. We expect the gap to shrink as adapter rank falls. This is a direct test of the tension raised by Adversarial Vulnerability from Interference Between Features in Superposition. That paper shows that cramming many features into few dimensions makes different networks share the same pattern of feature interference. Their transfer success rose from 5% to 98% when the networks shared that geometry. Low-rank adaptation leaves the backbone's geometry almost untouched by construction, so it should behave the same way.
The second claim is constructive. A "backbone perturbation bank" contains perturbations under the $\ell_\infty$, $\ell_2$ and $\ell_1$ norms, produced by a mix of feature-distortion attacks and attacks through cheap surrogate heads. Training an adapter on this bank with a plain robust loss, and no inner loop through the private model, gives transfer robustness comparable to full PEFT adversarial training. It also generalizes to attack algorithms that were not in the bank. Because the bank is built once per backbone and input distribution, its cost is shared across tasks and across PEFT families.
Research Program
The work has three parts.
Part one: the transfer audit. We fix public backbones (DINOv2, DeiT-III and OpenCLIP ViTs, the families used by Robustness Tokens). We adapt each backbone to several downstream tasks with four PEFT families at several ranks or token counts. We then attack each private model three ways, each with a different level of attacker knowledge. The weakest attacker uses a task-agnostic feature attack, which pushes the backbone's features as far as possible from the clean features. The middle attacker trains its own linear head, or its own same-family adapter, on public data, then runs multi-step PGD and AutoAttack through it. The strongest reference is the full white-box attack on the private model.
The audit removes the main limitation of Two Sides of the Same Coin, which tested mostly single-step FGSM on fully fine-tuned networks. It also follows the warning of Reliable Evaluation of Adversarial Transferability that transfer rates measured only between CNNs are inflated. Transfer across different architectures collapses, so we report transfer from a mismatched-architecture surrogate as a control. That control separates "shared backbone" from "generic transferability."
Part two: the bank defense. For each backbone, we precompute perturbations on a pool of unlabeled in-distribution images, for example the ImageNet training set. The attack recipes span the three norms, PGD and APGD optimizers, feature-distortion and head-based losses, and several random surrogate heads.
Downstream, the adapter minimizes
$$\mathcal{L} = \mathcal{L}_{ce}(f_{\theta}(x), y) + \lambda \, \mathbb{E}_{\delta \sim \mathcal{B}(x)} \, \mathcal{L}_{ce}(f_{\theta}(x+\delta), y),$$
where $f_\theta$ is the frozen backbone plus adapter parameters $\theta$, and $\mathcal{B}(x)$ is the bank of stored perturbations for image $x$. The weight $\lambda$ trades clean accuracy against robustness. Downstream images that are not in the pool get perturbations by nearest-neighbor reuse, or by a short attack on the backbone alone that is cached once per dataset. No gradient ever flows from the adapter's loss back to the pixels.
The same bank is used unchanged for every PEFT family. This is a deliberate test of the "architectural tethering" concern: defenses designed for one PEFT family often fail to port to others. For example, FullLoRA-AT is tied to low-rank linear layers, and the tokens of Robustness Tokens are tied to the input sequence. A defense that lives in the data rather than the parameterization should port by construction, and we check whether it does.
Part three: rank and geometry analysis. Following the interference measurements of Adversarial Vulnerability from Interference Between Features in Superposition, we track two quantities: the cosine similarity of input gradients between backbone surrogate and private model, and the overlap of their feature directions. We track both as adapter rank and bank training vary. The goal is to tell whether the defense works by actually moving the private model's geometry away from the backbone's, or merely by fitting the stored perturbations. If it only fits them, it should fail on held-out attack recipes.
Baselines
Undefended adapters. The naive reference is a clean PEFT model of each family, trained with the same data and steps. It sets the floor for every attack and gives the uncorrected $\gamma$.
Full PEFT adversarial training. This is the strongest competitor on robustness. FullLoRA-AT inserts learnable LayerNorms before LoRA matrices across all ViT blocks. It trains about 5% of parameters, uses about 75% of the training time of full adversarial fine-tuning on CIFAR-10, and reports robustness "often with only a 1-2% drop" relative to full adversarial fine-tuning. The file lists no per-attack robust accuracy for it, so we will reproduce it. We also run Hyper Adversarial Tuning, which trains a hypernetwork to generate LoRA weights for several adversarial-training objectives and then merges them. The file reports 18.26M trainable parameters for it but no robust accuracies on Lacuna.
Both baselines get the same backbones, data, number of epochs and hyperparameter search budget as our method. Their PGD inner loop uses the same step counts as the bank recipes. The bank defense beats them if it matches their transfer robust accuracy within about 2 points while using clearly less GPU time per task. Bank construction counts toward our cost, divided by the number of tasks that share it.
Gradient decoupling. Publishing Efficient On-device Models Increases Adversarial Vulnerability proposes Similarity-Unpairing, which fine-tunes a model so its input gradients stop aligning with a sibling model's. It reports cutting transferability by 50–80% (up to 90%) while keeping about 93% CIFAR-10 accuracy. Its stated obstacle was needing both parent and sibling during training. Here that obstacle disappears, because the parent is the public backbone the defender already holds. That makes it the most direct competitor on the transfer question, and we run it with the same adapter and budget. We charge it for the double backpropagation it requires.
Robustness tokens. Robustness Tokens is the minimal-parameter competitor. It recovers DINOv2-G white-box accuracy from 0.3% to 63.1% in 200 steps.
Ablations that isolate the key ingredient. The first replaces the shared bank with perturbations generated fresh in each step on the backbone only. This tests whether the stored, reused bank costs any robustness. The second restricts the bank to $\ell_\infty$ PGD. This tests whether diversity of attack recipes is what buys generalization.
Why not just…?
Why not just run standard PEFT adversarial training and assume it covers transfer? This is the most serious threat to the proposal. White-box robustness usually implies robustness to transferred attacks, so FullLoRA-AT may already solve the problem. If it does, our contribution reduces to cost: the same transfer robustness for less compute. We will report robust accuracy per GPU-hour so this is settled directly. Note that FullLoRA-AT still backpropagates through the frozen backbone to the pixels on every inner step. Freezing weights cuts optimizer memory but not the inner loop. Our cost advantage should therefore be several-fold, not marginal.
Why not just use test-time randomization? Mitigating Adversarial Effects Through Randomization restores Inception-v3 accuracy under DeepFool from 1.1% to 98.2% with random resizing and padding. However, its authors note it is weaker against transferable attacks and against attackers who average gradients over the random transformations. RobustBench found such heuristics overstated robustness by more than 10%. We include randomization as a baseline under transfer attacks that average over the random transformations, since that is exactly the threat model studied here.
Why not just keep robustness tokens secret? Robustness tokens are cheap, but the authors acknowledge the defense collapses if the token values leak. The tokens were also never tested against transfer attacks or other norms. Our audit tests them in exactly those conditions. If they already resist backbone transfer, we will report that as a result in its own right.
Proposed Experiments
Datasets and metrics. The main study adapts public ViT backbones to CIFAR-10, CIFAR-100, TinyImageNet and Imagenette. ImageNet-1k is used for bank construction and for one at-scale check. Robust accuracy is measured three ways. Transfer robust accuracy uses each of the three attacker settings, with the hard-transfer protocol of Reliable Evaluation of Adversarial Transferability. White-box robust accuracy uses AutoAttack via RobustBench, which is reported honestly even where our method is weaker. Robust accuracy is also measured under $\ell_1$ and $\ell_2$ attacks never seen in training. Alongside these we report $\gamma$, clean accuracy, and GPU-hours including the shared bank cost. For scale, Revisiting Adversarial Training for ImageNet reached 50.2% ImageNet $\ell_\infty$ robustness with full fine-tuning at about 20,000 GPU-hours. That is the full-update ceiling we are not trying to match.
Decisive checks. Four results decide the proposal.
- If undefended PEFT models show transfer robust accuracy close to white-box accuracy, while fully fine-tuned and cross-architecture controls do not, the diagnostic claim holds. If $\gamma$ is large for PEFT models, the threat is overstated, and the proposal reduces to a cost study.
- The bank defense must hold up against an attacker who trains its own adapter of the same family, using attack recipes absent from the bank. If it only resists recipes it was trained on, it is memorizing rather than generalizing, and that result kills the constructive claim.
- Gradient similarity between backbone and private model must fall alongside transfer success. If it does not, the defense is not working through the mechanism we claim.
- The same bank must transfer across all four PEFT families without retuning. This is the test of whether a data-level defense escapes the tethering that limits parameter-level ones.
References
- Two Sides of the Same Coin: White-box and Black-box Attacks for Transfer Learning
- Adversarial Vulnerability from Interference Between Features in Superposition
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Reliable Evaluation of Adversarial Transferability
- FullLoRA-AT: Efficiently Boosting the Robustness of Pretrained Vision Transformers
- Hyper Adversarial Tuning for Boosting Adversarial Robustness of Pretrained Large Vision Models
- Publishing Efficient On-device Models Increases Adversarial Vulnerability
- Mitigating Adversarial Effects Through Randomization
- RobustBench: a standardized adversarial robustness benchmark
- Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Models