Adversarial training makes a network hold up against worst-case input perturbations. It does this by training on inputs that an inner optimization has pushed to maximize the loss. Every such inner step costs a full forward and backward pass through the network, and the cost multiplies when you want robustness to several kinds of perturbation at once: small changes to every pixel ($\ell_\infty$), small total energy ($\ell_2$), a few large pixel changes ($\ell_1$), or natural corruptions. For large pretrained vision backbones, a common cheap alternative is to freeze the backbone and train only a small add-on, such as extra tokens or low-rank weight updates (LoRA). The natural extension to many threat models is one add-on per threat model. That repeats a pattern familiar from multilingual models: dedicated modules avoid interference, but their parameters and training runs grow linearly with the number of things supported.
This note proposes a test of whether robustness to different threat models lives in a shared, low-dimensional set of weight-update directions. The concrete design is a single dictionary of low-rank directions, shared across threats, with only a small coefficient vector per threat model. If the hypothesis holds, a frozen backbone gains specialist-level robustness to $K$ threat models with sub-linear parameters and adversarial compute, and a new threat can be added by fitting a handful of coefficients in a short single-step run. If it fails, we learn something equally useful. Failure would take one of two forms: a single union-trained adapter is already enough, so modularity buys nothing, or per-threat structure does not compress, so the linear cost is intrinsic.
Core Claim
For each adapted weight matrix $W$ in a frozen backbone, the update used under threat model $k$ is
$$\Delta W_k = U \,\mathrm{diag}(c_k)\, V^\top,$$
where $U, V \in \mathbb{R}^{d \times r}$ are rank-$r$ directions shared by all threats, and $c_k \in \mathbb{R}^r$ is the only threat-specific parameter. Per layer, this costs $2dr + Kr$ parameters. Independent per-threat LoRA of rank $r'$ costs $2dr'K$. The claim is that some $r$ well below $Kr'$ matches independent specialists on each threat's own robust accuracy. We also expect a union coefficient $c_\cup$, trained against the worst case over threats, to be at least as good as a single shared adapter with the same number of parameters.
The mechanism is soft sharing of capacity, borrowed from multilingual modeling. Soft Language Clustering for Multilingual Model Pre-training replaced per-language adapters with a pool of prompts mixed by soft weights. Its prompts clustered by language similarity, and it added under 0.4% parameters. That work only ever trained on clean data.
There is reason to expect structure across norms. Adversarial robustness against multiple lp-threat models at the price of one… showed that sampling only $\ell_1$ and $\ell_\infty$ attacks yields $\ell_2$ robustness nearly for free; the geometric argument is exact for linear classifiers. That is a hint that per-threat solutions overlap rather than being independent.
A deliberate design choice separates this from the obvious "prompt pool for threats" idea. We do not route on the input. In clean prompt pools, the mixing weights are computed from pooled input embeddings. An adversary controls those embeddings and can steer the mixture, the routing-collapse risk flagged in the research file. Here, the coefficients are selected by deployment context: a known threat, a chosen clean/robust trade-off, or the union setting when the threat is unknown. An input-conditioned router appears only as a stress-tested variant.
Relation to Prior Work
Two lines of work sit closest, and each covers one half of the problem.
Parameter-efficient defenses already exist, but each handles a single threat.
- Revisiting adapters with adversarial training co-trains a ViT-B with one classification token for clean inputs and one for adversarial inputs, 768 extra parameters in all. It reports 83.76% clean and 49.87% $\ell_\infty$-robust ImageNet accuracy, and interpolates the two tokens by hand. Its clean/adversarial split is binary, and the backbone is trained. We freeze the backbone and generalize the two-token soup to a $K$-way coefficient space.
- Robustness Tokens freezes DINOv2 and trains ten appended tokens in 200 steps, recovering 63.1% accuracy from 0.3% under a white-box attack. Its defense depends on the tokens staying secret, and it covers one threat. We treat all added parameters as public and attack through them. This makes the comparison stricter, and makes it likely that tokens alone are too small, which is why the proposal uses LoRA-scale updates.
- ADAPT uses a pool of decoy prompts to absorb shortcut patterns during adversarial prompt tuning of CLIP, then discards them. It shows that pooled prompt capacity can be shaped adversarially. It works under one attack type and doubles training memory.
Multi-norm training methods solve the compute side, but only for full networks.
- MSD computes steps for every norm at each attack iteration, so its cost grows with the number of norms.
- NCAT reaches comparable union robustness with one attack step per iteration. It uses a curriculum on the perturbation size and alternating norm sampling.
- E-AT (the Croce and Hein paper above) trains multi-norm robustness at single-norm cost and can quick-fine-tune a robust model to new norms in one ImageNet epoch.
All three update every weight, and each new configuration is a new full checkpoint. We reuse the NCAT/E-AT training recipe (one threat sampled per minibatch, single-step attacks, an $\epsilon$ curriculum) as the training method for the shared dictionary. The contribution is therefore not a faster attack. It is a test of whether that recipe can drive a compressible, modular parameterization on a frozen backbone.
Distillation from language experts, as in Efficient Multilingual ASR Finetuning via LoRA Language Experts, is the other way to collapse modules. That approach trains every expert first, so it keeps linear training cost, and it collapses specialists into one student. Our aim is to avoid training $K$ experts at all.
Research Program
Establishing the premise. On frozen ImageNet ViT-B backbones, we first train independent rank-$r'$ LoRA specialists for $\ell_\infty$, $\ell_2$, $\ell_1$, and ImageNet-C corruptions. We use two backbones: a clean self-supervised backbone, as in the Robustness Tokens setting, and an $\ell_\infty$-robust ViT drawn from RobustBench or from Revisiting Adversarial Training for ImageNet. We then measure how much of each specialist's robust accuracy survives when all specialists are projected onto a shared rank-$r$ basis fit to their stacked updates. Plotting robust accuracy against $r$ gives a direct compressibility curve.
Training the dictionary directly. We train $U$, $V$ and all $c_k$ jointly, sampling one threat per minibatch with a single-step attack. This makes the per-step cost equal to single-norm adversarial training. For $c_\cup$, the loss is computed on the sampled attack, following E-AT.
Adding a threat. We hold out one threat, either $\ell_1$ or corruptions. After training on the others, we freeze $U$ and $V$ and fit only $c_{new}$. We count backbone forward-backward passes until the new coefficients come within two points of a dedicated specialist.
Watching for fake robustness. NCAT reports that single-step $\ell_1$ training overfits catastrophically without a curriculum, and whether frozen-backbone modules are more or less prone to this is unknown. We track the gap between single-step training accuracy and multi-step APGD accuracy throughout training. We also run the attack-failure diagnostics from the pralab/IndicatorsOfAttackFailure codebase listed in the file.
Baselines
Naive references. The first is the frozen backbone with no module. The second, for the robust-backbone case, is the $\ell_\infty$-robust backbone evaluated as-is on all threats. That second reference measures how much union robustness is already inherited, which Revisiting Adversarial Training for ImageNet suggests is non-trivial with convolutional stems.
Independent specialists. One LoRA per threat, trained with multi-step attacks and the same total pass budget as our method. This is the linear-cost upper reference for per-threat accuracy. Beating it means matching its per-threat robust accuracy within one point at a smaller parameter count and fewer adversarial passes.
Strongest parameter-efficient competitor. A single shared LoRA with the same total parameter count as our dictionary ($2dr + Kr$), trained with E-AT alternating sampling. This is also the key ablation: it equals our method with every $c_k$ forced identical. Beating it means higher accuracy on each threat's specialist deployment, with no loss in union worst-case accuracy.
Strongest existing method overall. Full-network E-AT quick fine-tuning from the Croce and Hein paper. It reports "more than 51%" multiple-norm robustness on CIFAR-10. It also reports the first ImageNet model robust to the union of $\ell_1$, $\ell_2$ and $\ell_\infty$ from a single epoch of fine-tuning; for that ImageNet union accuracy, no reported result found on Lacuna. For the full-network CIFAR-10 union references, NCAT reports 43.7% worst-case accuracy on ResNet-18, against 43.9% for 50-step MSD in its own comparison, while MSD reports 47.0%. These are different architectures from ours, so they calibrate rather than compare directly. We rerun E-AT on our exact ViT and CIFAR-10 fine-tuning setup.
Fairness. All methods get the same backbone and data, the same budget of backbone forward-backward passes, and the same number of hyperparameter trials. All are evaluated with AutoAttack through RobustBench, with every added parameter visible to the attacker.
Why not just…?
Why not just quick-fine-tune the whole network with E-AT? This is the strongest practical competitor, and it may win on union accuracy, because it can move every weight. Its costs are different in kind. Every deployment configuration is a full checkpoint. It needs a robust starting model. And the E-AT paper itself notes the single-model union trades away per-norm optimality. We report union accuracy per unit of trainable parameters and per adversarial pass. We will state plainly if full E-AT dominates on absolute union accuracy even at matched compute.
Why not just train one shared adapter on the union? This is the real threat to the proposal. If E-AT's inherited-$\ell_2$ effect extends to all our threats, a single module may equal the specialists. The equal-parameter shared-LoRA baseline settles this directly. If per-threat specialist accuracy shows no gap over the shared module, the dictionary adds nothing, and we would report that union robustness on frozen backbones is not a capacity problem.
Why not just keep independent adapters, since they are small? For token-sized modules this objection is correct. Ten threat-specific token sets are a few tens of thousands of parameters, trivial next to the backbone. The argument here rests on LoRA-scale updates, which we expect to be needed once added parameters are public and attacked. It also rests on training compute: every independent adapter is another multi-step adversarial run. The held-out-threat experiment measures whether coefficient-only fitting actually saves those passes. If dedicated modules converge as fast as the 200 steps reported by Robustness Tokens, the compute argument weakens and only the storage argument at LoRA scale remains.
Why not route on the input, as prompt pools do? Routing on the input is the only way to pick specialists automatically when the threat is unknown, which Revisiting adapters names as future work. But the router is itself attackable. We test an input-conditioned soft router over the same dictionary under an adaptive attack that optimizes through the router. We expect it to be exploitable and report it as a finding, not the main method.
Decisive Checks
Three results decide the proposal. The first is the compressibility curve. If the projected specialists need $r$ close to $Kr'$ to keep accuracy, the shared-structure premise is false. The second is the specialist gap: the dictionary's per-threat coefficients must beat the equal-parameter shared adapter on the threat each coefficient serves. The third is the held-out-threat test: fitting only $c_{new}$ on a frozen basis must reach within two points of a dedicated specialist, using a small fraction of the specialist's adversarial passes, under AutoAttack with no training/evaluation gap indicating gradient masking.
We run each check on ImageNet-1K with both backbones, and on CIFAR-10 fine-tuning for contact with the multi-norm literature. We also report clean accuracy and ImageNet-C/-R shift accuracy so that robustness gains are not bought with clean performance.
References
- Soft Language Clustering for Multilingual Model Pre-training
- Adversarial robustness against multiple lp-threat models at the price of one and how to quickly fine-tune robust models to another threat model
- Revisiting adapters with adversarial training
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models
- Adversarial Robustness Against the Union of Multiple Perturbation Models
- Toward Efficient Robust Training against Union of $\ell_p$ Threat Models
- Efficient Multilingual ASR Finetuning via LoRA Language Experts
- Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Models
- RobustBench: a standardized adversarial robustness benchmark