Cheap Defense Against Attacks That Transfer From a Public Backbone

Most deployed vision models today are built by adapting a publicly released backbone such as DINOv2 or OpenCLIP. The adaptation is a small add-on trained on private data, for example a linear head, a low-rank weight update (LoRA), or a few learned tokens. This creates a specific and underexamined threat. An attacker who never sees the deployed model can download the public backbone and craft perturbations that scramble its internal features, meaning the intermediate activations inside the network. Because the private model reuses those features, the perturbations transfer. We propose to test a simple claim with a direct payoff. The damage these attacks do inside the public backbone concentrates in a small number of feature directions, and those directions can be estimated from the public model alone. If that holds, a deployer can buy protection against the whole family of public-backbone attacks with two things. The first is a one-time, shareable batch of precomputed perturbations. The second is a correction that acts only on those few directions. That combination costs almost nothing at training time and leaves clean accuracy intact, because nothing outside the attack subspace is touched.

This matters because the two usual remedies both fail practitioners. Adversarial training, which retrains the model on attacks generated on the fly, multiplies training cost by roughly an order of magnitude. It also typically costs more than 10% clean accuracy on ImageNet, a drop reported alongside A Self-supervised Approach for Adversarial Robustness. Test-time purifiers avoid retraining, but they add inference latency and are fragile under adaptive attack. Our proposal targets the threat that actually dominates the public-backbone setting and asks whether its structure makes it cheap to neutralize.

Core Claim

Consider feature-disruption perturbations $\delta$ crafted on a frozen public backbone $f$ to push its layer-$l$ features $f_l(x+\delta)$ away from the clean features $f_l(x)$. We claim the resulting displacements $\Delta_l(x) = f_l(x+\delta) - f_l(x)$ lie mostly in a low-dimensional subspace $U_r$, spanned by the top $r$ principal directions of the displacements, that is shared across images and across attack algorithms.

Two findings in the file motivate this. Adversarial Vulnerability from Interference Between Features in Superposition studies networks that pack more features than they have dimensions. It argues that attacks exploit fixed geometric overlaps between those packed features, and that models sharing this geometry share their attacks: synthetic transfer rose from 5% to 98% when data correlations were shared. A fixed interference geometry predicts a fixed, reusable set of attack directions. Robustness Tokens found that attacks on ViTs suppress a handful of "massive activation" channels, a few outlier channels that act as internal biases. That is another sign the damage is concentrated.

Neither paper measures the subspace directly for transfer attacks on real backbones, and neither builds a defense that exploits it. Our defense trains a small adapter $g$ on top of the frozen backbone. The adapter must undo the adversarial displacement only inside $U_r$, and it must leave clean features unchanged:

$$\mathcal{L} = \big\| P_r\big(g_l(x+\delta) - f_l(x)\big) \big\|^2 + \lambda \, \big\| g_l(x) - f_l(x) \big\|^2 + \mathcal{L}_{\text{task}},$$

The terms are as follows:

  • $P_r$ is the projection onto $U_r$.
  • $g_l$ is the adapted model's layer-$l$ feature.
  • $\lambda$ weights clean-feature invariance.
  • $\mathcal{L}_{\text{task}}$ is the ordinary downstream loss.

The perturbations $\delta$ are computed once on the public backbone and cached. The training loop therefore contains no inner attack, and the cache can be shared by every deployer of the same backbone.

Relation to Prior Work

Robustness Tokens supplies the method we build on. It appends about ten learned tokens to a frozen ViT, trains them in 200 steps to map attacked features back to clean ones, and keeps DINOv2-Giant clean accuracy at 85.2% versus 85.6% undefended. We change it in three ways:

  • Threat model. It evaluates white-box attacks that treat the tokens as a private key. It does not evaluate attacks crafted on the token-free public backbone, which is exactly what a realistic attacker holds.
  • Where the correction acts. It corrects all feature directions. We restrict correction to the estimated attack subspace.
  • Where the perturbations come from. It generates attacks during token training. We cache them from the public backbone.

TART supplies the principle behind restricting the correction. It shows that forcing invariance along every perturbation direction warps decision boundaries and costs clean accuracy, and that restricting invariance to chosen directions recovers accuracy: on CIFAR-10 with WideResNet, clean accuracy rose from 81.76% to 83.73% while AutoAttack accuracy went from 35.89% to 36.58%. TART chooses directions in pixel space using an autoencoder's estimate of the data manifold, which does not scale to ImageNet. We choose them in feature space using the attacks themselves.

Two feature-level training methods motivate the attack construction but do not answer our question:

  • Feature Scattering generates unsupervised feature-space perturbations for training with a single step.
  • Q-TART prunes training samples whose features are unstable, reports better resistance to transfer attacks on CNNs, and cuts training time by up to 17.9%.

Both require training the full network, and neither targets a known public parent model.

The threat itself is motivated by three lines of work. Naseer et al.'s label-free self-supervised perturbation attack (SSP) in A Self-supervised Approach for Adversarial Robustness shows that attacks which distort intermediate features transfer across architectures. Two Sides of the Same Coin and From Pretrain to Pain show that vulnerabilities of a public backbone survive fine-tuning. Adversarial Manipulation of Deep Representations supplies the second attack type we test, feature mimicry, where an image's features are pushed toward those of a chosen guide image.

The main constraint on the design comes from Understanding Robustness of Parameter-Efficient Tuning. On VTAB-1K it finds that linear probing on a frozen backbone is more robust than VPT (visual prompt tuning), adapters, or LoRA, and that VPT is especially fragile. A token-based defense therefore must not be assumed safe: it has to beat linear probing on the robustness axis.

Research Program

Stage 1: measure the subspace hypothesis directly. We will use DINOv2 (Small through Large) and OpenCLIP as public backbones. On ImageNet-1k training images we generate feature-disruption attacks: SSP-style divergence maximization at several depths, feature mimicry, and momentum and input-diversity variants from the transferability literature. We then compute how much variance of $\Delta_l$ the top $r$ directions capture, for $r$ from 1 to 256. The key test is whether a subspace estimated from one attack family and one image set captures the displacements of held-out attack families on held-out images. We also check whether the leading directions coincide with the massive-activation channels identified by Pulfer et al.

Stage 2: build the cheapest defense first. Before training anything, we test a training-free variant. It clips each coordinate of the features along $U_r$ to the range observed on clean data. This is the lowest-cost point on the cost curve.

Stage 3: train the adapter. We train the adapter under the loss above in three forms: robustness tokens, LoRA, and a small residual module. We sweep $r$ and $\lambda$. Throughout, we record training cost in GPU-hours and treat it as a primary metric, not an afterthought.

If Stage 1 shows the displacements are not low-rank, the proposal collapses to its unrestricted-correction ablation. That outcome is itself informative: it would say transfer attacks do not reuse a fixed interference geometry at realistic scale, contrary to what the superposition account suggests.

Baselines

The naive reference point is the undefended downstream model, adapted from the public backbone by linear probing and by LoRA. Its accuracy under public-backbone transfer attacks sets the size of the problem. Following Ruan et al., linear probing is also a serious competitor on robustness in its own right. We report it separately rather than folding it into "undefended."

The strongest existing low-cost defense is Robustness Tokens, run with its published objective and step budget on the same backbones and data. Its reported white-box numbers on DINOv2-Giant are 63.1% accuracy under attack, up from 0.3%, with clean accuracy of 85.2% versus 85.6%. On ADE20K segmentation it raises mIoU under attack from 4.7 to 23.3. We found no reported result on Lacuna for Robustness Tokens, or for any defense, against transfer attacks crafted on the public backbone and applied to a downstream adapted model. We will therefore run it ourselves.

Beating it means one of two outcomes at equal training compute and equal clean accuracy (within 0.5 points):

  • higher accuracy under public-backbone transfer attacks, or
  • equal transfer robustness at substantially lower training cost.

We add a second competitor from the purification line, NRP. It reports reducing ImageNet attack success from 79.8% to 31.9% under an ensemble input-diversity attack. We include it to price inference overhead against our approach, which adds no extra network pass.

For a heavy reference, we run fast single-step adversarial training on the downstream adapter, with perturbations computed through the private model, at matched wall-clock budget.

Two ablations isolate our ingredients. The first corrects all feature directions instead of only $U_r$, which tests whether the subspace restriction is what protects clean accuracy. The second trains the same adapter on the cached public-backbone perturbations with plain cross-entropy and no feature-level loss, which tests whether feature correction adds anything over data augmentation. All methods share the same training images, backbones, number of adapter parameters, and hyperparameter search budget, which is 20 configurations each, selected on a held-out validation split.

Why not just…?

Why not just adversarially train the downstream model? Full adversarial training is the most reliable defense, but its inner attack loop is exactly the overhead practitioners cannot afford on billion-parameter backbones. On ImageNet it has historically cost more than 10% clean accuracy, per the comparison in Naseer et al.. We include fast single-step adversarial training at matched wall-clock budget, so the proposal wins only if it matches that robustness against public-backbone attacks at a fraction of the cost.

Why not just use Robustness Tokens as published? This is a real threat. The published tokens may already correct transfer perturbations, since those perturbations also suppress massive activations. Our first experiment evaluates the published method against public-backbone transfer attacks. If it already matches our defense at the same cost, the contribution reduces to the subspace measurement and the finding that tokens transfer, and we will report it that way.

Why not just use linear probing, which Ruan et al. found most robust? Linear probing keeps the public features exactly, so a feature-disruption attack crafted on the public backbone hits it essentially white-box at the feature level. Its advantage in Ruan et al. was measured against attacks crafted on the tuned models themselves, not against feature attacks from the public parent. Our undefended linear-probing arm measures this directly. It also pays in clean accuracy on harder VTAB tasks.

Why not just augment training with the cached transfer examples? This is the most likely simpler competitor. It shares our amortized cost, and it may absorb most of the benefit. The plain cross-entropy ablation settles it. If that ablation matches the feature-level, subspace-restricted loss on both transfer robustness and clean accuracy, we will report that cached public-surrogate augmentation is sufficient and drop the subspace machinery from the recommended recipe.

Proposed Experiments

Data and backbones. We use ImageNet-1k classification, the 19 tasks of VTAB-1K, and ADE20K segmentation on DINOv2 and OpenCLIP backbones.

Attackers, from weakest to strongest. The attacker's knowledge increases through four levels:

  • attacks crafted on the public backbone alone;
  • attacks crafted on a public backbone plus the attacker's own LoRA, fine-tuned on public data for the same task, a stronger surrogate in the spirit of model-extraction transfer;
  • attack families held out from both subspace estimation and the cache;
  • full white-box AutoAttack through the adapter, run with RobustBench tooling and checked with the attack-failure diagnostics to rule out gradient masking, where a defense only appears robust because it hides useful gradients from the attacker.

Metrics. We report clean accuracy (or mIoU), accuracy under each attacker, cosine similarity between attacked and clean features, training GPU-hours including the one-time cache, and inference latency.

Decisive checks. Three results decide the proposal:

  • Subspace generalization. Does a subspace estimated from one attack family capture at least most of the displacement from held-out families? This is the hinge of the whole proposal.
  • Clean accuracy from the restriction. Does the subspace-restricted defense keep clean accuracy within 0.5 points of the undefended model, while the unrestricted ablation loses more?
  • Transfer robustness beyond the baselines. Does it beat both Robustness Tokens and cache-only augmentation under the attacker-fine-tuned surrogate?

We expect the defense to lose ground under full white-box attack, since its threat model assumes the adapter stays private. We will report that gap openly rather than claim general robustness.