Calibrated Latent Adversarial Training for Frozen Vision Transformers

A common way to make a large pretrained image model robust is to freeze its weights and adversarially train a small set of added parameters. The catch is that this saves less compute than it seems. Adversarial training means repeatedly searching for the small pixel change that most confuses the model, and every step of that search still back-propagates through the entire frozen network down to the pixels. Parameter-efficient methods shrink what gets stored and updated, but they leave this search, which dominates the cost, untouched.

We propose to move the search inside the network. The lower part of a Vision Transformer (ViT, an image model that splits the picture into patches and processes them with self-attention) stays frozen. Small adapters are added only to the upper blocks. The adversary perturbs the hidden activations at a chosen intermediate layer instead of the pixels. The perturbation is not an arbitrary norm ball: its shape is calibrated, at intervals, from the activation shifts that real pixel attacks produce at that layer. Each attack step then only needs gradients through the upper blocks, and the lower blocks never need a backward pass at all. Because the frozen lower layers and the original clean pathway are untouched, the clean-accuracy penalty of adversarial training should also shrink.

Core Claim

Pixel-space attacks on a pretrained ViT appear to produce structured, low-dimensional shifts in intermediate activations. If so, adapters trained against a latent adversary confined to that structure can reach pixel-space robustness, measured by AutoAttack, close to that of adapters trained with full pixel-space attacks. They would do so at a fraction of the compute and with a smaller clean-accuracy penalty.

Two findings in the literature make this plausible. First, Robustness Tokens, which appends 10 learnable tokens to a frozen ViT, reports that attacks succeed largely by suppressing a few "massive activations," the extreme outlier values in intermediate layers. That is a strongly structured failure mode. Second, FACM argues on CNNs that adversarial noise compounds with depth, so intermediate layers remain less corrupted. Both findings motivate the proposal; neither paper builds a training procedure on them.

Concretely, let $h_k(x)$ be the activation at module $k$, for instance the GeLU output inside a block's feed-forward network. Let $g_\phi$ be the upper blocks with adapter parameters $\phi$, $y$ the label, and $\mathcal{L}$ the loss. Training solves

$$\min_\phi \; \mathbb{E}_{(x,y)} \Big[ \max_{\delta^\top \Sigma_k^{-1} \delta \le 1} \mathcal{L}\big(g_\phi(h_k(x) + \delta), y\big) \Big],$$

where $\Sigma_k$ is a covariance matrix estimated from the shifts $h_k(x + \delta_{\text{pix}}) - h_k(x)$. Here $\delta_{\text{pix}}$ are pixel-space attacks computed against the current model on a small calibration batch, refreshed every $T$ steps.

Consider a $K$-step attack on an $L$-block network with the latent layer at depth $k$. The per-example attack cost drops roughly from $K \cdot L$ block forward-and-backward passes to $K \cdot (L-k)$, since only the upper blocks are traversed. The calibration batch adds a cost of order $1/T$ of a full pixel attack.

Research Program

The first stage is diagnostic and can falsify the idea cheaply. On an ImageNet-21k-pretrained ViT-B, as used in the module-probing study, we run pixel-space PGD (projected gradient descent, the standard iterative attack). At every block and every internal module we record the induced activation shift and measure three things. The effective rank tells us how many directions the shift occupies. Stability across images tests whether one $\Sigma_k$ can serve many inputs. Stability across training refreshes tests whether $\Sigma_k$ drifts slowly enough for a refresh every $T$ steps.

That probing study found that the feed-forward activation and the second LayerNorm give the most robust representations under natural distribution shift, and that the projection back down inside the feed-forward network discards information. We test whether the same modules are where adversarial shifts are lowest-rank. This would also give a way to choose $k$ without any out-of-distribution validation data, which Intermediate Layer Classifiers names as an open problem.

The second stage trains the adapters. We use the LayerNorm-plus-low-rank modules of FullLoRA-AT, which showed that plain low-rank adapters fail under adversarial training unless LayerNorm parameters can also shift. The modules are placed only above layer $k$. We add the clean-invariance term from Robustness Tokens, which keeps clean features close to the frozen model's, to limit the clean-accuracy penalty.

The third stage is honest evaluation. Robust accuracy is always measured by attacking the pixels of the full model, through the frozen lower layers and with the adapters in place, using AutoAttack. That paper found that reported robustness was overestimated for 48 of 49 models it re-evaluated.

Baselines

The naive reference is the frozen pretrained ViT with no defense, together with plain pixel-space PGD training of all adapters. These anchor the clean accuracy ceiling and the robustness and cost targets.

The strongest directly comparable method is FullLoRA-AT. It trains the same kind of adapters, throughout the network, with full pixel-space attacks. On CIFAR-10 it reports needing about 5% of the parameters and about 75% of the training time of full adversarial fine-tuning, and a 4.29% clean-accuracy gain over that baseline. We found no reported ImageNet number for it on Lacuna. We will rerun it with identical backbone, data, attack budget, adapter rank and tuning budget.

Beating it means matching its AutoAttack accuracy within one point at no more than half its measured GPU-hours, or exceeding it at equal GPU-hours.

On ImageNet we also compare against three other methods:

  • TORA-ViTs, which adds separate clean and robust adapters to a frozen ViT. It reports 83.7% clean and 38.0% PGD accuracy at its balanced setting.
  • AGAT, which drops background patches during single-step training. It reports 51.23% PGD-20 accuracy on ViT-B with a 35% training-time reduction.
  • The dual class-token method of Rebuffi et al., which co-trains the full backbone with separate clean and adversarial class tokens. It reports 83.76% clean and 49.87% robust accuracy. It serves as an upper reference that pays full training cost.

Two ablations isolate the key ingredient. The first replaces the calibrated ellipsoid with an isotropic latent ball of matched size. The second freezes $\Sigma_k$ at initialization instead of refreshing it.

Why not just…?

Why not just use single-step pixel attacks with adapters? This is the most serious threat, since one step is as cheap as our latent attack at moderate depth. AGAT shows that plain single-step training on ViTs suffers catastrophic overfitting: robustness to multi-step attacks collapses mid-training, and larger models are worse. AGAT's patch dropping stabilizes it. We will therefore include single-step training with AGAT-style dropping on the same adapters at matched GPU-hours. If it equals our method, the latent approach adds nothing and we will say so.

Why not just use Robustness Tokens, which converge in about 200 steps? Robustness Tokens reports 63.1% attacked accuracy on DINOv2-Giant, but its authors note that protection weakens if the attacker optimizes through the tokens. We will evaluate it under a full white-box attack that includes the tokens. Our method's robustness is measured only in that fully white-box setting.

Why not just reuse perturbations across epochs, or use fewer attack steps? This cuts cost without changing the threat model, so it is a fair competitor. We include cached-perturbation and 2-step pixel-attack variants at matched compute. Their weakness is that stale perturbations no longer target the current adapters. Our calibration refresh is designed to fix exactly that, and the comparison will show whether it matters.

Proposed Experiments

We run CIFAR-10 and CIFAR-100 with ε = 8/255, then ImageNet-1K ViT-B with ε = 4/255. For each method we report clean accuracy, PGD-20 and AutoAttack accuracy, and measured GPU-hours. We also report ImageNet-C, -R and -A to check that latent training does not erode natural robustness. The point of reference there is TORA-ViTs' reported ImageNet-C mean corruption error of 34.4% and ImageNet-R accuracy of 56.3%. We sweep the latent depth $k$ and module type, and plot robustness against cost.

Three outcomes would be decisive.

  1. Low rank must hold. The diagnostic must find pixel-induced shifts with low effective rank at some mid-depth module. If they are full-rank everywhere, the premise fails.
  2. Calibration must matter. The calibrated latent adversary must beat the isotropic ablation on AutoAttack. If not, calibration is not the operative ingredient.
  3. The cost advantage must survive competition. The full method must beat AGAT-style single-step training and FullLoRA-AT on the AutoAttack-accuracy-versus-GPU-hours curve, while losing less clean accuracy than FullLoRA-AT.