High-Pass Correction Adapters for Cheap Dense Robustness in Frozen Vision Transformers

Large pretrained vision transformers break easily under adversarial attack, meaning small, deliberately crafted pixel changes. Retraining them adversarially is expensive because every weight update needs an attack computed first. Robustness Tokens, a recent cheap fix, keeps the backbone frozen and learns about ten extra input vectors. That works surprisingly well for whole-image classification but much less well for pixel-level tasks such as semantic segmentation. We propose a different cheap fix aimed at those pixel-level tasks: small per-location correction branches, attached to a few layers of the frozen backbone, that operate only on the fine-detail ("high-pass") part of each feature map. That fine detail is what self-attention smooths away, and it is also where gradient-based attacks concentrate their energy. The branches are trained for a few hundred steps with the same cheap objective Robustness Tokens uses, but the loss is applied to every spatial location rather than to a pooled summary.

The motivation comes from joining two findings that have not been put together. The first is that self-attention behaves like a low-pass filter: it averages tokens and erases edges and textures. The second is that adversarial perturbations are mostly high-frequency. A frozen backbone therefore has no internal mechanism for repairing local, high-frequency damage. A handful of global tokens, which can only act through attention's averaging, is badly placed to undo damage that differs from patch to patch. If the hypothesis holds, robustness for dense prediction can be bought at roughly the cost of the Robustness Tokens recipe. If it fails, the experiments will show whether the bottleneck is spatial locality, frequency band, or simply parameter count.

Core Claim

In a frozen vision transformer under attack, the gap between clean and adversarial features is concentrated in the high spatial frequencies of the patch-token grid at intermediate layers. A correction module that is spatially local, restricted to that band, and trained with a per-location correction loss will recover more segmentation robustness than global robustness tokens or a parameter-matched generic adapter. It will do so at comparable training cost and with clean accuracy preserved.

The prediction is falsifiable in three ways:

  • If the spectral diagnostic shows the clean-versus-adversarial gap is spread evenly across frequencies, the band restriction is unmotivated.
  • If a plain per-location adapter of the same size does as well, the texture argument is wrong and the contribution reduces to "spatial adapters."
  • If ROB tokens trained with a dense loss do as well, locality is not the bottleneck.

Position Relative to Prior Work

The method is borrowed from Robustness Tokens. That work freezes DINOv2, DeiT-III, or OpenCLIP and trains about ten appended tokens with two losses. An invariance loss keeps clean outputs close to the original model's outputs. A correction loss pulls adversarial features back toward clean ones. It converges in about 200 steps and lifts DINOv2-G white-box ImageNet accuracy from 0.3% to 63.1%, with clean accuracy at 85.2% versus 85.6%. On ADE20K, however, mIoU under attack recovers only from 4.7 to 23.3. The paper also assumes the attacker cannot see the tokens. We reuse its training objective and frozen-backbone cost profile. We change where the correction acts (per location, at intermediate layers, in a frequency band rather than appended to the sequence), and we evaluate against attackers who know the module.

Three lines of evidence motivate the frequency targeting.

HAT targets clean accuracy through full training. We target adversarial robustness through frozen-backbone adaptation, so we do not compete with it on its own goal.

The module design draws on texture-preserving architectures that route fine detail around attention.

  • CRAFT feeds features from a convolutional high-frequency block into attention as queries for super-resolution.
  • CALIP and Cross-DINO bypass pooling or gate backbone features into the encoder.

None of these was evaluated under attack. Shortcuts that carry high-frequency content could equally carry adversarial noise straight through. Our module is therefore gated and trained explicitly to correct that band, not just pass it through.

One paper constrains the design and appears to cut against it. Mitigating low-frequency bias (HFDR) reports that adversarial training pushes CNNs to rely on low frequencies and ignore texture, which hurts robustness. HFDR adds a learned frequency disentanglement on top of full PGD training. We read this as a warning: a correction module could learn the lazy solution of deleting high frequencies, which would remove the attack but also the detail segmentation needs. The invariance loss is meant to block that solution, and we measure the module's spectral behavior directly to see whether it does.

FrequencyLowCut pooling shows that fixed frequency operations confer little robustness without an adversarial objective. This rules out a filter-only design.

Research Program

Phase 1: spectral diagnostic. For frozen DINOv2 backbones (Small through Large, plus Giant if compute allows), we generate PGD attacks on the segmentation loss over ADE20K images. At each layer, we take the 2D Fourier transform of the patch-token grid for clean and adversarial inputs. We record how the per-layer feature discrepancy is distributed across frequency bands and how that distribution changes with depth. This directly addresses the open question of whether robust-ViT vulnerability comes from attention's own low-pass behavior. It also decides which layers receive modules and where the band cutoff sits. If the discrepancy turns out to be low-frequency, the proposal pivots before any module is trained.

Phase 2: the correction module. At a small set $\mathcal{S}$ of layers chosen in Phase 1, the frozen block output is modified as

$$h_\ell' = \mathrm{Block}_\ell(h_\ell) + g_\ell \odot \phi_\ell\big(h_\ell - B(h_\ell)\big),$$

where:

  • $h_\ell$ is the patch-token grid entering layer $\ell$;
  • $B$ is a fixed local blur, so $h_\ell - B(h_\ell)$ is the high-pass residual;
  • $\phi_\ell$ is a small depthwise-plus-pointwise convolution;
  • $g_\ell$ is a per-location sigmoid gate computed from the same residual.

Only $\phi_\ell$ and $g_\ell$ are trained. The objective is the Robustness Tokens loss applied per location:

$$\mathcal{L} = \sum_{p} \big\| z_p(x) - z^0_p(x) \big\|^2 + \lambda \sum_{p} \big\| z_p(x+\delta) - z^0_p(x) \big\|^2,$$

where:

  • $z_p$ is the final patch feature at location $p$ of the adapted model;
  • $z^0_p$ is the same feature from the original frozen model;
  • $\delta$ is a PGD perturbation recomputed against the adapted model;
  • $\lambda$ trades off clean fidelity against correction.

The training loop is unchanged from the Robustness Tokens recipe. The cost is dominated by attack generation through the frozen backbone, so we report wall-clock GPU-hours including attacks, not just trainable-parameter counts.

Phase 3: adaptive evaluation. We consider two threat models. In the gray-box setting, matching the Robustness Tokens paper, the attacker sees the backbone but not the learned module. In the full white-box setting, the attacker differentiates through the module and gate. We also run an attack designed specifically against the gate: it adds a loss term that pushes the gate toward zero, which tests whether the defense can simply be switched off. Following RobustBench, which found that original papers overestimated robustness by more than 10%, classification robustness uses AutoAttack. Segmentation robustness uses many-step PGD with restarts.

Baselines

Frozen backbone (naive reference). This is the unmodified frozen backbone with its segmentation and classification heads. It sets the floor; the Robustness Tokens paper reports ADE20K mIoU under attack of 4.7.

Robustness Tokens (strongest existing method). This is the Robustness Tokens method, the strongest existing competitor. It is the only cited method that adds robustness to a frozen foundation ViT at comparable cost, and it already reports a dense number: 23.3 mIoU on ADE20K under attack. We will reproduce it on the same backbones, data, attack used during training, number of optimization steps, and hyperparameter search budget as our module. We will also try a version with more tokens so that its trainable parameter count matches ours. Beating it means a higher adversarial mIoU than our reproduced ROB-token number under the full white-box attack, with a margin larger than the spread across three seeds. Clean mIoU must stay within one point of the frozen backbone, and the gain must hold in both threat models. Matching its 63.1% ImageNet robust accuracy is a secondary condition: dense gains must not come at the cost of classification robustness.

Plain spatial adapter (key ablation). This baseline isolates the main ingredient. It is a per-location adapter with the same architecture, parameter count, layers, and dense loss, but it receives $h_\ell$ instead of the high-pass residual. If it matches the full module, the frequency argument is refuted. A second ablation keeps the high-pass input but removes the gate. A third keeps everything but uses a pooled rather than per-location loss.

Reference points for the cost axis. Adversarial low-rank (LoRA) fine-tuning of the backbone gives an upper-capacity reference for what modifying the backbone itself buys at the same step budget. HAT's reported 1.7x to 2.3x overhead and FLC's 1.26x overhead are not measured on the same task, so they serve only as context, not as direct comparisons.

Why not just…?

Why not just adversarially train the segmentation head? The head is cheap to train, but it sees features that the attack has already disrupted. The Robustness Tokens paper reports clean-versus-adversarial cosine similarity of 0.12 before correction, which is nearly orthogonal. A linear or shallow head is unlikely to undo damage of that size when it differs per location. That is a plausibility argument, not proof, so we run head-only adversarial training at the same step budget. If it matches our module, the backbone-side intervention is unnecessary and we will say so.

Why not just use Robustness Tokens with a dense loss and more tokens? This is the most serious threat, because tokens do influence every patch through attention. Our argument is that this influence is mediated by softmax weights that the attacker also controls, and that it is inherently averaging. The parameter-matched, dense-loss ROB-token baseline tests the argument directly. If it closes the gap, locality is not the bottleneck and the proposal reduces to a better training loss for the existing method.

Why not just low-pass filter the input or features? FLC pooling's authors report that fixed frequency operations without an adversarial objective give only marginal robustness. HFDR reports that static filtering discards useful information. Fixed filters are also easy to differentiate through, so an adaptive attacker can account for them. We include an input-blur baseline evaluated with attacks that pass through the blur, mainly to confirm that a filter alone does not produce the result.

Why not just do adversarial LoRA or full fine-tuning? With a large enough budget this will likely win on robustness, and that is fine. The question here is robustness at a fixed, small training cost on a frozen, shared backbone. We report it at equal steps and at its own convergence point so readers can see the full cost-robustness curve rather than a single favorable point.

Decisive Checks

The experiments run on ADE20K semantic segmentation with frozen DINOv2 backbones as the primary setting, and on ImageNet-1K classification as a secondary setting. Metrics are mIoU and top-1 accuracy on clean and attacked inputs, GPU-hours including attack generation, and per-layer spectral discrepancy.

Three outcomes would settle the proposal:

  1. The Phase 1 spectrum must show the clean-versus-adversarial gap concentrated at high token frequencies; otherwise the design premise fails.
  2. The full module must beat both reproduced Robustness Tokens and the parameter-matched plain adapter on adversarial mIoU under full white-box and gate-targeted attacks, at equal step budgets.
  3. A spectral check on the trained module must show it restores high-frequency feature energy on clean inputs rather than deleting it. This is the failure mode HFDR warns about. Restoration should also show up as preserved clean mIoU, with the largest per-class gains on thin, small-object classes.

If (2) holds only in the gray-box setting, the honest conclusion is that the module, like the tokens, is a secret key rather than a robustness mechanism.