Cheap Robust Fine-Tuning by Searching Only Low-Frequency Perturbations

Adversarial training protects an image classifier by repeatedly attacking it during training and learning to resist the attacks. It works, but it is expensive: every weight update needs an inner search, usually 7 to 20 gradient steps, to find a damaging perturbation. That multiplies training cost by roughly 10 to 30 times (TRADES). It also costs clean accuracy, because the model learns to ignore fine image detail that is still useful for classification (Huang et al.; Maiya et al.).

This note asks whether both costs can be cut by restricting the inner search to the coarse, smooth part of the image. In frequency terms, that is the low-frequency corner of the discrete cosine transform (DCT), the JPEG-style decomposition of an image into smooth and fine patterns. Several observations point the same way. Adversarial directions are dense in a low-frequency subspace with about 1/64th of the pixel dimensions (Guo et al.). Low-frequency attacks need fewer iterations to succeed (Sharma et al.). Attackers who adapt to a robust model move their perturbation energy into low frequencies (Huang et al.). ImageNet-scale models are biased toward low and mid frequencies even before any robust training (Maiya et al.).

Core Claim

When a pretrained backbone is robustly fine-tuned with a small adapter, a 2-step inner search confined to a low-frequency DCT subspace may give nearly the same worst-case robustness as a 10-step full-space search. This is measured by AutoAttack over the full pixel-space budget. The low-frequency version would cost about a fifth of the inner-loop compute. Because the adapter is never pushed to suppress high-frequency features, it should also lose less clean accuracy.

The claim is falsifiable in an obvious way. If the model is only ever attacked in low frequencies during training, an attacker who knows this can put the whole perturbation budget into the complementary high-frequency band. The proposal stands or falls on that attack.

Positioning

The closest prior work is the frequency-constrained adversarial training of Maiya et al.. They train models from scratch with frequency-weighted perturbations. They show that weighting low frequencies buys robustness at a clean-accuracy cost, and they suggest that band-targeted training could be cheaper. They do not evaluate against AutoAttack or band-aware adaptive attacks, they do not account for compute, and they do not study fine-tuning of pretrained models. Our work differs on all three points, and Maiya et al. motivate it directly.

Two methods supply the training recipe we build on. AutoLoRa attributes the clean-accuracy drop in robust fine-tuning to gradient conflict, meaning the natural-loss and adversarial-loss gradients point in opposing directions. It fixes this by giving the natural objective to a low-rank adapter that adds no inference cost, but it still runs full-space PGD and never examines frequency. ROB tokens freeze a vision transformer and train about 10 extra input tokens in 200 steps. They have not been tested against low-frequency attacks. We reuse both setups and change only the inner search.

PhaseAT also uses frequency to speed up adversarial training, but it reshapes the learning signal. It shifts high-frequency targets toward low frequencies and still does full inner maximization. Our proposal leaves the network alone and cuts the attack.

Three papers constrain the design. CSR and DiffBreak show that defenses which assume high-frequency noise collapse under low-frequency attacks; diffusion purification falls to about 0%. This is why we train against low frequencies rather than filter them out. Athalye et al. show that robustness claimed without adaptive evaluation is unreliable. This is why the band-complement attack is part of the core evaluation rather than an afterthought.

Research Program

The perturbation is parameterized as $\delta = \mathrm{IDCT}(M_r \odot z)$, clipped to the $\ell_\infty$ ball of radius $\epsilon$. Here $z$ holds DCT coefficients, $M_r$ keeps the top-left $r$ fraction of frequencies along each axis, and IDCT is the inverse transform back to pixels. The inner search runs $k$ signed-gradient steps on $z$. We sweep $r$ over {1/16, 1/8, 1/4, 1/2} and $k$ over {1, 2, 3}. The ratio $r$ is tuned per image resolution, because Maiya et al. find that the vulnerable band differs between 32-pixel and ImageNet-scale images.

The outer step is unchanged from AutoLoRa or ROB tokens. This keeps the comparison clean: the only thing that varies is where and how long the attacker searches during training.

Alongside the method we run one diagnostic that tests the mechanism. For a full-space PGD perturbation, we split it into DCT bands. For each band's component we compute the parameter gradient of the adversarial loss and its cosine similarity with the natural-loss gradient. If AutoLoRa's gradient conflict is concentrated in the high-band components, that explains why low-band-only training should preserve clean accuracy. If conflict is spread evenly across bands, the clean-accuracy part of the claim is weakened and we will report it that way.

Baselines

The naive reference points are two. The first is the pretrained backbone fine-tuned with no robust training, which should have near-zero robust accuracy. The second is vanilla robust fine-tuning with 10-step full-space PGD.

The strongest competitor is AutoLoRa with its standard full-space PGD inner loop. It directly targets both the clean-accuracy drop and the cost of robust fine-tuning, and it adds no inference latency. On CIFAR-100 with ResNet-18, AutoLoRa reports AutoAttack robust accuracy 3.41 points above vanilla robust fine-tuning and 2.03 points above the earlier best method, TWINS. The file gives no absolute value, so we reproduce it ourselves. We run AutoLoRa with the same backbones, data, adapter rank, epochs and tuning budget as our variant. We report results at matched epochs and also at matched total forward-backward passes.

On ImageNet with transformers, the reference is ROB tokens. On DINOv2-G they report 63.1% white-box robust accuracy and 85.2% clean accuracy, against 85.6% for the undefended model (ROB tokens).

For orientation only, we also cite full-network adversarial training on CIFAR-10 with ResNet-18. PhaseAT reports 52.1% AutoAttack and 86.2% clean accuracy, and TRADES reaches 48.5% AutoAttack in the same table (PhaseAT).

Two controls isolate the key ingredient. The first is a random DCT mask of the same size (Sharma et al.'s DCT_Random), which tests whether the gain comes from low frequency specifically or from any small subspace. The second is a full-space search with the same $k$, which tests whether the subspace matters at all once the step count is held fixed.

Beating the strongest baseline means the following. The method must stay within one point of full-space AutoLoRa on full-ball AutoAttack and on the band-complement attack, and also match or exceed its clean accuracy. It must do so with at most 40% of the inner-loop gradient evaluations.

Why not just…?

Why not just use cheap single-step full-space attacks? This is the most serious threat to the proposal. The file contains no reported result for single-step robust fine-tuning of adapters. We therefore run full-space $k=1$ and $k=2$ at identical cost. If they match low-frequency $k=2$ on AutoAttack, the subspace adds nothing and the claim reduces to "few steps suffice for adapters." We would report that instead.

Why not just low-pass filter the input at test time? Filters keep low-frequency perturbations intact. The winning input-transformation defenses from the NeurIPS 2017 competition gave almost no extra protection against low-frequency attacks (Sharma et al.). Non-differentiable filters are also bypassed by BPDA, which replaces the filter with a differentiable stand-in when computing gradients (Athalye et al.). We include a Gaussian-blur preprocessing baseline under BPDA plus the low-frequency attack for completeness.

Why not just evaluate existing full-space defenses against low-frequency attacks? A low-frequency perturbation inside the same $\ell_\infty$ ball is a special case of a full-space one. A model trained with full-space PGD may therefore already resist it, and on AutoLoRa we expect it largely will. That evaluation alone would be a useful audit of Gap 1, but it says nothing about the cost. Our contribution is the claim that the cheap subspace search is sufficient, not that full-space training is insufficient.

Proposed Experiments

We fine-tune ImageNet-pretrained ResNet-18 and ViT-B on CIFAR-10 and CIFAR-100 under the AutoLoRa protocol with $\epsilon = 8/255$. We fine-tune DINOv2 with ROB tokens on ImageNet-1K. Metrics are clean accuracy, AutoAttack accuracy, and gradient evaluations and wall-clock time per epoch.

The decisive adaptive suite has three parts. The first is a band-complement attack: APGD restricted to the DCT frequencies outside $M_r$, with the full $\epsilon$. The second is the low-frequency attacks of Sharma et al. (DCT_Low with the Momentum Iterative Method) and Guo et al. (LF-BA), run against every model. The third is a sweep of the attacker's subspace ratio, which locates where each model is weakest.

Robust accuracy under the band-complement attack must stay within one point of full-ball AutoAttack. If it drops substantially, training only on low frequencies has created a blind spot, and the method fails.

The per-band gradient-conflict analysis is run on both the vanilla and the low-frequency fine-tuning, and tests the explanation for any clean-accuracy gain.

A secondary check applies the low-frequency attack to ROB tokens and CSR. It tests whether the adaptive vulnerability shown for diffusion purification (DiffBreak) carries over to parameter-efficient and test-time defenses.