Adversarial training makes image classifiers resistant to small worst-case input perturbations, but it is expensive. Every weight update must first search for the most damaging perturbation of each training image, which multiplies training cost by roughly 10x or more. The cheapest variant takes one gradient step instead of many (FGSM, the fast gradient sign method). It suffers from a known failure called catastrophic overfitting: partway through training, robustness against stronger multi-step attacks drops abruptly to zero. The best existing fix, GradAlign, works but needs second-order "double" backpropagation, which gives back part of the speed it was meant to save. We propose a different fix. We hypothesize that catastrophic overfitting has a frequency signature in the convolution weights themselves: early filters acquire large gain at high spatial frequencies just before the failure. If so, a penalty on that high-frequency gain should prevent it. The penalty is computed from the weights alone, uses no data, and costs about as much as weight decay.
The opportunity is a single-step training recipe that reaches GradAlign-level robustness at large perturbation budgets with almost no added cost. The penalty has a tight ablation built in. By Parseval's identity, our penalty is ordinary weight decay with its weight redistributed across frequencies. If the method works, that ablation tells us whether the frequency targeting matters or whether the effect is just stronger regularization. The design also faces two specific risks. Removing high-frequency features may cost clean accuracy, and a model pushed toward low frequencies may become easier to fool with low-frequency attacks. Both are measured directly, not assumed away.
Core Claim
Three claims are tested. First, the rise of high-frequency energy in early-layer filters precedes the collapse of gradient alignment that defines catastrophic overfitting. Gradient alignment is the cosine similarity between the loss gradient at an input and at a nearby perturbed point. Second, penalizing that energy prevents the collapse at large perturbation budgets ($\epsilon = 16/255$ on CIFAR-10) at single-step cost. Third, the robustness this produces survives adaptive and frequency-targeted attacks rather than reflecting a new form of gradient masking.
The evidence for the first claim is indirect. Understanding and Improving Fast Adversarial Training diagnoses catastrophic overfitting as alignment dropping from about 0.95 to about 0.05. It also reports that even a single filter learning an edge-detecting Laplace-like pattern can trigger the failure, and a Laplace filter is a high-frequency filter. The frequency view of convolution supplies the method. Absum observes that convolutions behave like circulant matrices whose eigenvectors are Fourier modes, so a filter's sensitivity to each frequency is read directly off its Fourier transform. Absum penalizes the absolute sum of filter weights, which is the filter's response at zero frequency. It was never tested against white-box iterative attacks; its authors concede it does not replace adversarial training.
We target the opposite end of the spectrum and use the penalty inside adversarial training, not instead of it. Let $w_{l,c,c'}$ be the $k\times k$ filter in layer $l$ connecting input channel $c$ to output channel $c'$. Let $\hat w(\omega)$ be its discrete Fourier transform zero-padded to the feature-map size, and let $\mathcal{H}_\rho$ be the set of frequencies with normalized radius above a cutoff $\rho$. The penalty is
$$R(\theta) = \sum_{l} \lambda_l \sum_{c,c'} \sum_{\omega \in \mathcal{H}_\rho} |\hat w_{l,c,c'}(\omega)|^2 ,$$
where $\lambda_l$ is a per-layer strength. Because this is a fixed quadratic form in the weights, its gradient is a fixed linear map that can be precomputed once per kernel size. The cost is therefore that of weight decay, with no inner loop and no double backpropagation. Setting $\mathcal{H}_\rho$ to all frequencies recovers plain weight decay exactly, which gives the central ablation.
Research Program
The work proceeds in three stages.
Diagnosis. We train single-step FGSM with random initialization, following Fast is better than free, at budgets known to trigger catastrophic overfitting. At every epoch we log three quantities: high-band filter energy per layer, gradient alignment, and multi-step robust accuracy. The question is whether high-band energy rises before alignment collapses, by how many epochs, and in which layers. If the energy rise lags or is absent, the hypothesis is false and the project stops here, which is a useful negative result.
Intervention. We add $R(\theta)$ to single-step training and sweep the cutoff $\rho$, the per-layer strengths, and the set of layers penalized. Our prior is that the first one or two layers matter most, because deeper filters act on feature maps where "spatial frequency" is less meaningful; depth is an explicit ablation.
Stress-testing. This stage connects to the adaptive-attack and trade-off literatures. A Fourier Perspective on Model Robustness shows two relevant facts. Standard models earn clean accuracy partly from high-frequency features (over 50% ImageNet accuracy on high-pass-filtered images). Adversarial training shifts models toward low frequencies, which can worsen fog and contrast corruption robustness by up to 30%. Our penalty may simply accelerate this shift. We therefore measure clean accuracy, low-frequency corruption robustness, and low-frequency attack success, not just standard robust accuracy. Low Frequency Adversarial Perturbation shows that perturbations restricted to a low-frequency DCT subspace (a ratio of 1/8) find adversarial directions efficiently. A model that suppresses high-frequency gain is exactly the kind of model such attacks should exploit.
Baselines
The naive reference point is plain single-step FGSM with random initialization, using the recipe of Fast is better than free. That paper reports 45% robust accuracy against strong PGD on CIFAR-10 in 6 minutes at the standard budget. Understanding and Improving Fast Adversarial Training reports that it collapses to roughly 0% at $\epsilon = 16/255$. We run it with the same architecture (PreAct ResNet-18), schedule, and epoch count as our method. Beating it means non-zero, stable robustness at budgets where it collapses.
The strongest competitor is FGSM with GradAlign, because it is the published method that solves exactly this failure at single-step attack cost. The file reports only that it is "comparable to the much slower multi-step PGD training" at $\epsilon=16/255$. No reported numeric robust accuracy for GradAlign at that budget was found on Lacuna, so we reproduce it ourselves. GradAlign receives the same sweep budget over its regularization coefficient as we give $\lambda$ and $\rho$, on the same hardware. We report wall-clock time per epoch alongside accuracy. Beating it means matching its AutoAttack robust accuracy within about one point, at equal or better clean accuracy, with measurably lower wall-clock cost. Matching robustness at the same cost would be a tie, not a win.
The cheap-regularizer competitors are Spectral Normalization from Generalizable Adversarial Training via Spectral Normalization and Absum. Spectral Normalization bounds each layer's largest singular value with a reported 2% to 20% runtime overhead. Absum penalizes the zero-frequency response; on CIFAR-10 it reports 79.05% accuracy against the Single Fourier Attack, versus 69.09% for weight decay. We graft each onto the same single-step training with matched tuning budgets.
The key ablation is uniform weight decay tuned to the same total weight norm as our penalized models. This isolates whether the frequency targeting, not overall weight shrinkage, prevents the collapse. The upper reference is ten-step PGD adversarial training. DAT reports 82.78% clean accuracy for PGD adversarial training on CIFAR-10. We do not expect to exceed multi-step trade-off methods such as HAT or AR-AT, which reports 90.89% clean and 50.77% AutoAttack accuracy with a WideResNet. We report them only to locate the cost-accuracy frontier.
Why not just…?
Why not just use GradAlign? It works, and it is the bar to clear. Its weakness is cost: Understanding and Improving Fast Adversarial Training requires double backpropagation, which partly cancels the single-step saving. The file reports this qualitatively but gives no per-epoch number, so our timing comparison supplies it. If GradAlign's measured overhead turns out to be small on modern hardware, the practical case for our method weakens. The remaining contribution would be the diagnostic finding about where catastrophic overfitting originates in weight space, and we would say so.
Why not just turn up weight decay? This is the most serious threat, because our penalty is literally reweighted weight decay. Absum shows that uniform weight penalties strong enough to stop frequency attacks also spike training loss and cost accuracy. That result is for a different attack, though, so it does not settle this question. The norm-matched weight decay ablation does settle it. If uniform decay prevents catastrophic overfitting as well and at the same clean accuracy, frequency targeting adds nothing and the claim fails.
Why not just use Spectral Normalization? It is cheap and controls the network's overall sensitivity to input changes. However, it caps a layer's gain at every frequency equally, which should cost clean accuracy in the low and mid frequencies where semantic content lives; its bound $\beta$ needs tuning for this reason. It was also never evaluated with AutoAttack. Comparing at matched robustness and reporting clean accuracy, together with per-band sensitivity plots, will show whether band-selective control buys a better trade-off.
Why not just use smaller step sizes or early stopping on a PGD validation set? Fast is better than free relies on this kind of tuning, and it fails at large budgets according to Understanding and Improving Fast Adversarial Training. Early stopping also halts training at whatever robustness was reached before collapse. We include early-stopped FGSM as a baseline to show how much robustness is left on the table.
Proposed Experiments and Decisive Checks
Datasets and budgets. The main setting is CIFAR-10 at $\epsilon \in \{8, 12, 16\}/255$ under the $\ell_\infty$ norm (a cap on each pixel's change). We also run CIFAR-100, and SVHN up to $12/255$, the range where GradAlign was tested. Tiny ImageNet serves as a scale check if CIFAR results hold.
Evaluation. Robust accuracy uses AutoAttack through the RobustBench library and the evaluation scripts of the HAT codebase, plus PGD-50 with restarts. Following On Adaptive Attacks to Adversarial Example Defenses, we add attacks aimed at the plausible weak link. These include PGD restricted to the low-frequency DCT subspace from Low Frequency Adversarial Perturbation, the Single Fourier Attack, and the score-based Square attack. A gap between gradient-based and gradient-free attacks would signal masking. We also run the diagnostic checks in IndicatorsOfAttackFailure to catch optimization failures. Because the penalty touches only the weights and leaves the inference pipeline unchanged, no BPDA or expectation-over-transformation approximations are needed. Masking would therefore have to arise from loss-landscape shape, which these checks target.
Other metrics. We report clean accuracy, mean corruption error on CIFAR-10-C with fog and contrast broken out, wall-clock time per epoch, and gradient-alignment trajectories.
Decisive checks. Three results would decide the project. First, in the diagnostic runs, high-band energy must lead the alignment collapse across seeds; otherwise the mechanism is wrong. Second, at $\epsilon=16/255$, the penalized model must hold stable AutoAttack robustness near GradAlign's reproduced level, while the norm-matched weight decay ablation collapses. Third, low-frequency attacks and the fog and contrast corruptions must not degrade the penalized model by more than they degrade GradAlign and PGD-trained models. Failing the third check would show that the method trades one blind spot for another, which A Fourier Perspective on Model Robustness suggests is the likely failure mode.
References
- Understanding and Improving Fast Adversarial Training
- Absum: Simple Regularization Method for Reducing Structural Sensitivity of Convolutional Neural Networks
- Fast is better than free: Revisiting adversarial training
- A Fourier Perspective on Model Robustness in Computer Vision
- Low Frequency Adversarial Perturbation
- Generalizable Adversarial Training via Spectral Normalization
- DAT: Improving Adversarial Robustness via Generative Amplitude Mix-up in Frequency Domain
- Reducing Excessive Margin to Achieve a Better Accuracy vs. Robustness Trade-off
- Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off
- RobustBench: a standardized adversarial robustness benchmark
- On Adaptive Attacks to Adversarial Example Defenses