Decaying the Layers That Memorize: Weight Decay Aimed by Easy-Example Gradients in Adversarial Training

Adversarially trained networks are trained on inputs that an attacker has deliberately nudged to cause mistakes. Such networks show a peculiar failure called robust overfitting. Accuracy on attacked test images peaks early and then falls, even while the training loss keeps dropping. Two separate findings point to where this failure comes from. At the data level, robust overfitting is driven by the network continuing to fit adversarial examples it already classifies easily: removing the small-loss examples eliminates it, and removing the hard ones does not (Understanding Robust Overfitting of Adversarial Training and Beyond). At the parameter level, it lives almost entirely in the layers closest to the output. Freezing those layers removes it, and freezing early layers does not (On Intriguing Layer-Wise Properties of Robust Overfitting in Adversarial Training). Yet the most effective weight-decay fix, Adaptive Weight Decay (Improving Robustness with Adaptive Weight Decay), still applies one penalty strength to every layer. This is the uniform-regularization pathology that banded ridge regression and group norm constraints were designed to escape in other settings.

We propose to join the two findings into a single signal. For each layer, we measure how much of its current gradient comes from adversarial examples the network already fits. We then set that layer's weight decay in proportion to this share. The claim is concrete and falsifiable. A per-layer "easy-example gradient share" should rise first in the layers that are beginning to memorize, and it should rise at whatever point in training memorization starts. Decay driven by this signal should therefore locate and time regularization automatically. It should match or beat hand-split layer methods and global adaptive decay, including on datasets where the split point or timing differs from the CIFAR-10 default.

Core Claim

Adversarial training is the minimax procedure of Towards Deep Learning Models Resistant to Adversarial Attacks: an inner attack (projected gradient descent, PGD) finds a worst-case perturbation in a small ball around each input, and the outer step trains on it. In that setting, a layer that needs more regularization is one whose update is dominated by examples it has already fit.

We make this precise as follows. At step $t$, attack a minibatch and split it at the $q$-quantile of per-example adversarial loss. Let $g_l$ be the gradient of the full batch loss with respect to layer $l$'s weights $w_l$. Let $g_l^{\text{easy}}$ be the gradient from the below-quantile subset alone. Define the easy share $\rho_l(t) = \|g_l^{\text{easy}}\| / \|g_l\|$. The per-layer decay coefficient is

$$\lambda_l(t) = \alpha \,\frac{\|g_l\|}{\|w_l\|}\left(\frac{\rho_l(t)}{\bar\rho(t)}\right)^{\gamma},$$

where $\bar\rho(t)$ is the mean easy share across layers, $\alpha$ is a single target ratio playing the role of AWD's decay-over-gradient target, and $\gamma \ge 0$ controls how sharply decay is redistributed toward memorizing layers. Setting $\gamma = 0$ gives a layer-wise version of AWD, which balances each layer's decay against its own gradient. Additionally forcing the gradient ratio to be computed globally recovers AWD exactly. As in AWD, $\lambda_l$ is smoothed with an exponential moving average. We use a quantile rather than a fixed loss threshold on purpose. MLCAT's fixed $\ell_{\min}$ must be retuned for each dataset and each attack strength, and a quantile removes that dependence.

Research Program

The work has three parts, and the diagnostic comes first because the method is only justified if the diagnostic holds.

The first part is diagnostic. We train standard PGD adversarial training on PreAct ResNet-18 and log $\rho_l(t)$ for every layer. Alongside it we log two competing per-layer signals. One is the per-layer decay-to-gradient ratio from AWD. The other is the heavy-tail exponent of each weight matrix's eigenvalue spectrum, estimated with the Hill estimator as in AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs. That paper showed the exponent separates module types in language models but never examined adversarial training.

We then rerun the layer-freezing experiment of On Intriguing Layer-Wise Properties of Robust Overfitting in Adversarial Training at single-block granularity. Freezing a block and measuring the change in the best-to-last robust accuracy gap gives a ground-truth "responsibility" score for each block. The prediction is that $\rho_l$, measured before overfitting starts, ranks blocks by responsibility better than the other two signals. It should also rise at the learning-rate decay on CIFAR-10 and earlier on SVHN, where robust overfitting starts before the first decay and the learning-rate variant of the layer-split method fails.

The second part is the method itself: easy-share-weighted per-layer decay as defined above. The easy-subset backward pass is computed every $k$ steps. We sweep $k$ in the ablations. At $k=1$ the overhead is one extra backward pass on a fraction $q$ of the batch, which is small next to a multi-step PGD attack.

The third part tests the tension raised by AlphaDecay, which reported that global AWD hurt language-model perplexity (16.22 versus 14.66 for uniform decay). The question is whether per-layer allocation also matters in adversarially trained transformers. We run a small vision transformer under the same protocol and compare global AWD against the per-layer variants. This part is secondary and is reported whichever way it comes out.

This proposal sits in a specific place relative to prior work. Improving Robustness with Adaptive Weight Decay and Adaptive Weight Decay: On The Fly Weight Decay Tuning for Improving Robustness supply the balancing rule we generalize, and they acknowledge that their penalty is uniform across layers. The layer-wise paper motivates the layer localization but uses a fixed 50/50 split and perturbation or learning-rate tricks rather than a decay penalty. MLCAT supplies the easy-example mechanism but acts on the loss or on all weights, not selectively on layers. The group-penalty literature is the reason we treat layers as groups with separate penalties rather than one pool. That literature includes Feature-space selection with banded ridge regression, which fits separate ridge penalties per feature space, and MGNC-CNN, which applies separate norm constraints per embedding source. Both set their groups from prior knowledge. We assign the strength from a training signal instead.

Baselines

The naive reference is standard PGD adversarial training with a tuned fixed global weight decay, reported at both the best and the last checkpoint. The best checkpoint is exactly what early stopping would give. For this setup, the layer-wise paper reports a best-to-last AutoAttack gap of 7.86% on CIFAR-10 with PreAct ResNet-18. To count as progress, our method must close that gap while keeping the best-checkpoint accuracy, not merely flatten the curve at a lower level.

Global AWD is the closest method in spirit and the one whose limitation we target. It reports 54.04% AutoAttack accuracy on CIFAR-10 and 29.70% on CIFAR-100, but only when combined with the ASAM optimizer. The file does not state architecture details for those numbers, so we rerun AWD ourselves, with and without ASAM, rather than compare against the reported figures.

The strongest direct competitor on the layer question is $RAT_{WP}$, which applies adversarial weight perturbation only to the latter half of the network. It reports 49.19% AutoAttack accuracy on CIFAR-10 with PreAct ResNet-18 under $L_\infty$ and a best-to-last gap of 0.87%. MLCAT-WP reports a CIFAR-10 peak robust accuracy of 58.48% against 52.29% for standard training, with a gap of 0.83%. The file does not say which attack produced that peak, so we treat it only as a reference and rerun MLCAT-WP under AutoAttack.

Two ablations isolate our key ingredient. Layer-wise AWD ($\gamma=0$) tests whether per-layer balancing alone explains any gain. A static two-group decay, with separately tuned decay strengths for the front and back halves, tests whether the adaptive signal adds anything over a fixed split.

For fairness, every method uses the same data splits, a held-out validation set of attacked training images, the same 10-step PGD training attack, and the same epoch budget. Each method also gets the same tuning budget of an equal number of full runs over its own hyperparameters. The static split is additionally allowed to search the split point. All results are three seeds. Beating $RAT_{WP}$ means higher last-checkpoint AutoAttack accuracy at a comparable or smaller gap, on at least CIFAR-10 and SVHN, with clean accuracy no worse by more than seed noise.

Why not just…?

Why not just stop early? Early stopping recovers the best checkpoint of standard training but not the higher peak that MLCAT-WP and the layer-wise method report, and it requires frequent attacked validation runs. We report best-checkpoint standard training as a baseline, so any gain has to come from a higher peak or a better final model, not from choosing a checkpoint.

Why not just put a heavier fixed decay on the latter half of the network? This is the real threat. The layer-wise paper suggests the latter half is where the trouble lies, and a two-value decay costs almost nothing. We include it with the split point searched. If, on each dataset and architecture, the best static split matches our method, then the adaptive signal's only contribution is removing that search, and we will say so. We expect it to lose where the responsible layers or the timing differ from the default. The candidate cases are SVHN, where overfitting starts before the learning-rate decay, WideResNet-34-10, where the depth profile differs, and the vision transformer. We will also test whether a split tuned on CIFAR-10 transfers to those settings without retuning.

Why not just use layer-wise AWD, or AlphaDecay's spectral exponent, as the per-layer signal? Both are cheaper or already published, and either could turn out to be enough. The diagnostic phase settles this directly. It measures how well each signal predicts the freezing-derived responsibility score. It also compares the final methods, $\gamma=0$ against spectral-exponent-scaled decay against easy-share decay. If the easy share does not beat both at predicting responsibility, the main claim fails.

Proposed Experiments

The benchmarks are CIFAR-10, CIFAR-100, and SVHN under $L_\infty$ perturbations with radius 8/255. The architectures are PreAct ResNet-18 and WideResNet-34-10, plus a small vision transformer for the third part. All robustness numbers use AutoAttack (Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks) under the RobustBench protocol. We also use the attack-failure indicators in IndicatorsOfAttackFailure to rule out gradient masking, which matters because a method that changes weight norms per layer could distort gradients. We report clean accuracy, best and last AutoAttack accuracy, and their gap.

There are two decisive checks. The first is the diagnostic correlation: the rank correlation between $\rho_l$ measured before overfitting and the per-block responsibility score from freezing, compared against the two alternative signals. The second is the SVHN result. Easy-share decay should suppress robust overfitting there, where the learning-rate variant of the layer-split method fails, and it should do so without changing any setting tuned on CIFAR-10. A sweep over $\gamma$, the quantile $q$, and the recomputation interval $k$ completes the study. If the gain vanishes as $\gamma \to 0$, the redistribution of decay toward memorizing layers is the active ingredient.