Curvature-Gated Layer-Wise Step Sizes for Fast Adversarial Training

Adversarial training teaches a network to resist small, deliberately chosen input perturbations. It does this by attacking the model during training and learning from the attacked inputs. Its cost comes from the attack. Strong multi-step attacks multiply training time several-fold, while cheap one-step attacks tend to fail through catastrophic overfitting: within one or two epochs the model learns to defeat the one-step attack and loses nearly all resistance to stronger attacks. Most known fixes add extra forward or backward passes, which gives back much of the speed that one-step training was meant to save. Separately, large-batch training relies on layer-wise adaptive optimizers such as LARS and LAMB. These give each layer its own step size, set by the ratio of its weight norm to its gradient norm, so that no single layer blows up. Nobody has asked whether this optimizer-level control can also stop catastrophic overfitting. That question matters most in the large-batch regime, where practitioners already have to use these optimizers.

We propose that the per-layer step size should depend on how differently that layer's gradient behaves at the random starting point of the one-step attack and at the attacked point. The one-step attack already passes through both points, so this signal is a nearly free, per-layer finite-difference probe of the input–parameter curvature that recent work identifies as the trigger of catastrophic overfitting. A layer whose two gradients disagree sharply gets its step shrunk. We expect the effect to concentrate wherever shortcuts are forming, which is reported to be early layers at the onset of collapse and later layers during late-stage overfitting, with no hand-chosen layer split.

Core Claim

Plain trust ratios are ambiguous for catastrophic overfitting, and that ambiguity is the reason to study them. The LARS rule from ImageNet Training in Minutes shrinks a layer's step when its gradient is large. That should damp the gradient explosions that precede collapse; Understanding Catastrophic Overfitting in Single-step Adversarial Training reports the input-gradient norm spiking up to 35x before collapse. The same rule enlarges the step of layers with large weights. Layer-Aware Analysis of Catastrophic Overfitting finds that collapse is carried by large weights in early layers, and that pruning the largest 10% of those weights partly restores robustness. LARS could therefore accelerate the very shortcuts it is supposed to prevent.

Our claim is that the missing ingredient is a curvature signal, not a magnitude signal. Let $g_l^0$ be layer $l$'s parameter gradient at the random start $x+\delta_0$, and let $g_l^1$ be its gradient at the attacked input $x+\delta$. We define

$$\kappa_l = \frac{\|g_l^1 - g_l^0\|_2}{\|g_l^1\|_2 + \|g_l^0\|_2},$$

which lies in $[0,1]$. For small perturbations, $g_l^1 - g_l^0 \approx H^{(l)}_{\theta x}(\delta-\delta_0)$, where $H^{(l)}_{\theta x}$ is layer $l$'s slice of the mixed input–parameter Hessian. A Training-Dynamics View of Catastrophic Overfitting shows that the spectral norm of this mixed Hessian explodes before collapse, so $\kappa_l$ is a per-layer estimate of that quantity. The proposed update for layer weights $W_l$ is

$$W_l \leftarrow W_l - \eta \,\frac{\|W_l\|_2}{\|u_l\|_2}\, \frac{1}{1+\lambda\,\bar\kappa_l}\, u_l .$$

Here $u_l$ is the base update (the momentum-SGD direction, or the Adam direction as in LAMB from Large Batch Optimization for Deep Learning), $\bar\kappa_l$ is a running average of $\kappa_l$ normalized by its median across layers, $\eta$ is the global learning rate, and $\lambda$ is a single gate strength. The descent direction is $g_l^1$ alone. The probe gradient $g_l^0$ only sets the step size, which separates our method from gradient-averaging schemes.

Two existing lines of work come closest, and both leave this open. Distributed adversarial training to robustify deep neural networks at scale (DAT) applies a LAMB-style uniform trust ratio, but to multi-step training, and does not study catastrophic overfitting. LAP, from the Layer-Aware Analysis paper, puts adversarial weight perturbations mostly in early layers under a fixed decaying profile and trains with an ordinary optimizer. We change the optimizer instead of the objective, and we let the measured curvature decide which layers to slow down.

Research Program

The work has three stages. The first is diagnostic. Using R-FGSM (one-step attack with random start) on CIFAR-10, we log per-layer $\kappa_l$, trust ratios, and weight norms across runs that do and do not collapse. The goal is to establish two things: whether $\kappa_l$ rises in early layers before the robust-accuracy collapse, and whether it leads the 35x input-gradient spike. If $\kappa_l$ does not lead collapse by at least several hundred steps, the gating idea has no basis and we stop there. In the same stage we answer the plain question of whether LARS or LAMB, with no gate, makes collapse more or less frequent than SGD. That result is useful whatever its sign, because large-batch fast adversarial training on ImageNet currently runs with exactly these optimizers and no one knows how they interact with collapse.

The second stage tests the gated optimizer at standard batch sizes against loss-level fixes. The third pushes to large batches, where the method would offer the most. Fast is better than free already shows that one-step training is fragile under large learning rates, and DAT shows that multi-step training hits a "scaling wall" without layer-wise rates. Combining the two has never been mapped.

There is also a tension in prior work that this program can test. Collapse in one-step training is reported to start in early layers (Layer-Aware Analysis paper). Late-epoch robust overfitting in multi-step training is reported to live in the final layers, per On Intriguing Layer-Wise Properties of Robust Overfitting, whose static late-layer learning-rate fix fails on SVHN because overfitting there begins before the schedule changes. If $\kappa_l$ really tracks curvature, its peak should move from early layers to late layers over the course of training. A gate driven by $\kappa_l$ would then handle both failure modes with one rule, which a static split cannot.

Overhead is the key constraint, so we state it honestly. Extracting weight gradients during the attack's backward pass adds roughly one weight-gradient computation and no extra forward pass. We will also test computing $\kappa_l$ on a quarter of the batch, or every $k$ steps, and report wall-clock time against LAP's reported 7% overhead.

Baselines

The naive reference point is R-FGSM with momentum SGD and the cyclic schedule of Fast is better than free. The Layer-Aware Analysis paper reports that at $\epsilon=16/255$ this baseline collapses to near 0% AutoAttack accuracy. Beside it we run PGD-10 adversarial training as the expensive upper reference, since it defines the robustness that one-step methods try to reach cheaply.

The strongest existing competitors at the hard perturbation size are LAP and the Douglas-Rachford splitting method of Rethinking Fast Adversarial Training. R-LAP reports 19.55% AutoAttack accuracy on CIFAR-10 at $\epsilon=16/255$ (Layer-Aware Analysis paper), and the splitting method reports 19.58% under the same setting. LAP is the most direct rival because it is also layer-aware and nearly as cheap. At $\epsilon=8/255$, the strongest cheap regularizer is ELLE-A from Efficient local linearity regularization, which reports 44.32% AutoAttack accuracy. For collapse frequency, FGSM-PCO from Preventing Catastrophic Overfitting in Fast Adversarial Training reports zero collapses in 10 WideResNet-34-10 runs on CIFAR-100, where FGSM-MEP failed in 6 of 10. At large batch on ImageNet, the reference is DAT, which reports 38.45% robust and 63.75% clean accuracy with ResNet-50 on 36 GPUs. That result is for multi-step training, and the file gives no reported result for one-step training at matched large batch.

For fairness, we retrain every baseline ourselves in one codebase, built on the ELLE repository and its PGD-50-10 and AutoAttack evaluation scripts. All methods use the same architectures (PreActResNet-18 and WideResNet-34-10), epoch budgets, and data augmentation. Each method, ours included, gets the same hyperparameter search budget over learning rate and its own regularization strength, and we report means over 10 seeds. We count our method as beating the competition only if it meets two conditions together. Its AutoAttack accuracy must match or exceed retrained LAP and splitting at $\epsilon=16/255$, within seed noise. Its collapse rate must be no worse than FGSM-PCO's, and its wall-clock time must be no more than LAP's.

Three ablations isolate the key ingredient. The first is the trust ratio with no gate ($\lambda=0$), plain LARS or LAMB. The second keeps the gate's per-layer step-size profile but replaces $\kappa_l$ with a fixed early-to-late decay copied from LAP. The third gates on per-layer gradient norm instead of gradient disagreement. If the second or third ablation matches the full method, the curvature signal is not what does the work.

Why not just…?

Why not just use LARS or LAMB as they are? That is the first ablation, and it may help. Plain trust ratios normalize away gradient spikes, but they also enlarge steps for large-weight layers, which are the layers implicated in shortcut formation. Which effect dominates is unknown, and the diagnostic stage answers that before any new mechanism is claimed. If plain LAMB already removes collapse at matched cost, that becomes the headline result and the gate is dropped.

Why not just lower the learning rate, or clip gradients, in early layers? Static layer-wise schedules have a documented failure: the late-layer learning-rate fix of the Robust Overfitting paper breaks on SVHN because overfitting starts before the scheduled change. A fixed early-layer rule would also need to change sign over training if robust overfitting really moves to late layers. The LAP-profile ablation tests this directly. If that ablation equals the full method on CIFAR-10, CIFAR-100, and SVHN, the dynamic signal is unnecessary and we will say so.

Why not just average the two gradients, or use their disagreement as a loss penalty? This is the real threat. Averaging $g^0$ and $g^1$ uses exactly the same information and resembles a cheap smoothing step, and penalizing their gap is a parameter-space version of the curvature penalties in GradAlign-style methods or the mixed-Hessian regularizer. We run both variants: a direction-averaging update and a penalty that adds one backward pass. The distinction we test is that our gate changes only step size, never direction, so in layers where $\kappa_l$ stays low the model trains on the true adversarial gradient. If averaging matches us at equal cost, the contribution reduces to that simpler diagnostic, and we will report it that way.

Why not just detect collapse early and switch to a two-step attack? That is a reasonable engineering fallback, and $\kappa_l$ could serve as its trigger. However, switching raises cost exactly in the long-training and large-$\epsilon$ regimes where collapse recurs. We include it as a baseline at matched wall-clock time.

Proposed Experiments and Decisive Checks

On CIFAR-10 and CIFAR-100, we train PreActResNet-18 and WideResNet-34-10 at $\epsilon \in \{8/255, 16/255\}$ over 10 seeds. We report AutoAttack and PGD-50-10 accuracy, clean accuracy, collapse rate, and wall-clock time. SVHN tests the early-overfitting case on which static schedules fail. Tiny-ImageNet tests scale; the file has no reported number for any of these methods on it, so all comparisons there are our own reruns. Extended runs of 200 or more epochs check for delayed collapse, since a gate that only slows learning might postpone collapse rather than prevent it. The large-batch study sweeps batch size from 512 to 8192 on CIFAR-10 and runs ResNet-50 one-step training on ImageNet at a fixed epoch budget. It compares SGD, LAMB, gated LAMB, and DAT's multi-step training on robust accuracy per GPU-hour.

The decisive checks are four. First, $\kappa_l$ must rise before collapse, and first in early layers. Second, gated training must cut the collapse rate at $\epsilon=16/255$ to near zero while plain LAMB does not. Third, the LAP-profile and gradient-averaging ablations must fall short of the full method. Fourth, the location of $\kappa_l$'s peak must shift toward late layers in late training. If the first check fails, the mechanism is wrong. If either ablation in the third check matches the full method, the simpler method is the finding.