Linearity-Gated Perturbation Radii for Single-Step Adversarial Training

Adversarial training makes image classifiers resistant to small, deliberately crafted input changes. It does this by training on the worst perturbation found inside a small ball around each input, and finding that perturbation usually costs ten or more extra gradient passes per step. The cheap version takes one gradient step instead. It works until it suddenly fails: robust accuracy collapses to near zero within a few epochs, a failure called catastrophic overfitting. Collapse coincides with the network ceasing to behave linearly inside the perturbation ball. The fixes that work add penalty terms that need extra backward passes or a sensitive weighting coefficient, which gives back part of the speedup. Manifold learning met a structurally identical problem years ago. Methods that model data as locally flat patches break when one global neighborhood size is too large for curved or sparse regions. Adaptive Manifold Learning fixed this by shrinking each point's neighborhood until a first-order Taylor residual was small, then growing it back. That made the method insensitive to the initial neighborhood size.

We propose applying the same contract-and-expand rule to the perturbation radius in single-step adversarial training. Each training example keeps its own radius. At every visit, the network's departure from linearity along the attack direction is read off from two loss values the training step already computes. The radius shrinks where that residual is large and grows back toward the target radius where it is small. The method needs no extra forward or backward passes, no penalty term, and no offline geometry. Its one tolerance is scale-free, and we test whether a single setting of it holds across threat radii and architectures. In short: single-step training fails because it trusts a linear approximation outside the region where that approximation is valid, and manifold learning already has a cheap, per-point rule for finding that region.

Core Claim

Let $g_i$ be the input gradient of the loss at example $x_i$ (or at its random start point). Let $u_i = \mathrm{sign}(g_i)$ be the single-step attack direction, and let $\epsilon_i$ be the example's current radius. Along that direction, the loss behaves to second order as $L(x_i + t u_i) \approx L(x_i) + t\, g_i^\top u_i + \tfrac{c_i}{2} t^2$, where $c_i$ is the curvature along the attack direction. The training step already evaluates the loss at the start point and at the final perturbed point $x_i + \epsilon_i u_i$, so the residual $r_i = |L(x_i+\epsilon_i u_i) - L(x_i) - \epsilon_i g_i^\top u_i|$ is free. It gives $|c_i| \approx 2 r_i/\epsilon_i^2$. Requiring the curvature term to stay below a fraction $\tau$ of the linear term yields the largest trustworthy radius:

$$ \epsilon_i^{\star} = \min\!\Big(\epsilon_{\text{target}},\; \frac{2\tau\, g_i^\top u_i}{|c_i|}\Big), $$

where $\epsilon_{\text{target}}$ is the threat radius we ultimately want robustness at, and $\tau$ is a dimensionless tolerance. The next visit to example $i$ uses $\epsilon_i^\star$, smoothed with the previous value so the radius cannot jump. Because $g_i^\top u_i = \|g_i\|_1$, everything is already on hand after the attack step. The state cost is one float per training example.

The claim is that this per-example gating prevents catastrophic overfitting, and that it reaches robust accuracy at the full $\epsilon_{\text{target}}$ comparable to the best single-step regularizers at lower wall-clock cost. The second half of the claim is what separates gating from simply shrinking $\epsilon$. Robustness must be measured at the full target radius, and gating must beat a uniform shrink with the same average radius.

How This Sits Relative to Prior Work

The motivating evidence comes from the work that diagnosed catastrophic overfitting as a loss of linearity. Understanding and Improving Fast Adversarial Training showed that, during collapse, the cosine similarity between input gradients inside the ball falls from about 0.95 to about 0.05. It fixed this with GradAlign, a penalty that needs gradients of gradients. A Training-Dynamics View of Catastrophic Overfitting traced collapse to growth in the coupling between input and parameter gradients. It controlled that growth with a symmetric KL regularizer weighted by $\lambda$. Rethinking Fast Adversarial Training replaced penalties with an alternating splitting scheme (RAT). All three keep the radius fixed and change the loss. We do the reverse: keep the loss as plain single-step training and change where the attack is allowed to reach. The same diagnosis also predicts a check our method must pass. If the residual $r_i$ is a valid linearity signal, it should rise before robust accuracy collapses in unmodified single-step training.

The method template comes from Adaptive Manifold Learning. That paper prunes each local patch until its Taylor/SVD residual is small, then expands it to keep coverage. It reports that recovery no longer depended on whether the initial neighborhood was 15 or 30. We carry over its two-phase logic, its per-point state, and its claimed insensitivity to the starting scale. We replace its per-point SVDs, which the file notes scale poorly, with a scalar residual that costs nothing. The paper also carries a warning we take seriously: replacing $k$ with a tolerance can merely move the tuning burden. Our tolerance sweep is designed to show whether it does.

Four geometry-aware defenses constrain the design by showing what we should avoid. Manifold Regularization for Adversarial Robustness drops the adversary entirely. It samples within a fixed radius and adds smoothness terms, which makes it fast but leaves three interacting weights to tune and a failure mode in dense data regions. TART gates the perturbation radius by its tangential component, using an offline autoencoder and PCA with a global tangent dimension. Its training costs 1.24 times standard adversarial training before counting the offline work. Explaining the role of Intrinsic Dimensionality in Adversarial Training saves compute by perturbing only the lowest-intrinsic-dimension layer. For vision networks that layer is the first one, so the saving disappears. We work in input space, estimate geometry online, and pay nothing for it.

The closest relative in mechanism is Instance adaptive adversarial training (IAAT). It also keeps a radius per example, adjusted up or down according to whether a multi-step attack succeeded. IAAT's signal says nothing about whether the attack itself was trustworthy, and it requires multi-step attacks. Ours measures exactly that trustworthiness and runs with a single step. Because IAAT's rule could be ported to single-step training, we treat it as a baseline rather than a footnote.

Research Program

The first stage is diagnostic and could falsify the premise. We train ResNet-18 with plain single-step training (FGSM with random start) on CIFAR-10 at $\epsilon = 8/255$ and $16/255$. Along the way we log the distribution of the free residual $r_i$, the curvature estimate $|c_i|$, GradAlign's gradient cosine, and PGD robust accuracy. The premise holds only if the upper quantiles of the residual rise measurably before the collapse in robust accuracy, and track the cosine drop that GradAlign documented. If the residual moves only after collapse, the method has no early signal and we stop.

The second stage builds the gated trainer. Each example is attacked at its stored radius. The update uses the closed-form $\epsilon_i^\star$ above, with exponential smoothing and a floor so that no example drops to zero radius. That floor is the expansion phase borrowed from the adaptive manifold method, and it stops the radius distribution from silently shrinking. A within-step variant rescales the current perturbation down to $\epsilon_i^\star$ without recomputing the gradient, and tests whether waiting one epoch for the update matters. We also record how much of the training set sits at $\epsilon_{\text{target}}$ over time. If the method works, nearly all examples should reach the target radius by the end of training, and the remainder should be the high-curvature examples.

The third stage is the hyperparameter-sensitivity study, which is where the manifold-learning direction earns its place. For our method we sweep $\tau$ across an order of magnitude. For GradAlign and the KL regularizer we sweep $\lambda$ over a matched number of points. All sweeps are repeated at $\epsilon \in \{8, 12, 16\}/255$ and on two architectures. The quantity of interest is the spread of robust accuracy across each sweep, and whether a single setting picked at $8/255$ on ResNet-18 transfers unchanged to $16/255$ and to ViT-Small.

Baselines

The naive reference point is FGSM with random start at a fixed radius, with no regularizer, run for the same epochs and learning-rate schedule. It is expected to collapse at $16/255$, as Understanding and Improving Fast Adversarial Training reports. It fixes the compute floor that every method is measured against. Multi-step PGD-10 training serves as the expensive upper reference for robustness and cost. Beating it is not required. The goal is to approach its robustness at a fraction of its cost.

The strongest existing competitor at $8/255$ is the single-step mixed-Hessian KL regularizer from A Training-Dynamics View of Catastrophic Overfitting. On CIFAR-10 with ResNet-18 it reports 52.46% PGD-10 accuracy, against 51.79% for multi-step PGD-10 training, while running nearly 3 times faster. At large radii the strongest reported competitor is RAT from Rethinking Fast Adversarial Training. At $16/255$ on CIFAR-10 it reports 19.58% AutoAttack accuracy (versus 15.36% for GradAlign-style Fast-GA and 16.31% for local linearity regularization) and 1.7 hours of training versus 6.3 hours for PGD-AT. We also run GradAlign itself. The file gives no exact GradAlign number, only "comparable to multi-step PGD" at $16/255$. These methods are the strongest competitors because they attack the same failure, loss of linearity, with the same single-step budget.

For fairness, every method uses the same architecture, data augmentation, epochs, and learning-rate schedule, and all wall-clock times are measured on identical hardware. Each method gets the same tuning budget: a fixed number of trials on its single main hyperparameter ($\lambda$ for the regularizers, $\tau$ for ours), selected on a held-out validation split using PGD accuracy. All final numbers are AutoAttack through RobustBench, whose authors found that robustness claims made with weaker attacks were often overstated by more than 10 points.

Beating the strongest baseline means one of two things. The first is matching its AutoAttack accuracy at $8/255$ within seed noise and $16/255$ at measurably lower wall-clock time, given that our method adds no passes while the regularizers add at least one. The second is exceeding RAT's 19.58% at $16/255$ in our reproduction at no greater cost. A clearly flatter accuracy curve across the $\tau$ sweep than across the $\lambda$ sweeps would be a separate, independently reportable win.

Two ablations isolate the key ingredient, which is that the radius follows the per-example linearity signal. The matched-average ablation applies one global radius schedule equal to the batch mean of our per-example radii at each step, so the average perturbation size is identical. The shuffled ablation randomly permutes the per-example radii across the batch, preserving their distribution but breaking the link between radius and example. If either ablation matches the full method, the geometry is doing nothing, and the gain comes from a softer average radius.

Why not just…?

Why not just use a smaller global $\epsilon$, or a warm-up schedule that ramps it? This is the most serious threat, because our method could reduce to a covert shrink of the radius. That would echo the finding in Understanding and Improving Fast Adversarial Training that random start helps only by lowering the effective step size. Two features of the design address this. Evaluation is always at the full target radius, and the matched-average ablation gives a global schedule exactly the same mean radius as ours. If that schedule matches us, we will report that the per-example signal adds nothing, and the contribution would shrink to a recipe for setting a curriculum.

Why not just add GradAlign or the KL regularizer and tune $\lambda$? These methods work: the KL regularizer's 52.46% PGD-10 at $8/255$ is the bar. Their costs are the extra backward or forward passes and a weighting coefficient that the file flags as governing a delicate accuracy-robustness trade-off. Our advantage, if there is one, lies in cost and tuning stability rather than peak accuracy. The matched-budget sweep tests this directly. If the regularizers prove as insensitive to $\lambda$ as we hope our method is to $\tau$, and the wall-clock gap is small, then our case narrows to simplicity.

Why not just port IAAT's attack-success rule to single-step training? IAAT already shows that per-example radii shift the clean-robust trade-off, with roughly 4 points of clean accuracy on CIFAR-10 at matched robustness. A single-step version that shrinks the radius when FGSM flips the label is cheap. However, the success of a single-step attack is exactly the signal that becomes unreliable near collapse. A collapsing model resists FGSM easily, so this rule would grow radii at the worst possible moment. We run this port as a baseline at $16/255$. If it avoids collapse as well as linearity gating does, the geometric framing loses its justification.

Why not just stop early when PGD accuracy peaks? Both the GradAlign and RAT papers describe early stopping as a workaround that costs clean accuracy, and RAT notes that the loss jumps around irregularly before collapse, which makes the stopping point hard to pick. We report early-stopped FGSM with random start as a reference point, with stopping chosen on validation PGD accuracy.

Proposed Experiments and Decisive Checks

The main experiments run on CIFAR-10, CIFAR-100, and SVHN with ResNet-18 and ViT-Small, at $\epsilon \in \{8, 12, 16\}/255$ under the $\ell_\infty$ norm. We report clean accuracy, PGD-50 accuracy, AutoAttack accuracy, wall-clock time, and the number of runs, out of five seeds, that collapse. SVHN matters because GradAlign held only up to $12/255$ there. ViT-Small matters because the KL paper reports 32.59% PGD robustness on it while plain single-step training collapsed. An ImageNet run with ResNet-50, compared against RAT-BP's reported 3.29% PGD-50 versus 2.25% for Fast-AT, tests whether the per-example state and the residual signal still work at scale. The easyrobust toolkit supplies reference single-step implementations. A small toy based on TART's concentric-circles setup will show visually whether small radii concentrate where the decision boundary bends.

Three outcomes decide the proposal. First, the residual must rise before collapse in unmodified single-step training. Second, the full method must avoid collapse at $16/255$ in all seeds, beating both the matched-average and the shuffled ablation on AutoAttack accuracy at the full target radius. Third, its accuracy spread across the $\tau$ sweep must be no larger than the regularizers' spread across $\lambda$, at lower wall-clock time. If the first fails, the premise is wrong. If the second fails, gating is only a disguised curriculum. If only the third fails, the method still prevents collapse at zero added cost, but replaces one sensitive hyperparameter with another.