Adversarial training makes image classifiers robust by training them on worst-case perturbed inputs. It is expensive because finding each perturbation normally takes ten or more extra gradient computations. The cheap alternative takes a single gradient step to build each perturbation, but it has a well-known failure called catastrophic overfitting: partway through training, the model learns to defeat the one-step attack while its robustness to real multi-step attacks collapses to near zero. The fixes that work add back cost, through second derivatives, extra forward passes, or step-size searches. A separate line of work puts a network's final features and classifier weights on a unit sphere, so that predictions depend only on angles and not on vector length. This removes a "make the vectors longer" shortcut that networks otherwise use to lower their loss.
This note proposes that the sphere does more than remove that shortcut in single-step training. On the sphere, the sharpness of the classifier head is set by one scalar, the logit scale $s$, and not by learned feature lengths. A single-step training loop also already computes, at no extra cost, the quantities needed to tell whether the loss behaved linearly along the attack direction. We would read that free linearity signal at every step and use it to adjust $s$ on the fly. The goal is to suppress catastrophic overfitting with zero additional forward or backward passes, while also removing the hand-tuning of $s$ that the closest prior work leaves open.
Core Claim
The closest prior work is Boosting Adversarial Training with Hypersphere Embedding. It normalizes penultimate features and classifier weights, adds an angular margin $m$ under a fixed scale $s$, and reports that training time with fast single-step training is "nearly identical" to the baseline. It never examines catastrophic overfitting, gradient alignment, or curvature. It also states that $s$ and $m$ need careful manual tuning. Improving Adversarial Robustness with Hypersphere Embedding and Angular-based Regularizations adds angular compactness and class-separation penalties. It evaluates them only under expensive multi-step training and adds overhead. Neither paper treats the sphere as a handle on training stability, and neither adapts $s$.
Our claim has two parts. The first is diagnostic. In unconstrained single-step training, growth of the penultimate feature norm is one route by which the head sharpens and local linearity breaks. Normalization therefore changes when and whether catastrophic overfitting happens, but we do not expect it to prevent it on its own. The reason is that the normalization's own Jacobian scales as $1/\|f(x)\|$, which can amplify input gradients for low-norm features. The backbone can also still build jagged boundaries in angle. The second part is constructive. With features on the sphere the logits are $z_c = s\cos\theta_c$, where $\theta_c$ is the angle between the normalized feature and class weight $c$. The curvature of cross-entropy with respect to the cosines is exactly $s^2$ times the covariance of the softmax output. Lowering $s$ therefore directly flattens the head's contribution to input-space curvature, and with it the head's part of the mixed input–parameter Hessian. A Training-Dynamics View of Catastrophic Overfitting identifies the explosion of that Hessian as the trigger for catastrophic overfitting.
The free signal works as follows. In the single-step method with random initialization from Fast is better than free, the loop first evaluates the loss and input gradient at a random start $x+\eta$. It then takes a sign step of size $\alpha$ and evaluates the loss at the resulting point $x' = x+\eta+\alpha\,\mathrm{sign}(\nabla_x \mathcal{L}(x+\eta))$ for the training update. From these two existing passes we form the per-batch linearity ratio
$$\rho = \frac{\mathcal{L}(x') - \mathcal{L}(x+\eta)}{\alpha\,\|\nabla_x \mathcal{L}(x+\eta)\|_1},$$
which compares the loss increase actually achieved with the increase predicted by a first-order expansion. When the loss is locally linear, $\rho \approx 1$. Understanding Catastrophic Overfitting in Single-step Adversarial Training describes the onset of catastrophic overfitting as the attack overshooting a sharply curved boundary. That predicts $\rho$ falling well below 1. The controller updates $s \leftarrow \mathrm{clip}(s\,\exp(\kappa(\bar\rho - \rho^*)))$, where $\bar\rho$ is a running average, $\rho^*$ a target ratio, and $\kappa$ a gain. It costs one scalar division per batch.
The sphere is what makes this principled and not merely a heuristic. In an unconstrained network the model can undo any temperature change by growing feature norms. On the sphere, $s$ is the only source of head sharpness, and the bounded logits keep $\rho$ comparable across training.
Research Program
The first stage is measurement. On CIFAR-10 with a ResNet-18 at $\epsilon = 8/255$ and $16/255$, we would run single-step training with random initialization in three forms: an unconstrained network, Pang et al.'s hypersphere embedding at several fixed $s$, and the adaptive-$s$ variant. At every epoch we log the penultimate feature norms, $\rho$, and the input-gradient norm. As an offline diagnostic on held-out batches, we also log three quantities that are expensive and never used in training: the gradient alignment of Understanding and Improving Fast Adversarial Training, the mixed-Hessian spectral norm estimated by power iteration, and PGD accuracy. The diagnostic questions are as follows. Does feature-norm growth precede the collapse in alignment in unconstrained runs? Does $\rho$ fall before PGD accuracy collapses, and by how many iterations? Does normalization delay, prevent, or merely relocate the collapse?
The second stage tests the controller as a training method across $\epsilon \in \{8, 12, 16\}/255$, on CIFAR-10, CIFAR-100, SVHN, and Tiny-ImageNet. It also tests a ViT-Small on CIFAR-10, the setting where Yeom & Lee report that plain single-step training collapses. We would test sensitivity to $\rho^*$ and $\kappa$, and compare it with the sensitivity of a fixed $s$, because the motivation is to replace a delicate hyperparameter with a robust one. If budget allows, a final check runs on ImageNet at $\epsilon = 4/255$ against the single-step baseline of Wong et al.
Baselines
The naive reference point is plain single-step training with random initialization, cyclic learning rate, and mixed precision, from Fast is better than free. That paper reports 45% PGD accuracy on CIFAR-10 in 6 minutes at $8/255$. Understanding and Improving Fast Adversarial Training reports that the same method falls to about 0% at $16/255$. Beating it means avoiding collapse at large $\epsilon$ at essentially the same wall-clock cost.
The second reference is the same method with Pang et al.'s hypersphere embedding at fixed $s$ and $m$, tuned on a grid. Its runtime is reported as nearly identical to the baseline. Pang et al. report robustness numbers for this method only for multi-step TRADES (62.02% PGD-20 on CIFAR-10); there is no reported result on Lacuna for hypersphere embedding under single-step training at large $\epsilon$. This baseline is the ablation that isolates our key ingredient. If a well-tuned fixed $s$ matches the controller, then the adaptivity adds nothing.
The strongest existing competitor is the mixed-Hessian consistency regularizer of A Training-Dynamics View of Catastrophic Overfitting. It reports 52.46% PGD-10 on CIFAR-10, above multi-step PGD-10 training at 51.79%, at nearly 3x the speed of PGD-10, and 32.59% on ViT-Small. It is the strongest competitor because it targets the same curvature mechanism with one extra forward pass. A second strong competitor is ELLE-A from Efficient local linearity regularization to overcome catastrophic overfitting, which reports 44.32% AutoAttack accuracy on CIFAR-10. We would run it from the LIONS-EPFL/ELLE code.
We would also include three cheaper heuristics. The first is the adaptive checkpoint search of Kim et al., reporting 33.9% PGD-50 in 3.9 hours against 11.1 hours for PGD-7. The second is the FastAdvW warmup of Towards Understanding Fast Adversarial Training, reporting 53.09% in 40 minutes. The third is GradAlign, as the expensive reference at $16/255$; the file gives no reported number for it at that radius.
All methods would use the same architectures, epoch counts, and data augmentation, on the same GPU type. Each gets an equal hyperparameter-search budget, measured in full training runs, for its own trade-off weights or $s$. Results are reported over five seeds. Evaluation would use the ELLE repository's PGD-50-10 and AutoAttack scripts throughout. The indicators from pralab/IndicatorsOfAttackFailure would rule out gradient masking, which matters here because rescaling logits can fool weak attacks. Beating the strongest competitor means AutoAttack accuracy within one point of Yeom & Lee's regularizer, or above it, at lower wall-clock time and memory, with no catastrophic-overfitting seeds at $16/255$.
Why not just…?
Why not just use Pang et al.'s hypersphere embedding with a well-chosen fixed scale? This is the most serious threat, and it may be enough. If some fixed $s$ prevents collapse at every $\epsilon$ and dataset, the controller is unnecessary. The diagnostic stage alone would then answer the open question of whether normalization stops catastrophic overfitting. The fixed-$s$ grid tests this directly. We expect the safe $s$ to shift with $\epsilon$ and architecture, which would justify adapting it. If it does not shift, we would report that and drop the controller.
Why not just apply the same free ratio to temperature in an ordinary unconstrained network? That is cheaper still. However, an unconstrained network can grow its feature norms to undo any temperature reduction, so we predict the control loop will drift or fail to hold $\rho$. We would run exactly this ablation. If it works as well as the hypersphere version, the sphere is not essential, and the note's claim narrows to "free linearity feedback."
Why not just use the mixed-Hessian regularizer or ELLE, which already work? They do work. Their cost is an extra forward pass in the first case, and three extra forward passes plus memory in the case of ELLE. The comparison is about matching their robustness at strictly single-pass cost. If the controller loses more than a point of AutoAttack accuracy to them, the honest conclusion is that the free signal is a useful early-warning detector but not a replacement.
Why not just stop early, or warm up with multi-step training? Early stopping on PGD validation requires periodic multi-step evaluation, and warmup schedules are brittle. FastAdvW is strong at $8/255$, but the file has no result for it at $16/255$. We include both and report results across $\epsilon$.
Proposed Experiments and Decisive Checks
Three outcomes would settle the idea. First, $\rho$ must drop before PGD accuracy collapses in unconstrained and fixed-$s$ runs, with a lead of at least several hundred iterations. If it does not lead, the signal cannot be used for control. Second, at $\epsilon = 16/255$ on CIFAR-10 and $12/255$ on SVHN, the adaptive-$s$ method must avoid collapse on all five seeds. The fixed-$s$ grid's best setting must fail on some seeds or need retuning per $\epsilon$. Third, on CIFAR-10 at $8/255$, AutoAttack accuracy must come within one point of the mixed-Hessian regularizer at no more than the wall-clock time and peak memory of plain single-step training. The ViT-Small result is compared against the 32.59% reported by Yeom & Lee.
The offline mixed-Hessian measurements test the proposed mechanism. If robustness improves but the mixed-Hessian norm does not fall, the curvature explanation is wrong, and we would say so.
References
- Boosting Adversarial Training with Hypersphere Embedding
- Improving Adversarial Robustness with Hypersphere Embedding and Angular-based Regularizations
- A Training-Dynamics View of Catastrophic Overfitting: Understanding and Prevention
- Fast is better than free: Revisiting adversarial training
- Understanding Catastrophic Overfitting in Single-step Adversarial Training
- Understanding and Improving Fast Adversarial Training
- Efficient local linearity regularization to overcome catastrophic overfitting
- Towards Understanding Fast Adversarial Training