Adversarial training teaches a classifier to resist small, worst-case input changes. It does this by repeatedly finding those changes during training, and that search makes training five to ten times more expensive. It also lowers accuracy on ordinary, unaltered images. One well-supported explanation for the accuracy loss is that the model learns to stay confident far beyond the perturbation radius it was trained for. This "excessive margin" blunts the fine-grained boundaries that clean images need. The best fix so far, Helper-based Adversarial Training (Reducing Excessive Margin to Achieve a Better Accuracy vs. Robustness Trade-off), places extra "helper" points at twice the adversarial step and labels them with a separately trained non-robust teacher. That adds a full teacher training run on top of the already slow multi-step attack.
We propose to treat those helper points as synthetic outliers instead of teacher-labeled examples. The training rule becomes: "just outside the radius you must defend, stop being confident." The helper is built from the single gradient step that cheap one-step adversarial training already computes, and it is trained toward a uniform prediction. No teacher is needed and no extra attack iterations are run. The claim is that this one ingredient does two jobs at near-single-step cost. It caps the excessive margin that hurts clean accuracy. It also penalizes the confident, highly curved behavior inside the perturbation neighborhood that precedes catastrophic overfitting, the sudden collapse of robustness to zero that plagues one-step training. Because outlier-style targets flatten the model's outputs and can fake robustness by hiding gradients, the whole evaluation is built around defense-aware attacks.
Core Claim
Let $x$ be a training image with label $y$, and let $\delta$ be a one-step adversarial perturbation of size $\epsilon$ computed with a random start. Let $f_\theta$ be the classifier's logits, $u$ the uniform distribution over classes, and $t>1$ a helper scale. We train with
$$ \mathcal{L}(\theta) = \mathrm{CE}\big(f_\theta(x+\delta), y\big) + \beta\, \mathrm{KL}\big(u \,\|\, \mathrm{softmax}(f_\theta(x+t\delta))\big), $$
where $\beta$ weights the anchor term. The helper $x+t\delta$ reuses the direction of $\delta$, so building it costs nothing extra; the anchor term adds one forward and backward pass. The proposal predicts three things. First, at matched wall-clock time, this objective reaches a better clean-versus-AutoAttack trade-off than one-step methods without anchors. Second, it comes close to teacher-labeled HAT. Third, it does not suffer catastrophic overfitting at $\epsilon=16/255$.
The new element is how the helper is labeled and what that buys. HAT anchors the margin with a teacher's label and still runs multi-step PGD, so it keeps both bottlenecks. Our version removes the teacher and the inner loop.
A second, optional element borrows a technique from Geometrically Constrained Outlier Synthesis. That method synthesizes hard negatives in feature space inside a "conformal shell," the band between the 95th and 99th percentiles of a calibration nonconformity score. Instead of a fixed $t$, we pick $t$ per sample from a small grid so that the helper's penultimate feature lands in that shell. This requires only forward passes. The ablation of this choice asks whether calibrated distance beats a fixed $2\times$ step.
Positioning Against Prior Work
Three papers supply the motivation. HAT showed that adversarial training produces margins far exceeding $\epsilon$ and that anchoring a point at $2r$ recovers clean accuracy. We keep its geometry but replace its teacher label with an outlier label.
TART decomposes perturbations into on-manifold and off-manifold parts using an autoencoder, and sets $\epsilon=0$ when a perturbation is mostly off-manifold. It gains clean accuracy at 1.24x the cost of standard adversarial training, but it needs precomputed tangent matrices.
Which Models have Perceptually-Aligned Gradients? argues the opposite emphasis: robustness off the manifold is necessary, and only excessive robustness erodes clean accuracy. This apparent contradiction matters to us. TART discards off-manifold pressure, while Srinivas et al. say it is required. Our anchor does neither. It keeps full adversarial pressure inside the $\epsilon$-ball and only forbids confidence beyond it. That is precisely the "Bayes-aligned, not excessive" regime Srinivas et al. describe. Whether this resolves the tension is a measurable question, which we test with their diagnostic.
Two papers supply the efficiency side. RAT fixes catastrophic overfitting with operator splitting, but it does not address clean accuracy and is sensitive to its regularization weight. Robustness Tokens avoids retraining by learning prefix tokens on a frozen backbone, but its security depends on the tokens staying secret. We do not compete in the frozen-backbone regime.
Two papers constrain the design. Out-of-Distribution Detection with Adversarial Outlier Exposure shows that outlier training can preserve clean accuracy (94.4% on CIFAR-10). However, it never tests classification robustness against defense-aware attacks, so it gives no evidence that outlier targets yield real robustness. RobustBench reports that standardized AutoAttack exposed overestimates exceeding 10% in published defenses. That is why we treat flattened outputs as a gradient-masking risk to be ruled out, not as a result.
Research Program
We will implement the objective inside the official HAT codebase (imrahulr/hat). Doing so means PGD-AT, TRADES, MART, HAT and our method share data pipelines, schedules, and the eval-aa.py AutoAttack script. One-step baselines will be added using the same random-start FGSM routine.
The analysis has three parts.
The first part is margin measurement. Following HAT, we measure each test point's distance to the decision boundary along the attack direction. We then check whether the anchor brings the margin distribution back toward $\epsilon$.
The second part tracks catastrophic overfitting during training. Every epoch we record PGD-50 robust accuracy and local loss curvature inside the $\epsilon$-ball. The test is whether the anchor suppresses the curvature spike that precedes collapse.
The third part is an off-manifold diagnostic. Using the perceptual-alignment measures of Srinivas et al., we ask whether anchored models sit in their Bayes-aligned regime, whereas PGD-AT and TRADES sit in the excessive one.
Baselines
The naive reference points are standard training and plain one-step adversarial training with random start. Standard training fixes the clean-accuracy ceiling, around 94% for ResNet-18 on CIFAR-10 per Visual Prompting for Adversarial Robustness. Plain one-step training fixes the cost floor and shows the catastrophic-overfitting failure we claim to prevent. Our method must beat it on AutoAttack accuracy at $\epsilon=16/255$ with essentially the same per-epoch cost.
The strongest robustness-and-accuracy competitor is HAT run as published, with multi-step PGD and a standard teacher. It is the most direct statement of the excessive-margin fix. Rade and Moosavi-Dezfooli report about 2.17 points of clean accuracy over TRADES on PreAct ResNet-18 CIFAR-10 at about 49% AutoAttack accuracy. With extra diffusion-generated data and wider networks, they report 91.30% clean and 62.50% robust. We will retrain HAT ourselves on PreAct ResNet-18 with identical augmentation and epochs. Its reported cost will include teacher training. We do not expect to beat its absolute numbers. Beating it means landing within one AutoAttack point and one clean point of it while using at most about a third of its total GPU-hours.
The strongest efficiency competitor is RAT. On CIFAR-10 at $\epsilon=16/255$, it reports 19.58% AutoAttack accuracy, compared with 15.36% for GradAlign-style Fast-GA and 16.31% for LLR. Its training takes 1.7 hours against 6.3 for PGD-AT. The file contains no RAT result at $\epsilon=8/255$, so we will produce one. We run RAT from the authors' description with its $\lambda$ tuned over the same number of trials as our $\beta$. Beating it means higher AutoAttack accuracy at $16/255$, higher clean accuracy at both radii, and no more than 1.2x its wall-clock time.
For the clean-accuracy angle we also include TART's reported WideResNet-32-10 numbers as a reference: 81.76% clean and 35.89% AutoAttack for standard adversarial training, versus 83.73% and 36.58% for TART. Where compute allows, we rerun our method at that architecture for a like-for-like comparison.
The ablation that isolates the key ingredient is a one-step HAT. It uses the same random-start FGSM and the same $t\delta$ helper, but labels the helper with HAT's teacher instead of the uniform target. Comparing this against our method shows whether the outlier label matters or only the helper location does. A second ablation swaps the gradient-direction helper for Gaussian feature-space outliers in the style of VOS. This tests whether the helper must lie along the attack direction. A third compares a fixed $t=2$ against the conformal-shell $t$.
Every method gets the same hyperparameter-search budget, the same seeds (three per configuration), and the same evaluation code. Results are plotted against measured GPU-hours, not epochs.
Why not just…?
Why not just run HAT with a one-step attack? This is the most serious threat. If one-step HAT matches our method, the outlier label contributes nothing beyond dropping a teacher run, which costs roughly one standard training run. The ablation above settles it directly. We will report the outcome either way. If the teacher label is better, the honest conclusion is a teacher-for-compute trade, and the proposal narrows to the catastrophic-overfitting claim. That claim still distinguishes the two variants: one-step HAT's teacher labels are confident, so they need not suppress curvature.
Why not just use label smoothing or TRADES on the adversarial point? Smoothing the target at $x+\delta$ softens confidence inside the region we want defended, which should cost robustness. Our anchor softens only beyond it. TRADES additionally keeps the multi-step inner loop, and HAT shows it still produces excessive margin. We include label smoothing at $x+\delta$, with matched $\beta$, as an additional baseline. If it matches us, the geometric placement is unnecessary.
Why not just replace the inner attack with feature-space outliers, as in GCOS? This is the obvious combination of these directions, and we expect it to fail. Feature-space negatives constrain penultimate-layer geometry but say nothing about how an $\ell_\infty$-bounded change in pixels moves that feature. A model trained this way has no reason to resist input attacks, and any apparent resistance under weak attacks is likely masked gradients. We run a GCOS-only variant, with no input-space adversarial step, under full AutoAttack to document this, rather than building the proposal on it.
Why not just use RAT? RAT already stops catastrophic overfitting cheaply. However, its reported results concern robustness, not the clean-accuracy loss, and nothing in its objective targets excessive margin. The head-to-head above, which reports clean accuracy at both radii, decides whether the anchor adds value on top of stability. We also test RAT combined with the anchor.
Proposed Experiments
The main grid trains PreAct ResNet-18 on CIFAR-10 and CIFAR-100 at $\ell_\infty$ radii of $8/255$ and $16/255$, with a WideResNet confirmation on CIFAR-10. TinyImageNet-200 tests scale; HAT used it, but no number for it is reported on Lacuna. Metrics are clean accuracy, AutoAttack robust accuracy, total GPU-hours including any teacher, and the epoch at which robustness collapses, if it does.
The first decisive check is the Pareto plot of clean accuracy against AutoAttack accuracy at fixed GPU-hours. The proposal succeeds if anchored one-step training lies above every one-step baseline and near PGD-HAT at a fraction of the cost.
The second decisive check guards against fake robustness. Uniform targets can flatten logits and starve gradient-based attacks, so we go beyond standard AutoAttack. We also run the black-box Square attack alone, attacks on logit-margin and KL-to-uniform losses, and a step-count sweep up to PGD-1000. Every run is checked with the attack-failure indicators in pralab/IndicatorsOfAttackFailure. A gap of more than about two points between white-box and black-box robust accuracy, or robust accuracy that keeps falling as steps increase, would mean the gains are masking. That outcome would falsify the claim.
The third check tests the mechanism. The margin distribution should move toward $\epsilon$, and the in-ball curvature trace should stay flat where plain one-step training spikes.
As a secondary question, we report energy-based out-of-distribution AUROC and FPR95 on the CIFAR-100 protocol used by Adversarial Outlier Exposure, against Textures, TinyImageNet and LSUN. That paper reports 93.7% AUROC and 24.7% FPR95, versus 84.4% and 42.4% for standard outlier exposure. This shows whether a margin anchor also gives open-set detection for free. The proposal does not depend on this result.
References
- Reducing Excessive Margin to Achieve a Better Accuracy vs. Robustness Trade-off
- Geometrically Constrained Outlier Synthesis
- TART: Boosting Clean Accuracy Through Tangent Direction Guided Adversarial Training
- Which Models have Perceptually-Aligned Gradients? An Explanation via Off-Manifold Robustness
- Rethinking Fast Adversarial Training: A Splitting Technique to Overcome Catastrophic Overfitting
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Out-of-Distribution Detection with Adversarial Outlier Exposure
- RobustBench: a standardized adversarial robustness benchmark
- Visual Prompting for Adversarial Robustness