Free Noise Detection from the Fast Adversarial Training Step

Adversarial training makes a classifier resistant to small, worst-case input perturbations by training it on perturbed inputs. Its cheapest form, single-step "fast" adversarial training, computes one gradient-sign perturbation per example per epoch. That step already measures something about every training example: how much its loss rises, and whether its prediction flips, when it is pushed one step toward the decision boundary. We propose to treat this byproduct as a label-noise detector at no added cost. Logged over the first epochs alongside the ordinary training loss, it should separate mislabeled examples from examples that are correctly labeled but hard. Removing the flagged examples should let fast adversarial training reach on noisy data the robustness that current noise-aware methods reach only with expensive multi-step inner loops.

The opportunity exists because the two literatures each hold half of the answer. The noisy-label literature uses the "small-loss trick": networks fit clean examples before they memorize wrong labels, so early high-loss examples are probably mislabeled. That trick fails on hard clean examples and on human annotation noise. The adversarial-training literature has shown that a sample's distance to the decision boundary detects noise better than loss does. It has also shown that the most expensive part of adversarial training is the per-example attack, and that attack already probes this distance. No one has connected the two.

Core Claim

Understanding the Interaction of Adversarial Training with Noisy Labels introduced the "geometry value", the number of multi-step attack iterations needed to misclassify a sample. They showed it distinguishes wrong labels from correct ones, and rare examples from typical ones, better than cross-entropy loss, especially under pair-flipping noise. Computing it requires running the multi-step attack (projected gradient descent, PGD) to completion on every sample, which adds to the 10–100× overhead that fast methods were built to remove.

Our claim is that a two-number summary from the single step that fast adversarial training already takes recovers most of that detection power. The first number is the loss at the random starting point. The second is the loss gain from the gradient-sign step. The reason is mechanical.

For a mislabeled example on which the model predicts the true class, the loss on the wrong label is already high and nearly saturated. The single step pushes further away from a class the model does not predict, so the loss gain is small and the prediction rarely changes.

A hard but clean example behaves differently. It sits near the boundary with a moderate loss, so one step produces a large loss gain and often a flip.

Loss alone therefore confuses these two groups, while the joint signal separates them.

A secondary hypothesis connects this to the instability of fast adversarial training. Eliminating Catastrophic Overfitting Via Abnormal Adversarial Examples Regularization tracks "abnormal" examples, on which the single step lowers rather than raises the loss, and penalizes them to prevent catastrophic overfitting. Catastrophic overfitting is the sudden collapse of robustness to multi-step attacks. Mislabeled examples are natural candidates for producing abnormal or near-zero gains, so label noise may both trigger that collapse and contaminate the regularizer meant to stop it. Filtering noise first may stabilize fast training as a side effect.

Research Program

The method has three stages, all inside a single run of standard random-start fast adversarial training as in Fast is better than free.

Logging. During the first few epochs we record, for every example, three quantities that cost only bookkeeping because they come from the forward and backward passes already being computed. These are the loss at the random start, the loss after the gradient-sign step, and whether the predicted class flipped. The default window is the first three epochs, matching the early separation reported by Efficient Adversarial Training With Data Pruning and the two-epoch warm-up in Noisy Label Classification Using Label Noise Selection with Test-Time Augmentation Cross-Entropy and NoiseMix Learning.

Partitioning. We average these quantities over the window. We then fit a two-component Gaussian mixture in the two-dimensional space of start loss and step gain, in the spirit of the loss-based mixtures used by DivideMix but with the added geometric axis.

Training. Examples assigned to the high-loss, low-gain component are removed from the parameter update for the rest of training. A variant relabels them with the model's confident prediction instead. Everything else continues under a catastrophic-overfitting guard. The guard is the abnormal-example penalty from the paper above or gradient alignment from Understanding and Improving Fast Adversarial Training.

The analysis component asks three questions. First, how well does the single-step signal track the PGD geometry value? We measure this as rank correlation on a held-out subset where we pay for full PGD. Second, how does detection quality vary with noise type: symmetric, pair-flip, and human? Third, are abnormal adversarial examples disproportionately mislabeled?

Baselines

The naive reference is random-start fast adversarial training run directly on the noisy labels with the same catastrophic-overfitting guard. This measures what noise costs fast training when nothing is done about it. Beating it is necessary but not sufficient.

The closest efficiency competitor is the loss-based pruning of Efficient Adversarial Training With Data Pruning, which keeps low-loss examples and discards high-loss outliers at epoch 3. On noisy data it may already discard many wrong labels. Its reported results are on clean data: a 3× speedup using 20% of CIFAR-10, a 7–8% robust-accuracy drop when 40% is removed, and a 0.06% robustness gain on MNIST after pruning 60%. We will run it with the same epoch, the same fraction removed as our partition removes, and the same guard. It is beaten only if our method matches its noisy-sample recall while losing fewer clean hard examples, and converts that into higher robust accuracy on the minority and ambiguous classes.

The decoupled two-stage pipeline trains a standard (non-adversarial) model for a few epochs, filters by its loss, and then runs fast adversarial training on the cleaned set. It gets the same compute budget as our method, counting its warm-up epochs.

The strongest robustness reference is NRAT from Nrat: towards adversarial training with inherent label noise. It replaces cross-entropy with noise-robust active-passive losses inside multi-step TRADES. On CIFAR-10 with 40% symmetric noise it ends training at about 47% robust accuracy, while TRADES collapses from about 55% to about 26%. We will also port its loss into single-step training, so that loss design and filtering are compared at equal cost.

Omnipotent Adversarial Training is a high-cost reference. It uses a self-supervised "oracle" model to clean labels and reports 80.72% clean and 42.84% robust accuracy on imbalanced CIFAR with 60% noise, at roughly 40% more time than PGD adversarial training.

We report wall-clock time for every method on identical hardware. All hyperparameters get the same tuning budget on a clean validation split.

The ablations isolate the key ingredient by comparing three detectors: loss only, step-gain only, and the joint signal. We also include Zhu et al.'s full geometry value as a detection upper bound. It is used only for analysis, not as a method claim, because its cost is what we are trying to avoid.

On CIFAR-10N we found no reported result on Lacuna for adversarial training of any kind, so those comparisons will be established here. The only CIFAR-10N anchor in the file is standard cross-entropy reaching about 77% clean accuracy on the "Worst" labels versus about 93% on clean labels, from Learning with Noisy Labels Revisited.

Why not just…?

Why not just filter with a cheap standard model, then adversarially train the cleaned set? This is the most serious threat, because it is simple and nearly as cheap.

The evidence against it is indirect. Zhu et al. report that under standard training the gap between correct and incorrect training accuracy closes quickly. Under adversarial training the gap persists, and their geometry metric beats loss under pair-flipping noise. Human noise is memorized faster than synthetic noise because it sits on ambiguous images, so a standard model's early loss is a weaker signal precisely where it matters.

These are not head-to-head results. If the two-stage pipeline matches our detection quality on CIFAR-10N and Clothing1M at equal compute, the simple pipeline wins and we will say so.

Why not just use a noise-robust loss inside fast adversarial training, with no filtering? NRAT shows such losses prevent the collapse of both clean and robust accuracy that noisy labels cause late in training, but only within multi-step TRADES. Whether they survive single-step training, where catastrophic overfitting is a separate failure mode, is untested. The ported-loss baseline settles this. If it equals filtering, the contribution narrows to the analysis of noise and catastrophic overfitting.

Why not just apply existing loss-based pruning, which already discards high-loss points? Kaufmann et al. never checked whether the outliers they pruned were mislabeled or merely hard. On noisy data their rule is a loss-only detector with a fixed pruning fraction, which is exactly our loss-only ablation. The decisive measurements are recall of clean hard examples and per-class robust accuracy on minority classes, where loss-only selection should be visibly worse if our mechanism is right.

Proposed Experiments

Datasets and models. We use CIFAR-10 and CIFAR-100 with symmetric and pair-flip noise at 20–60%. We use CIFAR-10N (Aggregate, Random, and Worst) and CIFAR-100N, where the true noise mask is known. For scale we use Clothing1M, with Tiny-ImageNet as a check on large-perturbation stability. The backbones are ResNet-18 and WideResNet-28-10. Perturbation budgets are 8/255 and 16/255.

Metrics. Detection quality is measured as area under the ROC curve (AUROC) against the true noise mask, together with recall on clean examples in the top loss decile. Model quality is measured by clean accuracy, PGD-50 accuracy, and AutoAttack accuracy. Cost is measured as wall-clock time. Stability is measured by the rate of catastrophic overfitting across five seeds.

Decisive checks. Three outcomes would decide the proposal.

The first is detection. The joint single-step signal should reach AUROC significantly above loss-only detection on pair-flip noise and on CIFAR-10N Worst. It should also correlate strongly with the PGD geometry value, at essentially zero added time. If the joint signal is no better than loss, the core claim fails.

The second is robustness at low cost. On CIFAR-10 with 40% symmetric noise, the filtered fast training should approach NRAT's roughly 47% final robust accuracy at a fraction of its time, and should beat the two-stage pipeline and loss-based pruning on CIFAR-10N.

The third is the stability hypothesis. Abnormal adversarial examples should be enriched for mislabeled samples, and removing flagged samples should reduce catastrophic overfitting at 16/255 compared with the guarded naive baseline.