Robust Overfitting in Adversarial Training

536 concepts276 papers1 proposal

In standard deep learning, a foundational piece of wisdom is that models benefit from training for as long as possible. Even when a network has enough capacity to perfectly memorize the training set, its test performance typically stabilizes or continues to improve—a phenomenon often described as benign overfitting.

However, when we train models to be resilient against malicious inputs via adversarial training (AT), this wisdom completely breaks down. As first systematically demonstrated by Leslie Rice et al. in Overfitting in adversarially robust deep learning (icml 2020), adversarially trained models experience a severe phenomenon called robust overfitting. Early in the training process, the model's ability to defend against unseen adversarial attacks reaches a peak. But as training continues and training loss drops toward zero, test-time robustness plummets drastically—sometimes by over 20%—even while clean test accuracy remains stable.

Distinguishing Robust from Catastrophic Overfitting

To understand why robust overfitting happens, it helps to isolate it from other training failures. In fast, single-step adversarial training methods (such as the Fast Gradient Sign Method), models often suffer from "catastrophic overfitting." As Maksym Andriushchenko et al. show in Understanding and Improving Fast Adversarial Training (neurips 2020), catastrophic overfitting is a sudden, single-epoch collapse in robustness caused by a breakdown in the model's local linearity.

Robust overfitting, by contrast, is a gradual, late-stage decay that occurs across multi-step adversarial training regimes. It is not an optimization failure where the model stops learning; rather, it is a generalization failure where the model becomes highly specialized to the specific training perturbations while losing its defensive capabilities against novel attacks.

Why It Happens: The Core Mechanisms

Why does extended training destroy adversarial generalization? The literature points to three interconnected mechanisms:

1. Adversarial Memorization and the "Lookup Table" Problem

In standard training, a model learns to map semantic features (like whiskers and ears) to labels (like "cat"). In adversarial training, the model must also learn to ignore adversarial noise. However, as training progresses, overparameterized networks find a shortcut. Instead of learning a smooth, robust decision boundary, the network begins to memorize the specific noise patterns of the training-time perturbations.

Binghui Li et al. argue in On the Clean Generalization and Robust Overfitting in Adversarial Training from Two Theoretical Views: Representation Complexity and Training Dynamics (icml 2025) that higher model capacity actually facilitates this failure mode. Instead of building true structural invariance, a high-capacity network effectively stores a "lookup table" of the training attacks. As Shruthi Gowda et al. observe in Conserve-Update-Revise to Cure Generalization and Robustness Trade-off in Adversarial Training (iclr 2024), this memorization creates highly complex, convoluted decision boundaries that fit training noise perfectly but fail on unseen test perturbations.

2. Implicit Label Noise

Another intuitive framing treats adversarial training as learning under noisy labels. When an adversary perturbs an image, it can distort the true semantic meaning of the image. However, the adversarial training loop forces the model to classify this distorted image under its original label.

In Double Descent in Adversarial Training: An Implicit Label Noise Perspective (arxiv 2021), Chengyu Dong et al. demonstrate that robust overfitting follows an epoch-wise double descent curve, which is a signature dynamic of training on datasets with high label noise. The model initially learns the clean, underlying signal, but late-stage training forces it to fit the "noisy" semantic mismatches introduced by the adversary.

3. Escalating Gradient Variance and Overconfidence

The optimization dynamics of the training process itself can trigger this collapse. Ali Dabouei reveals in Revisiting Outer Optimization in Adversarial Training (eccv 2022) that robust overfitting is strongly accelerated by a massive spike in gradient variance in the outer optimization loop, typically occurring right after learning rate decay. This noisy gradient signal pushes the weights into configurations that overfit the training attacks.

Simultaneously, the model becomes overly confident. Minxing Zhang et al. show in Generating Less Certain Adversarial Examples Improves Robust Generalization (Accepted by TMLR 2024) that models exhibiting high certainty on training adversarial examples are highly prone to robust overfitting. The network fits the training attacks with such high logit certainty that its decision boundary loses the flexibility required to handle novel, unseen perturbations.

Solutions and Mitigation Strategies

Understanding the root causes of robust overfitting has led to several practical strategies to mitigate it:

  • Early Stopping: The simplest and most effective baseline, as noted by Leslie Rice et al., is to halt training at the peak of validation robustness before the late-stage collapse begins.
  • Data Diversity and Generative Models: Since robust overfitting stems from memorizing a finite set of attack patterns, expanding data volume and diversity breaks the lookup table. Zekai Wang et al. show in Better Diffusion Models Further Improve Adversarial Training (icml 2023) that incorporating high-quality synthetic data from diffusion models significantly delays or prevents the onset of robust overfitting. Similarly, Lin Li et al. demonstrate in Data augmentation alone can improve adversarial training (iclr 2023) that carefully balancing the hardness and diversity of standard data augmentations can eliminate the robust generalization gap entirely.
  • Geometry-Aware Regularization: Instead of treating all training points equally, methods like the one proposed by Jingfeng Zhang et al. in Geometry-aware Instance-reweighted Adversarial Training (iclr 2021) reweight instances based on their proximity to the decision boundary, preventing the model from overfitting to the hardest outliers or training noise.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 322 papers
  2. Version 1 · 2026-10-04 · 315 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_robust_overfitting_e2f9a6d7,
  title = {robust overfitting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robust-overfitting/rdr_e2f9a6d7d0e542f69a777ff0167c85b5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 2, frozen text):

@misc{lacuna_robust_overfitting_e2f9a6d7_v2,
  title = {robust overfitting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robust-overfitting/rdr_e2f9a6d7d0e542f69a777ff0167c85b5/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake