Exploring The Forgetting in Adversarial Training: A Novel Method for Enhancing Robustness

Xianglu Wang, Hu Ding

TL;DR

This paper reveals that Adversarial Training (AT) suffers from "catastrophic forgetting," a problem usually associated with Continual Learning, where a model forgets how to defend against earlier adversarial examples as it encounters new ones. To fix this, the authors introduce Adaptive Multi-teacher Self-distillation (AMS), a method that uses previous versions of the model as "teachers" to preserve old knowledge, significantly boosting robustness and reducing the common issue of robust overfitting.

What Problem Does This Paper Address?

Deep Neural Networks are notoriously fragile when faced with adversarial examples—input data with tiny, intentional perturbations that cause the model to fail. While Adversarial Training (AT) is the most effective defense, it faces two major hurdles:

  1. The Robustness Gap: There is still a significant performance gap between "clean" accuracy (on normal data) and "robust" accuracy (on attacked data).
  2. Robust Overfitting: Unlike standard training, where more training usually helps, AT models often reach a peak in robustness and then start performing worse as training continues.

The authors hypothesize that these issues stem from the "minimax" nature of AT. Because the adversarial examples are generated based on the current state of the model, the data distribution shifts every epoch. The authors argue that AT is essentially a Continual Learning (CL) problem where the model is constantly learning new "tasks" (new distributions of harder attacks) and, in the process, suffers from catastrophic forgetting of the defensive strategies it learned earlier.

Methods

The authors propose a framework called Adaptive Multi-teacher Self-distillation (AMS). The core idea is to treat previous versions of the model as a committee of teachers that guide the current model.

  • Stage-based Training: Training is divided into stages (e.g., every 20 epochs). At the end of each stage, the model's parameters are saved to a memory buffer.
  • Multi-teacher Distillation: During the current training stage, the model tries to match its output probabilities (logits) with those of all previous "teacher" models stored in the buffer. This encourages the model to retain the defensive knowledge it acquired in earlier stages.
  • Reweighting-based Loss Correction (RLC): This is the "Adaptive" part. Since adversarial training is hard, early-stage teachers are often wrong. Blindly following a "bad" teacher would hurt performance. AMS calculates a weight for each teacher based on its confidence in the true label of a sample. If a teacher misclassifies a sample, its influence on the loss function is minimized.

Datasets

The researchers evaluated their method on four standard computer vision benchmarks:

  • CIFAR-10 & CIFAR-100: Small-scale image classification.
  • SVHN (Street View House Numbers): Digit recognition in the wild.
  • TinyImageNet: A subset of ImageNet with 200 classes at 64x64 resolution.

They tested these across different perturbation models ($\ell_\infty$ and $\ell_2$ norms) and various architectures, including PreActResNet-18, WideResNet-34-10, Vision Transformers (ViT), and VGG16.

Key Insights and Ideas

  • Empirical Proof of Forgetting: The authors conducted a clever experiment: they collected adversarial samples that the model could correctly classify at Stage 1. By Stage 5, the model's accuracy on those same fixed samples dropped significantly (e.g., from 100% to ~75% on TinyImageNet). This confirms that the model "forgets" how to handle simpler attacks as it chases harder ones.
Progression of classification probabilities across training stages demonstrating catastrophic forgetting.
  • AT as a "CL-style" Problem: By framing AT as a sequence of evolving tasks, the authors bridge two previously distinct fields. This allows them to borrow successful strategies from Continual Learning (like replay and distillation) to solve robustness problems.
  • The "Bad Teacher" Problem: In standard Knowledge Distillation, the teacher is usually a superior, pre-trained model. In self-distillation for AT, the teachers are just "younger" versions of the student. The RLC mechanism is crucial because it allows the student to "disobey" the teacher when the teacher is clearly confused.

Key Results

  • Improved Robustness: AMS consistently improved the "Auto Attack" (AA) accuracy—the gold standard for robustness—across all datasets. For example, on CIFAR-10, it boosted TRADES (a popular AT method) from 53.00% to 54.31%.
  • State-of-the-Art on TinyImageNet: On the challenging TinyImageNet dataset, their method achieved 31.52% robust accuracy under Auto Attack, a significant result for this benchmark.
  • Mitigating Overfitting: One of the most striking results was the reduction in robust overfitting. In standard TRADES, the gap between the "best" and "final" robust accuracy was 3.28%. With AMS, this gap shrank to just 0.58%, meaning the model remains stable even with extended training.
  • Plug-and-Play: AMS proved to be a "unified enhancer," improving various base algorithms like PGD-10, TRADES, and AWP.

Limitations and Future Work

  • Computational Overhead: Storing multiple teacher models and performing forward passes through them increases memory usage and training time. While the authors suggest an interval of $m=20$ epochs to balance this, it is still more expensive than vanilla AT.
  • Theoretical Depth: While the empirical evidence for forgetting is strong, the authors note that the theoretical connection between the "forgetting" phenomenon and "robust overfitting" deserves deeper exploration.
  • Scaling to ImageNet: The experiments stopped at TinyImageNet; testing the scalability of maintaining multiple teachers on full-scale ImageNet (1000 classes) remains a challenge for future research.

Summary updated

Abstract

In recent years, there has been an explosion of research into developing robust deep neural networks against adversarial examples. As one of the most successful methods, Adversarial Training (AT) has been widely studied before, but there is still a gap to achieve promising clean and robust accuracy for many practical tasks. In this paper, we consider the AT problem from a new perspective which connects it to catastrophic forgetting in continual learning (CL). Catastrophic forgetting is a phenomenon in which neural networks forget old knowledge upon learning a new task. Although AT and CL are two different problems, we show that they actually share several key properties in their training processes. Specifically, we conduct an empirical study and find that this forgetting phenomenon indeed occurs in adversarial robust training across multiple datasets (SVHN, CIFAR-10, CIFAR-100, and TinyImageNet) and perturbation models ($\ell_{\infty}$ and $\ell_{2}$). Based on this observation, we propose a novel method called Adaptive Multi-teachers Self-distillation (AMS), which leverages a carefully designed adaptive regularizer to mitigate the forgetting by aligning model outputs between new and old ``stages''. Moreover, our approach can be used as a unified method to enhance multiple different AT algorithms. Our experiments demonstrate that our method can significantly enhance robust accuracy and meanwhile preserve high clean accuracy, under several popular adversarial attacks (e.g., PGD, CW, and Auto Attacks). As another benefit of our method, we discover that it can largely alleviate the robust overfitting issue of AT in our experiments.
BibTeX
@inproceedings{wang2025exploring,
  author = {Xianglu Wang and Hu Ding},
  title = {{Exploring The Forgetting in Adversarial Training: A Novel Method for Enhancing Robustness}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=fjPOt8QlqQ},
}
Figure