Enhancing Robustness in Incremental Learning with Adversarial Training

Seungju Cho, Hongsin Lee, Changick KimAAAI 2025 · OpenReview

Abstract

Adversarial training is one of the most effective approaches against adversarial attacks. However, adversarial training has primarily been studied in scenarios where data for all classes is provided, with limited research conducted in the context of incremental learning where knowledge is introduced sequentially. In this study, we investigate Adversarially Robust Class Incremental Learning (ARCIL), which deals with adversarial robustness in incremental learning. We first explore a series of baselines that integrate incremental learning with existing adversarial training methods, finding that they lead to conflicts between acquiring new knowledge and retaining past knowledge. Furthermore, we discover that training new knowledge causes the disappearance of a key characteristic in robust models: a flat loss landscape in input space. To address such issues, we propose a novel and robust baseline for ARCIL, named FLatness preserving Adversarial Incremental learning for Robustness (FLAIR). Experimental results demonstrate that FLAIR significantly outperforms other baselines. To the best of our knowledge, we are the first to comprehensively investigate the baselines, challenges, and solutions for ARCIL, which we believe represents a significant advance toward achieving real-world robustness.

Introduction

Deep Neural Networks (DNNs) are notoriously fragile; small, intentional perturbations to an input, known as adversarial attacks, can easily fool them. While Adversarial Training (AT) is the gold-standard defense, it typically assumes access to all training data simultaneously. In real-world scenarios, data often arrives sequentially—a setting known as Class Incremental Learning (CIL). Combining AT with CIL introduces a critical challenge: "catastrophic forgetting" of both class labels and adversarial robustness.

The authors identify three primary hurdles in Adversarially Robust Class Incremental Learning (ARCIL):

  1. Objective Conflict: Learning new classes often contradicts the goal of maintaining robustness for old ones.
  2. Flatness Forgetting: Robust models typically exhibit a "flat" loss landscape. Incremental learning tends to destroy this flatness, making the model more vulnerable.
  3. Data Scarcity: AT requires significant data, but incremental learning limits access to past samples.

The Challenge of Flatness Forgetting

A key insight of this work is the quantification of "Flatness Forgetting." As a model learns new tasks, its loss landscape for previously learned tasks becomes significantly "sharper." This loss of flatness directly correlates with a drop in adversarial robustness.

Comparison of loss landscapes: R-iCaRL (bottom-left) shows a jagged, sharp landscape after learning new tasks, while FLAIR (others) maintains a smooth, flat basin.

The FLAIR Framework

To address these challenges, the authors propose FLAIR (FLatness-preserving Adversarial Incremental learning for Robustness), which consists of three core components:

  • Adversarial Distillation with Separated Logits (ADSL): Instead of a standard softmax over all classes, FLAIR uses binary cross-entropy for new classes and a separate distillation loss for old classes. This prevents the "tug-of-war" where the model suppresses old class logits while learning new features.
  • Flatness Preserving Distillation (FPD): To maintain a flat loss landscape without the high cost of Hessian calculations, FLAIR distills the difference between the model's output on clean versus adversarial images from the previous task to the current one. This implicitly preserves curvature information.
  • Advanced Data Augmentation: Integrating techniques like RandAugment helps "fill the gaps" in the training distribution, which is particularly beneficial in data-scarce incremental settings.

Experimental Results

FLAIR was evaluated on benchmarks including Split CIFAR-10/100, SVHN, and TinyImageNet. It consistently outperformed all existing baselines in both rehearsal and non-rehearsal settings.

1st task PGD accuracy over sequential tasks. FLAIR (red/blue) maintains significantly higher robustness for the first task as more tasks are added compared to baselines.

On S-CIFAR100 (without a memory buffer), FLAIR achieved a PGD accuracy of 11.02%, compared to 4.82% for a naive PGD-AT baseline. Furthermore, FLAIR showed significantly better Robust Backward Transfer (R-BWT), indicating it retains nearly four times more robustness than standard methods on datasets like S-SVHN.

Conclusion

FLAIR represents a significant step toward achieving real-world robustness in dynamic learning environments. By preserving the flatness of the loss landscape and minimizing logit interference, it effectively mitigates the forgetting of adversarial robustness. Future work may explore computational optimizations for data augmentation and robustness against a wider variety of adaptive attacks.

Summary updated

Abstract

Adversarial training is one of the most effective approaches against adversarial attacks. However, adversarial training has primarily been studied in scenarios where data for all classes is provided, with limited research conducted in the context of incremental learning where knowledge is introduced sequentially. In this study, we investigate Adversarially Robust Class Incremental Learning (ARCIL), which deals with adversarial robustness in incremental learning. We first explore a series of baselines that integrate incremental learning with existing adversarial training methods, finding that they lead to conflicts between acquiring new knowledge and retaining past knowledge. Furthermore, we discover that training new knowledge causes the disappearance of a key characteristic in robust models: a flat loss landscape in input space. To address such issues, we propose a novel and robust baseline for ARCIL, named FLatness preserving Adversarial Incremental learning for Robustness (FLAIR). Experimental results demonstrate that FLAIR significantly outperforms other baselines. To the best of our knowledge, we are the first to comprehensively investigate the baselines, challenges, and solutions for ARCIL, which we believe represents a significant advance toward achieving real-world robustness.
BibTeX
@inproceedings{cho2025enhancing,
  author = {Seungju Cho and Hongsin Lee and Changick Kim},
  title = {{Enhancing Robustness in Incremental Learning with Adversarial Training}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=KXVA8QmuWN},
}
Figure