Investigating Catastrophic Overfitting in Fast Adversarial Training: A Self-fitting Perspective
Lists
Only you see this
Abstract
Although fast adversarial training provides an efficient approach for building robust networks, it may suffer from a serious problem known as catastrophic overfitting (CO), where multi-step robust accuracy suddenly collapses to zero. In this paper, we for the first time decouple single-step adversarial examples into data-information and self-information, which reveals an interesting phenomenon called "self-fitting". Self-fitting, i.e., the network learns the self-information embedded in single-step perturbations, naturally leads to the occurrence of CO. When self-fitting occurs, the network experiences an obvious "channel differentiation" phenomenon that some convolution channels accounting for recognizing self-information become dominant, while others for data-information are suppressed. In this way, the network can only recognize images with sufficient self-information and loses generalization ability to other types of data. Based on self-fitting, we provide new insights into the existing methods to mitigate CO and extend CO to multi-step adversarial training. Our findings reveal a self-learning mechanism in adversarial training and open up new perspectives for suppressing different kinds of information to mitigate CO.
BibTeX
@inproceedings{he2023investigating,
author = {Zhengbao He and Tao Li and Sizhe Chen and Xiaolin Huang},
title = {{Investigating Catastrophic Overfitting in Fast Adversarial Training: A Self-fitting Perspective}},
booktitle = {Workshops @ CVPR},
year = {2023},
url = {https://openreview.net/forum?id=GrsNyVZbreZ},
}Loading related content...