Abstract
Recently, FGSM adversarial training is found to be able to train a robust model which is comparable to the one trained by PGD but an order of magnitude faster. However, there is a failure mode called catastrophic overfitting (CO) that the classifier loses its robustness suddenly during the training and hardly recovers by itself. In this paper, we find CO is not only limited to FGSM, but also happens in $\mbox{DF}^{\infty}$-1 adversarial training. Then, we analyze the geometric properties for both FGSM and $\mbox{DF}^{\infty}$-1 and find they have totally different decision boundaries after CO. For FGSM, a new decision boundary is generated along the direction of perturbation and makes the small perturbation more effective than the large one. While for $\mbox{DF}^{\infty}$-1, there is no new decision boundary generated along the direction of perturbation, instead the perturbation generated by $\mbox{DF}^{\infty}$-1 becomes smaller after CO and thus loses its effectiveness. We also experimentally analyze three hypotheses on potential factors causing CO. And then based on the empirical analysis, we modify the RS-FGSM by not projecting perturbation back to the $l_\infty$ ball. By this small modification, we could achieve $47.56 \pm 0.37\% $ PGD-50-10 accuracy on CIFAR10 with $\epsilon=8/255$ in contrast to $43.57 \pm 0.30\% $ by RS-FGSM and also further extend the working range of $\epsilon$ from 8/255 to 11/255 on CIFAR10 without CO occurring.
Introduction
Adversarial training is the "gold standard" for making deep learning models robust against malicious inputs. However, the most effective method—Projected Gradient Descent (PGD)—is computationally expensive. Researchers have tried using the much faster Fast Gradient Sign Method (FGSM), only to run into a significant hurdle: Catastrophic Overfitting (CO).
CO is a failure mode where a model suddenly loses all robustness against strong attacks during training. This paper by Peilin Kang and Seyed-Mohsen Moosavi-Dezfooli investigates the geometry of why models suddenly lose their robustness during fast training and offers a surprisingly simple fix.
The Phenomenon of Catastrophic Overfitting
In adversarial training, a model is trained on "adversarial examples"—images tweaked with tiny, imperceptible perturbations designed to fool the network. While PGD training is robust, it is often 10x slower than standard training. FGSM is fast but prone to CO.
The primary symptom of CO is a sudden divergence in performance metrics. During training, the model's accuracy against the "weak" training attack (FGSM) may remain high or even shoot up to near 100%, while its accuracy against a "strong" evaluation attack (PGD) plummets to zero.
Geometric Analysis and Hypotheses
The authors employ several diagnostic tools to understand this failure:
- Geometric Analysis: Using DeepFool (DF) to visualize cross-sections of decision boundaries. By plotting decision regions along the direction of the adversarial perturbation versus the direction perpendicular to the boundary, they observed how the model's internal logic changes.
- Hypothesis Testing: Three variants of FGSM were designed to test common theories:
- Magnified-RS-FGSM: Tested if perturbation length ($l_2$ norm) is the cause.
- Boundary-RS-FGSM: Tested if perturbations need to span the entire continuous threat model.
- Diff-RS-FGSM: Tested if perturbation diversity prevents CO.
Key Insights
- CO is Universal: It isn't just an FGSM problem; it also occurs in $DF^{\infty}-1$ (a single-step DeepFool attack) and RS-FGSM (FGSM with random initialization).
- The "Inverted" Boundary: In FGSM, after CO occurs, a new decision boundary is generated along the direction of the perturbation. Paradoxically, small perturbations become more effective at fooling the model than large ones, making the training process "blind" to actual threats.
- Direction Over Length: Experiments with "Magnified-RS-FGSM" showed that the direction of the perturbation relative to random initialization is a more critical factor in triggering CO than simple length.
Proposed Solution: RS-FGSM-wo-Proj
Based on their empirical analysis, the authors propose a simple modification: RS-FGSM-wo-Proj. This method removes the step that "clips" or projects the perturbation back into the $l_\infty$ ball. By not projecting, the model is trained on examples that more accurately reflect the direction of the loss gradient, preventing the "distortion" of the decision boundary.
Results
The study primarily utilized the CIFAR-10 dataset and PreAct-ResNet18 architecture. Key results include:
- Improved Robustness: On CIFAR-10 ($\epsilon = 8/255$), RS-FGSM-wo-Proj achieved 47.56% PGD-50-10 accuracy, compared to 43.57% for standard RS-FGSM.
- Extended Working Range: The proposed method extended the stable training range from $\epsilon = 8/255$ up to 11/255 without CO occurring.
- Efficiency: The method maintains the speed of FGSM (one gradient step) while approaching the robustness levels of multi-step PGD training.
Conclusion
While the paper provides a clear description of the geometric distortion during CO, the underlying mathematical reason for why optimization creates this specific distortion remains an area for future research. Nevertheless, the "no-projection" fix offers a practical and efficient way to enhance the stability and robustness of fast adversarial training.
Summary updated