Efficient local linearity regularization to overcome catastrophic overfitting

Elias Abad Rocamora, Fanghui Liu, Grigorios Chrysos, Pablo M. Olmos, Volkan Cevher

Abstract

Single-step adversarial training (AT) offers a computationally efficient alternative to multi-step methods like PGD, yet it is frequently plagued by "catastrophic overfitting" (CO)—a sudden collapse in robust accuracy. This paper introduces Efficient Local Linearity Enforcement (ELLE), a novel regularization method that prevents CO by enforcing local linearity through interpolation rather than expensive second-order derivatives. ELLE achieves state-of-the-art robustness at less than half the computational cost of existing methods.

Introduction: The Challenge of Catastrophic Overfitting

Deep learning models are highly vulnerable to adversarial attacks—small, intentional perturbations that lead to misclassification. Adversarial Training (AT) is the primary defense, involving training on these perturbed images. While multi-step AT (e.g., PGD) is robust, it is computationally intensive. Single-step AT (e.g., FGSM) is faster but prone to Catastrophic Overfitting (CO), where the model's robust accuracy abruptly drops to zero during training.

Previous solutions to CO enforced "local linearity" to smooth the loss landscape, but these typically required Double Backpropagation, making them nearly three times slower than standard training.

Methodology: Efficient Local Linearity Enforcement (ELLE)

ELLE enforces local linearity without the heavy computational overhead of second-order gradients. The core principle is that for a linear function, the value at an interpolated point should equal the weighted average of the values at the endpoints.

  1. Sampling: For each training point, the algorithm samples two nearby points ($x_a, x_b$) and a random interpolation factor ($\alpha$).
  2. ELLE Term: It calculates the discrepancy between the actual loss at the interpolated point and the predicted linear loss.
  3. Optimization: Minimizing this discrepancy forces the loss landscape to remain smooth and linear.

The authors also propose ELLE-A (Adaptive), which monitors linearity error in real-time. If a spike in error—a precursor to CO—is detected, the regularization strength is automatically increased to stabilize the model.

Key Results and Performance

ELLE was evaluated on CIFAR-10, CIFAR-100, SVHN, and ImageNet. The results demonstrate that ELLE provides a superior balance of speed and robustness.

Training time comparison showing ELLE's efficiency relative to other methods.
  • Efficiency: ELLE is $2\times$ to $3\times$ faster per training step than methods like GradAlign or CURE.
  • Robustness: On CIFAR-10, ELLE-A achieved 44.32% AutoAttack accuracy, effectively preventing the 0% accuracy seen in standard fast training.
  • Scalability: On ImageNet, ELLE-A successfully prevented CO even at large perturbation sizes ($\epsilon = 8/255$).
  • Stability: ELLE remains robust across different architectures (e.g., ResNet vs. WideResNet) and long training schedules, avoiding "Delayed CO."

Discussion and Future Work

While ELLE is computationally efficient in terms of time, it requires three extra forward passes per sample, increasing VRAM usage. Future research may focus on reducing this memory footprint. Additionally, the authors note that at extreme perturbation levels, the inherent ambiguity of images suggests a natural limit to adversarial robustness.

Summary updated

Abstract

Catastrophic overfitting (CO) in single-step adversarial training (AT) results in abrupt drops in the adversarial test accuracy (even down to $0$%). For models trained with multi-step AT, it has been observed that the loss function behaves locally linearly with respect to the input, this is however lost in single-step AT. To address CO in single-step AT, several methods have been proposed to enforce local linearity of the loss via regularization. However, these regularization terms considerably slow down training due to *Double Backpropagation*. Instead, in this work, we introduce a regularization term, called ELLE, to mitigate CO *effectively* and *efficiently* in classical AT evaluations, as well as some more difficult regimes, e.g., large adversarial perturbations and long training schedules. Our regularization term can be theoretically linked to curvature of the loss function and is computationally cheaper than previous methods by avoiding *Double Backpropagation*. Our thorough experimental validation demonstrates that our work does not suffer from CO, even in challenging settings where previous works suffer from it. We also notice that adapting our regularization parameter during training (ELLE-A) greatly improves the performance, specially in large $\epsilon$ setups. Our implementation is available in https://github.com/LIONS-EPFL/ELLE.
BibTeX
@inproceedings{rocamora2024efficient,
  author = {Elias Abad Rocamora and Fanghui Liu and Grigorios Chrysos and Pablo M. Olmos and Volkan Cevher},
  title = {{Efficient local linearity regularization to overcome catastrophic overfitting}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=SZzQz8ikwg},
}
Figure