Abstract
Single-step adversarial training (AT) offers a computationally efficient alternative to multi-step methods like PGD, yet it is frequently plagued by "catastrophic overfitting" (CO)—a sudden collapse in robust accuracy. This paper introduces Efficient Local Linearity Enforcement (ELLE), a novel regularization method that prevents CO by enforcing local linearity through interpolation rather than expensive second-order derivatives. ELLE achieves state-of-the-art robustness at less than half the computational cost of existing methods.
Introduction: The Challenge of Catastrophic Overfitting
Deep learning models are highly vulnerable to adversarial attacks—small, intentional perturbations that lead to misclassification. Adversarial Training (AT) is the primary defense, involving training on these perturbed images. While multi-step AT (e.g., PGD) is robust, it is computationally intensive. Single-step AT (e.g., FGSM) is faster but prone to Catastrophic Overfitting (CO), where the model's robust accuracy abruptly drops to zero during training.
Previous solutions to CO enforced "local linearity" to smooth the loss landscape, but these typically required Double Backpropagation, making them nearly three times slower than standard training.
Methodology: Efficient Local Linearity Enforcement (ELLE)
ELLE enforces local linearity without the heavy computational overhead of second-order gradients. The core principle is that for a linear function, the value at an interpolated point should equal the weighted average of the values at the endpoints.
- Sampling: For each training point, the algorithm samples two nearby points ($x_a, x_b$) and a random interpolation factor ($\alpha$).
- ELLE Term: It calculates the discrepancy between the actual loss at the interpolated point and the predicted linear loss.
- Optimization: Minimizing this discrepancy forces the loss landscape to remain smooth and linear.
The authors also propose ELLE-A (Adaptive), which monitors linearity error in real-time. If a spike in error—a precursor to CO—is detected, the regularization strength is automatically increased to stabilize the model.
Key Results and Performance
ELLE was evaluated on CIFAR-10, CIFAR-100, SVHN, and ImageNet. The results demonstrate that ELLE provides a superior balance of speed and robustness.
- Efficiency: ELLE is $2\times$ to $3\times$ faster per training step than methods like GradAlign or CURE.
- Robustness: On CIFAR-10, ELLE-A achieved 44.32% AutoAttack accuracy, effectively preventing the 0% accuracy seen in standard fast training.
- Scalability: On ImageNet, ELLE-A successfully prevented CO even at large perturbation sizes ($\epsilon = 8/255$).
- Stability: ELLE remains robust across different architectures (e.g., ResNet vs. WideResNet) and long training schedules, avoiding "Delayed CO."
Discussion and Future Work
While ELLE is computationally efficient in terms of time, it requires three extra forward passes per sample, increasing VRAM usage. Future research may focus on reducing this memory footprint. Additionally, the authors note that at extreme perturbation levels, the inherent ambiguity of images suggests a natural limit to adversarial robustness.
Summary updated