Phase Transition from Clean Training to Adversarial Training

Yue Xing, Qifan Song, Guang ChengNeurIPS 2022 · OpenReview

Introduction: The Robustness-Accuracy Trade-off

Adversarial training (AT) is a cornerstone algorithm for building robust machine learning models capable of resisting intentional perturbations. However, it frequently incurs a steep cost to standard accuracy on clean data—for instance, causing a drop from 90% to 67% on the CIFAR-10 dataset. This notorious "robustness-accuracy trade-off" contradicts the theoretical guarantees provided by many existing studies, which often focus on very small attacks where the underlying math remains well-behaved.

This paper bridges the gap between theory and practice by asking a fundamental question: At what point does adversarial training stop being a simple variation of clean training and become a fundamentally different, more difficult optimization problem?

The authors identify an asymptotic phase transition governed by attack strength ($\epsilon$). Below a critical threshold ($\epsilon^*$), adversarial training behaves much like standard training. Above it, the model's behavior and performance deviate sharply.

Training Trajectories

Methods: Defining and Finding the Transition

To understand this phase transition, the researchers employed a mix of theoretical analysis and empirical techniques across standard computer vision benchmarks (CIFAR-10, CIFAR-100, SVHN, MNIST) and synthetic datasets.

  1. Theoretical Analysis: The authors mathematically prove the existence of a critical threshold $\epsilon^*$ in simple models, such as linear regression and two-layer neural networks. In these controlled settings, $\epsilon^*$ is closely related to the ratio of model dimensions to sample size ($\sqrt{d/n}$).
  2. Surrogate Measure for Deep Networks: Comparing entire optimization trajectories is computationally infeasible for deep neural networks. To bypass this, the authors propose a practical surrogate: $\epsilon^*$ is defined as the attack strength where the adversarial training loss equals the clean testing loss of a standard model.
  3. Loss Landscape Connectivity: Using path-finding techniques, the study demonstrates that models trained with attacks below $\epsilon^*$ remain in the same "loss basin" as clean models. Conversely, models trained with attacks above the threshold are pushed into entirely different regions of the parameter space.
Loss Landscape Connectivity
  1. Fast Approximation: Finding $\epsilon^*$ naively requires training dozens of models from scratch. The authors introduce a highly efficient method using adversarial fine-tuning and linear extrapolation, which accelerates the discovery of $\epsilon^*$ by approximately 80%.

Key Insights and Results

  • The "Clean-Like" Regime: If the attack strength $\epsilon$ is less than $\epsilon^*$, adversarial training is as well-behaved as clean training. If $\epsilon > \epsilon^*$, standard training intuitions no longer apply, and optimization becomes significantly harder.
  • Standard Attacks are Often "Too Strong": A striking finding is that commonly used attack strengths in the literature (e.g., $8/255$ for CIFAR-10) are significantly higher than the calculated $\epsilon^*$ (found to be $\approx 0.5/255$ for a PreActResNet18). This explains why the performance gap is so persistent in practice.
  • Model Capacity Matters: Wider neural networks exhibit a higher $\epsilon^*$. This confirms the intuition that adversarial training requires greater "model capacity" to succeed without sacrificing clean performance.
  • The Training Size Paradox: Counterintuitively, increasing the number of training samples can actually decrease $\epsilon^*$. As the model becomes better at fitting the clean data (reducing the generalization gap), the "room" for adversarial perturbations without deviating from the clean baseline shrinks.
  • Link to Catastrophic Overfitting: The paper links $\epsilon^*$ to "catastrophic overfitting" in Fast Gradient Sign Method (FGSM) training, showing that training instability begins precisely once the attack strength exceeds the phase transition threshold.

Limitations and Future Work

While this research provides a brilliant diagnostic tool for understanding the limits of robust models, it does not offer a direct "cure" for the accuracy drop. Instead, it explains why and when it happens.

The authors suggest that future research should focus on adjusting architectures or training schedules specifically for regimes where $\epsilon > \epsilon^*$. Additionally, $\epsilon^*$ could serve as a valuable metric for selecting architectures that are naturally more "robustness-ready" before undertaking expensive adversarial training. Finally, while the theoretical proofs cover linear models and two-layer networks, extending these proofs to deep, multi-layer ReLU networks remains an open challenge for the community.

Summary updated

Abstract

Adversarial training is one important algorithm to achieve robust machine learning models. However, numerous empirical results show a great performance degradation from clean training to adversarial training (e.g., 90+\% vs 67\% testing accuracy on CIFAR-10 dataset), which does not match the theoretical guarantee delivered by the existing studies. Such a gap inspires us to explore the existence of an (asymptotic) phase transition phenomenon with respect to the attack strength: adversarial training is as well behaved as clean training in the small-attack regime, but there is a sharp transition from clean training to adversarial training in the large-attack regime. We validate this conjecture in linear regression models, and conduct comprehensive experiments in deep neural networks.
BibTeX
@inproceedings{xing2022phase,
  author = {Yue Xing and Qifan Song and Guang Cheng},
  title = {{Phase Transition from Clean Training to Adversarial Training}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=gwsnBjNcVEe},
}
Figure