The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalization.

Yifei Min, Lin Chen, Amin Karbasi

Abstract

Adversarial training has shown its ability in producing models that are robust to perturbations on the input data, but usually at the expense of decrease in the standard accuracy. To mitigate this issue, it is commonly believed that more training data will eventually help such adversarially robust models generalize better on the benign/unperturbed test data. In this paper, however, we challenge this conventional belief and show that more training data can hurt the generalization of adversarially robust models in the classification problems. We first investigate the Gaussian mixture classification with a linear loss and identify three regimes based on the strength of the adversary. In the weak adversary regime, more data improves the generalization of adversarially robust models. In the medium adversary regime, with more training data, the generalization loss exhibits a double descent curve, which implies the existence of an intermediate stage where more training data hurts the generalization. In the strong adversary regime, more data almost immediately causes the generalization error to increase. Then we move to the analysis of a two-dimensional classification problem with a 0-1 loss. We prove that more data always hurts the generalization performance of adversarially trained models with large perturbations. To complement our theoretical results, we conduct empirical studies on Gaussian mixture classification, support vector machines (SVMs), and linear regression.

Introduction

In the world of machine learning, "more data is better" is a fundamental mantra. However, this research proves that for models trained to be adversarially robust, more data can actually hurt generalization on clean, unperturbed test sets. Adversarial training—training models on "perturbed" examples to make them harder to fool—is the gold standard for security in ML. However, it comes with a "robustness tax": these models usually perform worse on normal, clean data than their non-robust counterparts.

The common assumption was that this gap exists simply because robust models are data-hungry. The belief was that if we just provided enough training data, robust models would eventually "catch up" and generalize well to clean data. This paper challenges that belief, identifying scenarios where adding training samples leads to a "double descent" error curve or even a consistent decrease in standard accuracy.

Methodology

The authors tackle this question through a mix of rigorous mathematical proofs and empirical simulations. They focus on three primary settings:

  1. Gaussian Mixture Models (Linear Loss): They analytically derive the generalization error for a binary classification task where data comes from two Gaussian distributions.
  2. The "Manhattan Model" (0-1 Loss): To study the mathematically difficult 0-1 loss, they propose a simplified 2D conceptual model where data points sit on a grid.
  3. Empirical Validations: They run experiments on Support Vector Machines (SVMs) and Linear Regression to see if the theoretical regimes hold true in more common algorithmic settings.

Because this is a foundational theoretical paper, the authors use synthetic datasets (Gaussian mixtures, 2D grids, and Poisson-distributed data) where they can precisely control the signal-to-noise ratio and the strength of the adversary ($\epsilon$).

Theoretical Framework: The Three Regimes

The paper’s most significant contribution is identifying three distinct regimes based on the strength of the adversary ($\epsilon$):

  • The Weak Adversary Regime: Conventional wisdom holds. The adversary is small enough that more data consistently improves generalization on clean data.
  • The Medium Adversary Regime: In this regime, the error follows a double descent curve. Initially, more data helps. Then, as you add more data, the error actually increases (more data hurts). Finally, after a certain threshold, the error begins to drop again.
  • The Strong Adversary Regime: When the adversary is powerful, more data almost immediately causes the generalization error on clean data to rise. In this scenario, a larger training set makes the model worse at its primary task.
Conceptual illustration of the three regimes of generalization loss vs. training dataset size.

This happens because a strong adversary forces the model to prioritize "robust" features that might be less predictive of the actual label on clean data. As the model sees more data, it "overfits" to these robust but less accurate features.

Key Results and Empirical Evidence

  • Theorem 1 (Gaussian Mixtures): Formally proves the existence of the three regimes. Specifically, it shows that in the medium regime, the "increasing stage" of the error curve can be arbitrarily long depending on the adversary's strength.
  • Theorem 4 (Manhattan Model): Proves that under 0-1 loss, if the adversary is strong enough ($\epsilon > 2\mu$), the generalization error is guaranteed to increase as you add more training points.
  • SVM Findings: In experiments, even when the adversary was technically weaker than the data signal, the test loss increased with more data. This suggests that for complex models like SVMs, the "Strong Adversary" regime might be reached much sooner than expected.
Empirical validation of the three regimes showing test loss across different adversary strengths (epsilon).

Conclusion

The primary limitation is that the theoretical proofs are confined to relatively simple models. While the empirical results on SVMs suggest the phenomenon is widespread, further proof is needed for deep neural networks. The authors conclude that the current framework of adversarial training might be fundamentally flawed. Future work should look for new training paradigms that can achieve robustness without sacrificing the ability to learn from more data—essentially finding a way to eliminate the "robustness tax" entirely.

Summary updated

Abstract

Adversarial training has shown its ability in producing models that are robust to perturbations on the input data, but usually at the expense of decrease in the standard accuracy. To mitigate this issue, it is commonly believed that more training data will eventually help such adversarially robust models generalize better on the benign/unperturbed test data. In this paper, however, we challenge this conventional belief and show that more training data can hurt the generalization of adversarially robust models in the classification problems. We first investigate the Gaussian mixture classification with a linear loss and identify three regimes based on the strength of the adversary. In the weak adversary regime, more data improves the generalization of adversarially robust models. In the medium adversary regime, with more training data, the generalization loss exhibits a double descent curve, which implies the existence of an intermediate stage where more training data hurts the generalization. In the strong adversary regime, more data almost immediately causes the generalization error to increase. Then we move to the analysis of a two-dimensional classification problem with a 0-1 loss. We prove that more data always hurts the generalization performance of adversarially trained models with large perturbations. To complement our theoretical results, we conduct empirical studies on Gaussian mixture classification, support vector machines (SVMs), and linear regression.
BibTeX
@inproceedings{min2021curious,
  author = {Yifei Min and Lin Chen and Amin Karbasi},
  title = {{The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalization.}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=BK8iILKtXL},
}
Figure