Abstract
Adversarial training has shown its ability in producing models that are robust to perturbations on the input data, but usually at the expense of a decrease in the standard accuracy. To mitigate this, practitioners often assume that more data is the solution. This paper challenges this conventional wisdom, proving that for adversarially trained models, more training data can actually hurt generalization on clean test sets. By analyzing different "adversary regimes," the authors show that the benefit of more data depends entirely on the strength of the attacker.
Introduction
Adversarial training is the gold standard for defending models against "adversarial examples"—tiny, human-imperceptible perturbations designed to fool a classifier. However, this robustness comes at a price: a significant drop in "standard accuracy" (performance on clean, unperturbed data).
The common belief among practitioners is that this is a sample complexity problem—that we simply need more training data to bridge the gap between robust and standard models. This study asks: Does more training data always help adversarially robust models generalize better on benign data?
The answer, surprisingly, is no. Under certain conditions, adding more data to the training set can actually increase the generalization error.
Methods
The authors tackle this problem through a combination of rigorous mathematical proofs and empirical simulations. They focus on two primary theoretical frameworks:
- Gaussian Mixture Model with Linear Loss: They analyze a binary classification task where data comes from two Gaussian distributions. They derive the generalization error as a function of training set size ($n$) and the perturbation radius ($\epsilon$).
- The Manhattan Model: To study the mathematically "messy" 0-1 loss, they propose a simplified 2D model. This allows them to prove that the "more data hurts" phenomenon isn't just an artifact of specific loss functions like linear or hinge loss.
They categorize the interaction between data and robustness into three "regimes" based on the adversary's strength relative to the signal in the data.
Key Insights: The Three Regimes of Generalization
The most striking takeaway is the identification of three distinct regimes of generalization, which describe how test loss behaves as the training dataset grows.
- The Weak Adversary Regime: When the perturbation $\epsilon$ is very small, more data helps. The model behaves like a standard classifier, and the generalization error decreases monotonically.
- The Medium Adversary Regime (Double Descent): In this regime, the error curve is non-monotonic. First, error goes down (data helps), then it goes up (data hurts), and finally, it goes down again. This suggests a "danger zone" where a medium-sized dataset is worse than a small one.
- The Strong Adversary Regime: When the adversary is powerful enough to push data across the true decision boundary, more data almost immediately hurts. The learner is essentially being "fed" more corrupted signals that lead it further away from the true distribution.
Why does this happen? The authors hypothesize a trade-off: a larger dataset helps the model estimate the population risk better (the "learner wins"), but it also gives the adversary more points to manipulate and "pull" the decision boundary in the wrong direction (the "adversary wins"). When the adversary is strong, the latter effect dominates.
Key Results
Empirical simulations confirm these theoretical regimes across different perturbation radii ($\epsilon$).
- Sample-wise Non-monotonicity: The paper proves that generalization error does not always decrease with $n$. This is a rare example of "more data hurts" that isn't caused by model overparameterization, but by the training objective itself.
- Linear Regression: In 1D linear regression, the "hurt" threshold is distribution-dependent. For example, Poisson-distributed data is more robust to this phenomenon than Gaussian data because its discrete nature keeps points further apart.
- SVMs and 0-1 Loss: Tests on Support Vector Machines (SVMs) confirmed that even with different loss functions (hinge loss), the transition from the "weak" to "strong" regime occurs as $\epsilon$ increases, often well before the adversary reaches the actual signal strength.
Limitations and Future Work
The primary limitation is the simplicity of the models. The proofs rely on linear classifiers and synthetic distributions. While these provide clean mathematical insights, real-world deep neural networks have highly non-convex loss landscapes that might exhibit even more complex behaviors.
Future Directions: * Neural Networks: Extending these results to multi-layer perceptrons or CNNs. * New Training Frameworks: Since the current adversarial training framework (Min-Max optimization) leads to this paradox, new algorithmic approaches may be needed to achieve both robustness and high standard accuracy simultaneously. * Unlabeled Data: Exploring if unlabeled data could mitigate these negative effects, as the adversary cannot "label-flip" unlabeled points in the same way.
Summary updated