Adversarial Examples Are a Natural Consequence of Test Error in Noise

Nic Ford, Justin Gilmer, Nicholas Carlini, Ekin Dogus Cubuk

Abstract

Over the last few years, the phenomenon of adversarial examples --- maliciously constructed inputs that fool trained machine learning models --- has captured the attention of the research community, especially when the adversary is restricted to small modifications of a correctly handled input. Less surprisingly, image classifiers also lack human-level performance on randomly corrupted images, such as images with additive Gaussian noise. In this paper we provide both empirical and theoretical evidence that these are two manifestations of the same underlying phenomenon, establishing close connections between the adversarial robustness and corruption robustness research programs. This suggests that improving adversarial robustness should go hand in hand with improving performance in the presence of more general and realistic image corruptions. Based on our results we recommend that future adversarial defenses consider evaluating the robustness of their methods to distributional shift with benchmarks such as Imagenet-C.

Introduction

Adversarial examples—small, often imperceptible perturbations that cause misclassification—have long been viewed as a mysterious vulnerability of deep neural networks. Traditionally, the research community has treated adversarial robustness (worst-case perturbations) and corruption robustness (average-case random noise) as distinct challenges. This paper argues that they are, in fact, two sides of the same coin. Adversarial examples are not a pathological "bug" but a predictable geometric consequence of a model having any non-zero error rate when subjected to random Gaussian noise.

Theoretical Framework: The Geometry of Error

The core of the argument lies in high-dimensional geometry. In high dimensions, the volume of a sphere is concentrated near its equator. If a decision boundary misclassifies even a small fraction of a noisy distribution (e.g., 1% error under Gaussian noise), that boundary must be mathematically very close to the original clean input.

Geometric relationship between the distance to the decision boundary and error rate under Gaussian noise.

Using the Gaussian Isoperimetric Inequality, the authors show that for a given error volume, a "half-space" (a flat decision boundary) represents the most robust possible configuration. Any curvature in the boundary only brings errors closer to the input. Thus, if a model is not perfectly robust to random noise, it is mathematically destined to have adversarial examples nearby.

Empirical Validation

To test these theoretical bounds, the researchers evaluated naturally trained, noise-trained, and adversarially trained models on CIFAR-10 and ImageNet. They found that modern neural networks behave remarkably like the theoretical linear (half-space) models.

Empirical validation on CIFAR and ImageNet showing the linear relationship between noise robustness and distance to the decision boundary.

As shown in the figures, the distance to the nearest adversarial example (the decision boundary) is directly proportional to the noise level $\sigma$ at which the model maintains a low error rate. Adversarial training, by pushing the decision boundary further away, naturally improves robustness to Gaussian noise and other corruptions like blur.

Key Insights

  • The Robustness Link: Improving adversarial robustness requires reducing error rates under large additive noise to near zero. Conversely, training for noise robustness naturally increases the distance to the decision boundary.
  • Defenses as Sanity Checks: The authors suggest that any adversarial defense that does not also improve performance on simple Gaussian noise may be "obfuscating gradients"—tricking the attack algorithm rather than fixing the underlying geometric vulnerability.
  • Dataset Differences: While CIFAR-10 and ImageNet follow these geometric rules, MNIST is an outlier. Its classes are sufficiently separated that models can achieve near-perfect robustness, a luxury not afforded by more complex, high-dimensional natural image datasets.

Conclusion and Future Work

The study establishes that adversarial examples are a "natural" consequence of test error in noisy distributions. While Gaussian noise provides a clear mathematical link, other corruptions like fog or contrast shifts present different geometric challenges where adversarial training may even be detrimental. The authors urge the community to adopt corruption benchmarks like ImageNet-C as standard sanity checks for adversarial defenses, ensuring that improvements are grounded in fundamental geometric robustness rather than algorithmic artifacts.

Summary updated

Abstract

Over the last few years, the phenomenon of adversarial examples --- maliciously constructed inputs that fool trained machine learning models --- has captured the attention of the research community, especially when the adversary is restricted to small modifications of a correctly handled input. Less surprisingly, image classifiers also lack human-level performance on randomly corrupted images, such as images with additive Gaussian noise. In this paper we provide both empirical and theoretical evidence that these are two manifestations of the same underlying phenomenon, establishing close connections between the adversarial robustness and corruption robustness research programs. This suggests that improving adversarial robustness should go hand in hand with improving performance in the presence of more general and realistic image corruptions. Based on our results we recommend that future adversarial defenses consider evaluating the robustness of their methods to distributional shift with benchmarks such as Imagenet-C.
BibTeX
@inproceedings{ford2019adversarial,
  author = {Nic Ford and Justin Gilmer and Nicholas Carlini and Ekin Dogus Cubuk},
  title = {{Adversarial Examples Are a Natural Consequence of Test Error in Noise}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=BJVCvs-_Zr},
}
Figure