Increasing Confidence in Adversarial Robustness Evaluations

Roland S. Zimmermann, Wieland Brendel, Florian Tramèr, Nicholas Carlini

Abstract

Hundreds of defenses have been proposed to make deep neural networks robust against minimal (adversarial) input perturbations. However, only a handful of these defenses held up their claims because correctly evaluating robustness is extremely challenging: Weak attacks often fail to find adversarial examples even if they unknowingly exist, thereby making a vulnerable network look robust. In this paper, we propose a test to identify weak attacks, and thus weak defense evaluations. Our test slightly modifies a neural network to guarantee the existence of an adversarial example for every sample. Consequentially, any correct attack must succeed in breaking this modified network. For eleven out of thirteen previously-published defenses, the original evaluation of the defense fails our test, while stronger attacks that break these defenses pass it. We hope that attack unit tests - such as ours - will be a major component in future robustness evaluations and increase confidence in an empirical field that is currently riddled with skepticism.

Introduction

In the field of adversarial machine learning, a persistent "cat-and-mouse" game exists between researchers proposing new defenses and those developing stronger attacks. While standard model accuracy is straightforward to measure on a test set, proving adversarial robustness is significantly more complex. It requires demonstrating that no possible attack within a specific perturbation limit can succeed—a task more akin to proving a mathematical theorem than measuring a performance metric.

Often, a defense appears robust not because it is inherently secure, but because the attack used to evaluate it was too weak, poorly tuned, or hindered by gradient masking. This paper addresses this gap by introducing a "unit test" for adversarial attacks called the Binarization Test. By creating a version of a neural network that is "vulnerable by design," researchers can verify if an adversarial attack is actually strong enough to find existing flaws. If an attack cannot break the modified network, its evaluation of the original defense cannot be trusted.

The Binarization Test Methodology

The Binarization Test is an "active" robustness test. Rather than passively searching for artifacts in data, the authors purposefully inject a known vulnerability into the model to see if the attack can identify it. The process follows four key steps:

  1. Model Modification: The original feature extractor of a defense (the layers processing the input) is preserved, but the final classification layer is replaced with a new, linear binary classifier.
  2. Creating a Vulnerability: This new classifier is trained on a custom dataset consisting of two groups: points very close to a clean image (the "inner" set) and points placed exactly on the boundary of the allowed perturbation limit (the "boundary" set).
  3. The Unit Test: By construction, these boundary points are adversarial examples. If the attack used in the evaluation is competent, it must be able to find these boundary points and flip the model's decision.
  4. Detector Adaptation: For defenses that utilize a separate "detector" to flag suspicious inputs, the authors provide "Regular" and "Inverted" versions of the test to ensure the attack successfully targets both the classifier and the detector components.

This test is attack-agnostic; it does not matter whether the researcher uses gradient-based, black-box, or decision-based attacks. The only metric for success is whether the attack reaches the intended goal of finding the planted vulnerability.

Experimental Evaluation

The authors evaluated the Binarization Test against 13 previously published defenses using the CIFAR-10 dataset. These defenses were selected based on their high citation counts or their status as state-of-the-art at the time of publication. To ensure the test could be calibrated, the authors included a hyperparameter ($\kappa$) to control the "hardness" of the decision boundary, allowing the unit test's difficulty to match the expected difficulty of the original defense.

Key Results

The findings revealed a significant crisis in robustness evaluation standards: 11 out of the 13 defenses failed the Binarization Test.

  • Weak Evaluations: In the majority of cases, the original attacks used by the defense authors were unable to find the "planted" adversarial examples in the modified, vulnerable models.
  • Validation via Stronger Attacks: When these weak attacks were replaced with modern, rigorous benchmarks (such as AutoAttack or FAB), the stronger attacks passed the Binarization Test and subsequently broke the original defenses, reducing their claimed robust accuracy to near zero.
  • Identifying New Vulnerabilities: The test successfully identified flaws in two recent defenses (Sarkar et al. and Yang et al.) that had not yet been publicly broken. In one instance, a detector-based defense's reported robust accuracy plummeted from 99% to below 12%.

Discussion and Limitations

The Binarization Test represents a shift toward viewing robustness claims as theorems that require rigorous proof. By treating attacks as software components that require unit testing, the field can move away from skepticism and toward reproducible security claims.

However, the test has limitations. It is primarily designed for models with linear classification readouts; adapting it for non-standard architectures like k-Nearest Neighbors requires manual tailoring. Additionally, it may not catch "integration bugs" where an attack is technically correct but implemented with incorrect labels during the evaluation process.

Conclusion

The Binarization Test provides a necessary diagnostic tool for the machine learning community. By ensuring that the "evaluator" is capable of finding known vulnerabilities, researchers can have significantly higher confidence in the robustness claims of new defenses. The authors advocate for the adoption of attack unit tests as a standard requirement for future robustness research.

Summary updated

Abstract

Hundreds of defenses have been proposed to make deep neural networks robust against minimal (adversarial) input perturbations. However, only a handful of these defenses held up their claims because correctly evaluating robustness is extremely challenging: Weak attacks often fail to find adversarial examples even if they unknowingly exist, thereby making a vulnerable network look robust. In this paper, we propose a test to identify weak attacks, and thus weak defense evaluations. Our test slightly modifies a neural network to guarantee the existence of an adversarial example for every sample. Consequentially, any correct attack must succeed in breaking this modified network. For eleven out of thirteen previously-published defenses, the original evaluation of the defense fails our test, while stronger attacks that break these defenses pass it. We hope that attack unit tests - such as ours - will be a major component in future robustness evaluations and increase confidence in an empirical field that is currently riddled with skepticism.
BibTeX
@inproceedings{zimmermann2022increasing,
  author = {Roland S. Zimmermann and Wieland Brendel and Florian Tramèr and Nicholas Carlini},
  title = {{Increasing Confidence in Adversarial Robustness Evaluations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=NkK4i91VWp},
}
Figure