Increasing Confidence in Adversarial Robustness Evaluations

Roland S. Zimmermann, Wieland Brendel, Florian Tramer, Nicholas Carlini

Introduction

Evaluating whether a neural network is truly "robust" against adversarial attacks is notoriously difficult, often leading researchers to overestimate the effectiveness of their defenses. In the world of adversarial machine learning, there is a persistent cat-and-mouse game: a researcher proposes a new defense that supposedly makes a model robust to small input perturbations, and months later, another researcher develops a stronger attack that breaks it.

The core issue is that evaluating robustness is fundamentally different from evaluating standard accuracy. To measure robustness, one is essentially trying to prove a theorem: "No adversary can find a perturbation within distance $\epsilon$ that changes the model's output." If an attack is weak, it might fail to find an adversarial example even if one exists, making a vulnerable model look robust. This paper introduces an active "unit test" for adversarial attacks. By slightly modifying a network to guarantee that adversarial examples exist, the authors can definitively prove if an attack is too weak to find them.

Methodology: The Binarization Test

Current best practices for spotting "weak" evaluations are mostly passive, relying on researchers to spot red flags like vanishing gradients. The authors propose the Binarization Test to actively create a controlled environment where the ground truth is known.

The goal is to create a "vulnerable" version of a defense that is as similar as possible to the original:

  1. Feature Extraction: The original model is taken, and everything is frozen except the very last layer (the linear classification head).
  2. Dataset Construction: For a specific clean image, points are sampled from the "inner" region (very close to the image) and the "boundary" region (exactly at the limit of the allowed perturbation $\epsilon$).
  3. The Trap: A new, simple binary classifier is trained on these features to perfectly distinguish between the "inner" and "boundary" points.
  4. The Guarantee: By construction, this new model must have adversarial examples at the boundary. If you start at the clean image (inner), any point on the boundary is now a successful adversarial perturbation.
  5. The Test: The original researcher's attack is run on this modified model. If the attack cannot find these "planted" adversarial examples, it is proven to be too weak, invalidating the original robustness claims.
Overview of the Binarization Test

For defenses that use "detectors" (extra components that try to spot and reject adversarial inputs), the authors also developed Regular and Inverted versions of the test to ensure the attack properly targets both the classifier and the detector.

Key Insights

  • Robustness as a Proof: A defense proposal should be viewed as a theorem and its evaluation as the proof. A successful attack is simply a counter-example.
  • Unit Testing for Research: Just as software engineers use unit tests to ensure individual functions work, robustness researchers should use "attack unit tests" to ensure their evaluation tools are sharp enough for the job.
  • Attack-Agnosticism: The Binarization Test does not care how the attack works (gradients, random searches, etc.); it only cares if the attack finds the guaranteed example.
  • Identifying False Security: The test successfully reveals when defenses aren't actually robust but have merely made the optimization landscape "messy" enough to confuse standard attacks.

Key Results

The authors evaluated 13 different peer-reviewed defenses published at major conferences (NeurIPS, ICML, ICLR) using the CIFAR-10 dataset.

  • High Failure Rate: 11 out of 13 previously published defenses failed the Binarization Test.
  • Predictive Power: For 9 of those 11, the defenses had already been broken by the community in subsequent papers. This test would have caught the flaw immediately upon original publication.
  • New Vulnerabilities Discovered: For the remaining 2 defenses (Sarkar et al. and Yang et al.), this paper is the first to demonstrate that their original evaluations were insufficient. After failing the test, the authors applied stronger adaptive attacks, reducing the reported robustness significantly (e.g., from 99% down to below 12% for one model).
  • Validation of Strong Attacks: When known "strong" attacks (like AutoAttack or FAB) were used, they consistently passed the Binarization Test, proving the test's ability to distinguish between effective and ineffective evaluation methods.
Binarization Test Performance across Defenses

Limitations and Future Work

While powerful, the Binarization Test has limitations. It relies on hyperparameters (like the number of samples used to train the binary head) that dictate its "hardness." If the test is too easy, weak attacks might pass; if too hard, strong attacks might fail. The authors provide a "Random Attack Success Rate" (R-ASR) metric to help calibrate this.

Additionally, the current implementation is designed for models with linear classification readouts and requires custom adaptation for more exotic architectures. It also cannot catch every possible integration bug (e.g., attacking the predicted label instead of the ground truth).

Ultimately, the authors hope that active unit tests like the Binarization Test will become a standard requirement for any new defense paper submitted to major AI conferences, increasing confidence in an empirical field currently riddled with skepticism.

Summary updated

Abstract

Hundreds of defenses have been proposed to make deep neural networks robust against minimal (adversarial) input perturbations. However, only a handful of these defenses held up their claims because correctly evaluating robustness is extremely challenging: Weak attacks often fail to find adversarial examples even if they unknowingly exist, thereby making a vulnerable network look robust. In this paper, we propose a test to identify weak attacks and, thus, weak defense evaluations. Our test slightly modifies a neural network to guarantee the existence of an adversarial example for every sample. Consequentially, any correct attack must succeed in breaking this modified network. For eleven out of thirteen previously-published defenses, the original evaluation of the defense fails our test, while stronger attacks that break these defenses pass it. We hope that attack unit tests - such as ours - will be a major component in future robustness evaluations and increase confidence in an empirical field that is currently riddled with skepticism.
BibTeX
@inproceedings{zimmermann2022increasing,
  author = {Roland S. Zimmermann and Wieland Brendel and Florian Tramer and Nicholas Carlini},
  title = {{Increasing Confidence in Adversarial Robustness Evaluations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=NkK4i91VWp},
}
Figure