Adversarial training may be a double-edged sword

Ali Rahmati, Seyed-Mohsen Moosavi-Dezfooli, Huaiyu DaiarXiv 2021 · OpenReview

Abstract

Adversarial training has been shown as an effective approach to improve the robustness of image classifiers against white-box attacks. However, its effectiveness against black-box attacks is more nuanced. In this work, we demonstrate that some geometric consequences of adversarial training on the decision boundary of deep networks give an edge to certain types of black-box attacks. In particular, we define a metric called robustness gain to show that while adversarial training is an effective method to dramatically improve the robustness in white-box scenarios, it may not provide such a good robustness gain against the more realistic decision-based black-box attacks. Moreover, we show that even the minimal perturbation white-box attacks can converge faster against adversarially-trained neural networks compared to the regular ones.

Introduction

Deep neural networks are notoriously fragile, susceptible to "adversarial examples"—inputs with tiny, invisible perturbations that cause the model to misclassify. To combat this, researchers use Adversarial Training (AT), where the model is trained on these malicious examples to build "muscle memory" against them. While AT is the gold standard for defending against white-box attacks (where the attacker has full model access), its effectiveness in black-box settings (where the attacker only observes final labels) is less clear. This study investigates whether the geometric changes AT imposes on a model's decision boundary—specifically making it flatter and more linear—actually give black-box attackers a "road map" they didn't have before.

Methods

The researchers compared two models: a standard "Regular" ResNet-50 and an "Adversarially Trained" (Robust) ResNet-50. They evaluated these models using the ILSVRC2012 (ImageNet) validation set through three primary lenses:

  1. Robustness Gain ($\eta$): A metric defined as the ratio of the perturbation size ($\ell_2$ norm) needed to fool the robust model versus the regular model.
  2. Decision-Based Black-Box Attacks: They tested attacks like GeoDA, HSJA, and Boundary Attack (BA). These attacks do not see gradients; they "probe" the model with queries to find the boundary.
  3. Geometric Analysis: They analyzed the curvature of the decision boundary and the convergence speed of DeepFool, a white-box attack that relies on local linearization of the model's output.

Key Insights: The Flatness Trap

Adversarial training significantly reduces the curvature of the decision boundary. While a "curvy" boundary is usually seen as a sign of over-fitting, it actually makes it harder for black-box attackers to estimate the "normal vector" (the shortest path to the boundary). By flattening the boundary, AT makes these estimations much more accurate.

Furthermore, robust models behave more linearly than regular models. Because attacks like DeepFool work by approximating the model as a linear function, they "solve" the robust model in fewer steps. DeepFool required a median of only 2 iterations to fool the robust model, compared to 4 iterations for the regular model.

DeepFool performance comparison: (a) regular ResNet-50 shows more segments/non-linearity, (b) robust ResNet-50 shows fewer segments and faster convergence.

Results: Diminishing Returns in the Dark

The "robustness gain" provided by AT is massive in white-box scenarios (about 17x), but it shrinks significantly in black-box scenarios. The less information an attacker has (i.e., fewer queries allowed), the less benefit the user gets from having an adversarially trained model.

GeoDA’s Advantage: The GeoDA attack, which is explicitly built on the assumption of low boundary curvature, performs particularly well against robust models compared to other black-box attacks, confirming that AT creates the exact conditions GeoDA exploits. As the number of queries increases, the gain slowly rises, but it never reaches the efficiency seen in white-box defenses like DeepFool.

Black-box attack performance: (a) Median l2 norm vs. queries for robust and regular models; (b) Robustness gain across different black-box attacks compared to the white-box DeepFool baseline.

Conclusion

Adversarial training is a "double-edged sword": it increases a model's distance to the decision boundary (improving robustness), but it also flattens that boundary. This flatness makes it easier for decision-based black-box attacks to estimate the boundary's direction and allows white-box attacks like DeepFool to converge significantly faster. Future research must address whether we can design defense mechanisms that provide the distance of AT without the vulnerability of excessive flatness.

Summary updated

Abstract

Adversarial training has been shown as an effective approach to improve the robustness of image classifiers against white-box attacks. However, its effectiveness against black-box attacks is more nuanced. In this work, we demonstrate that some geometric consequences of adversarial training on the decision boundary of deep networks give an edge to certain types of black-box attacks. In particular, we define a metric called robustness gain to show that while adversarial training is an effective method to dramatically improve the robustness in white-box scenarios, it may not provide such a good robustness gain against the more realistic decision-based black-box attacks. Moreover, we show that even the minimal perturbation white-box attacks can converge faster against adversarially-trained neural networks compared to the regular ones.
BibTeX
@misc{rahmati2021adversarial,
  author = {Ali Rahmati and Seyed-Mohsen Moosavi-Dezfooli and Huaiyu Dai},
  title = {{Adversarial training may be a double-edged sword}},
  year = {2021},
  eprint = {2107.11671},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.11671},
}
Figure