A Geometric Perspective on the Transferability of Adversarial Directions

Zachary Charles, Harrison Rosenberg, Dimitris Papailiopoulos

TL;DR

This paper provides a rigorous geometric explanation for why adversarial attacks transfer between different machine learning models. The authors prove that for linear classifiers and two-layer ReLU networks, there exist specific "adversarial directions" that can simultaneously fool multiple models and data points, revealing that attacks on complex models are more likely to transfer to simpler ones than vice versa.

What Problem Does This Paper Address?

Adversarial examples—small, intentional perturbations that cause machine learning models to misclassify inputs—are a well-documented security flaw. A particularly puzzling phenomenon is transferability: an adversarial perturbation generated to fool Model A often successfully fools Model B, even if Model B has a different architecture or was trained on a different subset of data.

While empirical evidence for transferability is abundant, a theoretical "why" has been largely missing. This paper seeks to fill that gap by using geometry to understand the decision boundaries of linear classifiers and simple neural networks, proving that shared vulnerabilities are a mathematical necessity under certain conditions rather than just a fluke of training.

Methods

The authors approach the problem by defining adversarial directions—vectors that, when added to an input, push it across a decision boundary. Their methodology includes:

  • Geometric Analysis of Linear Classifiers: They utilize the theory of max-margin Support Vector Machines (SVMs) to show that the max-margin separator of a dataset provides a "universal" direction that is adversarial for any linear classifier that correctly separates that data.
  • Probabilistic Extensions: Using Rademacher complexity and Hoeffding’s inequality, they extend their proofs to "soft-margin" classifiers trained on independent samples from the same distribution, showing that shared adversarial directions exist with high probability.
  • ReLU Network Analysis: They analyze two-layer Rectified Linear Unit (ReLU) networks. By proving that certain decision regions in these networks are convex, they apply their geometric findings to show that these networks also possess transferable adversarial directions.
  • Asymmetry Proofs: They investigate the "directionality" of transfer, specifically whether an attack on a complex model (ReLU net) transfers to a simple one (linear) and vice versa.

Datasets

The paper validates its theoretical findings using the MNIST dataset of handwritten digits.

  • Experimental Setup: The authors trained 50 moderately-sized neural networks and 50 Linear SVMs.
  • Training Protocol: Each model was trained on a unique, randomly selected subset of 10,000 images from the MNIST training set to test how well attacks transfer across different data samples.

Key Insights and Ideas

  • The Max-Margin "Golden Key": For linear models, the max-margin direction is a fundamental vulnerability. If you find the direction that best separates the classes, its inverse is effectively a universal key that can unlock (fool) other linear separators.
  • Convexity as a Vulnerability: The transferability in ReLU networks is deeply tied to the convexity of their decision regions. If a model's "class A" region is convex, there is a guaranteed direction that leads toward "class B" for all points in that region.
  • The Complexity Asymmetry: One of the most striking findings is that transferability is not a two-way street. Adversarial directions for ReLU networks almost always transfer to linear classifiers. However, directions that fool linear classifiers often fail to fool ReLU networks.
  • Capacity and Robustness: This suggests that more complex models (higher capacity) generate "stronger" adversarial directions that encompass the vulnerabilities of simpler models.

Key Results

  • Theoretical Guarantee: The authors prove (Theorem 7) that for two linear classifiers trained independently on the same distribution, there is a shared adversarial direction that affects a significant fraction of the training data with high probability.
  • Empirical Validation on MNIST:
    • SVM to SVM: Attacks transferred well between different SVMs.
  • NN to NN: Attacks transferred well between different neural networks.
  • NN to SVM: Neural network attacks were highly effective at fooling SVMs.
  • SVM to NN: SVM attacks were significantly less effective at fooling neural networks, performing only slightly better than random noise in some cases.
  • Robustness to Noise: Both SVMs and NNs remained highly accurate when subjected to random perturbations, confirming that transferable adversarial directions are highly structured and non-random.

Limitations and Future Work

  • Architecture Depth: The theoretical proofs are primarily limited to linear classifiers and two-layer ReLU networks. While empirical results suggest these findings hold for deeper networks, the formal geometric proof for deep architectures remains an open challenge.
  • Norm Constraints: The study focuses on the direction of the attack. The authors note that future work should more strictly constrain the norm (the magnitude) of the perturbation to better align with "stealthy" adversarial attack research.
  • Kernel Methods: The authors suggest that their linear results could be extended to non-linear spaces using a kernel perspective, which might provide a bridge to understanding deeper, more complex architectures.
  • Generalizing Capacity: A major open question is whether the "complex-to-simple" transferability rule holds across all types of machine learning models beyond the ones studied here.

Summary updated

Abstract

State-of-the-art machine learning models frequently misclassify inputs that have been perturbed in an adversarial manner. Adversarial perturbations generated for a given input and a specific classifier often seem to be effective on other inputs and even different classifiers. In other words, adversarial perturbations seem to transfer between different inputs, models, and even different neural network architectures. In this work, we show that in the context of linear classifiers and two-layer ReLU networks, there provably exist directions that give rise to adversarial perturbations for many classifiers and data points simultaneously. We show that these "transferable adversarial directions" are guaranteed to exist for linear separators of a given set, and will exist with high probability for linear classifiers trained on independent sets drawn from the same distribution. We extend our results to large classes of two-layer ReLU networks. We further show that adversarial directions for ReLU networks transfer to linear classifiers while the reverse need not hold, suggesting that adversarial perturbations for more complex models are more likely to transfer to other classifiers. We validate our findings empirically, even for deeper ReLU networks.
BibTeX
@inproceedings{charles2019geometric,
  author = {Zachary Charles and Harrison Rosenberg and Dimitris Papailiopoulos},
  title = {{A Geometric Perspective on the Transferability of Adversarial Directions}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2019},
  url = {https://openreview.net/forum?id=BAl-4CFQ2J9},
}

Loading related content...

Figure