Rethinking Adversarial Transferability from a Data Distribution Perspective

Yao Zhu, Jiacheng Sun, Zhenguo LiICLR 2022 · OpenReview

TL;DR

This paper reveals that adversarial transferability—the ability of an attack to fool multiple different models—is deeply rooted in the underlying data distribution. By targeting "low-density" regions where models are poorly trained and aligning attack directions with the gradient of the ground truth distribution, the authors introduce the Intrinsic Adversarial Attack (IAA), which significantly outperforms existing state-of-the-art transfer attacks.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are notoriously vulnerable to adversarial examples: small, human-imperceptible perturbations that cause a model to misclassify an image. A particularly dangerous property of these perturbations is transferability, where an adversarial example generated for a "source" model successfully fools a "target" model that the attacker has no access to.

While previous research has largely focused on model architectures or "non-robust features," this paper asks a more fundamental question: Does the data distribution itself dictate transferability? The authors identify a gap in our quantitative understanding of how the location of a sample within the data manifold affects its vulnerability across different models.

Methods: Aligning Attacks with Data Distribution

The authors propose that the most transferable attacks are those that push data toward low-density regions of the ground truth distribution—areas where models haven't seen enough training data to form stable, generalized decision boundaries.

1. The AAI Metric

They introduce a new metric called Alignment between Adversarial attack and Intrinsic attack (AAI).

  • An Intrinsic Attack is defined as the direction that most effectively decreases the ground truth data density ($-\nabla_x \log p_D(x, y)$).
  • AAI measures how well a specific model's adversarial gradient aligns with this "model-agnostic" intrinsic direction.

2. Intrinsic Adversarial Attack (IAA)

To maximize this alignment and generate better attacks, the authors propose two structural modifications to the source model during the attack phase:

  • Activation Smoothing: They replace the standard ReLU activation with Softplus, a smoother approximation. This prevents the "noisy" and discontinuous gradients of ReLU from hiding the true direction toward low-density regions.
  • Layer Weighting: They observe that early layers capture local data statistics better than later layers. IAA uses decay parameters to reduce the influence of later layers, focusing the attack on more fundamental, distribution-relevant features.

The optimal hyperparameters for these modifications (the smoothness of Softplus and the weights of each block) are found using Bayesian optimization to maximize the AAI metric.

Datasets & The Data Density Hypothesis

The primary evaluation is conducted on a randomly selected 5,000-image subset of the ImageNet validation set. To support their hypothesis about data density, they categorize data into:

  • LDD (Low-Density Data): A subset of samples identified as being highly sensitive to random Gaussian noise.
  • HDD (High-Density Data): Samples robust to random noise.

They also utilize CIFAR-10-C and ImageNet-C to test robustness against common corruptions like blur and fog.

Key Insights and Ideas

  • The "Intriguing Phenomenon": The authors discovered that samples sensitive to simple Gaussian noise are "transferability goldmines." Adversarial versions of these low-density samples are much more likely to fool a wide array of target models.
Transferability success rates on Low-Density Data (LDD) versus High-Density Data (HDD).
  • Intrinsic vs. Model-Specific: Most attacks focus on the specific decision boundary of the source model. IAA shifts the focus to the intrinsic geometry of the data distribution, making the resulting perturbations much more universal.
  • Later Layers are "Distractors": In ResNet-style architectures, the final blocks often learn highly model-specific features that don't generalize. By "turning down the volume" on these layers during the attack, the gradient becomes more aligned with the general data distribution.
  • Smoothness is Essential: Discontinuous gradients (like those from ReLU) act as a form of "gradient masking" that hinders transferability. Smoothing the model reveals a clearer path to the low-density regions, leading to a distinct peak in attack success when the optimal smoothness parameter ($\beta$) is found.
Attack success rate across different target models peaks at an optimal activation smoothness parameter (beta).

Key Results

The IAA method achieves a new state-of-the-art in nearly all tested scenarios:

  • Untargeted Attacks: On a ResNet-50 source model attacking a VGG19 target, IAA achieved a 96.40% success rate, compared to just 53.00% for standard PGD and 82.72% for the previous best method (SGM).
  • Targeted Attacks: These are notoriously difficult to transfer. IAA improved the success rate from 0.40% (PGD) to 33.68% when transferring from ResNet-50 to VGG19.
  • Robust Models: IAA is highly effective even against models protected by Ensemble Adversarial Training, outperforming existing methods by significant margins (e.g., a ~31% improvement over PGD on IncV3ens3).
  • Real-World Impact: The authors successfully used IAA to fool commercial image search engines like Google Reverse Image Search and Baidu Reverse Image Search, demonstrating that these vulnerabilities exist in production-grade systems.

Limitations and Future Work

  • Computational Overhead: Finding the optimal structural hyperparameters via Bayesian optimization is computationally expensive (taking about 10 hours for a ResNet-50 model).
  • Estimation of Ground Truth: The method relies on approximating the gradient of the ground truth distribution. While the authors use a clever "sliced score matching" approach to avoid needing the actual density function, more direct or efficient ways to estimate this "intrinsic" direction could be explored.
  • Defense Implications: While the paper focuses on attacks, the insight that low-density regions are the primary source of vulnerability suggests that future defense research should focus on better "filling" or smoothing these regions during training.

Summary updated

Abstract

Adversarial transferability enables attackers to generate adversarial examples from the source model to attack the target model, which has raised security concerns about the deployment of DNNs in practice. In this paper, we rethink adversarial transferability from a data distribution perspective and further enhance transferability by score matching based optimization. We identify that some samples with injecting small Gaussian noise can fool different target models, and their adversarial examples under different source models have much stronger transferability. We hypothesize that these samples are in the low-density region of the ground truth distribution where models are not well trained. To improve the attack success rate of adversarial examples, we match the adversarial attacks with the directions which effectively decrease the ground truth density. We propose Intrinsic Adversarial Attack (IAA), which smooths the activation function and decreases the impact of the later layers of a given normal model, to increase the alignment of adversarial attack and the gradient of joint data distribution. We conduct comprehensive transferable attacks against multiple DNNs and show that our IAA can boost the transferability of the crafted attacks in all cases and go beyond state-of-the-art methods.
BibTeX
@inproceedings{zhu2022rethinking,
  author = {Yao Zhu and Jiacheng Sun and Zhenguo Li},
  title = {{Rethinking Adversarial Transferability from a Data Distribution Perspective}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=gVRhIEajG1k},
}
Figure