Enhance Transferability of Adversarial Examples with Model Architecture

Mingyuan Fan, Wenzhong Guo, Zuobin Ying, Ximeng Liu

TL;DR

Researchers have developed a new proxy model design called Multi-track Model Architecture (MMA) that significantly boosts the "transferability" of adversarial examples. By restructuring neural networks to prioritize shared low-level features over model-specific high-level ones, they achieved up to a 40% increase in attack success rates against black-box models compared to standard architectures.

What Problem Does This Paper Address?

Adversarial attacks—where tiny, invisible perturbations to an image cause a machine learning model to misclassify it—are a major security concern. In a "black-box" scenario, the attacker doesn't know the target model's architecture or weights. To get around this, attackers usually train a "proxy model," craft an attack against it (white-box), and hope the resulting adversarial example "transfers" to the target.

The problem is overfitting. Adversarial examples often become too specialized to the proxy model's unique high-level features. While they might achieve a 100% success rate on the proxy, they often fail miserably (less than 30% success) on the actual target because those high-level features aren't shared across different architectures.

Methods

The authors propose the Multi-track Model Architecture (MMA). Their core insight is that transferability stems from shared low-level features (like edges and basic textures) that most models learn similarly, whereas high-level semantic features are often model-specific.

Key components of MMA include:

  • Decomposition and Reconstruction: Instead of one deep "monolithic" network, MMA splits the architecture into multiple parallel "tracks" or small networks.
  • Multi-Head Output: The architecture features multiple "heads" at different depths. This allows the attacker to generate adversarial examples from various stages of the network.
  • Feature Sharing: Tracks are not entirely independent; networks placed "further back" in the architecture can utilize feature maps from those in the "front," creating a trade-off between learning shared features and introducing model-specific noise.
  • Ensemble Gradient: By crafting attacks based on the ensemble output of these multiple heads, the resulting adversarial noise is guided toward "vulnerable directions" that are common across diverse architectures.

Datasets

The researchers evaluated their approach using the CIFAR-10 dataset, a standard benchmark for image classification. They used three widely-used architectures as both proxy and target models for comparison:

  • MobileNet
  • GoogleNet
  • ResNet18

Key Insights and Ideas

  • The "Sweet Spot" of Depth: The authors found that transferability isn't linear with depth. In their tests, a depth of 3 blocks was the "crest." Increasing depth beyond that actually damaged transferability because the model began learning too many exclusive, high-level features that the target model didn't share.
  • Training Time Matters: Interestingly, Attack Success Rate (ASR) peaks early (around epoch 10) and then fluctuates or declines. This suggests that as a model "over-trains" and becomes more accurate, it becomes a worse proxy for transfer attacks because it has specialized too much.
  • Architecture as a Regularizer: Most previous work focused on better optimizers or loss functions. This paper argues that the architecture itself can act as a form of regularization to prevent adversarial overfitting.

Key Results

The MMA approach showed dominant performance across both single-step (FGSM) and multi-step (BIM) attacks:

  • Single-Step Attacks: MMA improved transferability by 2.32% to 17.96% over baseline models.
  • Multi-Step Attacks: The improvements were even more dramatic, with MMA achieving a maximum of 79.79% Attack Success Rate (ASR)—representing an improvement of up to 40% over standard state-of-the-art architectures.
  • Efficiency: Despite the "multi-track" nature, the authors demonstrated that the computational overhead (forward and backward propagation time) is comparable to standard baseline models, making it a practical choice for attackers.

Limitations and Future Work

  • Dataset Scale: The study was limited to CIFAR-10. It remains to be seen how well MMA scales to higher-resolution datasets like ImageNet, where high-level features are more complex.
  • Target Diversity: While they tested against three common architectures, the "black-box" world includes many more types of defenses (like adversarial training or input transformation) that weren't explicitly tested here.
  • Integration: The authors note that MMA is a "generic" technology. Future work could involve combining MMA with other transfer-boosting techniques, such as input diversity or advanced loss functions, to see if the gains are additive.

Summary updated

Abstract

Transferability of adversarial examples is of critical importance to launch black-box adversarial attacks, where attackers are only allowed to access the output of the target model. However, under such a challenging but practical setting, the crafted adversarial examples are always prone to overfitting to the proxy model employed, presenting poor transferability. In this paper, we suggest alleviating the overfitting issue from a novel perspective, i.e., designing a fitted model architecture. Specifically, delving the bottom of the cause of poor transferability, we arguably decompose and reconstruct the existing model architecture into an effective model architecture, namely multi-track model architecture (MMA). The adversarial examples crafted on the MMA can maximumly relieve the effect of model-specified features to it and toward the vulnerable directions adopted by diverse architectures. Extensive experimental evaluation demonstrates that the transferability of adversarial examples based on the MMA significantly surpass other state-of-the-art model architectures by up to 40% with comparable overhead.
BibTeX
@inproceedings{fan2023enhance,
  author = {Mingyuan Fan and Wenzhong Guo and Zuobin Ying and Ximeng Liu},
  title = {{Enhance Transferability of Adversarial Examples with Model Architecture}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10097192},
}

Loading related content...

Figure