Introducing Foundation Models as Surrogate Models: Advancing Towards More Practical Adversarial Attacks

Jiaming Zhang, Jitao Sang, Qi Yi, Changsheng XuarXiv 2023 · arXiv · OpenReview

Abstract

Recently, the no-box adversarial attack, in which the attacker lacks access to the model's architecture, weights, and training data, become the most practical and challenging attack setup. However, there is an unawareness of the potential and flexibility inherent in the surrogate model selection process on no-box setting. Inspired by the burgeoning interest in utilizing foundational models to address downstream tasks, this paper adopts an innovative idea that 1) recasting adversarial attack as a downstream task. Specifically, image noise generation to meet the emerging trend and 2) introducing foundational models as surrogate models. Harnessing the concept of non-robust features, we elaborate on two guiding principles for surrogate model selection to explain why the foundational model is an optimal choice for this role. However, paradoxically, we observe that these foundational models underperform. Analyzing this unexpected behavior within the feature space, we attribute the lackluster performance of foundational models (e.g., CLIP) to their significant representational capacity and, conversely, their lack of discriminative prowess. To mitigate this issue, we propose the use of a margin-based loss strategy for the fine-tuning of foundational models on target images. The experimental results verify that our approach, which employs the basic Fast Gradient Sign Method (FGSM) attack algorithm, outstrips the performance of other, more convoluted algorithms. We conclude by advocating for the research community to consider surrogate models as crucial determinants in the effectiveness of adversarial attacks in no-box settings.

Introduction: The Challenge of No-Box Attacks

In the field of adversarial machine learning, the "no-box" attack represents the most realistic yet difficult scenario. Unlike white-box or black-box attacks, a no-box attacker has zero knowledge of the target model’s architecture, parameters, or training data. To succeed, attackers typically employ a "surrogate model" to craft adversarial examples, relying on the transferability of these perturbations to the unknown target.

While previous research has focused heavily on refining attack algorithms, the choice and optimization of the surrogate model itself have been largely overlooked. This work identifies the surrogate model as the primary bottleneck and proposes leveraging large-scale foundation models, specifically CLIP, to overcome these limitations.

The MF-CLIP Framework

The proposed framework, MF-CLIP (Margin-based Fine-tuned CLIP), addresses the inherent lack of discriminative power in general-purpose foundation models. While models like CLIP possess vast representational knowledge, their internal feature spaces are often too compressed to provide the sharp decision boundaries necessary for generating potent adversarial noise.

The two-stage MF-CLIP framework: Stage 1 involves margin-aware fine-tuning of the surrogate model, and Stage 2 focuses on adversarial example generation using a U-Net generator.

The framework operates in two distinct stages:

  1. Margin-Aware Fine-Tuning: To enhance discriminative prowess, CLIP is fine-tuned using an Additive Angular Margin Loss. This process explicitly increases the distance between different class representations in the feature space, creating more exploitable decision boundaries.
  2. Adversarial Example Generation: Instead of relying on slow iterative methods, MF-CLIP utilizes a U-Net generator. This generator is trained to produce subtle, imperceptible perturbations that maximize the surrogate model's error. By leveraging the refined boundaries of the fine-tuned surrogate, the generated noise is significantly more likely to transfer to unseen target models.

Key Insights: Representation vs. Discrimination

A central finding of this research is that superior representation does not automatically equate to effective adversarial discrimination. "Vanilla" CLIP understands a wide array of concepts but fails as a surrogate because its internal class boundaries are "soft." By widening the angular margin, the adversarial perturbations target "non-robust features" that are more universal across different neural architectures. This explains why MF-CLIP demonstrates exceptional cross-architecture lethality, even doubling success rates against Vision Transformers (ViTs) compared to previous state-of-the-art methods.

Experimental Results and Evaluation

The efficacy of MF-CLIP was validated across seven diverse datasets, including fine-grained objects (Oxford Pets, Flowers102, Stanford Cars, Food101) and general scenes or actions (SUN397, EuroSAT, UCF101). The attacks were tested against various target architectures, including EfficientNet-B0, RegNetX, and ResNet-18.

Radar plot illustrating the superior performance of MF-CLIP across seven diverse datasets compared to existing state-of-the-art no-box attack methods.

Key results include:

  • Significant Performance Gains: MF-CLIP outperformed the best baselines by an average of 15.23% on standard models and 9.52% on adversarially trained models.
  • Robustness Across Domains: As shown in the radar plot, MF-CLIP consistently achieves higher success rates across all tested datasets, forming a nearly perfect outer ring of dominance.
  • Computational Efficiency: The U-Net generator approach is significantly faster and requires less memory than traditional iterative input augmentation attacks, making it highly practical for real-world scenarios.

Conclusion

This work demonstrates that the selection and fine-tuning of surrogate models are critical factors in the success of no-box adversarial attacks. By introducing foundation models as surrogates and applying margin-based optimization, MF-CLIP sets a new benchmark for attack effectiveness and transferability. Future work could explore extending this margin-optimization strategy to other modalities, such as audio or video, and investigating whether similar principles can be used to develop more robust defensive mechanisms.

Summary updated

Abstract

Recently, the no-box adversarial attack, in which the attacker lacks access to the model's architecture, weights, and training data, become the most practical and challenging attack setup. However, there is an unawareness of the potential and flexibility inherent in the surrogate model selection process on no-box setting. Inspired by the burgeoning interest in utilizing foundational models to address downstream tasks, this paper adopts an innovative idea that 1) recasting adversarial attack as a downstream task. Specifically, image noise generation to meet the emerging trend and 2) introducing foundational models as surrogate models. Harnessing the concept of non-robust features, we elaborate on two guiding principles for surrogate model selection to explain why the foundational model is an optimal choice for this role. However, paradoxically, we observe that these foundational models underperform. Analyzing this unexpected behavior within the feature space, we attribute the lackluster performance of foundational models (e.g., CLIP) to their significant representational capacity and, conversely, their lack of discriminative prowess. To mitigate this issue, we propose the use of a margin-based loss strategy for the fine-tuning of foundational models on target images. The experimental results verify that our approach, which employs the basic Fast Gradient Sign Method (FGSM) attack algorithm, outstrips the performance of other, more convoluted algorithms. We conclude by advocating for the research community to consider surrogate models as crucial determinants in the effectiveness of adversarial attacks in no-box settings. The implications of our work bear relevance for improving the efficacy of such adversarial attacks and the overall robustness of AI systems.
BibTeX
@misc{zhang2023introducing,
  author = {Jiaming Zhang and Jitao Sang and Qi Yi and Changsheng Xu},
  title = {{Introducing Foundation Models as Surrogate Models: Advancing Towards More Practical Adversarial Attacks}},
  year = {2023},
  eprint = {2307.06608},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.06608},
}
Figure