TL;DR
This paper challenges the conventional wisdom that more robust models lead to better adversarial transferability. Instead, it proposes a "Very Non-Robust Feature" (VNRF) hypothesis, suggesting that these specific features, while exploitable for attacks, actually harm transferability. The authors demonstrate that early stopping training and light adversarial training, which prune these VNRFs, significantly improve adversarial transferability.
What Problem Does This Paper Address?
Transferable Black-box Attacks (TBAs) are a significant security concern, allowing attackers to craft adversarial examples (AEs) on a surrogate model and then use them against a target model without direct access. A key component of TBAs is the transferability of AEs – how well they generalize across different models. Existing research largely attributes this transferability to "non-robust features" (NRFs) learned by models. The prevailing assumption is that longer training, which supposedly learns more NRFs, should lead to better transferability. However, this paper observes a counter-intuitive phenomenon: models trained for fewer epochs (early stopped) exhibit higher adversarial transferability, even though they are less accurate. This discrepancy suggests a more nuanced understanding of feature robustness and its impact on transferability is needed.
Methods
The core of this work lies in proposing and verifying the "Very Non-Robust Feature" (VNRF) hypothesis.
- VNRF Hypothesis: While NRFs are generally beneficial for transferability, "Very Non-Robust Features" (VNRFs) – features that are highly sensitive to even small perturbations – are hypothesized to harm transferability.
- Early Stopping: The authors train models and save checkpoints at various epochs. They then evaluate the transferability of AEs generated from these early-stopped models compared to fully trained models.
- Weight Filtering (L1-norm): To identify VNRFs, they leverage the L1-norm of convolutional filter weights. Filters with high L1-norms are considered more sensitive to input perturbations and thus potential VNRFs. They experiment with zeroing out these filters to see if it improves transferability.
- Light Adversarial Training: Motivated by the VNRF hypothesis, they propose "light" adversarial training. This technique aims to remove VNRFs while preserving more robust features, contrasting with "heavy" adversarial training that might remove all NRFs, potentially harming performance.
Datasets
The experiments were conducted on two standard image classification datasets:
- CIFAR-10: Used for training surrogate models and evaluating transferability.
- ImageNet: Used for evaluating transferability, often with pre-trained models.
The paper also mentions using a dataset from the NeurIPS 2017 adversarial challenge for ImageNet evaluation.
Key Insights and Ideas
- Early Stopping is Key: The most striking finding is that models trained for fewer epochs (early stopped) yield significantly better adversarial transferability than fully trained models, despite lower accuracy. This directly contradicts the idea that more training always equals better transferability.
- VNRFs are the Culprits: The paper introduces the VNRF hypothesis, proposing that not all non-robust features are equal. "Very Non-Robust Features" are identified as detrimental to transferability, likely because they are too specific to the surrogate model and don't generalize.
- Targeted Removal of VNRFs: The L1-norm filtering experiment provides empirical evidence for the VNRF hypothesis. Zeroing out filters with high L1-norms (suspected VNRFs) demonstrably improves transferability.
- Light Adversarial Training for the Win: This insight leads to a practical technique: light adversarial training. By carefully tuning the adversarial training process, they can remove VNRFs while retaining beneficial NRFs, leading to substantial improvements in transferability.
Key Results
- Early Stopping Boost: On CIFAR-10, early stopping improved transfer rates to over 98% for models like VGG16 and up to 99.9% for ResNet18. On ImageNet, early stopping also non-trivially improved transferability.
- VNRF Hypothesis Verification: Zeroing out filters with high L1-norms consistently improved transferability on CIFAR-10.
- Light Adversarial Training Effectiveness:
- On ImageNet, light adversarial training (with a small perturbation budget, e.g., $\epsilon=0.05$) improved transfer rates from 76.4% to 96.7% for I-FGSM on ResNet18.
- For non-CNN targets like ViT-B/16, transfer rates increased from 19.4% to 87.4% with light adversarial training.
- In the more challenging targeted setting, transfer rates increased from 0.4% to 38.4% for I-FGSM on ResNet18.
Limitations and Future Work
- Early Stop vs. Adversarial Training: The paper notes that early stopping's benefit on transferability is less pronounced for adversarially trained models, as they already focus on robust features.
- VNRF Definition Nuance: While the L1-norm filtering provides evidence, a more precise definition and identification of VNRFs could be explored.
- Transfer Learning Connection: The authors suggest that their findings on VNRFs harming transferability might also shed light on similar phenomena observed in transfer learning to new datasets, leaving this as a future research direction.
- Generalizability of Techniques: While effective on common architectures, further investigation into the generalizability of early stopping and light adversarial training across a wider range of models and tasks would be valuable.
Summary updated