Do Adversarially Robust ImageNet Models Transfer Better?

Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, Aleksander Madry

Conventional wisdom in deep learning suggests that the more accurate a pre-trained model is on a massive dataset like ImageNet, the better it will perform when transferred to a new, downstream task. However, recent research flips that script, demonstrating that adversarially robust models—despite often having lower initial ImageNet accuracy—consistently match or outperform their standard counterparts in transfer learning across classification, object detection, and instance segmentation.

The Problem with Standard Pre-training

Transfer learning is the backbone of modern computer vision. Practitioners typically take a model pre-trained on ImageNet and "fine-tune" it for a specific task, such as identifying bird species or detecting cars. Until now, the primary metric for choosing a "source" model was its top-1 accuracy on ImageNet.

However, ImageNet accuracy doesn't tell the whole story. Standard models are notoriously brittle, often fooled by "adversarial examples"—tiny, imperceptible perturbations to an image that cause catastrophic misclassification. The authors ask a fundamental question: Does the process of making a model robust to these adversarial attacks actually result in a "better" feature extractor for other tasks?

Methodology: Standard vs. Robust Models

The researchers compared standard ImageNet models (trained via Empirical Risk Minimization) against adversarially robust models across a variety of ResNet and WideResNet architectures.

  • Robust Training: These models are trained using a "min-max" optimization objective. During training, the algorithm looks for the worst-case small perturbation (within an $L_2$ ball of radius $\epsilon$) for every image and forces the model to classify it correctly anyway.
  • Transfer Settings: The models were tested under two main paradigms:
  • Fixed-feature: The pre-trained model is frozen, and only a new linear classifier is trained on top of its extracted features.
  • Full-network fine-tuning: Every weight in the network is updated for the new task.

The study utilized a comprehensive suite of benchmarks, including 12 downstream classification datasets (e.g., CIFAR-10/100, Caltech-101/256, Oxford Pets, Flowers), as well as object detection and segmentation tasks on PASCAL VOC and Microsoft COCO.

Key Insights: Why Robustness Helps

Why would a model with lower initial accuracy perform better on downstream tasks? The answer lies in the quality of the features it learns.

Robustness as a Feature Quality Filter: Robust training acts as a powerful prior. It forces the network to ignore "non-robust" high-frequency signals (which humans don't use but standard models rely on heavily) and focus on more fundamental, perceptually relevant shapes and structures. This results in "perceptually aligned gradients" and representations that are more easily invertible back into recognizable images.

Feature visualizations demonstrating that robust models learn perceptually aligned representations, unlike standard models.

The Accuracy-Robustness Trade-off: There is an inherent tension here. Higher ImageNet accuracy does help transfer, but robust training decreases ImageNet accuracy. The paper finds that the qualitative benefits of robust features often outweigh the penalty of lower initial accuracy.

Width is a Robust Model's Best Friend: In standard models, increasing network width often leads to diminishing returns in transfer performance. For robust models, however, increasing width leads to consistent and significant gains in downstream tasks.

The "Granularity" Hypothesis: The optimal level of robustness (the perturbation radius $\epsilon$) depends on the target dataset. Coarse-grained tasks (like CIFAR-10) benefit from high robustness, while fine-grained tasks (like Birdsnap) require a lower $\epsilon$ to preserve the subtle, high-frequency details needed for discrimination.

Key Results

The empirical results strongly support the use of robust models for transfer learning, particularly in fixed-feature settings.

  • Classification: In the fixed-feature setting, robust models outperformed standard models on nearly all 12 tasks. For example, on CIFAR-100, a robust ResNet-50 achieved 81.08% accuracy compared to the standard model's 60.14%.
Transfer learning accuracy across 12 datasets comparing Robust and Standard models.
  • Detection & Segmentation: Robust backbones consistently yielded higher Average Precision (AP). On PASCAL VOC object detection, the best robust backbone reached 53.87 AP, beating the standard backbone's 52.80 AP.
  • Texture vs. Adversarial Robustness: The authors also found that adversarial robustness is a more effective transfer prior than "texture invariance" (achieved by training on Stylized ImageNet), suggesting that adversarial training captures something more fundamental about image structure.

Limitations and Future Work

While promising, this approach is not without its challenges:

  • Computational Cost: Adversarial training is significantly more computationally expensive than standard training, which may limit its adoption for very large-scale foundational models.
  • Hyperparameter Sensitivity: The "optimal" robustness level ($\epsilon$) varies by task. Currently, there is no silver bullet for choosing $\epsilon$ without performing a grid search on the downstream dataset.
  • Theoretical Gap: While the paper provides strong empirical evidence that robust features are "better," the exact mathematical reason why these features generalize more effectively to new distributions remains an open research question.

Future work will likely focus on developing more efficient robust training methods and exploring how to automatically tune robustness levels for specific downstream applications. Ultimately, this research highlights that when it comes to pre-training, how a model learns is just as important as how accurate it is.

Summary updated

Abstract

Transfer learning is a widely-used paradigm in deep learning, where models pre-trained on standard datasets can be efficiently adapted to downstream tasks. Typically, better pre-trained models yield better transfer results, suggesting that initial accuracy is a key aspect of transfer learning performance. In this work, we identify another such aspect: we find that adversarially robust models, while less accurate, often perform better than their standard-trained counterparts when used for transfer learning. Specifically, we focus on adversarially robust ImageNet classifiers, and show that they yield improved accuracy on a standard suite of downstream classification tasks. Further analysis uncovers more differences between robust and standard models in the context of transfer learning. Our results are consistent with (and in fact, add to) recent hypotheses stating that robustness leads to improved feature representations. Code and models is available in the supplementary material.
BibTeX
@inproceedings{salman2020do,
  author = {Hadi Salman and Andrew Ilyas and Logan Engstrom and Ashish Kapoor and Aleksander Madry},
  title = {{Do Adversarially Robust ImageNet Models Transfer Better?}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=VJbAC7LnJb},
}
Figure