Abstract
Transfer learning is a widely-used paradigm in deep learning, where models pre-trained on standard datasets can be efficiently adapted to downstream tasks. Typically, better pre-trained models yield better transfer results, suggesting that initial accuracy is a key aspect of transfer learning performance. In this work, we identify another such aspect: we find that adversarially robust models, while less accurate, often perform better than their standard-trained counterparts when used for transfer learning. Specifically, we focus on adversarially robust ImageNet classifiers, and show that they yield improved accuracy on a standard suite of downstream classification tasks. Further analysis uncovers more differences between robust and standard models in the context of transfer learning. Our results are consistent with (and in fact, add to) recent hypotheses stating that robustness leads to improved feature representations.
1. Introduction
Transfer learning has become the standard operating procedure in computer vision. The typical workflow involves taking a model pre-trained on a massive dataset like ImageNet and adapting its knowledge to a smaller, specific task. Conventional wisdom suggests that higher accuracy on the source dataset (ImageNet) translates directly to better performance on target tasks. However, this research investigates whether accuracy is the sole determinant of transfer success. Specifically, the authors explore adversarial robustness—a model's invariance to small, imperceptible input perturbations—as a potential "prior" that might improve the quality and transferability of learned features.
2. Methodology
The study compares standard ImageNet models against adversarially robust versions across several architectures, including ResNet-18, ResNet-50, and WideResNet-50.
2.1 Robust Training
Instead of standard Empirical Risk Minimization (ERM), the authors employ a robust optimization objective. This min-max problem forces the model to minimize the worst-case loss within an $\epsilon$-sized ball around each training point, effectively training the model to ignore noise that does not change human perception.
2.2 Transfer Paradigms and Evaluation
The models were evaluated using two primary transfer methods:
- Fixed-feature: The pre-trained model is used as a frozen feature extractor, with only a new linear classifier trained for the target task.
- Full-network: The entire model is fine-tuned on the target task.
Performance was tested across a diverse suite of 12 classification datasets (e.g., CIFAR-10/100, Caltech-101/256, Oxford Flowers/Pets) as well as object detection (PASCAL VOC) and instance segmentation (MS COCO) tasks.
3. Key Insights: The Robustness Advantage
The core finding is that despite having lower "clean" accuracy on ImageNet, robust models consistently match or outperform standard models on downstream tasks. This advantage is particularly pronounced in the fixed-feature setting, suggesting that the internal representations of robust models are inherently more versatile.
3.1 Perceptual Alignment
A key reason for this success is that robust models exhibit "perceptually aligned" gradients. Visualizing the features learned by robust models reveals shapes and textures that are recognizable to humans, whereas standard models often produce grainy, unintelligible noise.
3.2 Accuracy vs. Robustness
The authors hypothesize that transfer performance is influenced by two separate "levers": accuracy and robustness. While standard models rely solely on high accuracy, robust models benefit from a "robustness gain" that often outweighs the loss in initial accuracy. Furthermore, robust models scale better with width; while standard models show diminishing returns, wider robust models continue to show significant transfer improvements.
4. Experimental Results
- Fixed-Feature Performance: On CIFAR-100, a robust ResNet-50 achieved 81.16% accuracy, a massive improvement over the 60.08% achieved by the standard version.
- Fine-Tuning: Robust models maintained a competitive edge or matched state-of-the-art results even when fine-tuned, despite starting from lower ImageNet baselines.
- Detection & Segmentation: Using a robust backbone for Faster R-CNN on PASCAL VOC improved Box AP from 52.80 to 53.87, with similar gains observed across COCO tasks.
5. Discussion and Limitations
While the empirical results are compelling, several challenges remain:
- Hyperparameter Sensitivity: The optimal level of robustness ($\epsilon$) varies by dataset and is likely tied to image resolution and "granularity." There is currently no universal formula for selecting the best $\epsilon$.
- Computational Cost: Adversarial training is significantly more resource-intensive than standard training, which may hinder its application to extremely large-scale models.
- Theoretical Understanding: A rigorous theoretical framework explaining why robustness improves feature transfer is still needed. Current hypotheses point toward "feature purification," but further investigation is required.
Summary updated