Abstract
Does progress on ImageNet transfer to real-world datasets? We investigate this question by evaluating ImageNet pre-trained models with varying accuracy (57% - 83%) on six practical image classification datasets. In particular, we study datasets collected with the goal of solving real-world tasks (e.g., classifying images from camera traps or satellites), as opposed to web-scraped benchmarks collected for comparing models. On multiple datasets, models with higher ImageNet accuracy do not consistently yield performance improvements. For certain tasks, interventions such as data augmentation improve performance even when architectures do not. We hope that future benchmarks will include more diverse datasets to encourage a more comprehensive approach to improving learning algorithms.
Introduction
For over a decade, ImageNet has served as the primary benchmark for computer vision. The underlying assumption in the field is that any architectural advancement or training technique that boosts ImageNet accuracy will inherently produce a "better" model capable of superior performance across all downstream tasks. However, ImageNet is composed of images scraped from the web, and there is growing concern that the community may be over-optimizing for this specific data distribution. This research investigates whether methodological progress on ImageNet actually translates to "real-world" tasks—data collected for specific, practical purposes rather than general internet content.
Methodology
To rigorously test the transferability of ImageNet progress, the researchers established a comprehensive evaluation testbed:
- Model Selection: 19 distinct architectures were evaluated, spanning from early models like AlexNet (57% ImageNet accuracy) to state-of-the-art systems like ConvNext and Vision Transformers (83% accuracy).
- Hyperparameter Optimization: To ensure fair comparison, an extensive grid search was performed over learning rates, weight decays, and optimizers for every model on every dataset.
- Linear Regression Analysis: The core metric for transferability was the slope of the linear regression between ImageNet accuracy and downstream task performance. A high slope indicates strong transfer, while a near-zero slope suggests that ImageNet improvements offer little benefit to the specific task.
Real-World Datasets
The study focused on six practical datasets that represent diverse, non-web-scraped domains:
- Caltech Camera Traps (CCT-20): Wildlife monitoring.
- APTOS 2019 Blindness: Diabetic retinopathy detection.
- Human Protein Atlas (HPA): Microscopic protein pattern classification.
- SIIM-ISIC Melanoma: Skin cancer identification.
- Cassava Leaf Disease: Agricultural disease diagnosis.
- EuroSAT: Satellite-based land use classification.
Results and Analysis
The findings reveal a significant "web-scraped bubble." While ImageNet accuracy remains a strong predictor for other web-scraped benchmarks (like CIFAR-10), the correlation is remarkably weak for real-world datasets.
Key Findings:
- Diminishing Returns: For tasks like EuroSAT and Cassava Leaf Disease, models with 80%+ ImageNet accuracy showed no significant improvement over those with 70% accuracy.
- The 0.05 Slope: On average, the slope of improvement for real-world tasks was less than 0.05. This implies that a substantial 10% increase in ImageNet accuracy might only yield a marginal 0.5% gain in a critical application like melanoma detection.
- Recipe vs. Architecture: Task-specific interventions, such as improved data augmentation or increased training data, often provided larger performance gains than upgrading to a more advanced ImageNet-optimized architecture.
- Pre-training Data Matters: Models like CLIP, trained on diverse image-text pairs rather than just ImageNet, often outperformed ImageNet-pretrained models, suggesting that the diversity of the source data is more critical than the architecture itself.
Discussion and Conclusion
The study highlights a critical limitation in current computer vision research: the specialization of recent innovations to the ImageNet distribution. While early progress (e.g., AlexNet to VGG) transferred broadly, modern gains appear increasingly localized to web-scraped data.
The authors conclude with a call for more diverse benchmarking. To develop learning algorithms that solve meaningful scientific and industrial problems, the community must look beyond ImageNet and incorporate a wider variety of data distributions into the standard evaluation pipeline.
Summary updated