Does progress on ImageNet transfer to real-world datasets?

Alex Fang, Simon Kornblith, Ludwig Schmidt

Introduction

For over a decade, the computer vision community has treated ImageNet accuracy as the ultimate benchmark. The prevailing wisdom has been that a model performing better on ImageNet will naturally serve as a superior "backbone" for almost any other vision task. While this held true during the transition from early models like AlexNet to ResNet, recent research reveals a "diminishing returns" problem. For highly optimized modern architectures achieving over 70% accuracy on ImageNet, further improvements often fail to translate into meaningful gains on specialized, real-world datasets.

The Problem: Web-Scraped vs. Real-World Data

Standard benchmarks like CIFAR-10 or PASCAL VOC share a similar visual distribution to ImageNet—they are largely composed of "web-scraped" natural images. However, practical applications often rely on datasets collected by domain experts for specific tasks, such as identifying diseases in crops or analyzing satellite imagery. The authors investigate whether the relentless optimization for ImageNet actually transfers to these specialized domains, or if the community is over-optimizing for internet images at the expense of scientific and medical applications.

Datasets and Methodology

To test this, the researchers conducted a massive empirical study evaluating 19 different model architectures—ranging from the 2012 AlexNet to modern Vision Transformers (ViTs) and ConvNexts. They evaluated these models across six diverse, non-web-scraped downstream tasks:

  1. Caltech Camera Traps (CCT-20): Identifying animals in the wild.
  2. APTOS 2019 Blindness: Detecting diabetic retinopathy from retinal images.
  3. Human Protein Atlas (HPA): Classifying protein patterns in microscopic images.
  4. SIIM-ISIC Melanoma: Identifying skin cancer.
  5. Cassava Leaf Disease: Diagnosing plant diseases from phone photos.
  6. EuroSAT: Land use classification from satellite imagery.
Sample images from the six real-world datasets evaluated in the study.

To ensure fair comparisons, the team performed extensive hyperparameter tuning for every model-dataset pair, including grid searches over learning rates, weight decay, and optimizers. They then analyzed the relationship between ImageNet top-1 accuracy and downstream performance using linear regression.

Key Insights: The 70% Saturation Point

The most striking finding is the divergence in transferability between older and newer models.

Scatter plots showing downstream performance versus ImageNet accuracy. The blue line indicates the flattened trajectory for models with >70% ImageNet accuracy.
  • Diminishing Returns: While there is a strong linear trend across the entire history of models (the green line), the gains become noisy and marginal once models exceed 70% ImageNet accuracy (the blue line). On four out of the six datasets, the slope of improvement for these modern models was less than 0.05. This means a massive 10% gain on ImageNet might yield less than a 0.5% gain on the target task.
  • Minimal Gains on Specific Tasks: On the EuroSAT (satellite) and Cassava (leaf disease) datasets, the correlation between recent ImageNet improvements and task accuracy was nearly zero.
  • Exceptions: The Caltech Camera Traps and Human Protein Atlas datasets showed more benefit from better architectures, suggesting that some complex real-world tasks still have room to grow with better backbones.
  • Augmentation Trumps Architecture: In several cases, task-specific interventions provided a larger performance boost than upgrading the model architecture. Adding simple data augmentations (like color jitter or affine transforms) often yielded better results than switching from an older ResNet to a cutting-edge EfficientNet.

Conclusion and Future Work

The study highlights a critical gap in current computer vision research: an over-reliance on ImageNet as a universal proxy for model quality. Because real-world datasets exhibit significant distribution shifts from standard web images, architectural tweaks that boost ImageNet scores do not reliably improve performance in medical, agricultural, or satellite domains.

While the study is limited to RGB image classification and does not explore object detection or non-RGB modalities, its core message is clear. The authors call for the community to move beyond a singular focus on ImageNet and incorporate more diverse, real-world datasets into standard benchmarking suites to encourage the development of truly robust learning algorithms.

Summary updated

Abstract

Does progress on ImageNet transfer to real-world datasets? We investigate this question by evaluating ImageNet pre-trained models with varying accuracy (57% - 83%) on six practical image classification datasets. In particular, we study datasets collected with the goal of solving real-world tasks (e.g., classifying images from camera traps or satellites), as opposed to web-scraped benchmarks collected for comparing models. On multiple datasets, models with higher ImageNet accuracy do not consistently yield performance improvements. For certain tasks, interventions such as data augmentation improve performance even when architectures do not. We hope that future benchmarks will include more diverse datasets to encourage a more comprehensive approach to improving learning algorithms.
BibTeX
@inproceedings{fang2023does,
  author = {Alex Fang and Simon Kornblith and Ludwig Schmidt},
  title = {{Does progress on ImageNet transfer to real-world datasets?}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=SS3CK3yx5Z},
}

Loading related content...

Figure