Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks

Micah Goldblum, Hossein Souri, Renkun Ni, Manli Shu, Viraj Prabhu, Gowthami Somepalli, Prithvijit Chattopadhyay, Mark Ibrahim, Adrien Bardes, Judy Hoffman, Rama Chellappa, Andrew Gordon Wilson, Tom Goldstein

TL;DR

The "Battle of the Backbones" (BoB) is a massive benchmarking effort that pits over 1,500 training runs of diverse pretrained vision models against each other. The results show that while transformers and self-supervised learning (SSL) are the current research darlings, supervised convolutional networks like ConvNeXt still reign supreme across most tasks, though SSL models become highly competitive when dataset sizes are equalized.

What Problem Does This Paper Address?

In modern computer vision, the "backbone"—the pretrained feature extractor—is the foundation of almost every system. A few years ago, the choice was simple: use a ResNet trained on ImageNet. Today, practitioners are overwhelmed by a "cornucopia of choices," including Vision Transformers (ViTs), self-supervised models (DINO, MAE), vision-language models (CLIP), and even generative models (Stable Diffusion).

Method papers usually only compare a new model against a few close competitors on a handful of tasks. This makes it nearly impossible for an engineer or researcher to know which backbone is truly the best for a specific application, like medical imaging, satellite classification, or object detection. BoB aims to provide a definitive, fair "gauntlet" to guide these decisions.

Methods

The authors conducted a comprehensive analysis using a ranking protocol based on Z-scores. By calculating how many standard deviations a model's performance sits above or below the mean for a specific task, they were able to aggregate results across wildly different metrics and datasets.

They evaluated backbones using two primary protocols:

  1. Linear Probing: Freezing the backbone and only training a task-specific head (testing the quality of the raw features).
  2. End-to-End Fine-tuning: Updating the entire model (testing the model's ultimate potential).

The study covers a diverse range of architectures (ResNet, ConvNeXt, ViT, SwinV2) and pretraining styles (Supervised, SSL, Vision-Language, Generative, and Depth-Estimation).

Datasets

The benchmark spans a massive array of tasks and domains:

  • Classification: 6 datasets including natural images (ImageNet-1k, CIFAR-100), satellite imagery (EuroSAT), and medical X-rays (CheXpert).
  • Detection & Segmentation: COCO and LVIS.
  • Out-of-Distribution (OOD): Robustness tests like ImageNet-A (adversarial), ImageNet-R (renditions), and syn-to-real transfers (VisDA).
  • Image Retrieval: 8 datasets ranging from fine-grained bird classification (CUB-200) to landmark recognition.

Key Insights and Ideas

  • CNNs are far from dead: Despite the hype surrounding transformers, the ConvNeXt-Base (a modern convolutional architecture) was the overall winner across the entire suite of tasks.
  • Spatial bias is a superpower: Hierarchical transformers like SwinV2, which incorporate spatial inductive biases similar to CNNs, significantly outperformed "vanilla" ViTs.
  • Performance is highly universal: The authors found a strong positive correlation ($\rho > 0.8$) between performance on different tasks. If a backbone is good at classification, it is likely excellent at detection and OOD generalization as well.
Figure 1: Performance is correlated across tasks. Performance for each model is reported in terms of standard deviations above/below the mean averages across datasets. Left: Comparison between classification and detection. Right: Comparison between classification and OOD classification.
  • The SSL "Disadvantage" is Data-Driven: Supervised models often look better simply because they are trained on larger datasets (like ImageNet-21k). In "apples-to-apples" comparisons on the same data scale, SSL models actually outperform their supervised counterparts in classification and retrieval.
  • Generative backbones lag behind: Backbones derived from generative models like Stable Diffusion performed poorly compared to discriminative models, suggesting that "reconstruction" isn't yet as effective a pretraining task as "discrimination" for general vision.

Key Results

  • The Champions: The top-performing backbones overall were Supervised ConvNeXt-Base (trained on ImageNet-21k), Supervised SwinV2-Base, and CLIP ViT-Base.
  • Budget Winners: For those on a tight computational budget, EfficientNet-B0 emerged as the best "tiny" backbone, followed by RegNetX-400MF.
  • Fine-tuning Sensitivity: ViTs benefit much more from end-to-end fine-tuning than CNNs, especially on dense prediction tasks like segmentation.
Transformers show a much larger performance jump when moving from frozen features to end-to-end fine-tuning compared to CNNs.

Limitations and Future Work

The study acknowledges a few key boundaries:

  • Model Size: The authors capped their study at "Base" size models (except for Stable Diffusion). It is possible that at "Huge" or "Giant" scales, transformers might eventually pull ahead of CNNs.
  • Task Vocabulary: While broad, the insights are limited to the specific tasks and datasets chosen.
  • SSL Scaling: The results suggest a clear path forward: the community should focus on training SSL models using more advanced architectures (like ConvNeXt or Swin) on much larger datasets to truly unlock their potential.

The authors have released their raw results and code, inviting the community to put new backbones through the same gauntlet at https://github.com/hsouri/Battle-of-the-Backbones.

Summary updated

Abstract

Neural network based computer vision systems are typically built on a backbone, a pretrained or randomly initialized feature extractor. Several years ago, the default option was an ImageNet-trained convolutional neural network. However, the recent past has seen the emergence of countless backbones pretrained using various algorithms and datasets. While this abundance of choice has led to performance increases for a range of systems, it is difficult for practitioners to make informed decisions about which backbone to choose. Battle of the Backbones (BoB) makes this choice easier by benchmarking a diverse suite of pretrained models, including vision-language models, those trained via self-supervised learning, and the Stable Diffusion backbone, across a diverse set of computer vision tasks ranging from classification to object detection to OOD generalization and more. Furthermore, BoB sheds light on promising directions for the research community to advance computer vision by illuminating strengths and weakness of existing approaches through a comprehensive analysis conducted on more than 1500 training runs. While vision transformers (ViTs) and self-supervised learning (SSL) are increasingly popular, we find that convolutional neural networks pretrained in a supervised fashion on large training sets still perform best on most tasks among the models we consider. Moreover, in apples-to-apples comparisons on the same architectures and similarly sized pretraining datasets, we find that SSL backbones are highly competitive, indicating that future works should perform SSL pretraining with advanced architectures and larger pretraining datasets. We release the raw results of our experiments along with code that allows researchers to put their own backbones through the gauntlet here: https://github.com/hsouri/Battle-of-the-Backbones
BibTeX
@inproceedings{goldblum2023battle,
  author = {Micah Goldblum and Hossein Souri and Renkun Ni and Manli Shu and Viraj Prabhu and Gowthami Somepalli and Prithvijit Chattopadhyay and Mark Ibrahim and Adrien Bardes and Judy Hoffman and Rama Chellappa and Andrew Gordon Wilson and Tom Goldstein},
  title = {{Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=1yOnfDpkVe},
}
Figure