Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks

Micah Goldblum, Hossein Souri, Renkun Ni, Manli Shu, Viraj Prabhu, Gowthami Somepalli, Prithvijit Chattopadhyay, Mark Ibrahim, Adrien Bardes, Judy Hoffman, Rama Chellappa, Andrew Gordon Wilson, Tom Goldstein

TL;DR

This paper, "Battle of the Backbones," provides a comprehensive benchmark of over 1500 pretrained models across a wide array of computer vision tasks. The key takeaway is that despite the rise of Vision Transformers (ViTs) and self-supervised learning (SSL), large-scale supervised pretraining on convolutional networks still holds the performance crown for most tasks, though SSL shows strong competitiveness when compared on equal footing.

What Problem Does This Paper Address?

Building modern computer vision systems often relies on using a "backbone" – a pre-trained feature extractor – that is then fine-tuned for a specific task. The landscape of available backbones has exploded, with models trained using various algorithms (supervised, self-supervised, vision-language) and on diverse datasets. This abundance of choice, while beneficial for performance, makes it incredibly difficult for practitioners to select the best backbone for their needs. There's a lack of a unified, large-scale comparison that directly pits these diverse models against each other across a broad spectrum of tasks. This paper aims to fill that gap by providing a systematic benchmark to guide both practitioners and researchers.

Methods

The authors conducted a "Battle of the Backbones" (BoB) by evaluating a diverse suite of publicly available pretrained models. This included:

  • Architectures: ResNet, ConvNeXt, Vision Transformer (ViT), Swin Transformer, and Stable Diffusion encoder.
  • Pretraining Algorithms: Supervised learning (on ImageNet-1k and ImageNet-21k), Self-Supervised Learning (SSL) methods like MoCo v3, VICReg, DINO, and MAE, and Vision-Language models like CLIP.
  • Tasks: The evaluation spanned image classification (natural, medical, satellite), object detection and segmentation, out-of-distribution (OOD) generalization, and image retrieval.
  • Evaluation Protocols: They used both end-to-end fine-tuning and linear probing (where only a linear classifier is trained on top of frozen backbone features).

To ensure fair comparisons, they performed moderate hyperparameter sweeps and used identically sized sweeps for each backbone on downstream tasks. They also considered "apples-to-apples" comparisons where possible, especially for SSL methods on similar dataset scales.

Datasets

The paper utilized a wide range of datasets for both pretraining and evaluation:

Pretraining Datasets: * ImageNet-1k and ImageNet-21k (for supervised and SSL pretraining) * LAION-2B (for CLIP and Stable Diffusion) * MiDaS (for depth estimation pretraining)

Evaluation Datasets: * Classification: ImageNet-1k, CIFAR-100, EuroSAT, Flowers-102, Chexpert. * Object Detection & Segmentation: COCO, LVIS v1. * Out-of-Distribution Generalization: ImageNet-A, ImageNet-V2, ImageNet-R, ImageNet-S, VisDA (synthetic-to-real), Sim10k (for object detection). * Image Retrieval: Oxford, Paris, INSTRE, CUB-200, iNaturalist, Objectnet, Google Landmarks v2, INRIA Copydays.

The paper notes that for some datasets (EuroSAT, Chexpert), only 10% of the labeled data was used to distinguish performance.

Key Insights and Ideas

  • Supervised Dominance (with a caveat): Despite the hype around ViTs and SSL, large-scale supervised pretraining on convolutional networks (like ConvNeXt and SwinV2) still leads in performance across most tasks. This is largely attributed to the larger datasets used for supervised pretraining.
  • SSL Competitiveness: When comparing SSL backbones against supervised ones on similarly sized pretraining datasets, SSL models become highly competitive and can even outperform supervised counterparts. This suggests that the future of SSL lies in using advanced architectures and larger datasets.
  • ViTs and Scale: Vision Transformers (ViTs) seem to benefit more significantly from scale (both in terms of pretraining data and model size) compared to CNNs. This implies that at very large scales, ViTs might eventually surpass CNNs.
  • Task Correlation: Performance across different tasks is highly correlated, suggesting a trend towards general-purpose "foundation models" for computer vision.
  • Speed vs. Performance: There's a strong negative correlation between throughput (speed) and average performance, meaning achieving higher performance often comes at the cost of speed.
  • Monocular Depth Estimation as Pretraining: The MiDaS model, trained for monocular depth estimation, showed surprisingly competitive performance across various tasks, suggesting depth estimation could be a powerful general-purpose pretraining strategy.

Key Results

  • Top Performers: For most tasks, supervised ConvNeXt-Base (trained on ImageNet-21k) and supervised SwinV2-Base (trained on ImageNet-21k, fine-tuned on ImageNet-1k) emerged as top performers. CLIP ViT-Base also performed very strongly, especially in vision-language tasks.
  • Classification: Supervised SwinV2-Base trained on IN-21k (finetuned on IN-1k) was the best, followed by CLIP ViT-Base and Supervised ConvNeXt-Base (IN-21k).
  • Object Detection & Segmentation: Supervised ConvNeXt-Base (IN-21k) outperformed others.
  • Image Retrieval: Supervised ConvNeXt-Base (IN-21k) was the top choice.
  • OOD Classification: Supervised ConvNeXt-Base (IN-21k) led the pack.
  • Small Backbones: For practitioners on a tight budget, ConvNeXt-Tiny and SwinV2-Tiny were strong contenders, with DINO ViT-Small also performing well.
  • Adversarial Robustness: Supervised models generally showed better adversarial robustness than SSL models, and ConvNeXt was noted as being particularly robust.

Limitations and Future Work

  • Scale: The benchmark did not include backbones larger than ConvNeXt-Base (except for Stable Diffusion), and it's acknowledged that larger models, particularly transformers, might shift the rankings.
  • Task/Backbone Vocabulary: The insights are contingent on the specific tasks and backbones evaluated. As new models and tasks emerge, these findings will need to evolve.
  • Focus on Performance: The study primarily focused on performance metrics, with less emphasis on other aspects like model biases.
  • Future Work: The authors suggest that future research should focus on advanced architectures for SSL, larger pretraining datasets, and exploring the potential of vision transformers at scale. They also highlight the need for continuous benchmarking as the field evolves. The release of their code and raw results aims to facilitate this ongoing research.

Summary updated

Abstract

Neural network based computer vision systems are typically built on a backbone, a pretrained or randomly initialized feature extractor. Several years ago, the default option was an ImageNet-trained convolutional neural network. However, the recent past has seen the emergence of countless backbones pretrained using various algorithms and datasets. While this abundance of choice has led to performance increases for a range of systems, it is difficult for practitioners to make informed decisions about which backbone to choose. Battle of the Backbones (BoB) makes this choice easier by benchmarking a diverse suite of pretrained models, including vision-language models, those trained via self-supervised learning, and the Stable Diffusion backbone, across a diverse set of computer vision tasks ranging from classification to object detection to OOD generalization and more. Furthermore, BoB sheds light on promising directions for the research community to advance computer vision by illuminating strengths and weakness of existing approaches through a comprehensive analysis conducted on more than 1500 training runs. While vision transformers (ViTs) and self-supervised learning (SSL) are increasingly popular, we find that convolutional neural networks pretrained in a supervised fashion on large training sets still perform best on most tasks among the models we consider. Moreover, in apples-to-apples comparisons on the same architectures and similarly sized pretraining datasets, we find that SSL backbones are highly competitive, indicating that future works should perform SSL pretraining with advanced architectures and larger pretraining datasets. We release the raw results of our experiments along with code that allows researchers to put their own backbones through the gauntlet here: https://github.com/hsouri/Battle-of-the-Backbones
BibTeX
@inproceedings{goldblum2023battle,
  author = {Micah Goldblum and Hossein Souri and Renkun Ni and Manli Shu and Viraj Prabhu and Gowthami Somepalli and Prithvijit Chattopadhyay and Mark Ibrahim and Adrien Bardes and Judy Hoffman and Rama Chellappa and Andrew Gordon Wilson and Tom Goldstein},
  title = {{Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=1yOnfDpkVe},
}
Figure