TL;DR
The "Battle of the Backbones" (BoB) is a massive benchmarking effort that pits over 1,500 training runs of diverse pretrained vision models against each other. The results show that while transformers and self-supervised learning (SSL) are the current research darlings, supervised convolutional networks like ConvNeXt still reign supreme across most tasks, though SSL models become highly competitive when dataset sizes are equalized.
What Problem Does This Paper Address?
In modern computer vision, the "backbone"—the pretrained feature extractor—is the foundation of almost every system. A few years ago, the choice was simple: use a ResNet trained on ImageNet. Today, practitioners are overwhelmed by a "cornucopia of choices," including Vision Transformers (ViTs), self-supervised models (DINO, MAE), vision-language models (CLIP), and even generative models (Stable Diffusion).
Method papers usually only compare a new model against a few close competitors on a handful of tasks. This makes it nearly impossible for an engineer or researcher to know which backbone is truly the best for a specific application, like medical imaging, satellite classification, or object detection. BoB aims to provide a definitive, fair "gauntlet" to guide these decisions.
Methods
The authors conducted a comprehensive analysis using a ranking protocol based on Z-scores. By calculating how many standard deviations a model's performance sits above or below the mean for a specific task, they were able to aggregate results across wildly different metrics and datasets.
They evaluated backbones using two primary protocols:
- Linear Probing: Freezing the backbone and only training a task-specific head (testing the quality of the raw features).
- End-to-End Fine-tuning: Updating the entire model (testing the model's ultimate potential).
The study covers a diverse range of architectures (ResNet, ConvNeXt, ViT, SwinV2) and pretraining styles (Supervised, SSL, Vision-Language, Generative, and Depth-Estimation).
Datasets
The benchmark spans a massive array of tasks and domains:
- Classification: 6 datasets including natural images (ImageNet-1k, CIFAR-100), satellite imagery (EuroSAT), and medical X-rays (CheXpert).
- Detection & Segmentation: COCO and LVIS.
- Out-of-Distribution (OOD): Robustness tests like ImageNet-A (adversarial), ImageNet-R (renditions), and syn-to-real transfers (VisDA).
- Image Retrieval: 8 datasets ranging from fine-grained bird classification (CUB-200) to landmark recognition.
Key Insights and Ideas
- CNNs are far from dead: Despite the hype surrounding transformers, the ConvNeXt-Base (a modern convolutional architecture) was the overall winner across the entire suite of tasks.
- Spatial bias is a superpower: Hierarchical transformers like SwinV2, which incorporate spatial inductive biases similar to CNNs, significantly outperformed "vanilla" ViTs.
- Performance is highly universal: The authors found a strong positive correlation ($\rho > 0.8$) between performance on different tasks. If a backbone is good at classification, it is likely excellent at detection and OOD generalization as well.
- The SSL "Disadvantage" is Data-Driven: Supervised models often look better simply because they are trained on larger datasets (like ImageNet-21k). In "apples-to-apples" comparisons on the same data scale, SSL models actually outperform their supervised counterparts in classification and retrieval.
- Generative backbones lag behind: Backbones derived from generative models like Stable Diffusion performed poorly compared to discriminative models, suggesting that "reconstruction" isn't yet as effective a pretraining task as "discrimination" for general vision.
Key Results
- The Champions: The top-performing backbones overall were Supervised ConvNeXt-Base (trained on ImageNet-21k), Supervised SwinV2-Base, and CLIP ViT-Base.
- Budget Winners: For those on a tight computational budget, EfficientNet-B0 emerged as the best "tiny" backbone, followed by RegNetX-400MF.
- Fine-tuning Sensitivity: ViTs benefit much more from end-to-end fine-tuning than CNNs, especially on dense prediction tasks like segmentation.
Limitations and Future Work
The study acknowledges a few key boundaries:
- Model Size: The authors capped their study at "Base" size models (except for Stable Diffusion). It is possible that at "Huge" or "Giant" scales, transformers might eventually pull ahead of CNNs.
- Task Vocabulary: While broad, the insights are limited to the specific tasks and datasets chosen.
- SSL Scaling: The results suggest a clear path forward: the community should focus on training SSL models using more advanced architectures (like ConvNeXt or Swin) on much larger datasets to truly unlock their potential.
The authors have released their raw results and code, inviting the community to put new backbones through the same gauntlet at https://github.com/hsouri/Battle-of-the-Backbones.
Summary updated