TL;DR
When we train deep learning models to high accuracy, do they all end up learning the same thing? This paper shows that while standard supervised models share similar biases, changing the training methodology—such as using contrastive learning or different pre-training datasets—produces models with highly uncorrelated errors. This diversity allows for massive ensembling gains and proves that even lower-accuracy models capture unique, valuable features that high-accuracy models miss.
What Problem Does This Paper Address?
In machine learning, it has long been observed that high-accuracy models tend to make the same mistakes. Previous research suggested a "dominance hierarchy," where higher-performing models learn a strict superset of the features discovered by weaker models. If true, this severely limits the utility of ensembling (since models fail on the same examples) and implies that lower-accuracy models are practically useless once a stronger model is available.
The authors of this paper ask a critical question: Is this lack of diversity an inherent property of deep learning, or is it a selection bias resulting from everyone training models the same way (i.e., via supervised learning on ImageNet)?
Methods
To investigate, the authors conducted a massive empirical study analyzing 82 different models. They fixed a standard ResNet-50 (trained with RandAugment) as their base model and categorized how other models diverged from it along five increasingly distinct axes:
- Reinits: Identical architecture and hyperparameters, differing only by random seed initialization.
- Hyperparameters: Same architecture, but varying dropout, learning rate, or weight decay.
- Architectures: Different structures (e.g., ViT, EfficientNet, VGG) trained on the same dataset with the same objective.
- Frameworks: Different optimization objectives on the same dataset (e.g., supervised ResNet vs. contrastive SimCLR).
- Datasets: Models trained on entirely different large-scale datasets (e.g., CLIP or BiT).
Crucially, to isolate the effect of methodology rather than raw performance, the authors restricted the bulk of their analysis to models within a narrow accuracy band (74–78% on ImageNet). They measured Error Inconsistency—the fraction of test examples where only one model in a pair predicts the correct class—to quantify prediction diversity.
Datasets
The models were trained and evaluated using several major benchmarks:
- ImageNet: Used as the primary baseline for in-distribution classification and accuracy control.
- Large-Scale Pre-training Datasets: WIT (WebImageText), JFT-300M, JFT-3B, and the ALIGN dataset.
- Pascal VOC: Used as a downstream multi-label classification task to evaluate how well the learned representations transfer to new domains.
Key Insights and Ideas
- Divergence Breeds Diversity: As training methodologies move from simple reinitialization to entirely different datasets and objectives, error inconsistency skyrockets. The errors become highly uncorrelated.
- Data Subdomain Specialization: Models trained differently naturally become "experts" in distinct subdomains. For example, the contrastive model CLIP-S specializes in anthropogenic (human-made) images, whereas a standard supervised ResNet-50 excels at identifying nature images.
- Overlapping, Not Supersetting: High-accuracy models do not learn a perfect superset of low-accuracy models. Instead, their feature representations overlap. Combining features from diverse models yields a representation that is more expressive yet more efficiently compressible than either model alone.
- The Value of "Weak" Models: A lower-accuracy model is not obsolete if it was trained differently enough. It can still provide unique information to rescue a high-accuracy model's mistakes.
Key Results
- Massive Ensemble Boosts: Ensembling a standard ResNet-50 (76.5% accuracy) with a highly divergent model like ALIGN-ZeroShot (75.5% accuracy) boosted top-1 ImageNet accuracy to 83.4%—a massive +7% absolute gain from two models of similar baseline strength.
- Rescuing the State-of-the-Art: By greedily ensembling a high-accuracy model (BiT-1k, 82.9%) with only lower-accuracy models (none higher than 77.4%), the authors created an ensemble that reached 86.7%.
- Downstream Super-Features: On the Pascal VOC transfer task, concatenating the features of the diverse ViT-G/14 and CLIP-S models achieved 91.36% mAP, outperforming the individual state-of-the-art Meta-Pseudo Labels (MPL) model (89.74% mAP), despite CLIP-S having much lower standalone ImageNet accuracy.
Limitations and Future Work
The most profound open question raised by the authors is a philosophical one: Why didn't standard Stochastic Gradient Descent (SGD) learn these diverse features on its own?
Currently, our training objectives strictly maximize standard classification accuracy, which forces models to exploit the easiest statistical shortcuts available in a dataset. The authors suggest that future work should focus on designing optimization objectives that explicitly reward the discovery of diverse "stepping stones" in feature space, rather than just optimizing for a single metric. Additionally, exploring how these diversity dynamics manifest in modalities outside of image classification (like NLP or audio) remains an open avenue.
Summary updated