No One Representation to Rule Them All: Overlapping Features of Training Methods

Raphael Gontijo-Lopes, Yann Dauphin, Ekin D. Cubuk

TL;DR

When we train deep learning models to high accuracy, do they all end up learning the same thing? This paper shows that while standard supervised models share similar biases, changing the training methodology—such as using contrastive learning or different pre-training datasets—produces models with highly uncorrelated errors. This diversity allows for massive ensembling gains and proves that even lower-accuracy models capture unique, valuable features that high-accuracy models miss.

What Problem Does This Paper Address?

In machine learning, it has long been observed that high-accuracy models tend to make the same mistakes. Previous research suggested a "dominance hierarchy," where higher-performing models learn a strict superset of the features discovered by weaker models. If true, this severely limits the utility of ensembling (since models fail on the same examples) and implies that lower-accuracy models are practically useless once a stronger model is available.

The authors of this paper ask a critical question: Is this lack of diversity an inherent property of deep learning, or is it a selection bias resulting from everyone training models the same way (i.e., via supervised learning on ImageNet)?

Methods

To investigate, the authors conducted a massive empirical study analyzing 82 different models. They fixed a standard ResNet-50 (trained with RandAugment) as their base model and categorized how other models diverged from it along five increasingly distinct axes:

  1. Reinits: Identical architecture and hyperparameters, differing only by random seed initialization.
  2. Hyperparameters: Same architecture, but varying dropout, learning rate, or weight decay.
  3. Architectures: Different structures (e.g., ViT, EfficientNet, VGG) trained on the same dataset with the same objective.
  4. Frameworks: Different optimization objectives on the same dataset (e.g., supervised ResNet vs. contrastive SimCLR).
  5. Datasets: Models trained on entirely different large-scale datasets (e.g., CLIP or BiT).

Crucially, to isolate the effect of methodology rather than raw performance, the authors restricted the bulk of their analysis to models within a narrow accuracy band (74–78% on ImageNet). They measured Error Inconsistency—the fraction of test examples where only one model in a pair predicts the correct class—to quantify prediction diversity.

Datasets

The models were trained and evaluated using several major benchmarks:

  • ImageNet: Used as the primary baseline for in-distribution classification and accuracy control.
  • Large-Scale Pre-training Datasets: WIT (WebImageText), JFT-300M, JFT-3B, and the ALIGN dataset.
  • Pascal VOC: Used as a downstream multi-label classification task to evaluate how well the learned representations transfer to new domains.

Key Insights and Ideas

  • Divergence Breeds Diversity: As training methodologies move from simple reinitialization to entirely different datasets and objectives, error inconsistency skyrockets. The errors become highly uncorrelated.
Error Inconsistency across training axes
  • Data Subdomain Specialization: Models trained differently naturally become "experts" in distinct subdomains. For example, the contrastive model CLIP-S specializes in anthropogenic (human-made) images, whereas a standard supervised ResNet-50 excels at identifying nature images.
  • Overlapping, Not Supersetting: High-accuracy models do not learn a perfect superset of low-accuracy models. Instead, their feature representations overlap. Combining features from diverse models yields a representation that is more expressive yet more efficiently compressible than either model alone.
  • The Value of "Weak" Models: A lower-accuracy model is not obsolete if it was trained differently enough. It can still provide unique information to rescue a high-accuracy model's mistakes.

Key Results

  • Massive Ensemble Boosts: Ensembling a standard ResNet-50 (76.5% accuracy) with a highly divergent model like ALIGN-ZeroShot (75.5% accuracy) boosted top-1 ImageNet accuracy to 83.4%—a massive +7% absolute gain from two models of similar baseline strength.
Figure 2: As uncorrelated errors increase, so does ensemble performance (left, center), and so does ensemble efficiency (right). Left: Error inconsistency is linearly correlated with the ensemble performance improvement, relative to the mean accuracy of the models in the ensemble. Stars represent averages over models in each category (w/ error bars). Center: Because we limited our analysis to models in the same 74-78% accuracy range, the increase in relative accuracy translates into absolute accuracy boost, with best performing ensembles comprising of models whose training methodologies are most different. Right: Surprisingly, the conversion rate – the rate at which these examples are converted into correct predictions by the ensemble – also increases. This indicates that the benefits of combining divergently-trained models go beyond increasing the number of examples that can become correct predictions, to also increasing how efficiently these examples do become correct predictions.
  • Rescuing the State-of-the-Art: By greedily ensembling a high-accuracy model (BiT-1k, 82.9%) with only lower-accuracy models (none higher than 77.4%), the authors created an ensemble that reached 86.7%.
  • Downstream Super-Features: On the Pascal VOC transfer task, concatenating the features of the diverse ViT-G/14 and CLIP-S models achieved 91.36% mAP, outperforming the individual state-of-the-art Meta-Pseudo Labels (MPL) model (89.74% mAP), despite CLIP-S having much lower standalone ImageNet accuracy.

Limitations and Future Work

The most profound open question raised by the authors is a philosophical one: Why didn't standard Stochastic Gradient Descent (SGD) learn these diverse features on its own?

Currently, our training objectives strictly maximize standard classification accuracy, which forces models to exploit the easiest statistical shortcuts available in a dataset. The authors suggest that future work should focus on designing optimization objectives that explicitly reward the discovery of diverse "stepping stones" in feature space, rather than just optimizing for a single metric. Additionally, exploring how these diversity dynamics manifest in modalities outside of image classification (like NLP or audio) remains an open avenue.

Summary updated

Abstract

Despite being able to capture a range of features of the data, high accuracy models trained with supervision tend to make similar predictions. This seemingly implies that high-performing models share similar biases regardless of training methodology, which would limit ensembling benefits and render low-accuracy models as having little practical use. Against this backdrop, recent work has developed quite different training techniques, such as large-scale contrastive learning, yielding competitively high accuracy on generalization and robustness benchmarks. This motivates us to revisit the assumption that models necessarily learn similar functions. We conduct a large-scale empirical study of models across hyper-parameters, architectures, frameworks, and datasets. We find that model pairs that diverge more in training methodology display categorically different generalization behavior, producing increasingly uncorrelated errors. We show these models specialize in subdomains of the data, leading to higher ensemble performance: with just 2 models (each with ImageNet accuracy ~76.5%), we can create ensembles with 83.4% (+7% boost). Surprisingly, we find that even significantly low-accuracy models can be used to improve high-accuracy models. Finally, we show diverging training methodology yield representations that capture overlapping (but not supersetting) feature sets which, when combined, lead to increased downstream performance.
BibTeX
@inproceedings{gontijolopes2022no,
  author = {Raphael Gontijo-Lopes and Yann Dauphin and Ekin D. Cubuk},
  title = {{No One Representation to Rule Them All: Overlapping Features of Training Methods}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=BK-4qbGgIE3},
}
Figure