Overview
Supervised learning models are traditionally the champions of ImageNet classification, but they often struggle to generalize to new tasks. Conversely, self-supervised learning (SSL) models are transfer-learning stars that tend to lag on raw ImageNet accuracy. This paper introduces t-ReX, a supervised training recipe that borrows the best techniques from SSL to create models that dominate both arenas, proving that the presence of labels does not have to hurt generalization.
The Generalization Gap
In computer vision, there has been a growing "generalization gap." If you train a model with labels (supervised learning), it becomes incredibly adept at that specific task (like ImageNet classification) but often fails when its features are applied to a new, unseen dataset. Paradoxically, models trained without labels (self-supervised learning) often produce more flexible, "transferable" features, even if their raw accuracy on the original task is lower.
The authors tackle a fundamental question: Why should having more information (labels) result in a worse general-purpose model? They aim to bridge this gap by creating a supervised model that maintains state-of-the-art ImageNet performance while matching or beating SSL models at transfer learning.
Methods
Rather than inventing a brand-new architecture, the researchers carefully integrated successful components from the SSL world into a standard supervised pipeline.
- Multi-crop Data Augmentation: Instead of showing the model one standard version of an image, they present multiple "global" views and several smaller "local" crops. This forces the network to learn how local details relate to the whole object, improving feature robustness.
- Expendable Projector Head: They add a Multi-Layer Perceptron (MLP) after the main encoder (e.g., ResNet50) during training. Crucially, this head is discarded after training is complete. Only the encoder's features are used for downstream transfer tasks.
- Online Class Means (OCM): This is the paper's most novel contribution. Instead of using a standard linear layer with fixed weights for classification, they use a memory bank to store recent image features. They then compute a "prototype" (an average feature vector) for each class on the fly. The model is trained to move an image's features closer to its respective class prototype.
Datasets
- Training: The models were trained on the standard ImageNet-1K (IN1K) dataset.
- Evaluation (Transfer): To test generalization, the authors used a massive suite of 13 transfer datasets, including ImageNet-CoG (five levels of increasingly difficult concept generalization) and small-scale benchmarks like Aircraft, Cars196, DTD (textures), EuroSAT (satellite), Flowers, Pets, Food101, and SUN397 (scenes).
- Robustness: They also tested on IN1K-v2, IN1K-Sketch, and long-tail datasets like iNaturalist 2018/2019.
Key Insights
- The Projector is a Trade-off Knob: The authors discovered that the design of the projector head acts as a control valve. Adding more layers to the projector significantly boosts transferability but slightly decreases ImageNet accuracy. This allows researchers to "tune" a model based on whether they want a specialized classifier or a general-purpose feature extractor.
- Sparsity vs. Information: Standard supervised models tend to produce very "sparse" features (many zeros), which are efficient for one specific task but lose general information. The projector head prevents this "over-specialization," resulting in denser features with higher information content that work better across different domains.
- Prototypes over Weights: Moving from fixed classifier weights to Online Class Means acts as a powerful regularizer. Because prototypes are calculated from a moving window of data, they are more dynamic and prevent the model from overfitting to the specific quirks of the training set.
Key Results
The authors produced two standout models, both based on a standard ResNet50 architecture:
- t-ReX: Optimized for transfer. It achieved a new State-of-the-Art (SotA) for transfer learning, outperforming top SSL methods like DINO and PAWS, while still beating them by 3-4% on ImageNet accuracy.
- t-ReX*: Optimized for ImageNet. It matched the performance of the highly-optimized RSB-A1 (one of the strongest supervised baselines) on ImageNet (80.2%) while significantly outperforming it on transfer tasks.
As shown in their Pareto front visualization, these models effectively push the envelope, proving you don't have to sacrifice transferability for target-task performance.
Limitations and Future Work
- Annotation Cost: While the paper proves supervision is powerful, it still relies on high-quality manual labels, which are expensive and time-consuming to produce compared to the "unlimited" data available for SSL.
- Architecture Scope: The study focused heavily on the ResNet50 architecture. While the authors suggest these techniques are "architecture-agnostic," extensive testing on Vision Transformers (ViTs) would be a natural next step.
- Computational Overhead: Using multi-crop augmentation increases the training time and memory requirements compared to standard single-crop supervised training.
Summary updated