Abstract
We consider the problem of training a deep neural network on a given classification task, e.g., ImageNet-1K (IN1K), so that it excels at both the training task as well as at other (future) transfer tasks. These two seemingly contradictory properties impose a trade-off between improving the model's generalization and maintaining its performance on the original task. Models trained with self-supervised learning tend to generalize better than their supervised counterparts for transfer learning; yet, they still lag behind supervised models on IN1K. In this paper, we propose a supervised learning setup that leverages the best of both worlds. We extensively analyze supervised training using multi-scale crops for data augmentation and an expendable projector head, and reveal that the design of the projector allows us to control the trade-off between performance on the training task and transferability. We further replace the last layer of class weights with class prototypes computed on the fly using a memory bank and derive two models: t-ReX that achieves a new state of the art for transfer learning and outperforms top methods such as DINO and PAWS on IN1K, and t-ReX* that matches the highly optimized RSB-A1 model on IN1K while performing better on transfer tasks.
Introduction: The Generalization Gap
A long-standing trade-off exists in computer vision: models trained with full supervision (e.g., ImageNet-1K) excel at their specific training task but often fall behind self-supervised learning (SSL) models when transferred to new, unseen datasets. Conversely, SSL models generalize well but typically lag by several percentage points in top-1 accuracy on the original ImageNet task.
The authors investigate why access to more information (labels) seemingly hinders a model's ability to generalize. They identify that standard supervised training often leads to "feature collapse" or over-specialization, where the model learns only the bare minimum needed to distinguish between the specific classes in its training set. This paper demonstrates that labels aren't the problem—training design is.
Proposed Methodology: Borrowing from Self-Supervision
The researchers propose an improved supervised training setup that integrates three key components typically found in SSL pipelines:
- Multi-crop Data Augmentation: Instead of just one or two views of an image, the model sees multiple "global" (large) and "local" (small) crops. This forces the encoder to learn features that are consistent across different scales and regions.
- Expendable Projector Head: A Multi-Layer Perceptron (MLP) is inserted between the ResNet50 encoder and the final loss function. Crucially, this projector is discarded after training. It acts as a "buffer" that absorbs task-specific noise, allowing the encoder to learn more generic, transferable representations.
- Online Class Means (OCM): Instead of learning a fixed weight matrix for classification, the last layer is replaced with class prototypes computed on-the-fly. Using a memory bank of recent embeddings, the model classifies an image based on its proximity to the average embedding (prototype) of each class.
Experimental Results and Performance Frontier
The authors present two primary versions of their model, both based on a standard ResNet50:
- t-ReX: Optimized for transfer learning. It achieves a new state-of-the-art in mean transfer accuracy (measured in log-odds) and outperforms top SSL methods like DINO on ImageNet-1K (78.0% vs 74.8%).
- t-ReX*: Optimized for ImageNet performance. It matches the highly optimized RSB-A1 baseline at 80.2% top-1 accuracy on ImageNet-1K while significantly outperforming it on transfer tasks.
The design of the expendable projector serves as a "control knob" for the trade-off between local accuracy and global generalization. Adding more hidden layers to the projector improves transferability at a slight cost to ImageNet accuracy. Furthermore, empirical analysis shows that increasing the number of local crops consistently improves performance on both axes, though gains begin to saturate around 8 crops.
Conclusion and Future Work
This work demonstrates that when trained correctly, supervised models can outperform SSL models like DINO and PAWS even on transfer tasks, effectively bridging the generalization gap. By reducing feature sparsity and preventing over-specialization through projectors and multi-crop augmentation, the t-ReX family establishes a new Pareto frontier for vision models.
Future work remains to explore the effectiveness of this method on noisier or smaller labeled datasets, as it currently relies on high-quality ImageNet-1K annotations. Additionally, while the study focuses on ResNet50, the components are expected to benefit Vision Transformers (ViTs), a direction for future empirical validation.
Summary updated