Improving the Generalization of Supervised Models

Mert Bülent Sariyildiz, Yannis Kalantidis, Karteek Alahari, Diane Larlus

Abstract

We consider the problem of training a deep neural network on a given classification task, e.g., ImageNet-1K (IN1K), so that it excels at that task as well as at other (future) transfer tasks. These two seemingly contradictory properties impose a trade-off between improving the model's generalization while maintaining its performance on the original task. Models trained with self-supervised learning (SSL) tend to generalize better than their supervised counterparts for transfer learning; yet, they still lag behind supervised models on IN1K. In this paper, we propose a supervised learning setup that leverages the best of both worlds. We enrich the common supervised training framework using two key components of recent SSL models: multi-scale crops for data augmentation and the use of an expendable projector head. We replace the last layer of class weights with class prototypes computed on the fly using a memory bank. We show that these three improvements lead to a more favorable trade-off between the IN1K training task and 13 transfer tasks. Over all the explored configurations, we single out two models: t-ReX that achieves a new state of the art for transfer learning and outperforms top methods such as DINO and PAWS on IN1K, and t-ReX* that matches the highly optimized RSB-A1 model on IN1K while performing better on transfer tasks.

Introduction

In computer vision, a long-standing tension exists between specialization and generalization. A ResNet50 trained via standard supervised learning on ImageNet-1K (IN1K) excels at its specific 1,000 classes but often underperforms when used as a feature extractor for downstream transfer tasks, such as satellite imagery or specialized biological classification. Conversely, self-supervised learning (SSL) models, which do not utilize labels during pre-training, often demonstrate superior generalization to these transfer tasks despite lagging behind supervised models on the original IN1K benchmark. This "generalization gap" raises a fundamental question: why should the inclusion of label information result in a less versatile general-purpose model? This work introduces t-ReX, a supervised training recipe designed to bridge this gap by integrating successful techniques from the SSL paradigm.

Methodology: The t-ReX Recipe

The authors identify and adapt three key components from SSL to enhance supervised training:

  1. Multi-crop Data Augmentation: Instead of a single global image crop, the model is presented with one global crop and several smaller "local" crops at varying resolutions. This encourages the model to learn local-to-global correspondences, ensuring that class-specific features are recognized independently of the full object's context.
  2. Expendable Projector Heads: Unlike standard supervised models where the encoder connects directly to a classifier, t-ReX inserts a Multi-Layer Perceptron (MLP) projector between the encoder and the loss function. This projector is discarded after training, leaving a more robust encoder for downstream tasks.
  3. Online Class Means (OCM): Rather than learning a static weight matrix for classification, the model maintains a memory bank of features to compute class "prototypes" (average feature vectors) on the fly. This prevents the model from over-specializing on the fixed geometry of a standard classification layer.
The t-ReX training pipeline incorporates multi-crop augmentation and an expendable projector head to improve feature representation.

Experimental Results

The proposed methods were evaluated on IN1K and a suite of 13 diverse transfer datasets, including ImageNet-CoG, Cars196, EuroSAT, and Flowers. The results demonstrate that t-ReX significantly shifts the Pareto frontier of the performance-vs-generalization trade-off.

Two primary configurations are highlighted:

  • t-ReX: Optimized for maximum transferability. It establishes a new state-of-the-art (SotA) for transfer learning, outperforming leading SSL methods like DINO and PAWS while maintaining superior IN1K accuracy (78.0% vs. DINO's 74.8%).
  • t-ReX*: Optimized for IN1K performance. It matches the highly-tuned "ResNet Strikes Back" (RSB-A1) model at 80.2% accuracy on IN1K while providing significantly better transfer performance.
Comparison of t-ReX models against previous state-of-the-art supervised and self-supervised methods, showing a superior trade-off between ImageNet-1K accuracy and transfer performance.

Discussion and Insights

A key finding is that the projector head acts as a "knob" for generalization. Increasing the depth of the projector boosts transfer performance by preventing "feature sparsity." Standard supervised models often collapse useful information into sparse, quasi-one-hot vectors optimized solely for the training labels. The projector head serves as a buffer, allowing the encoder to retain richer, denser representations that are more useful for unseen tasks. This proves that labels themselves are not detrimental to generalization; rather, the traditional way they are utilized in training objectives often limits the model's potential.

Conclusion and Limitations

t-ReX demonstrates that supervised models can achieve SSL-level generalization without sacrificing specialized performance. However, the method remains dependent on high-quality labeled datasets for initial training and has primarily been validated on the ResNet50 architecture. Future work will explore the applicability of these techniques to Vision Transformers (ViTs) and their robustness against large-scale, noisy web data.

Project page and pretrained models: https://europe.naverlabs.com/t-rex

Summary updated

Abstract

We consider the problem of training a deep neural network on a given classification task, e.g., ImageNet-1K (IN1K), so that it excels at that task as well as at other (future) transfer tasks. These two seemingly contradictory properties impose a trade-off between improving the model's generalization while maintaining its performance on the original task. Models trained with self-supervised learning (SSL) tend to generalize better than their supervised counterparts for transfer learning; yet, they still lag behind supervised models on IN1K. In this paper, we propose a supervised learning setup that leverages the best of both worlds. We enrich the common supervised training framework using two key components of recent SSL models: multi-scale crops for data augmentation and the use of an expendable projector head. We replace the last layer of class weights with class prototypes computed on the fly using a memory bank. We show that these three improvements lead to a more favorable trade-off between the IN1K training task and 13 transfer tasks. Over all the explored configurations, we single out two models: t-ReX that achieves a new state of the art for transfer learning and outperforms top methods such as DINO and PAWS on IN1K, and t-ReX* that matches the highly optimized RSB-A1 model on IN1K while performing better on transfer tasks. Project page and pretrained models: https://europe.naverlabs.com/t-rex
BibTeX
@inproceedings{sariyildiz2023improving,
  author = {Mert Bülent Sariyildiz and Yannis Kalantidis and Karteek Alahari and Diane Larlus},
  title = {{Improving the Generalization of Supervised Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=3Y5Uhf5KgGK},
}
Figure