ImageNet-21K Pretraining for the Masses

Tal Ridnik, Emanuel Ben Baruch, Asaf Noy, Lihi Zelnik-Manor

Abstract ImageNet-1K serves as the primary dataset for pretraining deep learning models for computer vision tasks. The ImageNet-21K dataset, which is larger and more diverse, is used less frequently due to its complexity, low accessibility, and an underestimation of its added value. This paper aims to close this gap and make high-quality, efficient pretraining on ImageNet-21K available for everyone. Via a dedicated preprocessing stage, utilization of the WordNet hierarchical structure, and a novel training scheme called semantic softmax, we show that various models significantly benefit from ImageNet-21K pretraining across numerous datasets and tasks, including small mobile-oriented models. We also outperform previous ImageNet-21K pretraining schemes for prominent models like ViT and Mixer. Our proposed pretraining pipeline is efficient, accessible, and leads to state-of-the-art reproducible results. The training code and pretrained models are available at: https://github.com/Alibaba-MIIL/ImageNet21K.

Introduction

In computer vision, pretraining on the ImageNet-1K dataset (1.2 million images, 1,000 classes) is the standard starting point for transfer learning. However, its much larger parent dataset, ImageNet-21K (14 million images, ~21,000 classes), is rarely used despite its potential. Historically, ImageNet-21K has been difficult to work with: it occupies 1.3TB of storage, lacks an official train/validation split, and suffers from inconsistent, non-mutually exclusive labels. For example, an image might be labeled "chair" while another identical image is labeled with its parent category, "furniture."

Because of these hurdles, only researchers with massive compute budgets have historically utilized ImageNet-21K. This paper democratizes access by introducing a streamlined preprocessing pipeline and a novel "semantic softmax" training scheme that makes ImageNet-21K pretraining accessible and highly effective for the broader community.

Methodology

The authors tackle the challenges of ImageNet-21K through a three-phase pipeline: data preparation, hierarchical label tree construction, and semantic softmax pretraining.

The proposed ImageNet-21K pretraining pipeline, consisting of data preparation, hierarchical label tree construction, and semantic softmax pretraining.

1. Preprocessing (ImageNet-21K-P)

To make the dataset manageable, the authors filtered out rare classes with fewer than 500 samples and established a standardized validation set of 50 images per class. By resizing images to 224x224, they reduced the storage footprint from 1.3TB to a highly manageable 250GB.

2. Semantic Softmax Training

Standard single-label training ignores the semantic relationships between classes. Conversely, treating the dataset as a multi-label problem with thousands of independent binary classifiers creates an optimization nightmare due to extreme class imbalance. Instead, the authors use the WordNet semantic tree to group labels into up to 11 hierarchical levels. They apply a separate softmax layer to each hierarchy level. For a given sample, only the relevant hierarchy levels are activated.

An example of semantic label generation for the 'Tunic' class, showing how labels are propagated up the hierarchy to activate specific softmax layers.

3. Semantic Knowledge Distillation (KD)

To handle missing labels (e.g., an image labeled "cow" but missing the "animal" tag), the authors introduce a teacher-student distillation loss. This loss is dynamically weighted based on the teacher model's confidence at each hierarchy level, ensuring robust feature learning even with noisy ground truth.

Key Insights and Results

A prevailing myth in deep learning is that massive datasets only benefit massive models. This paper proves that even tiny, mobile-optimized architectures like MobileNetV3 see massive downstream accuracy boosts from ImageNet-21K pretraining, reaching 78.0% accuracy on ImageNet-1K (a 2.8% improvement). Standard ResNet50 models reached 82.0%, far exceeding the typical 76.0% baseline.

The "Semantic Softmax" approach proved superior to naive multi-label training. The authors found that forcing a network to optimize 11,000+ independent binary classifiers causes extreme gradient interference and class imbalance, which can actually degrade downstream performance. By explicitly teaching the model the WordNet tree hierarchy, the model learns much richer visual representations.

The proposed pipeline also outperformed official pretraining weights for prominent models like ViT-B-16 and Mixer-B-16. For example, on Pascal-VOC, their ViT weights achieved 93.1% compared to Google's 78.7%. These improvements were consistent across seven diverse downstream datasets, including image retrieval and video action recognition.

Conclusion

By providing an efficient, accessible pipeline and a novel training scheme, this work makes ImageNet-21K a viable starting point for the broader computer vision community. While training still requires significant GPU resources (approximately 54 hours on 8 V100s), the resulting models set new benchmarks for transfer learning. Future work may explore incorporating complex multi-task optimization techniques to further yield even better pretraining quality.

Summary updated

Abstract

ImageNet-1K serves as the primary dataset for pretraining deep learning models for computer vision tasks. ImageNet-21K dataset, which is bigger and more diverse, is used less frequently for pretraining, mainly due to its complexity, low accessibility, and underestimation of its added value. This paper aims to close this gap, and make high-quality efficient pretraining on ImageNet-21K available for everyone. Via a dedicated preprocessing stage, utilization of WordNet hierarchical structure, and a novel training scheme called semantic softmax, we show that various models significantly benefit from ImageNet-21K pretraining on numerous datasets and tasks, including small mobile-oriented models. We also show that we outperform previous ImageNet-21K pretraining schemes for prominent new models like ViT and Mixer. Our proposed pretraining pipeline is efficient, accessible, and leads to SoTA reproducible results, from a publicly available dataset. The training code and pretrained models are available at: https://github.com/Alibaba-MIIL/ImageNet21K
BibTeX
@inproceedings{ridnik2021imagenet,
  author = {Tal Ridnik and Emanuel Ben Baruch and Asaf Noy and Lihi Zelnik-Manor},
  title = {{ImageNet-21K Pretraining for the Masses}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=g5_-3lC_Zkb},
}
Figure