Classification Done Right for Vision-Language Pre-Training

Zilong Huang, Qinghao Ye, Bingyi Kang, Jiashi Feng, Haoqi Fan

Abstract

We introduce SuperClass, a super simple classification method for vision-language pre-training on image-text data. Unlike its contrastive counterpart CLIP who contrast with a text encoder, SuperClass directly utilizes tokenized raw text as supervised classification labels, without the need for additional text filtering or selection. Due to the absence of the text encoding as contrastive target, SuperClass does not require a text encoder and does not need to maintain a large batch size as CLIP does. SuperClass demonstrated superior performance on various downstream tasks, including classic computer vision benchmarks and vision language downstream tasks. We further explored the scaling behavior of SuperClass on model size, training length, or data size, and reported encouraging results and comparisons to CLIP.

Introduction

For several years, Contrastive Language-Image Pre-training (CLIP) has been the dominant paradigm for teaching AI to understand images through text. However, CLIP comes with a "computational tax": it requires a separate text encoder and massive batch sizes (often over 64,000) to work effectively. This makes it resource-intensive and difficult for researchers with limited hardware to train from scratch.

Previous attempts to use classification (treating words as "labels") often relied on "gold labels"—hand-crafted, filtered vocabularies that removed "noisy" words. These methods were hard to scale and often threw away valuable information. SuperClass addresses this by achieving CLIP-level performance using a simple classification approach on raw, unfiltered web data.

Methodology

The SuperClass architecture is elegantly minimal. During pre-training, it uses only a Vision Transformer (ViT), removing the need for a text encoder entirely.

Comparison between CLIP's contrastive approach and SuperClass's classification approach.
  1. Text as Labels: Instead of encoding a caption into a vector, the text is tokenized using a standard subword tokenizer. These tokens are converted into a "K-hot" label vector.
  2. Softmax Loss: Surprisingly, a simple Softmax loss—typically used for single-label classification—outperformed multi-label losses like Binary Cross-Entropy (BCE). Treating labels as a probability distribution helps the model handle the inherent noise in web-scale image-text pairs.
  3. Online IDF Weighting: To prevent common "stop words" (like "the" or "a") from dominating, the authors use Inverse Document Frequency (IDF) to weight the loss. Rare, descriptive words get higher weights. Crucially, this is calculated "online" during training, making it highly portable.

Results and Scaling Behavior

The models were pre-trained on the DataComp-1B dataset (1.3 billion image-text pairs) and evaluated on a massive battery of tests, including ImageNet-1K, COCO, and multimodal benchmarks like VQAv2 and MMMU.

Scaling behavior of SuperClass vs. CLIP across different compute budgets and training samples.

Key Insights

  • The Power of Raw Tokens: Manual vocabulary cleaning is unnecessary. Subword tokens combined with IDF weighting allow the model to focus on informative parts of a caption automatically.
  • Superior Scaling: SuperClass demonstrates better scaling laws than CLIP. As model size (from ViT-S to ViT-L) or data volume increases, the performance gap between SuperClass and CLIP widens in favor of SuperClass.
  • Efficiency: Lacking a text encoder makes SuperClass more computationally efficient and removes the requirement for massive batch sizes that complicate CLIP training.

Performance Benchmarks

On ImageNet-1K linear probing, a SuperClass ViT-L achieved 85.0% accuracy, outperforming OpenAI's CLIP (82.7%) and the self-supervised DINOv2 (84.5%). Furthermore, when integrated into Large Language Model frameworks like LLaVA, SuperClass backbones consistently outperformed CLIP-based versions across nearly all benchmarks, including visual question answering and OCR tasks.

Conclusion and Future Work

SuperClass proves that a strikingly simple classification task can replace complex contrastive learning for vision-language pre-training. While it produces a powerful vision backbone, it currently ignores word order and relational structure (e.g., "dog chasing cat" vs. "cat chasing dog"). Additionally, it does not inherently produce text-image alignment for zero-shot tasks out of the box, requiring a secondary "Locked-image Tuning" (LiT) step. Future research will focus on incorporating structural language information into this simplified framework.

Summary updated

Abstract

We introduce SuperClass, a super simple classification method for vision-language pre-training on image-text data. Unlike its contrastive counterpart CLIP who contrast with a text encoder, SuperClass directly utilizes tokenized raw text as supervised classification labels, without the need for additional text filtering or selection. Due to the absence of the text encoding as contrastive target, SuperClass does not require a text encoder and does not need to maintain a large batch size as CLIP does. SuperClass demonstrated superior performance on various downstream tasks, including classic computer vision benchmarks and vision language downstream tasks. We further explored the scaling behavior of SuperClass on model size, training length, or data size, and reported encouraging results and comparisons to CLIP. https://github.com/x-cls/superclass
BibTeX
@inproceedings{huang2024classification,
  author = {Zilong Huang and Qinghao Ye and Bingyi Kang and Jiashi Feng and Haoqi Fan},
  title = {{Classification Done Right for Vision-Language Pre-Training}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=Hd2EOwKItm},
}
Figure