Classification Done Right for Vision-Language Pre-Training

Zilong Huang, Qinghao Ye, Bingyi Kang, Jiashi Feng, Haoqi Fan

TL;DR

SuperClass is a streamlined alternative to CLIP that replaces complex contrastive learning with a simple classification task. By treating raw text tokens as labels and applying smart weighting, it eliminates the need for a text encoder and massive batch sizes while outperforming CLIP on major benchmarks.

What Problem Does This Paper Address?

For the past few years, CLIP (Contrastive Language-Image Pre-training) has been the undisputed king of vision-language models. However, CLIP comes with a significant "compute tax": it requires a separate text encoder and massive batch sizes (often 64,000+) to work effectively. This makes it incredibly resource-intensive to train.

Previous attempts to simplify this using classification (treating words as labels) usually relied on "bag-of-words" approaches with heavy manual filtering—like removing stopwords or lemmatizing words. These methods often failed to scale or lost valuable information during the cleaning process. The authors of SuperClass set out to prove that a simple classification approach, if done correctly, can actually beat the contrastive giants.

Comparison of CLIP and SuperClass architectures

Methods

The core of SuperClass is its radical simplicity. Instead of contrasting image features against text features, it treats the text associated with an image as a set of classification labels.

  1. Raw Tokenization: Unlike prior work that curated "gold labels," SuperClass uses a standard subword tokenizer (like the one used in CLIP or BERT) on the raw, unfiltered text.
  2. K-hot Labels: If a caption contains $K$ unique tokens, those tokens become the target labels for that image.
  3. Softmax Loss: Surprisingly, the authors found that a simple Softmax loss—representing labels in a probabilistic manner—outperformed standard multi-label losses like Binary Cross-Entropy (BCE). This is likely because Softmax better handles the inherent noise and "non-exhaustive" nature of web captions (where a caption might not mention every object in an image).
  4. Online IDF Weighting: To handle the fact that some words (like "the" or "a") are less informative than others (like "alpaca"), they use Inverse Document Frequency (IDF). They implement an online version that updates during training, giving higher weights to rare, discriminative words.

Crucially, because there is no contrastive step, SuperClass does not require a text encoder during pre-training and can be trained with much smaller, more accessible batch sizes.

Datasets

  • Pre-training: The models were trained on DataComp-1B, a massive dataset of 1.3 billion image-text pairs.
  • Evaluation: The authors tested the resulting vision backbones on a wide array of benchmarks, including ImageNet-1K (Linear probing and zero-shot), Vision-Language Tasks (VQAv2, GQA, VizWiz, T-VQA, SQA, MMBench, MME, POPE, MMMU, and SEED), and COCO Captions.

Key Insights and Ideas

  • Manual Filtering is a Trap: The authors found that "cleaning" text by removing stopwords or numbers actually hurts performance. Raw text contains subtle signals that help the model learn better visual representations.
  • The "Text Encoder" is Optional: By moving the "knowledge" of the text into the classification head's weights, you can train a world-class vision backbone without ever running a text transformer during the pre-training phase.
  • Classification Scales Better: One of the most significant findings is that SuperClass exhibits better scaling behavior than CLIP. As you increase the model size (from ViT-S to ViT-L) or the amount of data, the performance gap between SuperClass and CLIP actually widens in favor of SuperClass.
Scaling behavior of SuperClass vs. CLIP across FLOPs and training samples

Key Results

SuperClass consistently outperforms or matches CLIP and other state-of-the-art self-supervised models:

  • ImageNet Linear Probing: A SuperClass ViT-L model achieved 85.0% top-1 accuracy, significantly higher than OpenAI's CLIP (82.7%) and even the specialized DINOv2 (84.5%).
  • Downstream Superiority: On vision-language benchmarks like VQAv2 and MMBench, SuperClass models (when paired with a language model like Vicuna) consistently beat CLIP-based counterparts trained on the same data.
  • Efficiency: SuperClass achieves these results while being more computationally efficient, as it avoids the forward and backward passes of a text encoder.

Limitations and Future Work

The authors identify one primary trade-off: Word Order. Because SuperClass treats text as a "bag of subwords," it completely ignores the structural relationships between words (e.g., it might not distinguish between "dog bites man" and "man bites dog").

While this doesn't seem to hurt visual representation learning, it means the model loses some "supervisory information." Future work could involve finding ways to incorporate linguistic structure into the classification framework without sacrificing its simplicity and efficiency. Additionally, for zero-shot tasks, SuperClass still requires a secondary "alignment" step (like Locked-image Tuning) to connect the vision backbone back to a text encoder for inference.

Summary updated

Abstract

We introduce SuperClass, a super simple classification method for vision-language pre-training on image-text data. Unlike its contrastive counterpart CLIP who contrast with a text encoder, SuperClass directly utilizes tokenized raw text as supervised classification labels, without the need for additional text filtering or selection. Due to the absence of the text encoding as contrastive target, SuperClass does not require a text encoder and does not need to maintain a large batch size as CLIP does. SuperClass demonstrated superior performance on various downstream tasks, including classic computer vision benchmarks and vision language downstream tasks. We further explored the scaling behavior of SuperClass on model size, training length, or data size, and reported encouraging results and comparisons to CLIP. https://github.com/x-cls/superclass
BibTeX
@inproceedings{huang2024classification,
  author = {Zilong Huang and Qinghao Ye and Bingyi Kang and Jiashi Feng and Haoqi Fan},
  title = {{Classification Done Right for Vision-Language Pre-Training}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=Hd2EOwKItm},
}
Figure