TL;DR
SuperClass is a streamlined alternative to CLIP that replaces complex contrastive learning with a simple classification task. By treating raw text tokens as labels and applying smart weighting, it eliminates the need for a text encoder and massive batch sizes while outperforming CLIP on major benchmarks.
What Problem Does This Paper Address?
For the past few years, CLIP (Contrastive Language-Image Pre-training) has been the undisputed king of vision-language models. However, CLIP comes with a significant "compute tax": it requires a separate text encoder and massive batch sizes (often 64,000+) to work effectively. This makes it incredibly resource-intensive to train.
Previous attempts to simplify this using classification (treating words as labels) usually relied on "bag-of-words" approaches with heavy manual filtering—like removing stopwords or lemmatizing words. These methods often failed to scale or lost valuable information during the cleaning process. The authors of SuperClass set out to prove that a simple classification approach, if done correctly, can actually beat the contrastive giants.
Methods
The core of SuperClass is its radical simplicity. Instead of contrasting image features against text features, it treats the text associated with an image as a set of classification labels.
- Raw Tokenization: Unlike prior work that curated "gold labels," SuperClass uses a standard subword tokenizer (like the one used in CLIP or BERT) on the raw, unfiltered text.
- K-hot Labels: If a caption contains $K$ unique tokens, those tokens become the target labels for that image.
- Softmax Loss: Surprisingly, the authors found that a simple Softmax loss—representing labels in a probabilistic manner—outperformed standard multi-label losses like Binary Cross-Entropy (BCE). This is likely because Softmax better handles the inherent noise and "non-exhaustive" nature of web captions (where a caption might not mention every object in an image).
- Online IDF Weighting: To handle the fact that some words (like "the" or "a") are less informative than others (like "alpaca"), they use Inverse Document Frequency (IDF). They implement an online version that updates during training, giving higher weights to rare, discriminative words.
Crucially, because there is no contrastive step, SuperClass does not require a text encoder during pre-training and can be trained with much smaller, more accessible batch sizes.
Datasets
- Pre-training: The models were trained on DataComp-1B, a massive dataset of 1.3 billion image-text pairs.
- Evaluation: The authors tested the resulting vision backbones on a wide array of benchmarks, including ImageNet-1K (Linear probing and zero-shot), Vision-Language Tasks (VQAv2, GQA, VizWiz, T-VQA, SQA, MMBench, MME, POPE, MMMU, and SEED), and COCO Captions.
Key Insights and Ideas
- Manual Filtering is a Trap: The authors found that "cleaning" text by removing stopwords or numbers actually hurts performance. Raw text contains subtle signals that help the model learn better visual representations.
- The "Text Encoder" is Optional: By moving the "knowledge" of the text into the classification head's weights, you can train a world-class vision backbone without ever running a text transformer during the pre-training phase.
- Classification Scales Better: One of the most significant findings is that SuperClass exhibits better scaling behavior than CLIP. As you increase the model size (from ViT-S to ViT-L) or the amount of data, the performance gap between SuperClass and CLIP actually widens in favor of SuperClass.
Key Results
SuperClass consistently outperforms or matches CLIP and other state-of-the-art self-supervised models:
- ImageNet Linear Probing: A SuperClass ViT-L model achieved 85.0% top-1 accuracy, significantly higher than OpenAI's CLIP (82.7%) and even the specialized DINOv2 (84.5%).
- Downstream Superiority: On vision-language benchmarks like VQAv2 and MMBench, SuperClass models (when paired with a language model like Vicuna) consistently beat CLIP-based counterparts trained on the same data.
- Efficiency: SuperClass achieves these results while being more computationally efficient, as it avoids the forward and backward passes of a text encoder.
Limitations and Future Work
The authors identify one primary trade-off: Word Order. Because SuperClass treats text as a "bag of subwords," it completely ignores the structural relationships between words (e.g., it might not distinguish between "dog bites man" and "man bites dog").
While this doesn't seem to hurt visual representation learning, it means the model loses some "supervisory information." Future work could involve finding ways to incorporate linguistic structure into the classification framework without sacrificing its simplicity and efficiency. Additionally, for zero-shot tasks, SuperClass still requires a secondary "alignment" step (like Locked-image Tuning) to connect the vision backbone back to a text encoder for inference.
Summary updated