What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights

Xin Wen, Bingchen Zhao, Yilun Chen, Jiangmiao Pang, Xiaojuan Qi

TL;DR

Researchers have long wondered why CLIP remains robust despite being trained on messy, "long-tailed" web data that usually cripples standard supervised models. This paper reveals that CLIP’s secret sauce is its dynamic classification pretext task and descriptive language supervision, and demonstrates that these insights can be "backported" to supervised and self-supervised models to make them equally robust to data imbalance.

What Problem Does This Paper Address?

In the real world, data is messy and imbalanced. A few "head" classes (like "dog" or "car") appear millions of times, while "tail" classes (like specific biological species) appear only a handful of times. Standard Supervised Learning (SL) is notoriously bad at this; it becomes heavily biased toward head classes, often ignoring the tail entirely.

Curiously, CLIP (Contrastive Language-Image Pre-training) seems to buck this trend. Even when trained on uncurated web data with extreme imbalances, it generalizes remarkably well to rare concepts. The authors set out to perform a controlled "autopsy" on CLIP to find out exactly which factors—supervision, task design, or data scale—contribute to this robustness.

Methods

The authors conducted a systematic controlled study, isolating variables one by one:

  1. Supervision Signal: They compared simple class templates (e.g., "a photo of a [class]") against descriptive natural language (titles, tags, and descriptions).
  2. Pretext Task: They compared standard SL (classifying into 1,000 fixed categories) against CLIP’s contrastive task. Crucially, they introduced Vocabulary Subsampling in SL—randomly picking a subset of classes for each training batch—to mimic CLIP’s behavior.
  3. Data Distribution: They created variants of datasets with different levels of intra-class diversity and distribution shifts.
  4. Scaling: They analyzed how increasing the number of images and the number of open-world concepts affects performance.

To measure bias, they used the Spearman correlation between a class's frequency in the training set and the model's accuracy/prediction rate for that class. A lower correlation indicates a more robust, less biased model.

Datasets

  • IN-Caps: A version of ImageNet where images are augmented with original Flickr metadata (titles, descriptions, tags) to provide descriptive supervision.
  • LAIONet: A filtered subset of the massive LAION-400M dataset, specifically mapped to ImageNet classes to allow for a direct "apples-to-apples" comparison with supervised models.
  • Standard Benchmarks: YFCC-15M, CC-12M, and MetaCLIP were used to validate findings at scale.
The curation process for IN-Caps and LAIONet allows for controlled experiments on data distribution.

Key Insights and Ideas

  • Dynamic Classification is Key: In standard SL, the model compares an image against all 1,000 classes every time. In CLIP, a batch of 1,024 images might only contain 600 unique concepts. This "dynamic classification" isolates tail classes from the overwhelming negative signal of dominant head classes, implicitly balancing the learning signal.
  • Descriptiveness Matters: Moving from simple templates to full-text captions significantly improves both accuracy and robustness. Descriptive text helps the model learn more nuanced features and prevents the "feature collapse" often seen in long-tailed SL.
  • Open-World Advantage: CLIP benefits from seeing concepts outside the target evaluation set. These "open-world" concepts share features with tail classes, providing a form of transfer learning that fixed-vocabulary SL cannot access.
  • Transferable Robustness: The most exciting finding is that you can make a standard ResNet or a DINO (self-supervised) model robust to imbalance simply by subsampling the vocabulary during training.
Figure 3: Results on IN-Caps about • text descriptiveness and ✖vocabulary size. 1) Increasing • text descriptiveness improves both robustness (a) and discriminability (b) of CLIP, but the tendency varies if using • less descriptive (template-based) supervision. 2) The gap between SL and CLIP (a) implies CLIP re-balances predictions, which is replicable by ✖subsampling the vocabulary SL trains with.

Key Results

  • Reduced Bias: CLIP models show a much weaker correlation between class frequency and accuracy compared to SL models. As shown in Figure 1, SL models are heavily biased (Spearman corr. ~0.8), while CLIP models are significantly more balanced.
  • SL Improvements: By applying vocabulary subsampling and using a frozen CLIP text encoder as a classifier, a standard SL model trained on "1-shot" tail data (only one example per class) achieved generalization performance similar to CLIP.
  • DINO Boost: Applying prototype subsampling to DINO (a self-supervised method) pre-trained on imbalanced web data significantly improved its transfer learning performance across 12 different datasets, effectively matching ImageNet-pretrained performance.
Figure 1: Per-class statistics of image-text datasets and models trained on top. (a) A highly imbalanced class distribution is shared across datasets.‡(b) Compared to supervised learning (✖SL), CLIP’s performance (measured by • accuracy) is less biased by data frequency, and the classifier is notably uncorrelated (measured by model’s number of • prediction per class). Besides, the correlation narrows as data scales up. Both aspects indicate implicit re-balancing mechanisms exist in CLIP.

Limitations and Future Work

  • Caption Noise: The study uses a rule-based pipeline to estimate concept frequency, which might be susceptible to noise in web captions.
  • Task Scope: The research focuses primarily on image classification. Whether these insights translate to more complex tasks like object detection or semantic segmentation remains an open question.
  • Generative Models: The authors note that while contrastive models like CLIP are robust, generative models (like LLMs) still seem to struggle significantly with long-tailed distributions, suggesting a need for similar "rebalancing" research in the generative space.

Summary updated

Abstract

Severe data imbalance naturally exists among web-scale vision-language datasets. Despite this, we find CLIP pre-trained thereupon exhibits notable robustness to the data imbalance compared to supervised learning, and demonstrates significant effectiveness in learning generalizable representations. With an aim to investigate the reasons behind this finding, we conduct controlled experiments to study various underlying factors, and reveal that CLIP's pretext task forms a dynamic classification problem wherein only a subset of classes is present in training. This isolates the bias from dominant classes and implicitly balances the learning signal. Furthermore, the robustness and discriminability of CLIP improve with more descriptive language supervision, larger data scale, and broader open-world concepts, which are inaccessible to supervised learning. Our study not only uncovers the mechanisms behind CLIP's generalizability beyond data imbalance but also provides transferable insights for the research community. The findings are validated in both supervised and self-supervised learning, enabling models trained on imbalanced data to achieve CLIP-level performance on diverse recognition tasks. Code and data are available at: https://github.com/CVMI-Lab/clip-beyond-tail.
BibTeX
@inproceedings{wen2024what,
  author = {Xin Wen and Bingchen Zhao and Yilun Chen and Jiangmiao Pang and Xiaojuan Qi},
  title = {{What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=PcyioHOmjq},
}
Figure