Abstract
Severe data imbalance naturally exists among web-scale vision-language datasets. Despite this, we find CLIP pre-trained thereupon exhibits notable robustness to the data imbalance compared to supervised learning, and demonstrates significant effectiveness in learning generalizable representations. With an aim to investigate the reasons behind this finding, we conduct controlled experiments to study various underlying factors, and reveal that CLIP's pretext task forms a dynamic classification problem wherein only a subset of classes is present in training. This isolates the bias from dominant classes and implicitly balances the learning signal. Furthermore, the robustness and discriminability of CLIP improve with more descriptive language supervision, larger data scale, and broader open-world concepts, which are inaccessible to supervised learning. Our study not only uncovers the mechanisms behind CLIP's generalizability beyond data imbalance but also provides transferable insights for the research community.
Introduction
Deep learning models typically require balanced datasets to perform well. However, foundation models like CLIP are trained on "wild" web-scale data characterized by extreme long-tail distributions—where a few "head" concepts appear millions of times while others are nearly absent. Surprisingly, CLIP maintains high performance on tail classes where traditional Supervised Learning (SL) fails. This study investigates why CLIP is robust to this imbalance and how these insights can be transferred to other learning paradigms.
Methodology and Datasets
To isolate the factors contributing to CLIP's robustness, the authors conducted a controlled study using ResNet-50 backbones for both CLIP and SL. They introduced two specialized datasets to study the impact of language and imbalance:
- IN-Caps (ImageNet-Captions): ImageNet images paired with rich metadata (titles, descriptions, tags) retrieved from Flickr to study the impact of language descriptiveness.
- LAIONet: A subset of LAION-400M filtered for ImageNet classes, preserving the natural web-scale imbalance.
The study focused on four key variables: language supervision quality, the contrastive pretext task as "dynamic classification," vocabulary subsampling, and feature space analysis via Neural Collapse metrics.
Mechanisms of Robustness
1. Dynamic Classification via Subsampling
In standard SL, rare classes are constantly penalized by the gradients of all other classes. CLIP’s contrastive loss, however, only compares images against the text present in a specific training batch. This "vocabulary subsampling" effectively isolates tail classes from dominant ones, preventing the "head" from overwhelming the learning signal.
2. Descriptive Language as a Regularizer
Richer captions (e.g., "A phone call at night" vs. "payphone") provide greater feature variation. This prevents the model from collapsing class representations into single points (Neural Collapse), improving generalization to diverse real-world examples.
3. Open-World Synergy
CLIP benefits from seeing "open-world" concepts outside the target evaluation set. These shared features provide a "privilege" that fixed-label supervised learning cannot access.
Key Results
Experiments demonstrate that the performance gap between SL and CLIP can be significantly narrowed by applying CLIP's principles to SL and Self-Supervised Learning (SSL).
- Closing the Gap: By implementing vocabulary subsampling in SL, the authors reduced its frequency bias, making its performance profile resemble CLIP’s. As shown in the figure above, smaller SL vocabulary sizes (subsampling) move the model toward lower frequency correlation.
- Improving SSL: Applying these insights to DINO trained on imbalanced data (LAIONet) improved transfer performance on 11 out of 12 downstream tasks, matching the performance of models trained on balanced ImageNet.
- Zero-Shot Success: CLIP models trained on imbalanced YFCC-15M achieved over 50% accuracy on classes with fewer than 10 training examples, a feat standard SL cannot match.
Conclusion
CLIP's robustness to data imbalance stems from its dynamic classification task and the regularizing effect of descriptive language. By transferring these mechanisms—specifically vocabulary subsampling and rich language anchors—to supervised and self-supervised models, researchers can achieve high performance even on severely imbalanced real-world data.
Summary updated