Generalization Beyond Data Imbalance: A Controlled Study on CLIP for Transferable Insights

Xin Wen, Bingchen Zhao, Yilun Chen, Jiangmiao Pang, Xiaojuan Qi

Abstract

Severe data imbalance naturally exists among web-scale vision-language datasets. Despite this, we find CLIP pre-trained thereupon exhibits notable robustness to the data imbalance compared to supervised learning, and demonstrates significant effectiveness in learning generalizable representations. With an aim to investigate the reasons behind this finding, we conduct controlled experiments to study various underlying factors, and reveal that CLIP's pretext task forms a dynamic classification problem wherein only a subset of classes is present in training. This isolates the bias from dominant classes and implicitly balances the learning signal. Furthermore, the robustness and discriminability of CLIP improve with more descriptive language supervision, larger data scale, and broader open-world concepts, which are inaccessible to supervised learning. Our study not only uncovers the mechanisms behind CLIP's generalizability beyond data imbalance but also provides transferable insights for the research community.

Introduction

Deep learning models typically require balanced datasets to perform well. However, foundation models like CLIP are trained on "wild" web-scale data characterized by extreme long-tail distributions—where a few "head" concepts appear millions of times while others are nearly absent. Surprisingly, CLIP maintains high performance on tail classes where traditional Supervised Learning (SL) fails. This study investigates why CLIP is robust to this imbalance and how these insights can be transferred to other learning paradigms.

Methodology and Datasets

To isolate the factors contributing to CLIP's robustness, the authors conducted a controlled study using ResNet-50 backbones for both CLIP and SL. They introduced two specialized datasets to study the impact of language and imbalance:

  • IN-Caps (ImageNet-Captions): ImageNet images paired with rich metadata (titles, descriptions, tags) retrieved from Flickr to study the impact of language descriptiveness.
  • LAIONet: A subset of LAION-400M filtered for ImageNet classes, preserving the natural web-scale imbalance.
Comparison of data construction for ImageNet-Captions (IN-Caps) and LAIONet.

The study focused on four key variables: language supervision quality, the contrastive pretext task as "dynamic classification," vocabulary subsampling, and feature space analysis via Neural Collapse metrics.

Mechanisms of Robustness

1. Dynamic Classification via Subsampling

In standard SL, rare classes are constantly penalized by the gradients of all other classes. CLIP’s contrastive loss, however, only compares images against the text present in a specific training batch. This "vocabulary subsampling" effectively isolates tail classes from dominant ones, preventing the "head" from overwhelming the learning signal.

2. Descriptive Language as a Regularizer

Richer captions (e.g., "A phone call at night" vs. "payphone") provide greater feature variation. This prevents the model from collapsing class representations into single points (Neural Collapse), improving generalization to diverse real-world examples.

3. Open-World Synergy

CLIP benefits from seeing "open-world" concepts outside the target evaluation set. These shared features provide a "privilege" that fixed-label supervised learning cannot access.

Key Results

Experiments demonstrate that the performance gap between SL and CLIP can be significantly narrowed by applying CLIP's principles to SL and Self-Supervised Learning (SSL).

Overall accuracy vs. frequency correlation for various training configurations.
  • Closing the Gap: By implementing vocabulary subsampling in SL, the authors reduced its frequency bias, making its performance profile resemble CLIP’s. As shown in the figure above, smaller SL vocabulary sizes (subsampling) move the model toward lower frequency correlation.
  • Improving SSL: Applying these insights to DINO trained on imbalanced data (LAIONet) improved transfer performance on 11 out of 12 downstream tasks, matching the performance of models trained on balanced ImageNet.
  • Zero-Shot Success: CLIP models trained on imbalanced YFCC-15M achieved over 50% accuracy on classes with fewer than 10 training examples, a feat standard SL cannot match.

Conclusion

CLIP's robustness to data imbalance stems from its dynamic classification task and the regularizing effect of descriptive language. By transferring these mechanisms—specifically vocabulary subsampling and rich language anchors—to supervised and self-supervised models, researchers can achieve high performance even on severely imbalanced real-world data.

Summary updated

Abstract

Severe data imbalance naturally exists among web-scale vision-language datasets. Despite this, we find CLIP pre-trained thereupon exhibits notable robustness to the data imbalance compared to supervised learning, and demonstrates significant effectiveness in learning generalizable representations. With an aim to investigate the reasons behind this finding, we conduct controlled experiments to study various underlying factors, and reveal that CLIP's pretext task forms a dynamic classification problem wherein only a subset of classes is present in training. This isolates the bias from dominant classes and implicitly balances the learning signal. Furthermore, the robustness and discriminability of CLIP improve with more descriptive language supervision, larger data scale, and broader open-world concepts, which are inaccessible to supervised learning. Our study not only uncovers the mechanisms behind CLIP's generalizability beyond data imbalance but also provides transferable insights for the research community. The findings are validated in both supervised and self-supervised learning, enabling models trained on imbalanced data to achieve CLIP-level performance on diverse recognition tasks. Code and data are available at: https://github.com/CVMI-Lab/clip-beyond-tail.
BibTeX
@inproceedings{wen2024generalization,
  author = {Xin Wen and Bingchen Zhao and Yilun Chen and Jiangmiao Pang and Xiaojuan Qi},
  title = {{Generalization Beyond Data Imbalance: A Controlled Study on CLIP for Transferable Insights}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=PcyioHOmjq},
}
Figure