TL;DR
This paper provides the first systematic, fully open-source study of scaling laws for contrastive language-image pre-training (CLIP). By training models on up to 2 billion public image-text pairs using the OpenCLIP repository, the authors demonstrate that multi-modal performance follows predictable power-law scaling across various downstream tasks, while revealing that the underlying data distribution significantly alters these scaling trajectories.
What Problem Does This Paper Address?
While scaling laws—the predictable mathematical relationships between compute, model size, data scale, and performance—have been widely mapped out for uni-modal domains like language and vision, multi-modal models have remained a blind spot. Previous attempts to study CLIP scaling relied heavily on proprietary datasets (like OpenAI's WIT-400M) and closed-source infrastructure. This lack of transparency has hindered reproducibility and prevented the broader research community from understanding how multi-modal models behave as they scale up. The authors address this gap by establishing reproducible, public baselines for CLIP scaling.
Methods
The authors systematically scaled three primary dimensions of contrastive learning:
- Model Size: Visual encoders ranged from ViT-B/32 (93M parameters) up to ViT-g/14 (1.37B parameters).
- Data Scale: Subsets of the public LAION dataset were used, scaling from LAION-80M to LAION-400M, and finally the full LAION-2B English subset.
- Compute (Samples Seen): Total training duration scaled from 3 billion to 34 billion samples seen.
Using the open-source OpenCLIP framework, they trained these configurations on up to 1,520 NVIDIA A100 GPUs. They then fit power-law functions ($E = \beta C^\alpha$, where $E$ is the error rate and $C$ is total compute) along the Pareto frontier to model performance across zero-shot classification, retrieval, linear probing, and fine-tuning.
Datasets
- Pre-training: LAION-80M, LAION-400M, and LAION-2B (all publicly available, extracted from Common Crawl).
- Downstream Evaluation: ImageNet-1k (along with robustness variants: ImageNet-V2, ImageNet-R, ImageNet-Sketch, ObjectNet, and ImageNet-A), the Visual Task Adaptation Benchmark (VTAB+), MS-COCO, and Flickr30K.
Key Insights and Ideas
- Power Laws Hold for Multi-Modal Learning: Just like LLMs, contrastive vision-language models improve predictably according to power laws across zero-shot classification, retrieval, and fine-tuning tasks.
- Data Distribution Dictates Scaling Behavior: A fascinating finding is that OpenAI's original CLIP models and OpenCLIP models exhibit entirely different scaling slopes despite sharing identical ViT architectures. OpenAI's CLIP scales better on zero-shot ImageNet classification, whereas OpenCLIP scales significantly better on image/text retrieval tasks.
- The Curation Hypothesis: The authors hypothesize this discrepancy stems from dataset curation. OpenAI's private WIT dataset likely has a higher affinity for ImageNet-style concepts, whereas LAION was filtered using a pre-trained CLIP model based on image-text similarity, inherently biasing it to favor retrieval-based tasks.
- Scale Bottlenecks: Scaling a single dimension in isolation can mask potential gains. For example, scaling data from 400M to 2B pairs showed no benefit for smaller models until the number of training samples seen was also scaled up to 34B, highlighting the necessity of co-scaling data and compute.
Key Results
- State-of-the-Art Open Models: The authors produced the largest public CLIP models at the time, including a ViT-H/14 model trained on LAION-2B that achieved 78.0% zero-shot accuracy on ImageNet and 73.4% Recall@5 on MS-COCO image retrieval.
- Predictive Power: Using their derived scaling laws, the authors successfully extrapolated performance to larger compute budgets. They predict that a giant ViT-G/14 model trained on 68 billion samples would achieve an 81.92% zero-shot top-1 accuracy on ImageNet purely from contrastive pre-training.
Limitations and Future Work
- Sparse Sampling at Scale: Due to the extreme financial and environmental costs of training large models, the data points at the highest end of the compute spectrum (e.g., ViT-g/14) are sparse, limiting the resolution of the Pareto frontier.
- Hyperparameter Tuning Constraints: It was impossible to perform exhaustive hyperparameter sweeps for the largest models, meaning their performance might be slightly sub-optimal relative to the theoretical ideal.
- Entangled Encoders: This study scaled the text and vision encoders together. Future work is needed to isolate and decouple the scaling laws of the text tower versus the vision tower independently.
Summary updated