Reproducible scaling laws for contrastive language-image learning

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, Jenia Jitsev

TL;DR

This paper provides the first systematic, fully open-source study of scaling laws for contrastive language-image pre-training (CLIP). By training models on up to 2 billion public image-text pairs using the OpenCLIP repository, the authors demonstrate that multi-modal performance follows predictable power-law scaling across various downstream tasks, while revealing that the underlying data distribution significantly alters these scaling trajectories.

What Problem Does This Paper Address?

While scaling laws—the predictable mathematical relationships between compute, model size, data scale, and performance—have been widely mapped out for uni-modal domains like language and vision, multi-modal models have remained a blind spot. Previous attempts to study CLIP scaling relied heavily on proprietary datasets (like OpenAI's WIT-400M) and closed-source infrastructure. This lack of transparency has hindered reproducibility and prevented the broader research community from understanding how multi-modal models behave as they scale up. The authors address this gap by establishing reproducible, public baselines for CLIP scaling.

Methods

The authors systematically scaled three primary dimensions of contrastive learning:

  1. Model Size: Visual encoders ranged from ViT-B/32 (93M parameters) up to ViT-g/14 (1.37B parameters).
  2. Data Scale: Subsets of the public LAION dataset were used, scaling from LAION-80M to LAION-400M, and finally the full LAION-2B English subset.
  3. Compute (Samples Seen): Total training duration scaled from 3 billion to 34 billion samples seen.

Using the open-source OpenCLIP framework, they trained these configurations on up to 1,520 NVIDIA A100 GPUs. They then fit power-law functions ($E = \beta C^\alpha$, where $E$ is the error rate and $C$ is total compute) along the Pareto frontier to model performance across zero-shot classification, retrieval, linear probing, and fine-tuning.

Datasets

  • Pre-training: LAION-80M, LAION-400M, and LAION-2B (all publicly available, extracted from Common Crawl).
  • Downstream Evaluation: ImageNet-1k (along with robustness variants: ImageNet-V2, ImageNet-R, ImageNet-Sketch, ObjectNet, and ImageNet-A), the Visual Task Adaptation Benchmark (VTAB+), MS-COCO, and Flickr30K.

Key Insights and Ideas

  • Power Laws Hold for Multi-Modal Learning: Just like LLMs, contrastive vision-language models improve predictably according to power laws across zero-shot classification, retrieval, and fine-tuning tasks.
  • Data Distribution Dictates Scaling Behavior: A fascinating finding is that OpenAI's original CLIP models and OpenCLIP models exhibit entirely different scaling slopes despite sharing identical ViT architectures. OpenAI's CLIP scales better on zero-shot ImageNet classification, whereas OpenCLIP scales significantly better on image/text retrieval tasks.
  • The Curation Hypothesis: The authors hypothesize this discrepancy stems from dataset curation. OpenAI's private WIT dataset likely has a higher affinity for ImageNet-style concepts, whereas LAION was filtered using a pre-trained CLIP model based on image-text similarity, inherently biasing it to favor retrieval-based tasks.
  • Scale Bottlenecks: Scaling a single dimension in isolation can mask potential gains. For example, scaling data from 400M to 2B pairs showed no benefit for smaller models until the number of training samples seen was also scaled up to 34B, highlighting the necessity of co-scaling data and compute.
Compute vs. Zero-Shot Classification Error

Key Results

  • State-of-the-Art Open Models: The authors produced the largest public CLIP models at the time, including a ViT-H/14 model trained on LAION-2B that achieved 78.0% zero-shot accuracy on ImageNet and 73.4% Recall@5 on MS-COCO image retrieval.
Compute vs. Zero-Shot Retrieval Performance
  • Predictive Power: Using their derived scaling laws, the authors successfully extrapolated performance to larger compute budgets. They predict that a giant ViT-G/14 model trained on 68 billion samples would achieve an 81.92% zero-shot top-1 accuracy on ImageNet purely from contrastive pre-training.
Zero-Shot Performance Comparison Table

Limitations and Future Work

  • Sparse Sampling at Scale: Due to the extreme financial and environmental costs of training large models, the data points at the highest end of the compute spectrum (e.g., ViT-g/14) are sparse, limiting the resolution of the Pareto frontier.
  • Hyperparameter Tuning Constraints: It was impossible to perform exhaustive hyperparameter sweeps for the largest models, meaning their performance might be slightly sub-optimal relative to the theoretical ideal.
  • Entangled Encoders: This study scaled the text and vision encoders together. Future work is needed to isolate and decouple the scaling laws of the text tower versus the vision tower independently.

Summary updated

Abstract

Scaling up neural networks has led to remarkable performance across a wide range of tasks. Moreover, performance often follows reliable scaling laws as a function of training set size, model size, and compute, which offers valuable guidance as large-scale experiments are becoming increasingly expensive. However, previous work on scaling laws has primarily used private data \& models or focused on uni-modal language or vision learning. To address these limitations, we investigate scaling laws for contrastive language-image pre-training (CLIP) with the public LAION dataset and the open-source OpenCLIP repository. Our large-scale experiments involve models trained on up to two billion image-text pairs and identify power law scaling for multiple downstream tasks including zero-shot classification, retrieval, linear probing, and end-to-end fine-tuning. We find that the training distribution plays a key role in scaling laws as the OpenAI and OpenCLIP models exhibit different scaling behavior despite identical model architectures and similar training recipes. We open-source our evaluation workflow and all models, including the largest public CLIP models, to ensure reproducibility and make scaling laws research more accessible. Source code and instructions to reproduce this study will be available at https://github.com/LAION-AI/scaling-laws-openclip
BibTeX
@inproceedings{cherti2022reproducible,
  author = {Mehdi Cherti and Romain Beaumont and Ross Wightman and Mitchell Wortsman and Gabriel Ilharco and Cade Gordon and Christoph Schuhmann and Ludwig Schmidt and Jenia Jitsev},
  title = {{Reproducible scaling laws for contrastive language-image learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
}
Figure