CLIPA-v2: Scaling CLIP Training with 81.1% Zero-shot ImageNet Accuracy within a $10,000 Budget

Xianhang Li, Zeyu Wang, Cihang XieR0-FoMo @ NeurIPS 2023 · OpenReview

TL;DR

CLIPA-v2 introduces an efficient method for training high-performance CLIP models by extending the "inverse scaling law" to the finetuning stage and scaling up experiments. This allows for achieving state-of-the-art zero-shot ImageNet accuracy (81.1%) with a remarkably low budget ($10,000), significantly reducing computational costs compared to previous methods.

What Problem Does This Paper Address?

Training large-scale vision-language models like CLIP is computationally expensive, creating a barrier for many researchers and institutions. While CLIP's ability to perform zero-shot classification and retrieval is revolutionary, its high training cost limits its accessibility. Previous work (CLIPA) identified an "inverse scaling law" where larger models could be trained effectively with fewer input tokens, reducing computational needs. However, the applicability of this law during the finetuning stage and its potential for even greater efficiency and performance at scale remained to be fully explored.

Methods

CLIPA-v2 builds upon the CLIPA framework by introducing two key contributions:

  • Inverse Scaling Law in Finetuning: The authors demonstrate that the inverse scaling law, which suggests using fewer input tokens for larger models, is also effective during the finetuning phase. This allows for further reductions in computational requirements without sacrificing performance.
  • Scaling Up Experiments: CLIPA-v2 explores the model's capabilities at a larger scale, investigating the impact of model size (up to H/14), data volume (up to ~13B image-text pairs), and training schedules. They also explore different masking strategies (random, grid, block) and progressive finetuning with increasing resolutions.

Datasets

  • Training: LAION-400M, LAION-2B, and DataComp-1B datasets were used for pretraining and finetuning.
  • Evaluation: ImageNet (for zero-shot top-1 accuracy) was the primary evaluation benchmark. Performance was also evaluated on COCO and Flickr30k for zero-shot retrieval tasks, and on IN-V2, IN-A, IN-R, ObjectNet, and IN-SK for zero-shot robustness.

Key Insights and Ideas

  • Finetuning Efficiency: The inverse scaling law isn't just for pretraining; it significantly reduces computational costs during finetuning, making CLIP model adaptation more accessible.
  • Larger Models, Fewer Tokens: Larger models (like H/14) are more robust to token reduction during finetuning, showing smaller performance drops compared to smaller models.
  • Data is Key (but efficiency matters): While scaling up data (e.g., to LAION-2B and DataComp-1B) improves performance, the CLIPA-v2 approach achieves high accuracy with less data and computation than prior methods.
  • Progressive Resolution Finetuning Pays Off: Gradually increasing input resolution during finetuning, combined with a smaller learning rate, yields a notable performance boost.
  • Random Masking is Optimal: Among the tested masking strategies for token reduction, random masking consistently provided the best performance.

Key Results

  • Budget-Friendly High Accuracy: CLIPA-v2 achieves an impressive 81.1% zero-shot ImageNet accuracy with a training budget of just $10,000.
  • Significant Cost Reduction: This performance surpasses the previous best CLIP model (OpenCLIP, 80.1%) by 1.0% while reducing computational cost by approximately 39x.
  • Further Boost: An additional $4,000 investment further elevates the accuracy to 81.8%.
  • Robustness: The 81.8% H/14 model demonstrates superior zero-shot robustness across various out-of-distribution datasets (IN-V2, IN-A, IN-R, ObjectNet, IN-SK) compared to OpenCLIP.
  • Retrieval Trade-off: While excelling in classification, CLIPA-v2 models trained on DataComp-1B showed slightly lower retrieval performance on COCO and Flickr30k compared to models trained on LAION-2B, suggesting a dataset-specific trade-off.

Limitations and Future Work

  • Retrieval Performance: The paper notes that while CLIPA-v2 excels in zero-shot classification, OpenCLIP models trained on LAION-2B performed better on zero-shot retrieval tasks. The authors hypothesize this is due to differences in training datasets and suggest further investigation into this aspect.
  • Open-Sourcing: The authors have open-sourced their training code and models to facilitate future research and broader adoption of advanced CLIP models.
  • Future Directions: The paper implicitly suggests further exploration of scaling laws, dataset curation, and efficient training strategies for vision-language models.

Summary updated

Abstract

The recent work CLIPA presents an inverse scaling law for CLIP training --- whereby the larger the image/text encoders used, the shorter the sequence length of image/text tokens that can be applied in training. This finding enables us to train high-performance CLIP models with significantly reduced computations. Building upon this work, we hereby present CLIPA-v2 with two key contributions. Technically, we find this inverse scaling law is also applicable in the finetuning stage, enabling further reduction in computational needs. Empirically, we explore CLIPA at scale, extending the experiments up to the H/14 model with approximately 13B image-text pairs seen during training. Our results are exciting --- by only allocating a budget of $\textdollar$10,000, our CLIP model achieves an impressive zero-shot ImageNet accuracy of 81.1%, surpassing the prior best CLIP model (from OpenCLIP, 80.1%) by 1.0\% and meanwhile reducing the computational cost by approximately $39\times$. Moreover, with an additional investment of $4,000, we can further elevate the zero-shot ImageNet accuracy to 81.8%. By upscaling a G/14 model, we've achieved an impressive state-of-the-art zero-shot ImageNet accuracy of 83.0%, relying solely on open-source data.
BibTeX
@inproceedings{li2023clipa,
  author = {Xianhang Li and Zeyu Wang and Cihang Xie},
  title = {{CLIPA-v2: Scaling CLIP Training with 81.1\% Zero-shot ImageNet Accuracy within a $10,000 Budget}},
  booktitle = {R0-FoMo @ NeurIPS},
  year = {2023},
  url = {https://openreview.net/forum?id=0hTtit3AAm},
}
Figure