TL;DR
CLIPA-v2 introduces an efficient method for training high-performance CLIP models by extending the "inverse scaling law" to the finetuning stage and scaling up experiments. This allows for achieving state-of-the-art zero-shot ImageNet accuracy (81.1%) with a remarkably low budget ($10,000), significantly reducing computational costs compared to previous methods.
What Problem Does This Paper Address?
Training large-scale vision-language models like CLIP is computationally expensive, creating a barrier for many researchers and institutions. While CLIP's ability to perform zero-shot classification and retrieval is revolutionary, its high training cost limits its accessibility. Previous work (CLIPA) identified an "inverse scaling law" where larger models could be trained effectively with fewer input tokens, reducing computational needs. However, the applicability of this law during the finetuning stage and its potential for even greater efficiency and performance at scale remained to be fully explored.
Methods
CLIPA-v2 builds upon the CLIPA framework by introducing two key contributions:
- Inverse Scaling Law in Finetuning: The authors demonstrate that the inverse scaling law, which suggests using fewer input tokens for larger models, is also effective during the finetuning phase. This allows for further reductions in computational requirements without sacrificing performance.
- Scaling Up Experiments: CLIPA-v2 explores the model's capabilities at a larger scale, investigating the impact of model size (up to H/14), data volume (up to ~13B image-text pairs), and training schedules. They also explore different masking strategies (random, grid, block) and progressive finetuning with increasing resolutions.
Datasets
- Training: LAION-400M, LAION-2B, and DataComp-1B datasets were used for pretraining and finetuning.
- Evaluation: ImageNet (for zero-shot top-1 accuracy) was the primary evaluation benchmark. Performance was also evaluated on COCO and Flickr30k for zero-shot retrieval tasks, and on IN-V2, IN-A, IN-R, ObjectNet, and IN-SK for zero-shot robustness.
Key Insights and Ideas
- Finetuning Efficiency: The inverse scaling law isn't just for pretraining; it significantly reduces computational costs during finetuning, making CLIP model adaptation more accessible.
- Larger Models, Fewer Tokens: Larger models (like H/14) are more robust to token reduction during finetuning, showing smaller performance drops compared to smaller models.
- Data is Key (but efficiency matters): While scaling up data (e.g., to LAION-2B and DataComp-1B) improves performance, the CLIPA-v2 approach achieves high accuracy with less data and computation than prior methods.
- Progressive Resolution Finetuning Pays Off: Gradually increasing input resolution during finetuning, combined with a smaller learning rate, yields a notable performance boost.
- Random Masking is Optimal: Among the tested masking strategies for token reduction, random masking consistently provided the best performance.
Key Results
- Budget-Friendly High Accuracy: CLIPA-v2 achieves an impressive 81.1% zero-shot ImageNet accuracy with a training budget of just $10,000.
- Significant Cost Reduction: This performance surpasses the previous best CLIP model (OpenCLIP, 80.1%) by 1.0% while reducing computational cost by approximately 39x.
- Further Boost: An additional $4,000 investment further elevates the accuracy to 81.8%.
- Robustness: The 81.8% H/14 model demonstrates superior zero-shot robustness across various out-of-distribution datasets (IN-V2, IN-A, IN-R, ObjectNet, IN-SK) compared to OpenCLIP.
- Retrieval Trade-off: While excelling in classification, CLIPA-v2 models trained on DataComp-1B showed slightly lower retrieval performance on COCO and Flickr30k compared to models trained on LAION-2B, suggesting a dataset-specific trade-off.
Limitations and Future Work
- Retrieval Performance: The paper notes that while CLIPA-v2 excels in zero-shot classification, OpenCLIP models trained on LAION-2B performed better on zero-shot retrieval tasks. The authors hypothesize this is due to differences in training datasets and suggest further investigation into this aspect.
- Open-Sourcing: The authors have open-sourced their training code and models to facilitate future research and broader adoption of advanced CLIP models.
- Future Directions: The paper implicitly suggests further exploration of scaling laws, dataset curation, and efficient training strategies for vision-language models.
Summary updated