DisCo-CLIP: A Distributed Contrastive Loss for Memory Efficient CLIP Training

Yihao Chen, Xianbiao Qi, Jianan Wang, Lei Zhang

Training state-of-the-art vision-language models like CLIP is famously resource-intensive. While the recipe for success—larger datasets and massive batch sizes—is well-known, the "memory wall" often stops researchers in their tracks. A new paper from the IDEA research team introduces DisCo-CLIP, a distributed contrastive loss that significantly slashes memory requirements, potentially democratizing large-scale CLIP training.

The Memory Wall in Contrastive Learning

The magic of CLIP (Contrastive Language-Image Pre-training) lies in its ability to learn from millions of image-text pairs by pushing "positive" pairs together and pulling "negative" pairs (unrelated ones) apart. To make this learning effective, very large batch sizes ($B$) are required to provide sufficient negative samples.

However, calculating the contrastive loss requires a similarity matrix of size $B \times B$. As the batch size grows, memory consumption increases quadratically. For a batch size of 65,536, the similarity matrix alone consumes 16GB of VRAM in Float32. When combined with the memory needed for model backbones like ViT-Large, even high-end GPUs like the A100 40GB quickly encounter Out-Of-Memory (OOM) errors. This has restricted large-scale CLIP training to organizations with massive compute clusters.

Distributed Contrastive Loss (DisCo)

The authors propose a distributed solution called Distributed Contrastive Loss (DisCo). In standard CLIP training, every GPU gathers all features from all other GPUs and independently calculates the entire $B \times B$ similarity matrix, leading to high redundancy. DisCo-CLIP decomposes this process:

  1. Decomposition: It splits the loss and gradient computation into intra-GPU (local data) and inter-GPU (remote data) parts.
  2. Communication over Computation: Instead of every GPU calculating the full matrix, each GPU only calculates a $B/N \times B$ slice.
  3. All-Reduce: The inter-GPU gradients are synchronized using an all_reduce operation.

Crucially, this is an exact mathematical equivalent to the original CLIP loss, not an approximation. As shown in the complexity analysis below, DisCo-CLIP reduces the memory cost of the contrastive loss from $O(B^2)$ to $O(B^2/N)$.

Complexity comparison of different methods. DisCo-CLIP reduces contrastive loss memory complexity to $O(B^2/N)$.

Key Results and Scaling Performance

The researchers evaluated DisCo-CLIP using large-scale datasets like LAION-400M and LAION-2B. The method demonstrated significant advantages in both memory efficiency and scaling:

  • Memory Savings: DisCo-CLIP reduced the memory footprint of the contrastive loss by roughly 97% when using 64 GPUs.
  • Scaling Capability: While standard CLIP implementations OOM at a batch size of 65,536 on 64 GPUs, DisCo-CLIP successfully trained a ViT-B/32 model with a massive batch size of 196,608.
  • Efficiency Gains: The reduction in redundant floating-point operations (FLOPs) led to a ~20.5% improvement in total training efficiency.
  • Accuracy Parity: Validation tests confirmed that DisCo-CLIP achieves the exact same zero-shot accuracy as standard CLIP, maintaining mathematical soundness.
Scaling capabilities of DisCo-CLIP compared to standard CLIP. DisCo-CLIP enables much larger batch sizes without Out-Of-Memory (OOM) errors.

Conclusion

DisCo-CLIP addresses the "memory wall" of large-scale contrastive learning by eliminating redundant computations across distributed GPUs. By enabling a 32K batch size on just 8 GPUs—a task previously requiring 128—this method makes high-performance CLIP training accessible to a much broader range of research labs. Future work may involve combining DisCo-CLIP with gradient checkpointing to further optimize backbone memory usage.

Summary updated

Abstract

We propose DisCo-CLIP, a distributed memory-efficient CLIP training approach, to reduce the memory consumption of contrastive loss when training contrastive learning models. Our approach decomposes the contrastive loss and its gradient computation into two parts, one to calculate the intra-GPU gradients and the other to compute the inter-GPU gradients. According to our decomposition, only the intra-GPU gradients are computed on the current GPU, while the inter-GPU gradients are collected via all_reduce from other GPUs instead of being repeatedly computed on every GPU. In this way, we can reduce the GPU memory consumption of contrastive loss computation from $\bigO(B^2)$ to $\bigO(\frac{B^2}{N})$, where $B$ and $N$ are the batch size and the number of GPUs used for training. Such a distributed solution is mathematically equivalent to the original non-distributed contrastive loss computation, without sacrificing any computation accuracy. It is particularly efficient for large-batch CLIP training. For instance, DisCo-CLIP can enable contrastive training of a ViT-B/32 model with a batch size of 32K or 196K using 8 or 64 A100 40GB GPUs, compared with the original CLIP solution which requires 128 A100 40GB GPUs to train a ViT-B/32 model with a batch size of 32K. The code will be released at https://github.com/IDEA-Research/DisCo-CLIP
BibTeX
@inproceedings{chen2023disco,
  author = {Yihao Chen and Xianbiao Qi and Jianan Wang and Lei Zhang},
  title = {{DisCo-CLIP: A Distributed Contrastive Loss for Memory Efficient CLIP Training}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.02169},
}
Figure