Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shift

Yihao Xue, Siddharth Joshi, Dang Nguyen, Baharan Mirzasoleiman

Introduction

Multi-modal models like CLIP have demonstrated a remarkable ability to maintain high performance across diverse and unexpected distribution shifts—a feat that standard Supervised Learning (SL) models often fail to achieve. While the community has frequently attributed this robustness to the sheer scale of training data, this research provides a rigorous theoretical explanation identifying two fundamental drivers: intra-class contrasting and inter-class feature sharing.

Deep learning models are typically "lazy," latching onto the simplest features to minimize loss. In SL, this often results in the model picking up domain-dependent "spurious" features (e.g., a green grass background) instead of the "core" features (e.g., the cow). When the background changes, the model fails. Multi-modal Contrastive Learning (MMCL) avoids this trap not just through data volume, but through the specific nature of its loss function and the descriptive richness of its captions.

Theoretical Framework: MMCL vs. SL

The authors model data as a set of underlying feature vectors containing both core features (the object) and spurious features (the background). By comparing the mathematical objectives of MMCL and SL, they identify why the former is inherently more robust.

1. Intra-class Contrasting

In standard SL, all images of a "cow" are mapped to the same label. The model has no incentive to distinguish between different cows; it only needs to find a common denominator, which is often a stable spurious feature like the background. In contrast, MMCL attempts to match a specific image of a cow to its unique caption. This forces the model to attend to the unique variations within the class—high-variance core features—rather than settling for the uniform spurious features shared across the class.

2. Inter-class Feature Sharing

MMCL utilizes captions that often describe secondary details. If a caption for a "wolf" mentions a "tree" in the background, the model learns the "tree" feature independently. When the model later encounters a "tree" class, it has already decoupled the tree from other contexts. This cross-pollination of features prevents the model from treating a specific background as an inseparable part of a specific object category.

The Role of Caption Richness

A central finding of the paper is that robustness is a direct product of "caption richness." If captions are reduced to simple class labels, MMCL loses its advantage. The authors introduce a parameterization of caption content to demonstrate how the presence of core feature variance in text directly correlates with out-of-distribution (OOD) performance.

Figure 1: Visualization of simulated captions in the theoretical framework. The vectors represent how captions emphasize core features (e.g., digit shape) versus spurious features (e.g., background color) based on the parameters $\pi_{core}$ and $\pi_{spu}$.

As shown in the simulated captions above, when the probability of the caption describing the core feature ($\pi_{core}$) is high, the model is forced to represent those features in its embedding space. Conversely, if the caption only reflects spurious correlations ($\pi_{spu}$), the model's robustness degrades.

Empirical Validation

The theory was validated using both synthetic and real-world datasets:

  • Semi-synthetic MNIST: Digits were correlated with specific background colors during training but randomized during testing. MMCL maintained up to 81% accuracy under shift, while SL dropped to near-random performance.
  • Real-world Evaluation: Models trained on MSCOCO and CC3M were tested on six shifted versions of ImageNet (e.g., ImageNet-Sketch, ObjectNet). CLIP-style training consistently yielded a superior OOD-to-ID (In-Distribution) accuracy trade-off compared to SL or uni-modal contrastive methods like SimCLR.

Key Insights

  • Variance is a Virtue: In MMCL, high-variance features described in text (e.g., "a white cow") become the easiest for the model to learn, effectively overshadowing stable but irrelevant spurious features.
  • The Zero-shot Advantage: The robustness of CLIP is often fragile; fine-tuning the model with a linear head using standard labels can re-introduce the "lazy" preference for spurious correlations, destroying the zero-shot robustness.
  • The Power of "Also": Captions mentioning non-primary objects (e.g., "a dog on a beach") allow the model to learn the "beach" feature as a separate entity, preventing it from becoming a spurious cue for the "dog" label.

Conclusion

This work moves beyond the "more data" argument to show that the structure of the contrastive loss and the descriptive nature of language are what truly immunize models against distribution shifts. Future work may involve developing automated metrics for "caption richness" to better curate web-scale datasets for robustness rather than just volume.

Summary updated

BibTeX
@inproceedings{xue2024understanding,
  author = {Yihao Xue and Siddharth Joshi and Dang Nguyen and Baharan Mirzasoleiman},
  title = {{Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shift}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=rtl4XnJYBh},
}
Figure