Abstract
Multimodal contrastive learning is a methodology for linking different data modalities, such as image and text, by aligning their representations within a common latent space. In this work, we interpret contrastive learning through a probabilistic lens, viewing it as the optimization of encoders that define conditional probability distributions consistent with observed data. This framework allows us to generalize classical contrastive methods by introducing novel probabilistic loss functions and alternative alignment metrics. By analyzing these generalizations in a multivariate Gaussian setting, we characterize the ability of specific loss functions to approximate statistics like conditional means and covariances. Our findings are validated through numerical experiments on synthetic Gaussians, the MNIST dataset, and a complex data assimilation application in oceanography.
Introduction
Contrastive learning has become the backbone of modern multimodal AI, powering models like CLIP that bridge the gap between vision and language. Despite its empirical success, the field has largely relied on the geometric heuristic of "pushing similar samples together and pulling dissimilar ones apart." While intuitive, this approach lacks a unified probabilistic language, making it difficult to understand what these models truly learn about the underlying data distribution.
This paper provides a rigorous mathematical foundation for contrastive learning by reframing it as a problem of matching probability distributions. We demonstrate that standard contrastive methods are essentially "mode-seekers"—excellent at finding the most likely match (the mean) but often failing to capture the full statistical structure, such as the variance or uncertainty, of the data.
A Probabilistic Framework: Tilting and Alignment
We move away from simple geometric intuition toward a concept we call Probabilistic Tilting. In this view, contrastive learning is a method to "tilt" the independent product of two data modalities into a joint distribution that reflects their real-world dependencies.
Novel Loss Functions
We introduce two primary generalizations of the contrastive objective:
- Conditional Loss ($J_{cond}$): Focuses on matching conditional distributions (e.g., the probability of a specific text given an image).
- Joint Loss ($J_{joint}$): Aims to match the full joint distribution of both modalities.
We prove that the standard InfoNCE loss used in models like CLIP is actually a specific discrete approximation of a more general probabilistic objective. By exploring unnormalized encoders and quadratic forms (L2-distance) instead of the standard cosine similarity, we can better capture the statistical nuances of the data.
The Gaussian Setting and the "Variance Gap"
To make the mathematics tractable, we analyze these objectives in the multivariate Gaussian setting. Here, identifying the optimal latent space becomes a low-rank matrix approximation problem. This allows us to derive closed-form solutions for the best possible encoders.
One of our most significant findings is the "Variance Gap." We prove that standard contrastive learning using cosine distance correctly identifies the mean of the data but consistently overestimates the variance. This explains why standard CLIP-like models struggle with generative tasks; they can find the "best" match but fail to represent the diverse range of possible valid matches. In contrast, our proposed "Positive Quadratic Form" loss can perfectly match both the mean and the covariance.
Experimental Validation
MNIST: Diversity vs. Classification
Using the MNIST handwritten digit dataset, we tested how different losses affect model behavior. We found that a "one-sided" conditional loss, optimized for classification, often leads to "mode collapse," where the model repeatedly samples the same few images. However, using a "two-sided" conditional loss allows the model to sample a diverse set of variations for a single digit, respecting the true distribution of the data.
Lagrangian Ocean Data Assimilation
To demonstrate the framework's utility in the physical sciences, we applied it to a data assimilation task: reconstructing ocean velocity fields (Eulerian data) from the paths of floating sensors (Lagrangian data). This is a complex inverse problem usually solved with heavy differential equations. Our transformer-based contrastive model achieved nearly 100% accuracy in R@5 retrieval, proving that contrastive learning can effectively learn the physical relationships between disparate scientific data types.
Conclusion
By reframing contrastive learning as a probabilistic distribution-matching problem, we provide a path toward more robust and versatile AI models. While our detailed proofs currently rely on Gaussian assumptions, the empirical success on MNIST and oceanographic data suggests these insights are broadly applicable. Future work will focus on extending this theory to non-linear settings and multimodal data (beyond two modalities) to further reduce mode collapse in large-scale generative models.