The "Law'' of the Unconscious Contrastive Learner: Probabilistic Alignment of Unpaired Modalities

Yongwei Che, Benjamin Eysenbach

Abstract

Modern multimodal AI relies heavily on contrastive learning to align disparate data types. A curious phenomenon has emerged: models trained on separate pairs (e.g., Image-Text and Audio-Text) often exhibit zero-shot alignment between modalities they never saw together (Image-Audio). This paper formalizes this phenomenon as the "Law of the Unconscious Contrastive Learner." We provide a theoretical proof for why simple dot products between unpaired embeddings can be mathematically principled under specific geometric conditions and introduce a robust LogSumExp (LSE) method to handle cases where these conditions fail.

Introduction

In the current landscape of multimodal research, models like CLIP (Contrastive Language-Image Pre-training) have set the standard for cross-modal retrieval. However, a significant bottleneck remains: the scarcity of multi-way datasets. While we possess vast repositories of (Image, Text) and (Audio, Text) pairs, we rarely have high-quality triplets containing all three.

Despite this, practitioners frequently use a "plug-and-play" approach, calculating the similarity between image and audio embeddings—which were never co-trained—using a simple dot product. Empirically, this works surprisingly well. This paper investigates the theoretical underpinnings of this emergent alignment, asking whether this "unconscious" learning captures true underlying probability distributions or is merely a lucky heuristic.

The "Law" of the Unconscious Contrastive Learner

The authors bridge the gap between Contrastive Learning and Probabilistic Graphical Models to explain this phenomenon. Their theoretical framework rests on the insight that contrastive models do not merely match features; they implicitly model the ratio of the conditional distribution to the marginal distribution: $p(B|A)/p(B)$.

Theoretical Foundations

The paper proves that the "naive" dot product between two unpaired representations is a monotonic function of the true probability ratio, provided two key assumptions hold:

  1. Conditional Independence: The modalities (e.g., Image and Audio) must be conditionally independent given a "bridge" modality (e.g., Text).
  2. Representation Uniformity: The learned representations must be uniformly distributed on a hypersphere.

When these conditions are met, the geometry of the embedding space naturally aligns unpaired modalities, justifying the common practice of direct dot-product comparison.

Beyond the Law: The LogSumExp (LSE) Method

Real-world embeddings often violate the assumption of uniformity, appearing "clumpy" or non-uniform. In such cases, the standard dot product becomes an unreliable measure of similarity. To address this, the authors propose the LogSumExp (LSE) method, a more general algorithm for probabilistic inference.

Instead of a direct comparison, LSE uses a Monte Carlo approximation that integrates over the intermediate bridge modality. By summing the exponentiated similarities across a bank of bridge samples (such as a set of text descriptions), the method can recover the correct probability ratio even when the "Law" is violated. This approach allows the model to maintain the full distribution of possible matches, which is particularly crucial in ambiguous scenarios.

Empirical Evaluation

The researchers validated their theory across synthetic environments and large-scale real-world datasets.

Zero-Shot Audio-Visual Alignment

Using LSE to bridge CLIP (Vision-Language) and CLAP (Audio-Language), the authors evaluated performance on AudioSet. The results were stark: the LSE method achieved 62% Recall@10, whereas the naive direct evaluation baseline managed only 14%. This suggests that for models not explicitly trained together, the LSE method is essential for unlocking latent alignment.

Handling Ambiguity in Reinforcement Learning

In navigation tasks within the PointMaze environment, agents were given ambiguous language instructions (e.g., "the first column"). Standard dot-product methods often failed by pulling the agent toward the "mean" or average position of all possible interpretations, frequently resulting in the agent hitting a wall. Conversely, the LSE method correctly identified the distinct peaks of the probability distribution, boosting success rates by 20-30% over baseline methods.

Discussion and Limitations

The "Law of the Unconscious Contrastive Learner" provides a much-needed theoretical anchor for multimodal alignment. However, the LSE method does introduce higher computational costs, as it requires processing a matrix of intermediate representations. Furthermore, the theory's reliance on conditional independence means that if two modalities share a direct correlation not captured by the bridge, the alignment remains an approximation.

Future work should focus on developing diagnostic tools to help practitioners determine when the geometry of their embeddings necessitates the LSE approach over the simpler, but more fragile, dot product. By understanding the probabilistic nature of contrastive learners, we can better design systems that generalize across the vast, unpaired landscape of human data.

Summary updated

Abstract

While internet-scale data often come in pairs (e.g., audio+image, image+text), we often want to perform inferences over modalities unseen together in the training data (e.g., audio+text). Prior work has addressed this issue by learning multiple contrastive embedding spaces between existing modality pairs, implicitly hoping that unseen modality pairs will end up being aligned. This theoretical paper proves that this hope is well founded, under certain assumptions. Starting with the proper Bayesian approach of integrating out intermediate modalities, we show that directly comparing the representations of data from unpaired modalities can recover the same likelihood ratio. Our analysis builds on prior work on the geometry and probabilistic interpretation of contrastive representations, showing how these representations can answer many of the same inferences as probabilistic graphical models. Our analysis suggests two new ways of using contrastive representations: in settings with pre-trained contrastive models, and for handling language ambiguity in reinforcement learning. Our numerical experiments study the importance of our assumptions and demonstrate these new applications.
BibTeX
@inproceedings{che2025law,
  author = {Yongwei Che and Benjamin Eysenbach},
  title = {{The "Law'' of the Unconscious Contrastive Learner: Probabilistic Alignment of Unpaired Modalities}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=DsIOUoZkVk},
}
Figure