In traditional vision-language models like CLIP, we treat understanding as a game of "connect the dots." We attempt to map an image and its corresponding caption to the exact same point in a high-dimensional mathematical space. If the points are close, the model "understands" the relationship; if they are far, it does not.
However, real-world semantics are rarely point-like. As noted by Yatai Ji and colleagues in their work on MAP, a single image of a "meal" is simultaneously an "avocado sandwich," "healthy food," and "lunch." Conversely, a single word like "bank" could refer to a river's edge or a financial institution. Forcing these multi-faceted, hierarchical, and often ambiguous concepts into rigid, deterministic vectors creates a "representation bottleneck" that fails to capture the inherent uncertainty of human language and vision.
To solve this, a growing body of research is shifting the paradigm from point embeddings to probabilistic distributions.
From Points to Clouds
Instead of representing an input as a single coordinate, these models map images and text to probability distributions—typically multivariate Gaussians. In this framework, an embedding is defined by two components:
- The Mean ($\mu$): Represents the "core" semantic concept (the center of the cloud).
- The Variance ($\sigma$): Represents the "uncertainty" or "breadth" of the concept (the size of the cloud).
This approach, pioneered in works like Hedged Instance Embedding (HIB) by Seong Joon Oh and Kevin Murphy, allows a model to "hedge its bets." If an image is blurry, occluded, or contains a broad category (like "animal"), the model assigns it a high variance, creating a large "cloud" in the latent space that can overlap with many specific sub-concepts (like "dog," "cat," or "elephant").
Capturing One-to-Many Relationships
The primary advantage of probabilistic modeling is its ability to handle one-to-many and many-to-many relationships. In standard retrieval, if a model is forced to map one caption to five different valid images, the deterministic loss functions will pull the caption toward the average of those images—a "mean" point that might not actually represent any of them well.
By using distributions, models like PCME++, developed by Sanghyuk Chun, allow the embedding space to reflect semantic overlap. As Chun demonstrates in ProLIP, when a model is pre-trained on billion-scale datasets using these "fuzzy" regions, it naturally learns to represent aleatoric uncertainty—the randomness inherent in how different people describe the same visual scene.
Robustness to Noise and Bias
Probabilistic embeddings also act as a natural filter for noise. In video-text retrieval, Huaiwen Zhang and others introduced DVTR to address "soft positives." Often, a "negative" sample in a dataset (a video not explicitly labeled for a query) might actually be a good match. Deterministic models penalize these matches harshly. Probabilistic models, however, can estimate the uncertainty of these matches, dynamically down-weighting the penalty for samples that fall within the "uncertainty cloud" of the query.
This robustness extends to specialized domains:
- Medical Imaging: Qingtao Pan and Shuo Li developed DuSSS, which uses Gaussian distributions to manage the high ambiguity between complex medical scans and their textual descriptions, preventing the model from overfitting to noisy pseudo-labels.
- Sign Language: UPRet by Xuan Wu applies this to sign language videos, where a single sign often has multiple valid textual interpretations depending on context.
- Drone Imagery: MSAM uses distributions to handle the tiny, ambiguous objects found in high-altitude aerial footage.
Measuring Similarity Between Clouds
When embeddings are distributions, we can no longer use simple cosine similarity. Instead, researchers employ metrics that measure the overlap or "distance" between two probability distributions:
- Expected Likelihood Kernel: Used in PCME to calculate how likely a point sampled from the image distribution is to have been generated by the text distribution.
- Wasserstein Distance / Optimal Transport: Used in frameworks like UPRet and Implicit Counterfactual Learning to provide a more robust geometric alignment between multimodal "clouds."
By embracing uncertainty rather than trying to eliminate it, these models achieve a more nuanced and human-like understanding of the world, where meaning is not a fixed point, but a range of possibilities.