Probabilistic Embeddings for Multimodal Ambiguity

86 concepts71 papers1 proposal

Imagine you are training a model to connect images and text. You feed it a photo of a golden retriever playing in a park, paired with the caption "a golden retriever catching a frisbee." A standard vision-language model like CLIP will try to project both the image and the text into a shared embedding space, forcing them to land on the exact same mathematical coordinate.

But what happens when the model encounters the caption "a dog" or "a sunny day"? Both are entirely accurate descriptions of the same image. Conversely, the phrase "a dog" could validly match millions of completely different images, from a tiny chihuahua on a couch to a husky in the snow.

This is the one-to-many and many-to-many mapping problem. Real-world semantic relationships are inherently ambiguous and multi-faceted. When we force a model to map these complex, overlapping concepts to single, deterministic points in space, we create a fundamental geometric conflict.

The Flaw of Deterministic Points

In traditional contrastive learning, models assume a strict one-to-one correspondence between modalities. As Ji et al. observe, this deterministic approach fails to account for the inherent noise and compositionality of real-world data; a single image region might simultaneously represent a "billboard," "zebras," and "mountains."

When a model is forced to compress an entire spectrum of valid interpretations into a single vector, it suffers from semantic collapse. To satisfy multiple conflicting training signals at once, the model often resorts to producing blurred or averaged representations. As Reed et al. noted in early text-to-image synthesis research, this averaging prevents models from capturing distinct, sharp visual details.

Furthermore, deterministic models struggle with "soft positives"—unlabeled pairs in a dataset that are semantically valid matches. If a model randomly samples a "negative" text that actually describes the image well, a rigid point-based loss will aggressively penalize the model for a correct semantic association. Zhang et al. and Byun et al. demonstrate that this lack of uncertainty representation makes it incredibly difficult for models to distinguish true negatives from missing labels during training.

The Solution: Mapping to Distributions

To resolve this geometric bottleneck, researchers have shifted from mapping data to fixed points to mapping data to probability distributions. Instead of predicting a single vector $\mathbf{z}$, a neural network outputs the parameters of a distribution—typically the mean ($\mathbf{\mu}$) and the variance ($\mathbf{\sigma}^2$) of a Gaussian.

Deterministic Embedding:
Input (Image/Text)  ───►  [  Vector Point (μ)  ]

Probabilistic Embedding:
Input (Image/Text)  ───►  [  Mean (μ)  ]  ──┐
                    ───►  [ Variance (σ²) ]  ──┴─► [ Distribution Region ]

This simple shift changes how the embedding space behaves:

  • The Mean ($\mathbf{\mu}$) represents the core semantic concept (e.g., the general category of an object).
  • The Variance ($\mathbf{\sigma}^2$) represents the aleatoric uncertainty or semantic breadth of the input.

A highly specific caption, such as "a three-year-old golden retriever catching a red frisbee on a beach," leaves little room for interpretation. The model will assign it a small variance, collapsing it into a tight, localized region of the embedding space.

Conversely, a vague or polysemous input like "an animal" or a loose hand-drawn sketch will be assigned a large variance. This creates a wide, expansive distribution that naturally overlaps with the tighter distributions of its many valid sub-concepts (like dogs, cats, or birds).

This foundational framework, termed Probabilistic Cross-Modal Embedding (PCME), was introduced by Chun et al.. By allowing models to "hedge their bets" via variance, PCME preserves the multi-faceted nature of language and vision without sacrificing specificity.

Measuring Similarity in Distribution Space

If representations are now volumes instead of points, how do we calculate the similarity between an image and a text query? We can no longer rely on a simple dot product or cosine similarity. Instead, we must measure the overlap between two distributions.

Chun et al. achieve this by calculating the expected dot product between the two Gaussian distributions. This mathematical formulation yields a closed-form solution that rewards two things: how close the centers (means) of the distributions are, and how well their uncertainties align. In a follow-up single-author paper, Chun introduced PCME++, which refines these distance metrics to efficiently filter out noisy web-scale prompts and handle false negatives in a closed-form mathematical solution.

This probabilistic alignment has been adapted across various complex retrieval tasks:

  • Text-Video Retrieval: Videos often suffer from content asymmetry, where a short caption only describes a fraction of a long video. Fang et al. proposed Probabilistic Token Aggregation (ProTA), which models individual tokens as distributions to capture the shifting semantic range of video frames over time.
  • Person Retrieval: Descriptions of people are notoriously ambiguous. Deng et al. developed UANet, which uses probability distributions to handle the polysemy of human descriptions, preventing retrieval errors caused by subtle variations in phrasing or camera angles.

Compositionality and Expressive Landscapes

Representing concepts as distributions also unlocks powerful algebraic properties for complex queries. In Composed Image Retrieval, a user searches for an image using a reference image combined with a text modifier (e.g., a photo of a dress + the text "make it blue").

Deterministic models struggle to merge these distinct modalities cleanly. However, Neculai et al. demonstrated that by treating queries as Gaussians, models can use a probabilistic composition rule to fuse an arbitrary number of multimodal inputs. The resulting distribution accurately encapsulates the joint uncertainty of both the visual query and the textual modification.

When the underlying data relationships are too complex for a standard Gaussian distribution, researchers leverage even more flexible probabilistic structures. Mahajan et al. introduced Latent Normalizing Flows (LNFMM) to map cross-domain data. Instead of a rigid sphere of uncertainty, normalizing flows warp simple distributions into complex, multi-modal landscapes. This allows a single image embedding to map to an intricate, highly diverse space of multiple distinct, valid textual captions, capturing the true many-to-many essence of human expression.

Key Foundational Work

Sense discovery via co-clustering on images and text (2015) — contributed 2 concept elements to this direction.
Multi-label Cross-Modal Retrieval (2015) — contributed 1 concept elements to this direction.
Generative Adversarial Text to Image Synthesis (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_deterministic_embedding_limitation_d308d6df,
  title = {deterministic embedding limitation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/deterministic-embedding-limitation/rdr_d308d6dfb1da49e49d6a981116aba2dd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_deterministic_embedding_limitation_d308d6df_v1,
  title = {deterministic embedding limitation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/deterministic-embedding-limitation/rdr_d308d6dfb1da49e49d6a981116aba2dd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake