Stochastic Latent Embeddings for Multimodal Semantic Ambiguity

This direction is merged into another direction: deterministic embedding limitation. Its last version stays readable and citable here.

Version 1 of 118 papers18 conceptsfrozen 2026-10-04

In standard vision-language models like CLIP, an image and its caption are treated as two points that should be pulled together into the exact same location in a high-dimensional mathematical space. This "point-to-point" approach assumes a strict one-to-one relationship: one image has one perfect description, and one description matches one perfect image.

However, real-world data is rarely that simple. As Yatai Ji and Junjie Wang point out in their work on MAP, a single image of a "meal" is simultaneously an "avocado sandwich" and "food." Conversely, a single caption like "a dog in a park" could describe thousands of different images. Forcing these diverse, hierarchical, and often noisy relationships into single deterministic points creates a "black box" that struggles with ambiguity.

From Points to Distributions

To solve this, a growing body of research suggests moving away from deterministic vectors toward probabilistic embeddings. Instead of a single point, models represent images and text as probability distributions—typically multivariate Gaussians.

In this framework, the mean of the distribution represents the central semantic concept, while the variance (or "cloud" around the point) represents uncertainty or breadth. As Sanghyuk Chun demonstrates in ProLIP and PCME++, this allows the model to capture "many-to-many" relationships. A general term like "animal" would have a high-variance distribution (a large cloud) that physically encompasses the lower-variance distributions (smaller clouds) of specific terms like "beagle" or "tabby cat."

Why Probabilistic Modeling Works

The shift to distributions provides three primary advantages over deterministic models:

  1. Handling Semantic Hierarchy: Deterministic points cannot easily represent that one concept is a subset of another. By using distributions, models can use the overlap between "clouds" to represent levels of detail. Andrei Neculai and Zeynep Akata showed in their Multimodal Probabilistic Composer that this allows for complex queries, such as combining multiple images and text snippets to find a specific target.
  2. Quantifying Uncertainty: When a model is "unsure" because an image is blurry or a caption is vague, the variance of the distribution increases. Uddeshya Upadhyay and colleagues introduced ProbVLM, a probabilistic adapter that allows frozen, pre-trained models to estimate this uncertainty without needing to be retrained from scratch.
  3. Reducing Model "Laziness": Standard models often take shortcuts, like identifying a bird based on a grassy background rather than the bird's features. Mayank Nautiyal and Li Ju argue in PARIC that probabilistic regularization prevents the model from collapsing onto these simple, deterministic correlations, forcing it to account for the variety of visual configurations a text description might imply.

Controllability and Diversity

This probabilistic perspective also transforms how models generate content. In image captioning, the "one-to-many" problem means there are many valid ways to describe a scene. Aditya Deshpande and Alex Schwing addressed this by using Part-of-Speech sequences as "blueprints" to guide the model through the distribution of possible captions, producing diverse and high-quality descriptions much faster than traditional search methods.

Similarly, Marcella Cornia and Rita Cucchiara developed frameworks to control and ground captions, allowing users to steer the model toward specific objects in an image. By acknowledging that an image contains multiple valid narratives, these models move from being "black boxes" to steerable tools that can prioritize different perspectives or levels of detail.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_stochastic_latent_embeddings_for_multimo_85b77182,
  title = {Stochastic Latent Embeddings for Multimodal Semantic Ambiguity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stochastic-latent-embeddings-for-multimodal-semantic-ambiguity/rdr_85b771826ed84f3c90baf78e6e215aa9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_stochastic_latent_embeddings_for_multimo_85b77182_v1,
  title = {Stochastic Latent Embeddings for Multimodal Semantic Ambiguity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stochastic-latent-embeddings-for-multimodal-semantic-ambiguity/rdr_85b771826ed84f3c90baf78e6e215aa9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}