Modeling Multiplicity in Cross-Modal Alignment

This direction is merged into another direction: deterministic embedding limitation. Its last version stays readable and citable here.

Version 1 of 142 papers48 conceptsfrozen 2026-10-04

Traditional multimodal models, such as CLIP, operate on a fundamental assumption: every image has exactly one corresponding caption, and every caption describes exactly one image. This "one-to-one" or bijective assumption is the engine behind contrastive learning, where models are trained to pull paired points together and push all others apart.

However, as argued in Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learning, this assumption is a mathematical convenience that contradicts reality. In the real world, a single image of a "golden retriever in a park" could be validly described as "a dog," "a sunny day," or "pet photography." Conversely, the word "apple" could refer to a fruit or a tech company. This inherent many-to-many relationship is known as multiplicity.

The Point-to-Point Bottleneck

When we force a model to map an image and its caption to the exact same point in a shared embedding space, we create a "semantic collapse." If an image has three valid captions, a deterministic model tries to find a single point that represents the average of all three. This forces the model to ignore the unique nuances of each description, leading to what researchers call the "modality gap"—a persistent distance between visual and textual clusters that prevents perfect alignment.

As Sanghyuk Chun notes in Probabilistic Embeddings for Cross-Modal Retrieval, forcing these complex relationships into single points makes the model brittle. It cannot "hedge its bets" when faced with an ambiguous query, leading to poor performance in real-world retrieval where a single search term might match thousands of diverse images.

From Points to Distributions

One powerful solution to multiplicity is to stop treating embeddings as fixed points and start treating them as probability distributions. In the PCME and ProLIP frameworks developed by Sanghyuk Chun and Wonjae Kim, each image and text string is mapped to a Gaussian distribution.

  • Mean ($\mu$): Represents the core semantic concept.
  • Variance ($\sigma$): Represents the level of ambiguity or "spread."

A generic caption like "a dog" would have a high variance, covering a large volume of the embedding space to encompass many different dog breeds. A specific caption like "a 3-year-old black labrador catching a red frisbee" would have a low variance, occupying a tiny, precise region. This allows the model to naturally represent one-to-many relationships: the "dog" distribution physically overlaps with thousands of specific "labrador" or "beagle" points. This approach has been extended to specialized domains, such as drone surveillance, where MSAM uses distributions to handle the high semantic ambiguity of aerial footage.

Multiple Embeddings and Mixture Tokens

Another strategy is to allow a single instance to have multiple "senses" or facets. Mohammad Soleymani introduced PIE-Nets, which generate multiple diverse embeddings for a single image to capture different possible meanings.

Building on this, Samuel Lavoie and Mark Ibrahim proposed Llip (Latent Language Image Pretraining). Instead of a static image vector, Llip uses "mixture tokens" and cross-attention to let the image representation change based on the text it is being compared to. If the text mentions "color," the image embedding emphasizes color features; if the text mentions "action," the embedding shifts to emphasize motion. This move from one-to-one to one-to-many mappings resulted in a +6.0% boost in retrieval tasks, proving that flexibility is more effective than rigid alignment.

Softening the Alignment

If the data is inherently many-to-many, then the training loss should be too. Standard contrastive learning uses a "hard" target: the paired item is a 1, and everything else is a 0. But in a large batch, there are often "false negatives"—other images that are semantically similar to the query but aren't the official pair.

Yuting Gao and Wei Liu addressed this with SoftCLIP. By using intra-modal similarity to create "soft" targets, the model acknowledges that an image might be 100% related to its caption, but also 40% related to another caption in the same batch. Similarly, Jaeseok Byun introduced MAFA to manage these false negatives, preventing the model from being penalized for identifying valid but unlabeled cross-modal associations.

Why Multiplicity Matters

Addressing multiplicity isn't just about higher accuracy; it's about making AI more robust and interpretable. When models like ProbVLM by Uddeshya Upadhyay and Zeynep Akata incorporate uncertainty, they can signal when a query is too vague to provide a definitive answer.

By moving away from the "one-point bottleneck" toward probabilistic distributions (PCME++) or weighted point sets (WPSE), the field is shifting toward a more realistic representation of human knowledge—one where meaning is not a single coordinate, but a diverse and overlapping landscape.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_modeling_multiplicity_in_cross_modal_ali_3938569c,
  title = {Modeling Multiplicity in Cross-Modal Alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modeling-multiplicity-in-cross-modal-alignment/rdr_3938569c8f6b4c3084f57a566e0dc556},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_modeling_multiplicity_in_cross_modal_ali_3938569c_v1,
  title = {Modeling Multiplicity in Cross-Modal Alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modeling-multiplicity-in-cross-modal-alignment/rdr_3938569c8f6b4c3084f57a566e0dc556/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}