When we train multimodal models to connect different senses—such as matching an image of a sunset with the phrase "golden hour"—our instinct is often to treat representations as discrete points in a shared mathematical space. We use contrastive losses to pull paired points closer together and push unpaired points apart.
However, real-world concepts are rarely single points. An image of a meal carries layers of meaning: it represents "food," an "avocado sandwich," and a specific texture or lighting condition all at once. Forcing these complex, multi-faceted concepts into rigid, deterministic vectors creates a fundamental tension. It leads to the "modality gap," where different data types remain trapped in isolated clusters, unable to truly blend.
To solve this, a powerful paradigm has emerged: latent distribution alignment. Instead of mapping inputs to static points, models map them to probability distributions within a shared latent space. By minimizing the statistical distance between these distributions—primarily using Optimal Transport and the Wasserstein distance—models achieve a smoother, more geometrically coherent alignment that fundamentally improves generalization and robustness.
The Geometry of the Modality Gap
Even in highly successful contrastive models like CLIP, the latent space exhibits a persistent separation between modalities. Levi et al. discovered that CLIP’s latent space is structured as two separate, non-origin-centered ellipsoid shells—one for images and one for text. This geometric separation isn't a failure; it is a functional mechanism that manages uncertainty by embedding common, ambiguous concepts closer to each modality's mean, mitigating false negatives during training.
To measure and bridge this gap without destroying the unique geometric properties of each modality, we need a way to quantify the distance between entire clouds of data. Zhao et al. demonstrated that the 1-Wasserstein Distance serves as a robust metric for measuring this modality gap. Because it operates on distributions rather than individual pairs, it provides a unified framework to evaluate alignment quality across both embedding-based retrieval models and logit-based generative models.
Why Wasserstein Distance?
When aligning two distinct probability distributions in a latent space, the choice of distance metric dictates how smoothly the model learns. Traditional metrics like the Kullback-Leibler (KL) divergence—used by Bai et al. to align feature and label spaces—require the distributions to have overlapping support. If the visual and textual distributions are completely disjoint early in training, KL divergence can suffer from vanishing gradients.
In contrast, the Wasserstein distance (often called the Earth Mover's Distance) measures the minimum "work" required to transport the probability mass of one distribution to match the shape of another.
Point-Based Alignment (Rigid, sensitive to noise)
[Image Point] ---> <--- [Text Point]
Distribution Alignment via Wasserstein Distance (Smooth, geometry-preserving)
[Image Distribution (Cloud A)] === (Optimal Transport) ===> [Text Distribution (Cloud B)]
As Theodoridis et al. showed, using the Wasserstein distance as a loss function produces significantly smoother and more stable transitions between latent distributions than standard divergence metrics. This geometric flexibility is crucial when distributions do not perfectly overlap. For instance, Chen et al. leveraged Earth Mover's Distance to align visual and semantic distributions on a shared unit hyper-sphere, providing superior robustness against the inherent noise of cross-modal data in generalized zero-shot learning.
To make this optimization computationally tractable over high-dimensional spaces, researchers employ several variants:
- Sliced Wasserstein Distance: Liu et al. utilized this approach to align feature distributions from disparate aerial sensors (Electro-Optical and Synthetic Aperture Radar), providing a stable, mathematically grounded alternative to unstable adversarial training.
- Entropy-Regularized Wasserstein Distance: Cemgil demonstrated that incorporating entropy regularization into the Wasserstein objective creates a smoother mapping from the input space, yielding unsupervised representations that are significantly more robust to adversarial attacks.
- 2-Wasserstein Distance: When dealing with probabilistic embeddings that model uncertainty explicitly, Ji et al. used the 2-Wasserstein distance to quantify the overlap between entire probability distributions, enabling more nuanced cross-modal retrieval by capturing semantic ambiguity.
Cross-Reconstruction as a Distributional Constraint
How do we practically force two different encoders to map data into the same region of a shared latent space? One of the most effective mechanisms is cross-reconstruction: forcing the model to decode or generate features of one modality using the latent representation of another.
Zhang et al. proved that this cross-reconstruction task acts as an implicit distribution constraint, naturally reducing the Wasserstein distance between the latent spaces. When an encoder is forced to produce a representation that a completely different modality's decoder can understand, it has no choice but to map diverse data types into a shared, compatible region of the feature space.
[Image Input] ---> [Visual Encoder] ---> [Shared Latent Space] ---> [Text Decoder] ---> [Predicted Text Features]
^
(Wasserstein Alignment)
v
[Text Input] ---> [Text Encoder] ---> [Shared Latent Space] ---> [Visual Decoder] ---> [Predicted Visual Features]
This dual constraint—matching latent distributions while enforcing cross-reconstruction—ensures that the latent space remains both discriminative and consistent across diverse input sources, as shown by Ma et al. and Xing et al..
This alignment is so tight that it can handle completely missing modalities during inference. For example, Zhou et al. designed a survival prediction framework where expensive genomic data is often missing at test time. Because their model minimizes the Wasserstein distance between visual (pathology) and genomic latent distributions during training, the visual features act as a reliable proxy, allowing the model to "reconstruct" functional genomic embeddings directly from tissue slides.
Theoretical Guarantees: Bounding Generalization Error
Beyond intuitive geometric appeal, aligning latent distributions has rigorous theoretical benefits for multimodal fusion. A core challenge in multimodal learning is "distribution incoherence"—the statistical divergence between how different data types behave in intermediate layers.
Li et al. provided a formal proof showing that distribution incoherence is a primary driver of generalization error in multimodal intermediate fusion. By explicitly minimizing the Wasserstein distance between the latent distributions of different modalities, models directly reduce the generalization error upper bound. This mathematical guarantee explains why distribution matching leads to significantly more robust performance on unseen data.
Whether it is aligning audio and video frames (Zhu et al.), fusing silhouette and skeleton data for gait recognition (Min et al.), or mapping linguistic and visual embeddings into a structural knowledge graph (Cao et al.), treating cross-modal alignment as a problem of optimal transport ensures that the intrinsic geometry of each individual modality is preserved, while creating a unified, coherent space for joint analysis.
Key Foundational Work
Wasserstein CNN: Learning Invariant Features for NIR-VIS Face Recognition (2019) — contributed 1 concept elements to this direction.
Learning Robust Visual-Semantic Embeddings (2017) — contributed 1 concept elements to this direction.