The central promise of multimodal models like CLIP or CLAP is the creation of a unified semantic space. By training on pairs of different modalities—such as images and text, or audio and text—using contrastive learning, these models aim to map different representations of the same concept to the same mathematical location. Intuitively, the embedding for an image of a dog and the embedding for the word "dog" should perfectly overlap.
In practice, they do not. Instead, the latent space fractures. Embeddings from different modalities segregate into completely isolated regions, creating a persistent geometric separation known as the modality gap.
Understanding why this gap emerges, why it harms downstream performance, and how recent research aims to bridge it reveals a fundamental characteristic of how neural networks organize multimodal information.
The Illusion of a Shared Space
When we visualize a joint embedding space, we expect to see concepts clustered together regardless of their source modality. However, as Grassucci et al. observe in Closing the Modality Gap Aligns Group-Wise Semantics, different data types form distinct, modality-specific clusters. This spatial separation occurs even when the model successfully aligns pairs for cross-modal retrieval.
This geometric anomaly is remarkably structured. In The Double-Ellipsoid Geometry of CLIP, Levi et al. discovered that CLIP’s latent space is organized as two separate, non-origin-centered ellipsoid shells—one exclusively for images and one for text. Within this space, embeddings are confined to a narrow cone, leaving the vast majority of the vector space entirely empty.
As a result of this separation, Tran et al. point out in Aggregating Image and Text Quantized Correlated Components that an embedding is frequently closer to completely unrelated samples of its own modality than to its paired counterpart from another modality.
Expected Space: Actual Space (The Modality Gap):
[Image: Dog] [Image: Dog] [Text: Dog]
[Text: Dog] [Image: Cat] [Text: Cat]
[Image: Cat] [Image: Car] [Text: Car]
[Text: Cat] (Vision Zone) (Text Zone)
Why the Gap Emerges
The root cause of the modality gap lies in the objectives we use to train these models. Contrastive learning functions like a ranking mechanism: it forces the dot product of a true pair (e.g., an image and its actual caption) to be higher than the dot product of random negative pairs.
Crucially, this objective can be satisfied without forcing the absolute coordinates of the two modalities to overlap. As Yin et al. demonstrate in Distributional Vision-Language Alignment by Cauchy-Schwarz Divergence, achieving high mutual information between modalities does not guarantee that their global distributions will occupy the same region in the latent space. The model preserves the structural differences inherent to the architecture of each individual encoder.
Furthermore, optimizing purely for cross-modal alignment leaves the internal structure of each individual modality uncalibrated. In Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion, Mistretta et al. show that contrastive models suffer from "intra-modal misalignment." Because the model is never explicitly trained to compare images to images or text to text, items with similar semantic content are often poorly positioned relative to one another within their native embedding spaces.
The Cost of Misalignment
The modality gap creates severe bottlenecks when we attempt to use these models for zero-shot transfer or cross-modal generation.
Consider the task of training an audio captioning model using only text data, as explored by Deshmukh et al. in Training Audio Captioning Models without Audio. If a text decoder is trained exclusively on text embeddings, it learns to interpret vectors from the "text zone" of the latent space. During inference, if we feed it an audio embedding representing the exact same concept, the decoder fails. Because the audio embedding resides in a completely different neighborhood, the decoder encounters a distribution shift it cannot interpret, leading to hallucinations or semantic inconsistencies.
This exact failure mode repeats across various architectures and modalities, including target sound extraction (Ma et al.), zero-shot audio captioning (Li et al.; Zhang et al.), and multimodal autoencoders (Chandar et al.).
Strategies for Bridging the Gap
To resolve this misalignment, researchers have developed three primary intuition-driven approaches: global distribution alignment, strategic noise injection, and generative transport.
1. Global Distribution Alignment
Instead of relying solely on pairwise contrastive losses, some approaches force the entire point clouds of different modalities to merge.
In Distributional Vision-Language Alignment by Cauchy-Schwarz Divergence, Yin et al. supplement contrastive learning with a divergence loss that minimizes the statistical distance between the overall text and image distributions.
Other methods modify the geometry of the space during fine-tuning. For example, Oh et al. introduce Geodesic Multi-Modal Mixup for Robust Fine-Tuning, which blends image and text embeddings along the surface of a hypersphere. This creates synthetic "hard negatives" that lie directly inside the unutilized gap, forcing the model to populate and smooth out the empty space between modalities. Similarly, Mordacq et al. utilize Minimum Spanning Tree regularization in T-REGS: Minimum Spanning Tree Regularization for Self-Supervised Learning to maximize embedding uniformity and prevent dimensional collapse.
2. Perturbation and Noise Injection
If the downstream model cannot cross the gap, we can artificially expand the training distribution to blur the boundary between modalities.
In I can't believe there's no images! : Learning Visual Tasks Using Only Language Supervision, Gu et al. train vision models using only text by injecting strategic noise into the text embeddings. This noise acts as a proxy for the spatial displacement of the modality gap. By forcing the downstream decoder to be invariant to these spatial shifts, it successfully generalizes to actual image embeddings at test time.
A similar intuition is applied by Saijo et al. in Leveraging Audio-Only Data for Text-Queried Target Sound Extraction, where high-rate dropout is applied to audio embeddings to bridge the representation discrepancy between training and inference.
3. Generative Transport
Rather than modifying the latent space or adding noise, we can treat the modality gap as a structured translation problem.
In Diffusion Bridge: Leveraging Diffusion Model to Reduce the Modality Gap Between Text and Vision for Zero-Shot Image Captioning, Lee et al. frame the displacement between image and text representations as a form of Gaussian noise. They train a Denoising Diffusion Probabilistic Model (DDPM) to perform a reverse diffusion process that physically refines and transports an image embedding across the gap, transforming it into a valid text-like representation before passing it to a text decoder.
[Image Embedding] ---> (Diffusion Bridge / Denoising) ---> [Text-like Representation]
(Vision Zone) (Text Zone)
By understanding that the modality gap is a geometric consequence of relative objective functions, these diverse strategies allow models to achieve true semantic fusion—ensuring that cross-modal inference behaves as though the shared space were truly shared.
Key Foundational Work
Aggregating Image and Text Quantized Correlated Components (2016) — contributed 1 concept elements to this direction.
Correlational Neural Networks (2015) — contributed 1 concept elements to this direction.