Contrastive models like CLIP and CLAP have revolutionized multimodal AI by mapping different data types—such as text, images, and audio—into a single, shared embedding space. In an ideal shared space, a text vector representing "a barking dog" would sit at the exact same coordinates as an audio recording or an image of a barking dog.
However, reality is more complicated. In practice, these modalities do not perfectly overlap. Instead, they cluster into distinct geometric regions separated by a persistent displacement known as the modality gap. Text embeddings tend to be highly concentrated and discrete, while visual or auditory embeddings are continuous, noisy, and spread out over a wider distribution.
This gap creates a major hurdle for zero-shot transfer. If you train a downstream model (such as a text-to-speech decoder or an image captioner) using only text embeddings, the model overfits to the clean, narrow distribution of the text space. When you present it with an image or audio embedding at inference time, the slight distributional shift causes the model to fail.
To solve this without relying on expensive paired datasets, a powerful strategy has emerged: stochastic embedding augmentation. By intentionally injecting noise into latent vectors during training, researchers can simulate the variance of the target modality, forcing models to become robust to the modality gap.
The Geometry of Latent Noise
To understand why adding random noise to a clean embedding space makes a model smarter, we have to look at the geometry of the latent space.
When a decoder is trained on unaugmented text embeddings, it learns to map precise, isolated points in the latent space to specific textual outputs. This makes the decoder fragile. If an input vector shifts even slightly—as happens when transitioning from a text encoder to an image encoder—the decoder encounters a coordinate it has never seen before, leading to degraded performance.
Without Noise:
[Text Vector] • ------> Decoder ------> Correct Output
[Image Vector] * ------> Decoder ------> Error / Misalignment
With Noise Injection:
. .
. • . (Spherical Region of Uncertainty)
. .
[Perturbed Vector] ------> Decoder ------> Robust Output (Covers Image Vector *)
Injecting zero-mean Gaussian noise into the embeddings during training changes this dynamic entirely. As Nukrai et al. demonstrate in Text-Only Training for Image Captioning using Noise-Injected CLIP, adding noise transforms an isolated semantic point into a spherical region of uncertainty.
Instead of learning a brittle point-to-point mapping, the decoder is forced to map an entire neighborhood or "buffer zone" of the latent space to the target output. The model can no longer rely on exact coordinate matches; it must learn to tolerate the inherent variance and jitter typical of cross-modal inputs.
Why It Works: Simulating the Shift
Stochastic perturbation is more than a simple regularization trick like dropout; it acts as a direct simulation of the target modality's distribution.
In contrastive latent spaces, the discrepancy between different modalities is not completely chaotic. In fact, Longtian Qiu demonstrates in Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training that the difference between image and text embeddings closely follows a zero-mean Gaussian distribution. This means that adding Gaussian noise to a text embedding during a text-only training phase mathematically mimics the distribution shift the model will encounter when processing actual visual features at inference time.
This perspective is pushed to its logical conclusion by Lee et al. in Diffusion Bridge: Leveraging Diffusion Model to Reduce the Modality Gap Between Text and Vision for Zero-Shot Image Captioning. They treat the modality gap itself as a structured Gaussian noise process, utilizing a denoising diffusion framework to iteratively refine shifted image embeddings back into the text-like distribution that the decoder expects.
Applications Across Modalities
Because the modality gap is a universal property of contrastive representation learning, latent noise injection has proven successful across a wide range of zero-shot tasks.
Image Captioning and Vision Tasks
Training an image captioner usually requires millions of paired image-text examples. However, by injecting stochastic noise into text embeddings, models can learn visual tasks using zero images during training.
Gu et al. leverage this in I can't believe there's no images! : Learning Visual Tasks Using Only Language Supervision, showing that text-space perturbations spread out representations enough to allow a text-only decoder to seamlessly generalize to image inputs at test time. Similarly, Lee et al. in IFCap and Yang et al. in MultiCapCLIP use Gaussian noise injection to bridge feature distribution shifts, enabling robust zero-shot multilingual and visual captioning.
Audio and Speech Processing
The continuous nature of audio signals makes the audio-text modality gap particularly wide. To train audio language models without paired data, Deshmukh et al. in Training Audio Captioning Models without Audio and Wang et al. in MATS inject synthetic noise into text embeddings. This perturbation successfully simulates the fuzziness of audio-derived vectors, allowing text-trained decoders to process raw audio streams during inference.
For speech translation, Xu et al. in Rethinking and Improving Multi-task Learning for End-to-end Speech Translation introduce synthetic noise to make discrete text features behave more like continuous, noisy speech representations, yielding tighter cross-task alignment.
Cross-Modal Retrieval and Classification
In Composed Image Retrieval (CIR), where a user searches for an image using a combination of a reference image and a text modifier, text encoders often suffer from task discrepancy. By introducing random noise into text embeddings during training, frameworks like RTD by Byun et al. (Reducing Task Discrepancy of Text Encoders for Zero-Shot Composed Image Retrieval) mimic the distribution of projected image features.
Furthermore, the authors of iSEARLE and Bao et al. in MLLM-I2W demonstrate that adding Gaussian noise creates overlapping feature distributions, preventing embeddings from clustering into isolated, modality-specific groups. For open-vocabulary image classification, Allgeuer et al. (Unconstrained Open Vocabulary Image Classification...) apply heavy angular and Gaussian noise to text vectors, forcing the resulting decoder to generalize across the modality gap at blindingly fast speeds.
Conclusion
Stochastic embedding augmentation works because it embraces the imperfect alignment of contrastive spaces. Rather than attempting to eliminate the modality gap through costly joint training on billions of pairs, it treats the gap as a geometric property that can be simulated. By blurring precise text vectors into broader, noisy regions of uncertainty, researchers can train highly versatile, zero-shot models on a single modality that generalize effortlessly across the multi-modal landscape.
Key Foundational Work
I can't believe there's no images! : Learning Visual Tasks Using Only Language Supervision (2023) — contributed 2 concept elements to this direction.
Training Audio Captioning Models without Audio (2023) — contributed 2 concept elements to this direction.
MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning (2023) — contributed 2 concept elements to this direction.