Contrastive foundation models like CLIP (for vision-language) and CLAP (for audio-language) are trained to pull representations of matching pairs (e.g., an image of a cat and the text "a cat") close together in a shared vector space, while pushing mismatched pairs apart.
The profound implication of this alignment is geometric: if an image embedding and a text embedding occupy the same neighborhood in a shared latent space, any downstream model optimized to understand the text embedding should, in theory, seamlessly understand the image embedding as well. This insight enables cross-modal embedding substitution: training a model entirely within one modality (where data might be abundant or entirely synthetic) and deploying it on another modality zero-shot.
The Modality Swap Trick in Generative Tasks
To understand why this works, consider the task of automated audio captioning. Traditionally, training an audio captioner requires thousands of hours of audio paired with human-written descriptions. However, Deshmukh et al. demonstrated in Training Audio Captioning Models without Audio that you can train an audio captioning model using zero audio files.
Training Phase (Text Only):
[Text Description] ──> [Frozen CLAP Text Encoder] ──> [Text Embedding] ──> [Language Decoder] ──> [Reconstructed Text]
Inference Phase (Zero-Shot Audio):
[Audio Input] ──> [Frozen CLAP Audio Encoder] ──> [Audio Embedding] ──> [Language Decoder] ──> [Generated Caption]
During training, the authors feed text descriptions into a frozen CLAP text encoder. They then train a language decoder to reconstruct the original sentence from this text embedding—essentially building a text-to-text autoencoder. At inference time, they perform an embedding substitution: they pass an actual audio file through the CLAP audio encoder and feed the resulting audio embedding into the decoder. Because CLAP aligns audio and text embeddings into the same semantic space, the decoder treats the audio embedding as if it were a text representation and generates an accurate caption.
This exact proxy mechanism works symmetrically across other modalities. For example, Yang et al. in MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning and Gong et al. in Visual Zero-Shot E-Commerce Product Attribute Value Extraction use text-only training to build zero-shot visual and product attribute captioners.
Conversely, Zhou et al. flip this dynamic for text-to-image generation in Towards Language-Free Training for Text-to-Image Generation. By training a generative model to map CLIP image embeddings to pixels using only uncaptioned images, they create a model that can synthesize images from text prompts at inference time by substituting the image embedding with a text embedding.
Bridging the Modality Gap
If shared spaces were perfectly aligned, embedding substitution would work flawlessly out of the box. However, contrastive models suffer from a systematic shift known as the modality gap—a phenomenon where text embeddings and their corresponding image or audio embeddings occupy distinct, parallel clusters within the shared space rather than overlapping completely.
To prevent downstream models from overfitting to the specific geometric quirks of the training modality, researchers introduce regularization techniques to bridge this gap:
- Strategic Noise Injection: Deshmukh et al. and Gu et al. (in I can't believe there's no images! : Learning Visual Tasks Using Only Language Supervision) inject calibrated noise into the text embeddings during the training phase. This blurring forces the decoder or task head to learn robust semantic features rather than exact vector coordinates, ensuring it generalizes when a substituted audio or visual embedding is presented at test time.
- Soft and Hard Prompting: In Zero-Shot Audio Captioning Using Soft and Hard Prompts, Zhang et al. utilize a combination of embedding-level augmentations ("soft prompts") and retrieved acoustic labels ("hard prompts") to smooth out the transition between text-only training and audio-based inference.
Discriminative Tasks and Zero-Shot Classifiers
The power of embedding substitution extends beyond generation to complex classification and dense prediction tasks. Because text is highly malleable and easy to generate synthetically using Large Language Models (LLMs), researchers can train highly specialized visual classifiers without a single training image.
For instance, in Zero-shot Visual Commonsense Immorality Prediction, Jeong et al. trained a visual commonsense morality classifier entirely on text-based ethical scenarios. By optimizing a classification head on text vectors from CLIP, the model successfully identified immoral visual content at inference time when presented with aligned image vectors.
In a similar vein, Zhang et al. in Auditing Gender Presentation Differences in Text-to-Image Models trained linear visual attribute classifiers purely on text prompts to audit gender presentation biases in text-to-image models.
When domain shifts occur, text can even simulate visual variations. Cho et al. developed PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization, which "hallucinates" diverse visual styles using learnable text prompts in CLIP's space. This allows them to train a robust visual classifier for unseen domains without using any source images, demonstrating that a sufficiently rich text latent space can act as a proxy for the physical variations of the visual world.
Key Foundational Work
Hand sign language recognition using multi-view hand skeleton (2020) — contributed 1 concept elements to this direction.
Cross-Modal and Hierarchical Modeling of Video and Text (2018) — contributed 1 concept elements to this direction.
Learning Multimodal Representations for Unseen Activities (2020) — contributed 1 concept elements to this direction.