When querying a vision-language model like CLIP or a large language model (LLM) for zero-shot classification, a fundamental challenge emerges: prompt brittleness. If you ask a model to classify an image using the prompt "a photo of a dog", the resulting text embedding does not capture a pure, abstract concept of "dog." Instead, it captures a mixture of the core concept and the specific linguistic noise introduced by that exact phrasing, syntax, and template structure. Swap "dog" for "canine", or change the template to "an image of a dog", and the vector shifts significantly in the latent space.
To solve this, researchers have converged on a simple yet powerful geometric insight: embedding aggregation. Instead of relying on a single prompt, we can generate a diverse set of alternative phrasings—using different templates, synonyms, or detailed descriptions—and average their embeddings or predictions.
In a high-dimensional vector space, the idiosyncratic noise of any single phrasing points in a somewhat random direction. When you compute the mean of dozens of diverse prompt vectors, these random variations cancel each other out. The resulting vector converges on a stable conceptual centroid that represents the pure semantic core of the target concept.
[Prompt 1: "A photo of a canine"] ---> Vector 1 \
[Prompt 2: "An image of a dog"] ---> Vector 2 +---> [AVERAGE] ---> Stable Conceptual Centroid
[Prompt 3: "A furry barking pet"] ---> Vector 3 /
Mitigating Template and Synonym Bias
The most direct application of this intuition is stabilizing zero-shot and open-vocabulary classifiers against minor lexical variations. Standard word-level embeddings are highly volatile; Olga Majewska et al. demonstrate that averaging a word's embeddings across multiple diverse sentence contexts significantly mitigates single-instance noise and stabilizes its semantic profile. This principle applies equally to visual data; Zhuosheng Zhang et al. show that aggregating multiple representative images for a single topic provides a far more robust cross-modal semantic signal than retrieving a single image.
In zero-shot image classification, Taihong Xiao et al. show that passing category names through a wide variety of natural language templates, then averaging and normalizing the resulting embeddings, creates a generalized weight vector that captures a stable class representation rather than one biased by a single prompt context.
This averaging strategy is equally vital for handling synonyms and fine-grained domain shifts:
- Synonym Robustness: Adriana Kovashka et al. and Muzammal Naseer et al. both demonstrate that averaging the text embeddings of a primary class label with its synonyms creates a more stable semantic anchor. This prevents vision-language models from failing when a user swaps synonymous terms during open-vocabulary object detection.
- Compositional Ensembling: For industrial anomaly detection, Jongheon Jeong et al. utilize a library of descriptive state words and templates to build a compositional prompt ensemble. Averaging these embeddings allows the model to capture various defect types without task-specific training.
- Domain and Population Generalization: Zeyi Huang et al. average multiple text prompt templates to create a domain-agnostic training target, forcing student models to ignore stylistic noise. Similarly, Louisa Fay et al. combine embeddings from multiple text prompts and augmented image views to dramatically enhance zero-shot performance across diverse medical patient populations.
Generative Expansion via Language Models
Simple hand-crafted templates can only capture surface-level variations. To build truly robust centroids for complex concepts, researchers leverage the generative power of LLMs to expand a single label into a rich cloud of descriptive text.
A single label like "cooking" misses crucial visual and semantic sub-components. By asking an LLM to generate descriptions of constituent movements and environmental contexts, Massimo Bosetti et al. encode and average these multi-faceted descriptors into a single representative vector, significantly boosting zero-shot video action recognition without any video training data. Similarly, Muhammad Jehanzeb Mirza et al. use LLMs to automatically generate diverse visual perspectives of a category, demonstrating that the averaged text features cover a much broader and more accurate representational space than any single hand-crafted prompt.
This generative ensembling works across a wide range of tasks:
- Disambiguating Compound Nouns: Vision-language models often struggle to separate compound nouns (like "peanut butter") from their individual parts. Sonal Kumar et al. solve this by averaging the model's similarity scores across multiple LLM-generated, context-rich captions, ensuring the model evaluates the holistic meaning rather than isolated keywords.
- Text-Based Motion Retrieval: Léore Bensabath et al. show that averaging the embeddings of multiple LLM-paraphrased descriptions during training captures the shared semantic core of a motion concept, allowing the model to generalize across highly divergent linguistic styles.
- Entity and Relation Extraction: Gabriele Picco et al. generate variations of class descriptions via expansion, summarization, and paraphrasing, showing that leveraging different semantic perspectives drastically improves zero-shot classification stability. For open-vocabulary named entity recognition, Zhuoran Jin et al. represent entity types by mean-pooling the output vectors of multi-word definitions to capture aggregate meaning.
- Pure Text Embeddings: Raghuveer Thirukovalluru et al. average the embeddings of an input sentence and its LLM-generated variations. This multi-view aggregation reduces the noise associated with any single embedding estimate and stabilizes representations across different hidden layers of an LLM.
- Visual Question Answering: To handle the risk of incomplete image descriptions in knowledge-based VQA, He Zhu et al. employ multiple diverse captioning architectures to generate a set of distinct textual representations, ensuring a broader range of visual details is captured before reasoning.
Latent Space Concept Synthesis and Alignment
Centroid averaging is not limited to classification; it is also a critical mechanism for discovering precise directions in latent space for image editing and cross-modal alignment.
If you want to modify an image to swap a source concept (e.g., "cat") for a target concept ("dog") using a diffusion model, computing a simple vector subtraction between single words (embed("dog") - embed("cat")) introduces severe prompt-specific bias. To find a clean, generalized edit direction, Gaurav Parmar et al. generate a large set of diverse descriptive sentences for both concepts and calculate the mean difference between their respective embedding clouds. This averaging process filters out the stylistic noise of individual sentences, yielding a robust semantic edit vector that generalizes perfectly across different input images.
[Source Sentences] ---> [Embed & Average] ---> Source Centroid \
+---> Clean Edit Direction
[Target Sentences] ---> [Embed & Average] ---> Target Centroid /
Similarly, the authors of Concept Lancet synthesize stable concept vectors by averaging numerous diverse textual descriptions generated by language models, reducing the noise inherent in raw text embeddings to perform precise compositional image edits. To train concept activation vectors with language guidance, Bingde Hu et al. aggregate signals from multiple augmented text prompts to mitigate phrasing sensitivity and provide a stable target for training.
This stabilization is equally powerful when aligning entirely different modalities. For instance, Eric Yang Yu enhances out-of-distribution robustness by generating multiple captions for an image using a variety of text prompts and calculating the centroid of their embeddings. When decoding brain activity, Nizhuan Wang generates multiple descriptions of a visual stimulus using an LLM and aggregates them into a conceptual centroid. This centroid provides a highly stable semantic anchor that makes cross-modal alignment between EEG signals and visual representations far more consistent.
Test-Time Adaptation and Prediction-Level Fusion
While pre-computing embedding centroids is highly effective, ensembling can also be applied dynamically at test-time to adapt model weights or fuse final predictions.
Instead of averaging embeddings before they influence the model, the authors of WATT adapt CLIP at test-time by averaging the model weights derived from multiple distinct prompt templates. This weight-averaging approach captures complementary features from different semantic views of a class, leading to superior generalization under domain shifts.
Alternatively, ensembling can occur at the prediction level by averaging the model's output scores across multiple prompt variations:
- Gaze Following: Anshul Gupta et al. query vision-language models with multiple semantically similar prompts and average their output confidence scores, creating a reliable consensus that stabilizes zero-shot performance.
- Few-Shot Text Classification: Jirui Qi et al. aggregate predictions from multiple augmented views and mask slots of a single sentence, diluting the influence of any single misleading lexical cue across the ensemble.
- Domain Generalization: Yunlong Tang 0004 et al. aggregate predictions from multiple models trained on diverse prompt templates, ensuring that classification remains robust across different ways of describing the same visual style.
Ultimately, whether applied to embeddings, model weights, or output predictions, the core mechanism remains identical: multi-prompt aggregation acts as a statistical filter. By sampling a concept from multiple linguistic directions, foundation models can successfully bypass the arbitrary noise of human phrasing and operate on stable, generalized representations.
Key Foundational Work
Deep correlation for matching images and text (2015) — contributed 1 concept elements to this direction.
Manual Clustering and Spatial Arrangement of Verbs for Multilingual Evaluation and Typology Analysis (2020) — contributed 1 concept elements to this direction.
Neural Machine Translation with Universal Visual Representation (2020) — contributed 1 concept elements to this direction.