Paraphrase-Weighted Product of Experts for Stable Zero-Shot Vision-Language Prediction

Models that match images to text, such as CLIP, classify an image by comparing it with text descriptions of each class. They are brittle in a specific way: rewording "a photo of a dog" as "a picture of a dog", or adding a typo, can change the predicted label. The standard fix is to average the embeddings of many hand-written prompt templates and hope the wording noise cancels. We propose instead to treat each prompt template as an expert, meaning a noisy witness that reports where a class sits in image space, together with an error bar. The error bar is measured, without labels, by how far that template's embedding moves under small intent-preserving rewordings. The experts are then combined with a product of experts, a fusion rule that multiplies the witnesses' probability distributions so that confident witnesses count more and wobbly ones count less.

The mechanism has two parts. First, rewordings mostly move text embeddings along directions that images barely use. These directions are tied to the modality gap, the well-documented fact that image and text embeddings of the same concept sit in separate regions of the shared space. Second, a fusion rule that knows which directions are noisy can ignore them, while plain averaging cannot. If both hold, prompt sensitivity is partly a geometric artifact of the gap, and it can be corrected with closed-form statistics from unlabeled data rather than with more prompts or with fine-tuning.

Core Claim

We claim that, after a simple gap correction, the variation of a class's text embedding across paraphrases is strongly anisotropic, meaning it is concentrated in a few directions. We further claim that weighting each direction by its inverse paraphrase variance, the "precision" in product-of-experts terms, reduces prediction flips across prompt wordings and raises worst-case accuracy compared with mean ensembling, centering alone, and zero-shot prompt reweighting. A specific prediction follows from the Gaussian product: the fused class distribution narrows as more templates are added, because precisions add. Accuracy should therefore keep improving with the number of reliable templates in the regime where plain averaging saturates.

The three directions play different roles. Product-of-experts fusion supplies the method. Prompt sensitivity is both the target failure and, following On the Relation between Sensitivity and Accuracy in In-Context Learning, a usable uncertainty signal. The modality gap supplies the explanation for why the noise is structured.

Relation to Prior Work

Four lines of work motivate the proposal. ProSA ties prompt sensitivity to flat, low-confidence output distributions. POSIX gives an index of sensitivity based on how log-likelihoods shift across prompt variants. SENSEL, from On the Relation between Sensitivity and Accuracy in In-Context Learning, shows that instability across perturbed prompts predicts errors, with Pearson correlation reaching −0.40, and uses it to decide when to abstain. All three are text-only and diagnostic. None of them fuses the perturbed variants into a better representation, which is what we do.

On the geometry side, Mind the Gap explains the gap through a "cone effect", in which each encoder tower squeezes its outputs into a narrow cone, together with contrastive repulsion. Closing the Modality Gap for Mixed Modality Search, known as GR-CLIP, removes per-modality means after training to repair ranking. Closing the Modality Gap Aligns Group-Wise Semantics closes the gap during training to improve clustering. These works treat each text prompt as a single fixed point and the gap as a uniform shift. We keep GR-CLIP's centering as a first step and add what it omits: a covariance, estimated per template, that describes how wording moves the embedding.

Two papers supply methods. Generalized Product-of-Experts, or gPoE, raises each expert's distribution to a reliability power $\alpha$ so that an overconfident corrupted sensor cannot dominate the fused result. It has only been tested on RGB and depth sensors. We reuse the reliability exponent for text templates, where the analogue of a corrupted sensor is a template that is consistent across rewordings but systematically misleading, such as "a blurry photo of a {}". Diversity Covariance-Aware Prompt Learning, or DCA, replaces cosine similarity with a Mahalanobis distance, a distance that stretches or shrinks each direction according to a covariance. Its covariance comes from labeled few-shot image features and its prompts are learned. Ours comes from unlabeled natural-language paraphrases and needs no labels.

The closest competitor is ZPE. It weights templates using scalar logit corrections for word-frequency bias. It works per template and per class and never looks inside the embedding space. Our method can downweight a template in some directions while trusting it in others.

One tension should be stated plainly. Mind the Gap reports that enlarging the gap slightly improves CIFAR10 zero-shot accuracy and reduces demographic harm on FairFace. We therefore do not close the gap during training. We center only at scoring time, and we ablate that step on exactly those benchmarks.

Research Program

The method is defined as follows. For class $c$ and template $t$, let $\tilde e_{c,t}$ be the text embedding after GR-CLIP-style mean correction, which is shifted so that it lives in image-centered coordinates. For each template we generate intent-preserving perturbations: synonym swaps, typos, reordering and LLM paraphrases. Following FairVLM's practice, we filter these perturbations for semantic fidelity. From the spread of the perturbations, pooled across classes and shrunk toward a scaled identity as in DCA, we estimate a covariance $\Sigma_t$. Each template becomes a Gaussian expert over the location of the class prototype in image space. The generalized product gives the fused class prototype $\mu_c$ and its precision $\Lambda_c$:

$$\Lambda_c = \sum_t \alpha_t \Sigma_t^{-1}, \qquad \mu_c = \Lambda_c^{-1}\sum_t \alpha_t \Sigma_t^{-1}\tilde e_{c,t},$$

Here $\alpha_t \in [0,1]$ is the reliability of template $t$. We score a centered image embedding $\tilde z$ by its log-likelihood under $\mathcal N(\mu_c, \Lambda_c^{-1} + \Sigma_{\text{img}})$, where $\Sigma_{\text{img}}$ is the covariance of unlabeled image embeddings.

The exponents $\alpha_t$ are fitted without labels. We maximize the likelihood of the unlabeled test images under the resulting mixture of class Gaussians. This is the same transductive access to test images that ZPE uses.

When every $\Sigma_t$ equals a common isotropic covariance and every $\alpha_t$ equals 1, the method reduces exactly to "center and average". This makes the contribution isolable: any gain comes from the anisotropic precision and the reliability exponents. The fused variance $\Lambda_c^{-1}$ also gives a per-prediction confidence, which we compare directly with SENSEL's instability gate.

The analysis half of the program tests the mechanism. For CLIP ViT-B/16 we decompose each paraphrase-induced change in the text embedding into two parts. One part lies in the top principal subspace of image embeddings. The other lies in its complement, which includes the gap direction. We then measure what fraction of label flips each part causes when it is applied alone.

Baselines

The naive reference point is the single template "a photo of a {}" with cosine scoring. The standard practice is CLIP's hand-crafted ensemble of 80 templates. ZPE reports 68.31% top-1 on ImageNet with ViT-B/16 for this ensemble and 67.59% for an equal average over its pool. The strongest label-free competitor is ZPE itself, which the same paper reports at 68.60% ImageNet top-1 with ViT-B/16 and 71.15% average across its fine-grained suite. Beating ZPE means exceeding 68.60% on ImageNet with the same backbone and the same template pool while also lowering flip rate. We will rerun ZPE ourselves on our perturbation-augmented pools, so that both methods see identical templates and identical unlabeled images and neither gets a tuning advantage. ZPE's LAION reference sample will be matched in size to our unlabeled pool used for estimating image statistics.

A second strong baseline is GR-CLIP centering followed by mean ensembling. We run it with the same perturbation pool, so that the only remaining difference is our covariance and exponents. Closing the Modality Gap for Mixed Modality Search reports up to 26 percentage points of NDCG@10 improvement over CLIP on MixBench. No absolute NDCG value is given, so we will reproduce it.

DCA serves only as a labeled-data reference, because it trains prompts with few-shot labels. The file reports its gains only relative to CoOp (4.76–8.46%), so no directly comparable absolute number is available.

Three ablations isolate the key ingredient. The isotropic-covariance ablation is equivalent to centering plus averaging. The $\alpha_t = 1$ ablation is plain product of experts and exposes the overconfidence failure described for gPoE. The no-centering ablation tests the "gap as feature" objection.

Why not just…?

Why not just average more prompts? Averaging reduces noise only in proportion to the number of templates, and it gives every direction equal weight. If template noise is shared across templates, as systematic wording biases would be, averaging cannot remove it. Its accuracy should plateau while the precision-weighted product keeps improving. We test this directly by plotting accuracy and flip rate against the number of templates, from 1 to more than 200, for both methods. If the two curves coincide, the covariance idea is wrong.

Why not just center the embeddings, as GR-CLIP does? This is the most serious threat. If the harmful paraphrase noise lies mainly along the mean-offset direction, centering already removes it. The subspace-decomposition analysis settles the question. If most flips come from a single mean direction, centering is sufficient and we will report that result. Our method is justified only if substantial flip-inducing variance remains, spread over many directions, after centering.

Why not just reweight templates with ZPE? ZPE assigns one scalar weight per template. That weight cannot express that a template is reliable about shape-related directions but unreliable about background-related ones. We run ZPE on the same pools. If ZPE matches us on flip rate as well as accuracy, then direction-specific weighting adds nothing.

Why not just use SENSEL's variance to abstain? Abstaining leaves the underlying prediction unchanged. We compare the area under the F1-coverage curve when using our fused variance versus paraphrase instability. The comparison is meaningful because SENSEL reports +4.1 points on text-only GPT-J, although its vision-language performance is unmeasured.

Proposed Experiments

We evaluate on ImageNet and on the fine-grained suite shared by ZPE and DCA: OxfordPets, Flowers102, Caltech101, Cars196, DTD, EuroSAT, Food-101, Resisc45 and Sun397. We add CIFAR10 and FairFace, where Mind the Gap reports a CIFAR10 improvement from 0.9013 to 0.9081 by enlarging the gap and a drop in offensive misclassification of White subjects from 15.7% to 13.7%. For retrieval under paraphrased queries we use MixBench.

The metrics are top-1 accuracy, worst-template accuracy, flip rate across held-out perturbations, a POSIX-style sensitivity index computed on class probabilities, NDCG@10 together with its variance under query rewording, and selective-prediction area under the curve. The perturbations used for evaluation come from a different generator than those used to estimate covariances. This prevents the method from fitting its own test noise.

Three results are decisive. First, after centering, a meaningful share of flip-causing paraphrase variance must lie outside the mean direction. Second, the full method must beat centering plus averaging on flip rate and worst-template accuracy with the same template pool, and must exceed ZPE's reported 68.60% on ImageNet. Third, on FairFace, the scoring-time centering must not reverse the bias improvement that Mind the Gap attributes to the gap. If the first check fails, the proposal collapses to GR-CLIP and we will say so.