Image-text models like CLIP place every image and every caption at a single point in a shared vector space. Two separate literatures have found problems with this. One shows that single points cannot represent ambiguity: a photo matches many captions, and a vague caption matches many photos. Its fix is to represent each input as a small Gaussian cloud whose spread encodes ambiguity. The other shows that points are easy to push around with imperceptible pixel changes. Its fix, adversarial training (retraining on worst-case perturbed inputs found by many gradient steps), is expensive enough that most work now tries to avoid full retraining. This note proposes a test of whether the spread of a probabilistic embedding can serve as a cheap, single-forward-pass reliability signal under attack. We attach a small trainable module to a frozen CLIP and train it so that predicted variance goes up when an input has been perturbed. We then check whether this signal survives an attacker who knows about it.
The motivation is that the obvious combination does not work the way it first appears. Replacing a deterministic target with a Gaussian one does not change the gradient an attacker follows on the mean. A query image's own variance also cannot change the order of retrieval results. What variance can do is flag unreliable queries, and a UCAT study found that attacks do the reverse at the logit level, making CLIP "arrogantly wrong." The proposal therefore targets calibrated abstention and cheap training, not a claim that Gaussian clouds absorb perturbations.
Core Claim
PCME++ (Improved Probabilistic Image-Text Representations) scores an image distribution $\mathcal{N}(\mu_v,\sigma_v^2 I)$ against a text distribution $\mathcal{N}(\mu_t,\sigma_t^2 I)$ with a closed-form distance, where $\mu$ is the predicted mean vector and $\sigma^2$ the predicted per-dimension variance:
$$d(Z_v,Z_t)=\|\mu_v-\mu_t\|_2^2+\|\sigma_v^2+\sigma_t^2\|_1 .$$
This form has two consequences that the research file's candidate gaps overlook. First, the image variance $\sigma_v^2$ adds the same constant to the distance of every candidate caption. It therefore leaves image-to-text rankings unchanged, so query variance cannot act as an adversarial "buffer" for ranking. Second, an attacker who moves only $\mu_v$ against a text distribution follows exactly the gradient toward $\mu_t$. That gradient is the same one SAFT's attack follows against the averaged description embedding (Semantic-aware Adversarial Fine-tuning for CLIP). "Replace the prompt ensemble with a text volume" (Gap 2 in the file) therefore buys nothing on the mean term.
The claim is that the useful role of distributional embeddings under attack lies elsewhere. Variance should become a calibrated indicator of whether a query's embedding can be trusted. It can be trained on a frozen backbone at roughly the cost of robustness tokens. Its value stands or falls on whether an adaptive attacker can simultaneously move the mean and suppress the variance.
Research Program
The model is a frozen CLIP ViT image encoder. Two small additions are prepended to or attached to it. The first is a set of learnable tokens in the style of Robustness Tokens: about ten vectors added to the patch sequence, which restored DINOv2-G from 0.3% to 63.1% white-box ImageNet accuracy in around 200 optimization steps. The second is a PCME++-style variance head that reads the token outputs. On the text side we use the frozen CLIP text encoder plus a lightweight variance head. This is a deliberately cheap version of PCME++, which trains its backbones.
Training combines three terms. The first is the PCME++ clean objective: pairwise binary cross-entropy on the closed-form distance, with pseudo-positives. It keeps the means useful for retrieval and lets text variance absorb caption ambiguity, which is the soft-positive benefit shown in PCME. The second is a mean-restoration term on perturbed images, in the ROB-token and FARE style, that pulls $\mu_v(x+\delta)$ back toward $\mu_v(x)$. The third term is new: a variance-ordering hinge that requires $\|\sigma_v^2(x+\delta)\|_1$ to exceed $\|\sigma_v^2(x)\|_1$ by a margin that scales with the residual mean displacement $\|\mu_v(x+\delta)-\mu_v(x)\|$.
This third term is the feature-space, open-vocabulary counterpart of UCAT's logit-level reverse-KL objective (Calibrating Uncertainty for Zero-Shot Adversarial CLIP). UCAT needs a fixed class list to build its Dirichlet distribution. Our signal needs none, so it applies directly to retrieval over arbitrary captions.
The perturbations $\delta$ during training come from one or two PGD steps. These steps still require gradients through the frozen backbone, so the cost saving comes from updating only a few hundred thousand parameters and from converging quickly. It does not come from avoiding input gradients. We will report wall-clock training time honestly against multi-step full fine-tuning.
At test time the system returns rankings from the means and an abstain-or-flag decision from $\|\sigma_v^2\|_1$ in one forward pass. By contrast, ATAC needs several augmented passes per query.
The analysis half of the program addresses the tension in Gap 4 directly. The ranking argument above covers image-query variance. For attacks on gallery items, an attacker who wants a perturbed image to surface for a target caption has an incentive to shrink that item's variance, because larger variance increases distance. The variance-ordering term pushes the other way. We will measure which force wins under white-box optimization.
Baselines
The naive reference is frozen CLIP with no defense, scored by cosine similarity. It fixes the attacked floor on every benchmark. UCAT reports 0.51% average accuracy for vanilla CLIP under AutoAttack across its 16-dataset suite.
The strongest robustness competitor for zero-shot classification is AdvSimplex (Improving Zero-Shot Adversarial Robustness in Vision-Language Models by Closed-form Alignment of Adversarial Path Simplices). It fine-tunes CLIP by aligning clean features to whole adversarial-path triangles, using a second-order closed-form bound. It reports 36.48% ImageNet robust accuracy, against 24.57% for FARE and 27.03% for TeCoA, at 61.28% clean accuracy and 4.1 hours per epoch. We rerun it with its public settings on the same fine-tuning data and attack budget. We do not expect to beat its robust accuracy with a frozen backbone. The comparison is robust accuracy per GPU-hour, together with the reliability metrics below, which AdvSimplex does not provide.
The strongest competitor on uncertainty is UCAT, which reports 30.58% AutoAttack accuracy and 54.17% clean accuracy averaged over 16 datasets. On classification we run it with its public configuration. For retrieval we build its Dirichlet over the top-$k$ gallery similarities, which gives the fairest open-vocabulary adaptation.
For retrieval robustness, the multimodal MAT line (Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships) reports Flickr30k TR@1 under the SGA multimodal attack rising from 0.6% to 36.4% with MAT. We quote this as context only, because our main threat model is image-only. For image-only attacked COCO retrieval with probabilistic models we found no reported result on Lacuna, so we will establish that baseline ourselves.
The key ablation uses identical tokens, data, attack steps and tuning budget, and varies only the objective. One version removes the variance head and uses the cosine margin between the top-1 and top-2 candidates as the confidence score. A second keeps the variance head but drops the variance-ordering term, which gives plain PCME++-on-tokens. A third keeps the ordering term but drops the text variance. The method beats the field if it matches ROB-style deterministic tokens on robust recall while clearly exceeding every confidence baseline on attack-detection AUROC and on the area under the risk-coverage curve under adaptive attack. It must do this at no more than one forward pass at inference.
Why not just…?
Why not just use deterministic robustness tokens with a confidence score from the similarity margin? This is the most serious threat. The margin is free and could flag attacks as well as learned variance does. However, UCAT's finding that attacks inflate confidence suggests similarity-based confidence is exactly what adversaries corrupt. The first ablation above settles the question directly. If the margin matches variance on detection AUROC under adaptive attack, we will report that the distributional machinery is unnecessary.
Why not just use test-time augmentation correction like ATAC? ATAC needs no training and outperforms earlier test-time defenses by nearly 50% on average across 13 datasets. Its costs are several forward passes per query, a small clean-accuracy loss, and sensitivity to the choice of augmentations, and the file notes that its gating degrades when attackers optimize through it. We compare both methods at matched inference FLOPs, and also give ATAC its full pass budget, under the same adaptive attacker.
Why not just ensemble many captions or descriptions, as in SAFT or MAT+? Our own analysis shows that on the mean term an attack against a text Gaussian is equivalent to an attack against the averaged embedding. We do not claim to improve on SAFT's robustness through the volume. Multi-caption augmentation does improve robustness in MAT+ (9.6% to 17.7% TR@1). We will therefore also train our tokens on multiple captions, to separate the benefit of extra captions from the benefit of learned text variance on graded-similarity benchmarks.
Why not just prepend a defensive prompt? The Adversarial Possibility prompt raised LLaVA-7B from 12.54% to 91.46% under APGD on COCO (Bhagwatkar et al.). It depends on an instruction-following language model, however, and does not exist for a bare CLIP retrieval encoder, which is the setting here.
Proposed Experiments
Tokens and heads are trained on COCO captions, with Tiny-ImageNet as a second fine-tuning set to match SAFT's zero-shot protocol. Evaluation covers COCO 5K and Flickr30k retrieval (R@1 and PMRP), CxC and ECCV Caption for graded similarity, and the 16-dataset zero-shot suite under AutoAttack. The suite includes EuroSAT and PCAM, where UCAT's calibration breaks down.
Metrics are clean and robust recall or accuracy, attack-detection AUROC from $\|\sigma_v^2\|_1$, selective-retrieval risk-coverage area, and training GPU-hours and inference passes.
The decisive check is an adaptive attacker that jointly maximizes retrieval loss and minimizes predicted image variance, swept over the weight on the variance term. The idea succeeds only if detection AUROC stays well above the margin baseline across that sweep, and robust recall is not traded away to hold it. A second check targets gallery poisoning to answer the Gap 4 question: whether perturbed gallery images can shrink their variance to intrude into results. A third check is the EuroSAT and PCAM split, which tests whether a feature-space signal holds where logit-level calibration was reported to fail.
References
- Improved Probabilistic Image-Text Representations
- Semantic-aware Adversarial Fine-tuning for CLIP
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Probabilistic Embeddings for Cross-Modal Retrieval
- Calibrating Uncertainty for Zero-Shot Adversarial CLIP
- ATAC: Augmentation-Based Test-Time Adversarial Correction for CLIP
- Improving Zero-Shot Adversarial Robustness in Vision-Language Models by Closed-form Alignment of Adversarial Path Simplices
- Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships
- Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design