Defending CLIP Without Labels by Preserving How Images Relate to Text

Vision-language models such as CLIP match images to text by comparing embeddings. Their zero-shot accuracy collapses under small, gradient-crafted image perturbations: on one nine-dataset benchmark, CLIP fell to 1.0% accuracy under attack (COLA). The usual repair is adversarial training, which means retraining the model on its own worst-case inputs. That repair has three costs. It needs class labels or class-name prompts. It erodes clean accuracy and zero-shot transfer. And it multiplies training compute by the number of attack steps. We propose a defense that removes all three costs together. The backbone stays frozen and only a few learnable input tokens are trained. The training signal comes from unlabeled images and an arbitrary bank of text, with no class names. The defense target is relational: an attacked image should keep the same pattern of similarities to many texts that the clean, frozen model gives it. It should not be pulled onto one "correct" text.

The bet is that the target of invariance matters more than the parameterization. Unimodal feature matching, as in Robustness Tokens, protects the image embedding without regard to the text side. Prompt-anchored methods such as TeCoA drive the adversarial embedding toward a single class text, which is a forced alignment. Matching the full image-to-text similarity profile protects exactly the relationship that zero-shot classification and retrieval read out. It also leaves the natural image-text gap where the pretrained model put it. If this holds, cheap label-free tokens should match supervised prompt tuning on robustness, keep clean zero-shot accuracy close to the original model, and transfer to open-vocabulary retrieval, where class-prompt defenses cannot be used.

Core Claim

Let $f$ be the frozen CLIP image encoder and $\phi$ a small set of learnable tokens prepended to its input patches. Let $T=\{t_1,\dots,t_M\}$ be frozen text embeddings of a generic, unlabeled caption bank. Define the similarity profile

$$p_\phi(x)=\mathrm{softmax}\big(f_\phi(x)^\top T/\tau\big),$$

where $\tau$ is CLIP's temperature and $p_0$ denotes the profile of the original model without tokens. We train only $\phi$ with the objective

$$\min_\phi \; \mathbb{E}_x\Big[\max_{\|\delta\|_\infty\le\epsilon}\mathrm{KL}\big(p_0(x)\,\|\,p_\phi(x+\delta)\big)+\lambda\,\mathrm{KL}\big(p_0(x)\,\|\,p_\phi(x)\big)\Big].$$

Here $\lambda$ weights clean preservation. Structurally this is TRADES-style self-distillation, and we say so plainly. Two things change. The "classes" are an open, label-free text bank. The teacher is the frozen model itself. The inner attack therefore targets cross-modal misalignment directly, which is the kind of damage CMLA showed can break cross-modal links while single-modality features look intact. The target profile is also not collapsed onto a single text. That choice follows Wasserstein Modality Alignment, which found that forcing modality distance to zero discards non-redundant information and invites shortcut learning. We transfer that lesson from random corruptions, where it was shown, to worst-case attacks, where it was not tested.

Research Program

Training regime. We prepend tokens to the image encoder and leave both encoders frozen. Training data is a few hundred thousand unlabeled images drawn from paired or interleaved web documents, such as a subset of PIN-200M. The caption bank is sampled from the same corpus and deliberately contains no evaluation class names. Because nothing in training references a label space, the method is label-independent in the strong sense that Gap 1 of the literature leaves open. TeCoA's pseudo-labels, by contrast, still rely on a candidate class list.

Compute. Compute is controlled in two ways. First, gradients flow only into the tokens, so each attack step is a forward and backward pass through a frozen network with no optimizer state for the backbone. This is the regime in which Robustness Tokens converge in 200 steps. Second, we test "perturbation replay." A per-image perturbation is stored and refined by one or two attack steps each time the image is revisited, rather than recomputed with ten-step projected gradient descent (PGD). This sits between the static precomputed bank of AdvPT, whose attacks never see the changing tokens, and fully online attacks. We report all cost as forward-backward passes and GPU-hours.

Analysis. We will measure whether relational targets preserve the clean model's geometry better than feature targets. The measures are the rank correlation between clean and defended similarity profiles on held-out text, and the spread of image-text cosine gaps before and after defense. These indicate whether the non-zero-gap intuition from WMA actually operates here.

Baselines

Undefended CLIP. The naive reference point is CLIP itself with no defense, evaluated with identical attacks. Its reported 1.0% robust accuracy on COLA's nine-dataset suite (COLA) sets the floor.

Robustness Tokens on CLIP. The most direct competitor is Robustness Tokens applied to CLIP's image encoder. That method trains the same kind of prepended tokens with an invariance loss on clean images and a loss that pulls adversarial image features back to clean ones. On DINOv2-G it reported 63.1% robust ImageNet accuracy against 0.3% undefended, while clean accuracy moved only from 85.6% to 85.2%. We run it with the same token count, training images, attack budget, and step count as our method. That makes it the decisive ablation: the only difference is whether the target is the image feature or the image-to-text profile. Beating it means higher robust zero-shot accuracy and retrieval recall at equal clean accuracy and equal compute.

TeCoA. The strongest supervised competitor is TeCoA. It fine-tunes CLIP adversarially toward class-text embeddings on ImageNet. It reported 38.18% average robust accuracy across unseen tasks, versus about 10% for standard adversarially trained CLIP, and it controls the trade-off by interpolating weights between clean and robust checkpoints. We run both its visual-prompt-tuning variant, matched to our token budget, and its full fine-tuning variant. We also show its interpolation curve, and compare whole clean-versus-robust curves rather than single points. Matching TeCoA's robust transfer without labels, at a fraction of its compute, would count as success. Exceeding it on datasets far from ImageNet would be a stronger result.

APT. Among prompt-tuning defenses, APT learns a single text-side prompt word under supervision, using its public code. It reports +13% accuracy and +8.5% robustness on average across 15 datasets. It shows what supervision buys at a similar parameter count.

Forced-alignment ablation. This variant replaces our profile target with a contrastive loss that pulls the attacked image onto its paired caption alone. It is label-free, but it is a zero-gap target. It isolates the WMA-style contribution.

Test-time defenses. For completeness we evaluate ATAC and COLA, which need no training, under the same adaptive attacks.

Why not just…?

Why not just run Robustness Tokens on CLIP's image encoder? This is the real threat. If pulling attacked image features back to clean features already restores the image-text relationship, our relational target adds nothing. There is a reason to expect otherwise. Feature-cosine recovery is measured in image space, which is not the space zero-shot classification reads out. Robustness Tokens also leaned on keeping token weights secret, while we evaluate full white-box attacks with the tokens known. Still, only the matched ablation in Baselines settles it. If the two methods tie on zero-shot and retrieval robustness, we will report that the gain comes from cheap tokens, not from cross-modal targets.

Why not just use TeCoA with zero-shot pseudo-labels on unlabeled data? Pseudo-labels still require a closed list of class prompts at training time. Pulling toward one prompt is the forced alignment that WMA found harmful. TeCoA also reports sharp drops on domains far from CLIP pretraining, such as PCAM and HatefulMemes. Our comparison on those same far-domain datasets, and on retrieval where no class list exists, will show whether the open text bank helps.

Why not just use a training-free test-time defense like ATAC or COLA? COLA builds its projection subspace from dataset class names, so it does not apply to open-vocabulary retrieval. ATAC is sensitive to its choice of augmentations. Neither method's reported numbers come with attacks that are aware of the defense. We will attack both by differentiating through the defense, and report whether their gains survive.

Proposed Experiments

Zero-shot classification. We evaluate ImageNet-1k and its shifted variants, plus the fine-grained suite (Flowers-102, Pets, Caltech-101, StanfordCars, FGVC Aircraft, DTD, Food-101, EuroSAT) and the far-domain sets on which TeCoA struggles. Attacks are $\ell_\infty$ with $\epsilon=1/255$ and $4/255$, using PGD-100 and AutoAttack. One adaptive variant attacks the full defended pipeline, tokens included. Separately, a label-free attack in the spirit of the Feature Collapse Attack of On the Adversarial Vulnerability of Label-Free Test-Time Adaptation attacks the similarity profile directly, which is the attack our own objective invites.

Retrieval. Flickr30K image-text retrieval uses clean recall@k and robust recall@k. The robust evaluation includes universal cross-modal perturbations from CPGC.

Reporting. Every method is plotted on a clean-versus-robust curve by sweeping $\lambda$ or checkpoint interpolation, at matched forward-backward pass budgets.

Decisive checks. First, the relational target should beat the matched Robustness Tokens and forced-alignment ablations on robust retrieval at equal clean recall. Second, label-free tokens should land within a few points of TeCoA's 38.18% unseen-task robust average at substantially lower GPU-hours. Third, removing evaluation class names from the caption bank should not change results. That would confirm no hidden label leakage. Fourth, perturbation replay with one or two steps should come within about one point of ten-step online attacks. A failure on the first check would mean the cross-modal target is unnecessary. A failure on the fourth would mean the overhead claim rests only on freezing the backbone.