Does Robust Fine-Tuning of CLIP Leak Its Training Images?

Companies that deploy vision-language models such as CLIP, which scores how well an image matches a text description, often adapt them on small private image collections. Examples include clinical scans, product catalogs, and satellite tiles. Because these models are easy to fool with imperceptible pixel changes (adversarial examples), practitioners increasingly adapt them with adversarial training, which fine-tunes on deliberately perturbed images. This proposal asks what that hardening costs in privacy. Specifically, does it make it easier for an outsider to tell whether a given image was in the private adaptation set, which is called membership inference? In ordinary image classifiers the answer is known and uncomfortable. Evaluating Membership Inference Through Adversarial Robustness showed that adversarial training widens the gap between how far training images and unseen images sit from the decision boundary. On CIFAR-10 with VGG-16, its boundary-distance attack rose from 0.576 to 0.722 inference accuracy once the model was adversarially trained. Nobody has checked whether the same thing happens for the parameter-efficient, text-anchored robust adaptation methods now standard for CLIP.

The core claim is specific and falsifiable. Membership leakage in robustly adapted vision-language models is driven by robust overfitting: the gap between how robust the model is on its training images and how robust it is on held-out images. The ordinary clean-accuracy gap is not what drives it. If this is right, three things follow. First, an attack that scores images by their loss after a fixed-budget perturbation should expose members far better than standard loss-based attacks. Second, methods that shrink robust overfitting, such as the decoy prompts of ADAPT, should leak less at matched robustness. Third, choosing a checkpoint or an interpolation weight by held-out robust gap should give a better three-way trade-off between clean accuracy, robustness, and privacy than anything tuned on two axes alone.

Where this sits

The file points to a real tension that no one has resolved. On one side, membership-inference studies of large vision-language models report that parameter-efficient adaptation leaks much less than full fine-tuning. Membership Inference Attacks against Large Vision-Language Models introduces the VL-MIA benchmark and an entropy-based score. Membership Inference Attacks Against Vision-Language Models uses set-level temperature sensitivity. Both find that adapter-style training (MiniGPT-4, LLaMA-Adapter) leaks less than full language-model fine-tuning. That suggests small prompt vectors are safe. On the other side, the robustness-privacy paradox from CNNs suggests any adversarial objective amplifies the member/non-member margin gap. Membership Inference Attacks via Adversarial Examples similarly shows that members need larger perturbations to flip. Neither side has studied adversarially driven prompt or encoder tuning for contrastive zero-shot models. The two findings motivate the question. This work also differs from both: it makes a causal-style claim, that robust overfitting is the mediator, and tests it across methods, training time, and interpolation paths.

Three robust-adaptation methods supply the systems under study. Understanding Zero-shot Adversarial Robustness for Large-Scale Models introduces TeCoA, which fully fine-tunes the image encoder so perturbed images match frozen text embeddings, and proposes weight interpolation back toward clean CLIP. Adversarial Prompt Tuning for Vision-Language Models introduces AdvPT, which freezes the image encoder and tunes only text prompt vectors against a bank of precomputed adversarial image features. Learning What Not to Learn introduces ADAPT, which adds decoy prompts that soak up perturbation shortcuts that do not generalize. ADAPT diagnoses robust overfitting to seen classes, but it measures that overfitting at the class level, not at the level of individual images. It never asks whether individual training images become identifiable. Here ADAPT serves as the natural "treatment" for the mediator hypothesis.

Several papers constrain the design. Learnability and Privacy Vulnerability are Entangled in a Few Critical Weights finds that leakage and accuracy share under 1% of weights, and that noisy whole-model defenses such as DP-SGD waste utility. That is a warning that privacy fixes applied to all parameters may be poor baselines on CLIP too. How Does Data Augmentation Affect Privacy in Machine Learning? shows that sets of transformed views leak more than single views, with ImageNet attack success rising from 68.3% to 75.2%. Its set-level moment attack is therefore a required competitor. It also suggests that a PGD trajectory, which is itself a set of perturbed views, may be an especially strong fingerprint. Finally, White-box vs Black-box argues that thresholding the loss is near Bayes-optimal under its assumptions. Any claim that a perturbation-based score beats the plain loss must therefore be demonstrated, not assumed.

Research Program

The setting fixes a threat model in which a model owner adapts CLIP (ViT-B/16, with ViT-L/14 as a scale check) on a private labeled set. Each downstream dataset is split into a member half, used for adaptation, and a non-member half drawn from the same distribution. Prompt tuning is run in the 16-shot per class regime common for these methods, and TeCoA is run in the full-data regime. The attacker gets the adapted model's image-text similarity scores. In the stronger variant, the attacker also gets gradients. In the weaker variant, the attacker instead transfers perturbations computed on a shadow model adapted on disjoint data from the same distribution.

The main attack is deliberately cheap so it scales to foundation models, unlike the multi-direction Carlini-Wagner search of $I_{dd}$. It runs a fixed-budget PGD attack on image $x$ with true class $y$ and records the adversarial cosine margin

$$ m_\epsilon(x) = \min_{\|\delta\|_\infty \le \epsilon} \Big[\cos(f(x+\delta), t_y) - \max_{c \ne y} \cos(f(x+\delta), t_c)\Big], $$

where $f$ is the adapted image encoder, $t_c$ is the (possibly prompt-tuned) text embedding of class $c$, and $\epsilon$ is the perturbation budget. A larger $m_\epsilon$ means the image stays correctly matched under attack. The membership score is $m_\epsilon$ calibrated against shadow models, in the style of per-example likelihood-ratio tests. A set variant feeds moments of the margin along the PGD trajectory into a small permutation-invariant classifier, borrowing $M_{\text{moments}}$ from Yu et al.

The mediator analysis records, for each training run, four quantities at many checkpoints and at many TeCoA interpolation weights. These are clean accuracy, robust accuracy on held-out images, the member-minus-non-member gap in $m_\epsilon$ (the robust generalization gap), and membership AUC. The hypothesis predicts that AUC is explained by the robust gap much better than by the clean-loss gap. It also predicts that ADAPT sits at lower AUC than AdvPT at equal held-out robustness. Finally, it predicts that AdvPT leaks less than TeCoA per member image: with a frozen encoder, prompts can store class-level shortcuts but have little capacity for image-specific ones. If this last prediction fails, the "prompts are too small to leak" intuition is refuted directly.

The constructive output is modest by design. It is a selection rule that picks the interpolation weight or stopping epoch that maximizes robust accuracy subject to a cap on the held-out robust gap. It is compared against the same model family tuned on clean and robust accuracy alone.

Baselines

The naive reference points are zero-shot CLIP with no adaptation and clean (non-adversarial) fine-tuning or prompt tuning on the same member set. The first shows the floor of leakage on data never trained on. The second isolates what the adversarial objective adds on top of ordinary adaptation. Both use identical member/non-member splits, and their clean accuracy is reported so comparisons are made at matched clean accuracy wherever possible.

On the attack side, the strongest simple competitor is the loss or confidence threshold, calibrated with the same shadow models. Sablayrolles et al. argue it is near-optimal, so the robust-margin attack counts as a contribution only if it beats this baseline in AUC and in true-positive rate at 1% false-positive rate on the adversarially adapted models. It must also not merely match the baseline on clean-adapted ones. The second attack competitor is the set-level $M_{\text{moments}}$ score over ordinary augmentations. The third is a scaled-down $I_{dd}$ boundary-distance search given the same number of gradient evaluations. For reference, the strongest reported membership number in the file is the $I_{dd}$ result of 0.722 inference accuracy on PGD-trained VGG-16 for CIFAR-10 (Evaluating Membership Inference Through Adversarial Robustness). For CLIP-family adaptation, VL-MIA reports AUC 0.815 on GPT-4 images, but that is a different model, data, and threat model. No reported result was found on Lacuna for membership inference against TeCoA, AdvPT, or ADAPT.

On the defense side, the strongest existing robust methods are TeCoA and ADAPT. The file reports TeCoA at 38.18% average robust accuracy on unseen tasks versus about 10% for plain adversarially trained CLIP. It reports ADAPT at a cross-dataset harmonic mean of 17.21% and AdvPT raising Flowers PGD-40 accuracy from 6.4% to 37.4%. These methods are reproduced with their published hyperparameters and each is given the same tuning budget. The privacy baseline is differentially private prompt tuning, clipping and noising per-example gradients of the prompt vectors only, swept over several privacy budgets. The ablation that isolates the key ingredient swaps the fixed-$\epsilon$ adversarial margin for the clean margin, holding everything else fixed. If AUC does not drop, the adversarial part of the attack is doing nothing. The selection rule beats the baselines if, at equal held-out robust accuracy (within one point), it lowers membership AUC by a margin larger than the seed-to-seed spread while keeping clean accuracy within one point.

Why not just…?

Why not just use the loss threshold, since it is supposedly Bayes-optimal? The optimality argument in White-box vs Black-box holds under its distributional assumptions. On adversarially trained CNNs, however, the entropy score reached only 0.578 against 0.722 for the boundary-distance attack, a large empirical gap. Whether that gap survives in text-anchored CLIP adaptation is exactly what the clean-margin ablation tests. If the loss threshold matches the robust-margin attack, the paper reports that leakage in these models does not depend on the attack, and the mediator analysis proceeds with the loss score.

Why not just train with differential privacy? For full-encoder TeCoA, the file's evidence that noise on every weight costs utility is strong (Learnability and Privacy Vulnerability are Entangled in a Few Critical Weights). For prompt tuning, however, per-example clipping of a few thousand parameters is cheap, and this alternative is a genuine threat. It is therefore run as a first-class baseline across budgets. If DP prompt tuning dominates the selection rule on all three axes, the paper recommends it and reports the mediator analysis as explanation rather than defense.

Why not just interpolate weights, as TeCoA already does? Interpolation is part of the program, not a rival to it. The open question is which point on the path to pick. Currently it is picked on clean and robust accuracy alone, and the proposal measures whether the third axis, membership AUC, moves monotonically along the path or has its own minimum. If AUC simply tracks clean-model similarity, the TeCoA recipe already suffices and the selection rule adds nothing. The experiment would show this directly.

Why not just use a training-free test-time defense like AOM or ATAC, so nothing private is trained? AOM and ATAC defend a frozen pretrained model. They do not help an owner who must adapt to private domains. The file notes TeCoA's weakness on far-from-pretraining data such as PCAM, and that is precisely where adaptation matters. Frozen CLIP with AOM is included as a reference point on the medical and satellite datasets to quantify how much accuracy is given up by refusing to adapt.

Proposed Experiments

The datasets come from the suites in the file: Flowers-102, OxfordPets, EuroSAT, PCAM, CIFAR-10/100, and ImageNet subsets, split into member and non-member halves. The base-to-new protocol of ADAPT is also used, so class-level and image-level overfitting can be separated. Robust accuracy is measured with PGD and AutoAttack at the budgets used by TeCoA and AdvPT. Privacy is reported as AUC and true-positive rate at 1% false-positive rate, averaged over at least three seeds and several shadow models.

The decisive checks are three. First, across all runs, checkpoints, and interpolation weights, does the robust generalization gap predict membership AUC substantially better than the clean-loss gap, in held-out variance explained? If not, the central claim is false. Second, does the ordering of leakage at matched robustness follow TeCoA > AdvPT > ADAPT, or does parameter-efficient adversarial tuning turn out to leak as much as full fine-tuning? Either outcome resolves the tension between the VLM membership papers and the CNN paradox. Third, does the robust-gap selection rule beat both clean/robust-only interpolation and DP prompt tuning on the three-way frontier? A secondary transfer check asks whether the attack keeps most of its power when perturbations come from a shadow model rather than the target's own gradients, which determines whether the risk is practical for deployed models.