Services that let people search photo collections with text often store and share image embeddings: fixed-length vectors from a vision-language model such as CLIP, a pair of encoders trained so that matching images and captions land close together. Those vectors carry far more than a text query needs, including who is in the photo. This proposal rests on a simple observation. Text-to-image search reads an image embedding only through its inner product with text embeddings. Any part of the image vector that points in directions text embeddings never occupy therefore adds nothing to search and serves only as a side channel. We propose to estimate the subspace that text embeddings actually occupy, using public captions, and to project image embeddings onto it before release. Text search scores are then preserved up to a computable error bound, while image-only content such as a person's visual identity is discarded.
This also turns a confused debate into a testable claim. A line of work treats the "modality gap", the observed separation between image and text clusters, either as a defect to close or as a buffer that protects people. We argue that most of the gap is a constant offset. A constant offset carries no information about any individual, so moving it can neither create nor remove privacy leakage. What matters for privacy is the per-image residual that text cannot express. That residual is exactly what projection removes, and it is also what gap-closing losses that pull each image toward its caption implicitly squeeze out.
Core Claim
The privacy leakage of a released image embedding splits into two parts. One part is cross-modally expressible: information, such as perceived gender, that captions also encode. Removing it necessarily costs search utility, so this part is the true privacy-utility trade-off. The other part is modality-private: information, such as facial and bodily identity cues, that lives outside the text-occupied subspace. This part can be removed at near-zero search cost.
Write $v$ for an image embedding, $t$ for a text query embedding, and $P$ for orthogonal projection onto a subspace $S$ estimated from public caption embeddings: the caption mean direction plus the top-$k$ principal directions of caption variation. The score change is then bounded by
$$|\langle v,t\rangle-\langle Pv,t\rangle| = |\langle (I-P)v,(I-P)t\rangle| \le \|(I-P)v\|\,\|(I-P)t\|.$$
The equality holds because $P$ is symmetric and idempotent. The inequality is Cauchy-Schwarz. A query's retrieval ranking is unchanged whenever its score margin between the correct image and competing images exceeds twice this bound. That margin condition is measurable before release. It is what makes the method a controlled trade-off rather than a heuristic: $k$ sets the frontier, and the residual text energy $\|(I-P)t\|$ tells the operator how much search could suffer.
The surprising predictions follow directly. Post-hoc gap removal by mean-centering, as in I0T: Embedding Standardization Method Towards Zero Modality Gap, should leave a retrained attacker's success unchanged, because it is a translation. Pair-pulling objectives, such as the Align True Pairs loss in Closing the Modality Gap Aligns Group-Wise Semantics, should reduce identity leakage, because they shrink the image residual relative to the caption. This reverses the intuitive worry, listed as an open gap in the file, that closing the gap must increase cross-modal leakage.
How This Sits Against Prior Work
The motivation comes from two findings. Privacy Enhanced Multimodal Neural Representations for Emotion Recognition shows that fused audio-text representations leak more speaker gender and identity than unimodal ones and scrubs them with a gradient-reversal adversary. Understanding and Controlling User Linkability in Decentralized Learning shows that public text can link anonymous image-model updates with 0.76 AUC and that differential-privacy noise ruins utility before it stops linkage. Both establish that leakage crosses modalities. Neither asks which part of a representation the downstream cross-modal task actually reads; we answer that question geometrically and without attribute labels.
The geometric picture comes from the modality-gap literature. Mind the Gap traces the gap to encoder "cones" at initialization and to contrastive temperature. I0T locates it in a few fixed peak dimensions per modality, and ModGap and Mitigate the Gap: Improving Cross-Modal Alignment in CLIP close it to improve clustering and zero-shot accuracy. None of these papers measures privacy.
ModGap's own result constrains our design and also supports it. Retrieval is rank-based and insensitive to the gap, which is why a projection that preserves inner products with text can preserve search.
One finding appears to contradict us. Mind the Gap reports that widening the gap from 0.82 to 0.97 reduced FairFace denigration harms, for example from 15.7% to 13.7% for White subjects. That effect concerns the outputs of zero-shot classification under a fixed prompt and temperature, not what an attacker can recover. A shift changes how uncalibrated scores land without changing the information present. We test this distinction directly rather than assume it.
The trade-off formulation borrows from Privacy-Preserving Representation Learning on Graphs: A Mutual Information Perspective. That work retrains encoders with a mutual-information upper bound, but only on single-modality graphs. Our method needs no retraining and no sensitive labels. In exchange, it cannot touch cross-modally expressible attributes, and we report that limitation as a result rather than hide it.
Research Program
We first build the estimator. Subspaces are estimated from caption embeddings of public corpora: Conceptual Captions CC3M, and MS-COCO train captions when evaluating on COCO. They are always disjoint from the evaluation queries, so the mismatch between estimated and true query distributions is part of what we measure. Encoders are OpenAI-style CLIP checkpoints and MetaCLIP (facebookresearch/MetaCLIP), plus AlignCLIP-style and ModGap-style trained variants, so we can ask how training-time gap closure changes the size of the modality-private residual.
Second, we measure the decomposition. For each attribute, we define its cross-modal expressibility as how well it can be predicted from the paired caption embedding alone. We then test the core prediction: across attributes and encoders, the leakage that survives projection should track expressibility. Identity should fall sharply. Gender, which captions state routinely, should mostly survive.
Third, we map the full frontier as $k$ varies, with search utility on one axis and attacker success on the other. Every attack is retrained on the released (projected) embeddings, because a defense evaluated only against stale attackers is meaningless.
Baselines
The naive reference point is releasing raw CLIP embeddings. On Flickr30K, I0T reports CLIP retrieval accuracy of 69.60% image-to-text and 67.10% text-to-image, and 73.30% and 76.30% after I0T_post centering. Those are the utility numbers to hold. For identity linkage on PIPA, and for attribute probes on raw CLIP embeddings of FairFace, no reported result was found on Lacuna, so we establish both ourselves.
I0T_post centering is the first gap-based baseline. It closes the centroid distance from 0.7642 to 0.0102 and is the most direct embodiment of "fix the gap". Our prediction is that it leaves retrained-attacker success unchanged.
The strongest existing privacy competitor is adversarial scrubbing with a gradient-reversal adversary, as in Jaiswal and Mower Provost. That paper kept emotion recognition at 0.68 UAR while pushing gender leakage toward the 0.5 chance level on IEMOCAP. We also run a mutual-information variant using the vCLUB upper bound from Wang et al., which on Cora cut link-prediction AUC from about 80% to 50% at a two-point accuracy cost. We apply both as a learned post-hoc map on frozen CLIP embeddings. They get the same public data, the same compute, and the same hyperparameter search budget as our method, and they are given identity or attribute labels that our method never sees.
Gaussian noise on released embeddings is the differential-privacy-style baseline, matched to our method at equal retrieval loss.
The key ablation isolates the ingredient that matters: projection onto the top-$k$ principal directions of the images themselves, at the same $k$. If image PCA traces the same frontier, our claim that the text subspace is special is false.
Beating a baseline means one of two things. Either we achieve lower retrained-attacker success at equal text-to-image R@1, within 0.5 points, or we achieve higher R@1 at equal attacker success, across at least two encoders.
Why not just…?
Why not just train an adversarial scrubber for the attribute you care about? For a known, labeled attribute this is a real threat, and it may beat projection on attributes captions express, since projection cannot remove those by design. Its weaknesses are different. It needs labels for every attribute to be protected. It must be retrained per encoder. Adversarial min-max training is unstable, and Jaiswal and Mower Provost found leakage concentrated unevenly across modalities. Most importantly, it protects only against the attack it was trained on. Our head-to-head experiment reports, attribute by attribute, where each method wins. We expect the scrubber to win on gender and the projection to win on unlabeled identity linkage. If the scrubber also wins on identity at equal R@1, the practical case for projection shrinks to "label-free and instant", and we will say so.
Why not just add noise to the embeddings? Orekondy et al. found that differential-privacy noise destroyed utility before it suppressed linkage. Isotropic noise spends its budget equally on the directions search uses and the directions it ignores. Our matched-utility comparison quantifies that waste directly.
Why not just close the gap with centering or ModGap? Centering is a translation, and a retrained probe absorbs any translation into its bias term. We therefore predict no privacy change, and a measurable change would refute our account. ModGap-style pair alignment is different and may genuinely help, because it shrinks the image-only residual during training. We treat it as a training-time cousin of our method, and we measure how much residual it leaves for projection to remove.
Why not just blur faces before encoding? Unimodal sanitization misses identity carried by clothing, body shape, and context, which is precisely what PIPA-style person recognition exploits. Blurring may also hurt captions about faces. We run blurring, projection, and their combination on PIPA. If blurring alone matches projection, the residual-subspace story adds little for identity.
Proposed Experiments
Utility is measured as text-to-image and image-to-text R@1 and R@5 on Flickr30K and the MS-COCO 1K test set. We also report ModGap's clustering V-Measure on MS-COCO, which tests whether projection harms group-wise semantics.
Identity leakage uses PIPA with people-level splits. The attacker trains a Siamese verifier, in the style of Orekondy et al., on the released embeddings of training identities and reports verification AUC on unseen identities. A second, cross-modal attacker holds one reference photo of a target and ranks the released index against it. Attribute leakage is measured with retrained MLP probes on FairFace embeddings for race, gender, and age, using the dataset's balanced groups (FairFace).
As a secondary check that the idea is not specific to images, we apply the same projection to video and audio embeddings on MSR-VTT. There, the subspace is estimated from text, and the utility measure is text-to-video retrieval.
Four decisive checks would settle the proposal. First, at a $k$ where Flickr30K text-to-image R@1 drops by under one point, PIPA verification AUC should fall substantially toward 0.5. Second, text-subspace projection should dominate image-PCA projection at every matched $k$. Third, I0T_post centering should leave retrained-attacker AUC within noise of raw CLIP, which falsifies the "gap as privacy cushion" reading if it fails. Fourth, across FairFace attributes, post-projection leakage should rank in the same order as cross-modal expressibility. If gender vanishes as easily as identity, the decomposition is wrong.
We also report the margin bound against observed rank changes, query by query. That comparison shows whether operators can trust the bound to predict search damage before they release anything.
References
- I0T: Embedding Standardization Method Towards Zero Modality Gap
- Closing the Modality Gap Aligns Group-Wise Semantics
- Privacy Enhanced Multimodal Neural Representations for Emotion Recognition
- Understanding and Controlling User Linkability in Decentralized Learning.
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Mitigate the Gap: Improving Cross-Modal Alignment in CLIP
- Privacy-Preserving Representation Learning on Graphs: A Mutual Information Perspective
- FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age