How the Modality Gap Shapes Faceless Identity Leakage in CLIP

Image–text models such as CLIP (a pair of neural networks trained so that a photo and its caption land near each other in a shared vector space) are known to name people from their photos even after the face has been hidden. Hair, build, clothing and posture are enough. At the same time, a separate literature has found that image vectors and text vectors in these models sit in two separated clusters, called the modality gap, and it argues over whether to close that gap for better alignment or keep it for robustness. No one has asked what the gap does to privacy. This note proposes to measure how the size of the gap changes a model's ability to link a face-obscured photo to a person's name, and then to test a small, training-free defense. The defense uses the text side of the model, the embeddings of names themselves, to decide which image-side directions to remove.

The central prediction is specific and can be proven wrong. The gap should matter for distance-based attacks, such as thresholding a similarity score or clustering faceless photos around name anchors. It should barely matter for rank-based attacks, such as asking "which of these 500 names is closest?". So widening the gap, the cheap knob earlier work treats as a fairness lever, would give a false sense of protection. Closing the gap would make threshold and clustering attacks stronger. A defense that works under both attack types has to remove identity information and not just move clusters around.

Core Claim

The claim rests on two findings from the gap literature. Closing the Modality Gap Aligns Group-Wise Semantics shows that cross-modal retrieval depends on rank and is therefore largely insensitive to the gap. Clustering depends on distance and is highly sensitive to it: their ModGap losses shrink the gap from 0.47 to 0.03 and raise the MSCOCO clustering V-Measure (a score for how well clusters match true categories) from about 13.0 to 23.63. Mind the Gap shows the gap forms at initialization and is sustained by the contrastive loss. It also shows that widening the gap from 0.82 to 0.97 reduced offensive misclassifications on FairFace. Together these motivate a split: the gap should modulate threshold-based and clustering-based identity leakage, but not top-1 name retrieval.

Our first contribution is to measure that split directly on faceless inputs. Our second is name-anchored cross-modal erasure. It estimates a low-dimensional "identity subspace" from the text embeddings of person names and the matching faceless image embeddings, then projects that subspace out of both modalities consistently. The goal is to cut the link between faceless images and names while keeping ordinary semantic alignment intact.

Relation to Prior Work

Three kinds of work motivate the problem. First, The Phantom Menace tests five vision-language models built on CLIP-style encoders. It finds that 10×10 face pixelation leaves LLaVA-1.5 celebrity recognition near 9%, that blurring raised BLIP-2 leakage from 4.6% to 6.3%, and that fine-tuning on fully anonymized data does not erase identities learned in pretraining. That study treats the model as a black box. We look inside the encoder's geometry, which it leaves unexamined.

Second, Faceless Person Recognition and On Hair Recognition in the Wild by Machine show that body, clothing and hair identify people. On PIPA (People In Photo Albums) with blacked-out heads, within-event recognition exceeds 80%. That work is purely visual; we study the step where such cues are converted into text identities.

Third, A Study of Face Obfuscation in ImageNet shows that models trained on face-blurred data readily rely on everything else in the image.

Other work constrains the design. Joint Vision-Language Social Bias Removal for CLIP finds that demographic attributes are encoded asymmetrically in the two encoders, and that editing one side alone breaks alignment. This is why our erasure is applied to both modalities. That work removes coarse categories such as gender and race using counterfactual prompts. We target individual identity, which has hundreds of classes and no natural counterfactual. Mitigate the Gap shows that naively shifting the mean of one modality distorts fine-grained structure within that modality. This rules out gap translation as a privacy method unless it is shown to preserve utility, so we include it as a baseline rather than a proposal. Is the Modality Gap a Bug or a Feature? proves that the gap is orthogonal to both modality subspaces, and that translating along that orthogonal direction improves noise robustness. If the gap is truly orthogonal, translating along it should leave identity rankings unchanged, which is exactly our prediction.

RaSa is the opposing technology. It is trained to maximize matching between text descriptions and full-body photos of people, reaching 76.51% Rank-1 on CUHK-PEDES. We use it as an upper bound on how much non-facial identity can be extracted cross-modally. DeepPrivacy2 supplies the strongest pixel-level anonymizer, which we compare against.

Research Program

Stage 1: build faceless test sets. We build faceless probe sets at three strengths of obfuscation: face pixelation, head blackout, and full-body synthetic replacement with DeepPrivacy2. We apply these to two sources. The first is a celebrity set following the Phantom Menace protocol of 500 identities with multiple images each. If their set is not released, we rebuild it with the same construction. The second is PIPA, using both its within-event and across-day splits.

Stage 2: define the attacks. We define three attacks on the shared embedding space. The rank attack takes an image and ranks all candidate name prompts ("a photo of [name]") by cosine similarity. The threshold attack declares a match whenever similarity exceeds a calibrated value, and is scored by area under the ROC curve at a fixed false-positive rate. The clustering attack clusters faceless images together with name anchors and scores the result by V-Measure against true identities. For PIPA, which has no names, the rank and threshold attacks become image-to-image linking of faceless photos across events.

Stage 3: vary the gap. We vary the gap on the same starting model, an openly released CLIP ViT-B/32, in two ways. The first is a continuous post-hoc translation along the mean-difference direction. The second is short, matched-budget fine-tuning under three objectives: the standard contrastive loss, an AlignCLIP-style loss (shared projection plus intra-modality separation), and ModGap's align-true-pairs and centroid-uniformity losses. The fine-tuning data is a fixed image–caption corpus with no celebrity names. Because celebrity knowledge comes from pretraining, this tests how changing the geometry reshapes identity knowledge the model already has.

Stage 4: the defense. Name-anchored erasure works as follows. Let $T \in \mathbb{R}^{k \times d}$ hold the embeddings of $k$ held-in name prompts in a $d$-dimensional shared space. Let $V$ hold the mean faceless-image embedding of each of those people. We take the top-$r$ singular directions $U_r$ of the stacked, centered matrix $[T; V]$ and project both modalities with

$$P = I - U_r U_r^\top ,$$

where $I$ is the identity matrix and $r$ is chosen on a validation split.

Stacking both modalities does two things. It makes the removed subspace the one shared by names and faceless photos, rather than one side's quirks. It also means the same operator is applied to both encoders, which respects the asymmetry lesson of the joint debiasing work. We test whether erasure fitted on held-in identities generalizes to held-out identities. Generalization is necessary for the defense to mean anything, since a deployer cannot enumerate everyone.

Finally, we insert the erased encoder into LLaVA-1.5. We report two variants: one with the original connector (the small network that maps image features into the language model) and one where the connector is briefly retrained on non-identity data. We then measure Phantom Menace-style name disclosure.

Baselines

The naive reference point is the unmodified CLIP encoder on each faceless probe set, under all three attacks. It sets the leakage ceiling that every defense must lower. We found no reported result on Lacuna for CLIP name retrieval from faceless images, so we will measure it ourselves. At the language-model level, the reference is LLaVA-1.5 at roughly 9% recognition under 10×10 pixelation and BLIP-2 at 6.3% under blur, both reported in The Phantom Menace.

Pixel-level anonymization. The strongest existing method is DeepPrivacy2 full-body synthesis, which replaces the whole person with a generated one. It is the most thorough input-side defense in the file. It is run on identical images, and leakage is measured with the same attacks. Its utility cost is known from Does Image Anonymization Impact Computer Vision Training?: COCO detection AP is maintained at 20.3 under realistic full-body synthesis, versus 0.7 under masking. For linking faceless photos on PIPA, the reference is the album-level inference of Faceless Person Recognition, which raised single-head accuracy from 15% to 39%. Beating DeepPrivacy2 means matching or lowering its rank-attack leakage while staying within 1% of unmodified CLIP on ImageNet-1K zero-shot accuracy and MSCOCO image–text retrieval Recall@5. We use the 1% tolerance because the joint debiasing work reports staying within that margin on ImageNet-1K. The two defenses can also be combined, and we report that combination.

Gap manipulation. The gap baselines are post-hoc translation and the orthogonal-direction translation of Is the Modality Gap a Bug or a Feature?. Each is swept over the same range and given the same validation budget as our choice of $r$.

Upper bound on extractable identity. On CUHK-PEDES we report RaSa's retrieval under face and head obfuscation. This tells us how much faceless identity a model trained for that purpose can pull out, and gives context for the leakage we see.

Ablations. The ablations isolate the key ingredient. The first applies erasure fitted on image embeddings only, which tests whether the text anchors matter. The second uses a random subspace of equal rank, which tests whether any rank-$r$ projection would do. The third applies $P$ to the image side only, which tests the symmetry requirement.

Why not just…?

Why not just blur, pixelate, or black out the head? The file's evidence is direct. Pixelation leaves LLaVA-1.5 near 9%, and blur raises BLIP-2 leakage. On PIPA, recognition stays above 80% within events even with blacked-out heads. We keep these conditions as probe inputs rather than as defenses.

Why not just widen the modality gap, since Liang et al. found that it reduced harmful misclassifications? This is the alternative our central prediction targets. If the ranking over names is insensitive to the gap, as the gap-insensitivity result for retrieval implies, then widening the gap will lower threshold-attack scores while top-1 name accuracy stays flat. Stage 3 settles the question. If widening the gap does cut rank-attack leakage without loss of utility, we will report that the simple knob is sufficient, and our defense becomes unnecessary.

Why not just apply standard linear concept erasure to the image embeddings alone? This is a real threat, and it is why the image-only fit and the image-only projection are ablations rather than straw men. The joint debiasing results suggest that one-sided edits damage cross-modal alignment. The text anchors should also carry the stable "which names exist" structure that faceless images express only noisily. If the image-only variants match our method on both leakage and utility, the contribution reduces to the audit plus a standard tool, and we will say so.

Why not just fine-tune on anonymized data? The Phantom Menace reports that this does not remove identities learned in pretraining. Disclosure persists at about 1% under generic prompts and tracks how often a person appeared in LAION. Our matched-budget fine-tuning arm replicates this condition at the encoder level.

Decisive Checks

The first decisive check is the dissociation itself. Over the range of post-hoc gap translations, top-1 name accuracy should move by less than a few points, while threshold and clustering leakage should move substantially. ModGap-style gap closure should increase clustering leakage.

The second check is generalization. Erasure fitted on 250 identities must reduce rank-attack leakage on the other 250 by a clear margin over the random-subspace control, while keeping ImageNet-1K zero-shot accuracy and COCO retrieval within 1% of unmodified CLIP.

The third check is transfer to the full vision-language model. LLaVA-1.5 with the erased encoder must disclose names less often than under pixelation alone.

Failing the first check falsifies the geometric account. Failing the second means identity in CLIP is not low-rank in a way that transfers across people, which is itself a useful negative result for anyone hoping to anonymize embeddings after training.