Concealing Visual Identity by Generalizing, Not Erasing

When a photo contains something private, such as a recognizable car, a branded device, or a specific breed of pet, the usual fix is to black it out or blur it. For vision-language models (systems that read an image and answer questions or write captions about it), erasure backfires. The model sees an empty region and fills it with a plausible invention: an erased person becomes a fire hydrant. Text privacy research settled a similar problem years ago. Instead of deleting "16F" or "Starbucks," it replaces them with "teenage girl" and "coffee shop," moving each detail up a category hierarchy so that identity disappears and meaning survives. This note proposes the visual counterpart. We would re-encode a sensitive object so that a vision-language model perceives a generic member of its parent category ("a car," "a dog") rather than either the specific instance or an empty hole.

The central bet is that hallucination after concealment depends on how the concealed region is filled, not merely on whether something was removed. Filling it with a representation that is consistent with the category and with the scene should give the model nothing to invent, while still denying it the fine-grained identity. If that holds, abstraction level becomes a single dial that trades privacy against usefulness, much as it does in text. In the text setting, Language Models are Advanced Anonymizers shows that generalization lowers an attacker's attribute-inference accuracy from 86.7% to 41.6% while keeping text readable. We ask whether the same holds for images.

Core Claim and Where It Sits

The closest work is Do Not Leave a Gap. It motivates this proposal and supplies most of its method. The paper shows that suppressing visual tokens creates a "representational void" that the language decoder fills with invented objects. Its fix, Background-Consistent Re-encoding, optimizes image pixels so that the target object's internal features blend into the surrounding background. It does this with three losses: one matches feature statistics to the background, one projects the target's features onto a dictionary of background features, and one keeps the rest of the image unchanged. The authors report 92–98% concealment success, preservation of up to 86% of non-target objects, and up to 3× fewer grounded hallucinations than attention suppression.

Background-Consistent Re-encoding still treats concealment as all or nothing: the object should become background. The authors also note a failure they did not design for, "partial semantic substitution," in which the target drifts into a semantically adjacent object. For example, an accessory gets re-encoded as part of a shirt. We read that failure as evidence that the feature space resists voids and prefers semantic neighbors. We therefore propose to make substitution deliberate. The target should become a prototype of its parent category, not background. Our claim has three parts, and each can be refuted. First, parent-category re-encoding yields fewer grounded hallucinations than background re-encoding. Second, it keeps the parent category available for captioning and question answering. Third, it drives fine-grained identity leakage to near chance among sibling categories.

The text sanitization literature supplies the design principle rather than the method. NAP2 formalizes "obscuring" as generalizing to a broader category, and reports that it beats differential-privacy rewriting on privacy entailment (93.81% versus below 63%) while keeping text natural. Reducing Privacy Risks in Online Self-Disclosures shows the same pattern for demographic rewriting. Neither paper touches images. We also take one constraint from the text work. Staab et al. note a "core message" dilemma: when the question is about the sensitive attribute itself, abstraction necessarily destroys the answer. We expect the same limit here and will measure it rather than hide it.

Research Program

The method, Hypernym-Consistent Re-encoding, keeps the white-box pixel optimization and background-preservation loss of Background-Consistent Re-encoding. It replaces the background dictionary with a category prototype dictionary and adds an explicit identity-erasure term. Let $x'$ be the edited image and $f(x')_R$ the model's middle-layer features for the target region $R$. For a target whose fine-grained label is $c$ with parent category $h$, let $D_h$ be a dictionary of region features pooled from many instances of all siblings of $c$ under $h$. Let $p_{\text{leaf}}$ be a linear probe that predicts the sibling label from those features. We minimize

$$ \mathcal{L}(x') = \mathcal{L}_{\text{pres}}(x') + \lambda_1 \, \big\| f(x')_R - \Pi_{D_h}\big(f(x')_R\big) \big\|^2 + \lambda_2 \, \mathrm{KL}\big(p_{\text{leaf}}(f(x')_R) \,\|\, U_h\big), $$

In this objective, $\Pi_{D_h}$ is projection onto the span of the dictionary, $U_h$ is the uniform distribution over the siblings of $c$, and $\lambda_1, \lambda_2$ weight the two terms. The second term keeps the region looking like "some $h$." The third makes the region uninformative about which $h$.

We place the probe in the middle layers because SHARP found hallucination-relevant information linearly separable at layers 10–15 of LLaVA-style models. That is where we expect identity to be most recoverable. Setting $h$ to the root of the hierarchy and building $D_h$ from background patches recovers Background-Consistent Re-encoding as a special case. This lets us sweep abstraction level on one axis: leaf, parent, supercategory, background, and void.

We use hierarchies that already exist with the data. For ImageNet, the WordNet tree supplies fine-grained leaves such as dog breeds and car models, which serve as proxies for identity. For COCO, object categories sit under supercategories. These proxies do not match real privacy targets exactly, and we will say so. They do, however, give exact ground truth at every level.

Baselines

The naive reference points are doing nothing and erasing. Leaving the image unedited sets the ceiling for utility and the floor for privacy. Black-box masking and Gaussian blur of the bounding box, run through the same models, set the "void" condition. The void-driven hallucination story predicts these will produce the most grounded hallucinations.

The strongest existing method is Background-Consistent Re-encoding from Do Not Leave a Gap. Its reported results are 92–98% concealment success, up to 86% non-target preservation, and up to 3× fewer grounded hallucinations than attention suppression. The file reports no exact SSIM or LPIPS numbers for it. We will re-run it with its released settings on the same images, the same three models (LLaVA-1.5, BLIP-2, InstructBLIP), the same GLIP-based grounding check, and the same number of optimization steps and hyperparameter search budget as our method. Beating it means two things together. At matched fine-grained leakage and matched perceptual distortion, our method should show lower grounded hallucination and higher parent-category retention. It should also not reduce non-target preservation.

We compare against attention or token suppression as a second, weaker white-box point, since that is the failure case the prior paper documented.

The key ablation isolates the prototype dictionary. We run our full objective with $D_h$ replaced by background features, and separately with the identity-erasure term removed. If replacing the dictionary alone accounts for the hallucination gain, the claim about semantic adjacency is supported. If the gain requires the probe term, the story is instead about identity removal.

Why not just…?

Why not just inpaint a generic object of the parent category with an image generator? This is the most serious threat. It is black-box, cheap, and makes the region contain a real "car" in pixels, so descriptions of it are not hallucinations. We will run it as a full baseline, using a text-conditioned inpainter prompted with the parent category and given the same mask. Our hypothesis is that inpainting swaps more than identity. It changes the object's color, pose, size, and spatial relation to its neighbors, which is exactly the "uncontrolled semantic drift" the text literature warns about. Feature-space editing, in contrast, changes only what the probe needs. We will test this with MME-style attribute questions about the concealed object: existence, color, position, and count, answered before and after editing. If inpainting matches our method on attribute preservation and leakage, we will report that honestly. In that case the contribution narrows to the abstraction-level measurement protocol and the finding that parent-level filling beats voids and backgrounds.

Why not just let the model see everything and sanitize its output text? A text anonymizer in the style of Language Models are Advanced Anonymizers could rewrite "a 2019 Tesla Model 3" to "a car" after generation. That protects one caption from one trusted model, but the released image still carries the identity. Any other model, or a targeted prompt, can recover it. The prior concealment paper already found that targeted probing extracts traces even from its own re-encoded images. Our threat model is releasing the image, so we evaluate output rewriting under open probing by a second model. We expect near-baseline leakage.

Why not just blur or mask more aggressively? The file's evidence is that removing visual information creates the void that drives invention. Do Not Leave a Gap found up to 3× more grounded hallucination under suppression, and text masking produced readability of 0.38 against 0.89 for generalization in Staab et al. The void conditions in our sweep test this directly.

Proposed Experiments

We use about 1,000 ImageNet images with fine-grained WordNet leaves and COCO images with supercategory labels. This follows the prior paper's scale and use of GLIP as an external detector. For each image and method we measure five quantities:

  • Fine-grained leakage: accuracy of an attacker at recovering the leaf label, using two attackers. The black-box attacker is an open question plus multiple choice over siblings, posed to the three models and to a held-out model not used during optimization. The white-box attacker is a freshly trained middle-layer linear probe.
  • Parent retention: the rate at which captions and answers correctly mention the parent category.
  • Grounded hallucination: CHAIR-style sentence-level and object-level rates, where GLIP verifies objects against the original image.
  • Existence probing: POPE-style questions for random, popular, and adversarial negatives on the edited image.
  • Scene fidelity: non-target preservation and SSIM/LPIPS.

The decisive check is the abstraction sweep, plotted as leakage and hallucination against level: leaf, parent, supercategory, background, and void. The hypothesis predicts that hallucination is lowest at parent level and rises toward both the background and void ends. Leakage at parent level should stay near $1/|\text{siblings}|$. A flat hallucination curve, or one lowest at background, would refute the claim.

A second check is transfer to the held-out model, since white-box methods that fail to transfer cannot protect a released image. A third is the core-message subset, meaning questions whose answer is the identity itself, where we expect and will report unavoidable utility loss. Given the documented gap between yes/no probing and free-form captioning in Evaluation and Analysis of Hallucination in LVLMs, we report both protocols and do not treat either alone as conclusive.