Do Multimodal Judges Hallucinate Privacy Leaks?

Privacy researchers increasingly ask a vision-language model to decide whether a reconstructed image reveals someone's identity. A vision-language model is a chatbot-style model that reads images as well as text. The reconstructions in question come from model inversion attacks, which try to rebuild private training images, such as a specific person's face, from a trained classifier. The switch to these judges was a correction. The older practice scored attacks with a second face classifier, and those classifiers were fooled by noise patterns that do not actually depict the person. But the hallucination literature shows that vision-language models answer "yes" too readily, guess from scene context, and change their answers when a prompt is reworded.

This proposal tests whether vision-language privacy judges declare an identity match because the face itself matches, or because the hair color, skin tone, pose, and background look plausibly similar. It then builds a corrected audit that separates two cases: leakage of a person's identity, and leakage of the cheap attributes that a generative image prior would produce anyway. The practical stakes are concrete. Attack rankings, defense claims, and regulatory risk assessments now rest on these judges. If the judges hallucinate, the field has swapped one false-positive problem for a subtler one.

Core Claim

Revisiting Model Inversion Evaluation re-scored 26 inversion setups with general-purpose vision-language judges (Gemini 2.0, Qwen2.5VL). It found that standard attack accuracy is badly inflated. For the IFGMI attack on FaceScrub, the reported success rate fell from 95.85% to 34.46%. That paper treats the judge's verdict as close to ground truth.

Our claim is that part of the remaining 34.46%-style success is itself a judge artifact, and that the artifact can be measured with counterfactual images. If the judge's "same person" verdict survives deleting the face, it is a context guess, not face recognition. If the verdict also fires for an impostor who shares the victim's attributes, it is attribute matching, not identity matching. Either outcome is a judge false positive in the same spirit as the classifier false positives Ho et al. exposed.

The new contribution is not the use of a vision-language judge, which Ho et al. already introduced. It is the following:

  1. Importing the hallucination community's controlled-intervention methodology into privacy auditing.
  2. A decomposition of measured leakage into an identity-level part and an attribute-level part.
  3. A test of whether the corrected score agrees with humans better than the raw judge does.

Research Program

The motivating evidence comes from three hallucination papers.

  • Behind the Magic, MERLIM inpainted objects out of images. Models kept claiming the objects were present, with precision dropping about 8%, and accuracy swung by up to 26.5% across rephrasings of the same question.
  • Evaluating Object Hallucination in Large Vision-Language Models introduced POPE, which asks yes/no questions about whether an object is present. It showed that negatives chosen to co-occur with the true objects hurt models most, and that some models say "yes" nearly 99% of the time.
  • MLLM-as-a-Judge found that model judges cluster high scores and that chain-of-thought prompting makes them invent visual flaws. Even GPT-4V reached only 0.490 Pearson agreement with humans in scoring and about 79% agreement in pairwise comparison.

None of these papers studied privacy judgments. We carry their two key moves, removing the evidence and using co-occurrence-matched negatives, into identity verification of reconstructions.

The first component is a counterfactual probe suite built around each (reconstruction, victim reference) pair that a judge evaluates. It has four variants:

  • Face-ablated: the facial region is inpainted to a neutral fill, while hair, clothing, background, and color statistics are kept. This is a direct analogue of MERLIM's inpainting.
  • Context-ablated: the face is tightly cropped and the surroundings are replaced with a constant background.
  • Attribute-matched impostors: a different identity from the same dataset that shares the victim's annotated attributes. For CelebA we use its 40 binary attributes; for FaceScrub, gender and approximate age. This is the identity-verification analogue of POPE's adversarial split.
  • Prior-only reconstructions: images sampled from the attack's own generative model with no inversion signal at all. They give the judge's "yes" rate on images that carry zero private information.

The second component turns these probes into a corrected score. A reconstruction counts as an identity-level leak only if all three conditions hold:

  1. The judge accepts the true pair.
  2. The judge rejects the face-ablated version.
  3. The judge prefers the victim over attribute-matched impostors in a forced choice.

A reconstruction that passes the first condition but fails the others counts as attribute-level leakage. This is still privacy-relevant, in the sense that PerceptAnon showed context can identify people, but it is a weaker claim than reconstructing the person. Reporting the two separately also settles an apparent tension with PerceptAnon. Context leakage is real, but it should be named as context leakage rather than credited to an attack as identity recovery.

The third component is human ground truth. Its design follows the matching study used to validate Crafter, where annotators matched reconstructions to identities and reached a Macro-F1 of 0.251 against a chance level of 0.200. It also draws on the recognizability annotations of SemSim, which showed that a small human-labeled set (as few as 50 samples) suffices to train a useful metric. Annotators will perform forced-choice identification among the victim and attribute-matched impostors. This measures identity recognition directly and is less sensitive to rater leniency than yes/no ratings.

Baselines

The naive reference is the standard attack-accuracy protocol: a held-out face classifier trained on the private identities, run exactly as in the original attack papers. It is included to reproduce the inflation Ho et al. reported, so that our numbers are anchored to theirs.

The strongest existing method is Ho et al.'s judge itself. This means Qwen2.5VL, as the open model in their study, plus Gemini 2.0 where access allows, prompted with their published protocol. We run it on the same reconstructions from the same attack setups (FaceScrub, CelebA, Stanford Dogs) that they evaluated. The reported reference point is IFGMI on FaceScrub, at 95.85% standard accuracy versus 34.46% judge-assessed success. No reported result was found on Lacuna for how well Ho et al.'s judge agrees with human identification on reconstructions. That gap is the measurement this proposal supplies. Beating this baseline means our corrected score achieves higher rank correlation and higher forced-choice agreement with human identification. Specifically, the improvement must exceed the variation of the raw judge across five paraphrased prompts. We include that bound because MERLIM shows prompt variance alone can be large.

The hallucination-mitigated judge applies decoding-time anti-hallucination methods to the raw judge, with no counterfactual probes. We use RITUAL, which fuses predictions from the original and a randomly transformed image and raised LLaVA-1.5 POPE accuracy from 84.13% to 88.87%. Where we can access hidden states of an open judge, we also use SHARP, which steers middle-layer activations away from text-prior directions and reached 82.8% POPE adversarial accuracy on LLaVA-Next versus 79.5% without steering. These run with the same prompts and the same images, and with compute matched to our probe suite by counting forward passes.

The ablation that isolates the key ingredient keeps the full probe pipeline but replaces the attribute-matched impostors with random impostors. If random impostors give the same correction, then co-occurrence-style hard negatives are not what matters, and the POPE analogy fails.

Why not just…?

Why not just score reconstructions with an independent, off-the-shelf face recognition embedding? This is the most serious threat, because a verifier trained on different identities and architecture than the target might not share its adversarial weaknesses. Ho et al.'s false positives arose from an evaluator sharing the target's task and data distribution, and the file reports no test of fully independent verifiers. We therefore run such a verifier on every probe. We also run it on the prior-only and face-ablated sets. If its human agreement matches our corrected judge, the honest conclusion is that a vision-language judge is unnecessary for faces. The probes would then remain useful only for non-face domains like Stanford Dogs, where no strong identity verifier exists.

Why not just ensemble several vision-language judges and average over prompt paraphrases? Ensembling reduces prompt variance, which MERLIM shows is large. But it cannot remove a bias that all judges share, and the language-prior bias POPE documents appears across model families. The experiment that decides this is the prior-only set. If an ensemble still says "match" on attack-prior samples that carry no private signal at a rate well above zero, averaging has not fixed the problem.

Why not just apply anti-hallucination decoding to the judge? RITUAL and SHARP were designed for object presence, and they plausibly reduce yes-bias here too. However, flips and crops leave the hair, skin tone, and background intact, which are exactly the context cues we suspect drive false matches. Steering suppresses text priors, not visual-attribute shortcuts. This is tested directly as a baseline. If it closes most of the gap to human agreement, we will report that cheaper fix as the main recommendation.

Why not just use humans? Humans are the ground truth here, but auditing 26 attack setups, plus every new defense, by human study does not scale. The question is whether an automatic score can stand in for humans, and the human subset exists to answer it.

Proposed Experiments

The first study is diagnostic. For each judge, attack, and dataset we measure three things: the "same person" rate on face-ablated images, the rate on attribute-matched impostors, and the rate on prior-only samples. If the judges are grounded, all three should sit near the judge's false-match rate on unrelated identities. If the face-ablated and impostor rates are substantially elevated, the judges are hallucinating identity from context and attributes, mirroring MERLIM's object-removal result.

The second study re-scores the attacks Ho et al. evaluated, including IFGMI, PPA, PLGMI and LOMMA. Each is reported with three numbers: standard accuracy, raw judge success, and corrected identity-level success. The attribute-level remainder is reported separately. We also check whether attack rankings change, since rankings matter more to the field than absolute numbers.

The third study is the decisive check: human forced-choice identification on a stratified subset of a few thousand reconstructions. We compare Kendall's τ and forced-choice agreement with humans for five scores: the standard accuracy classifier, the independent face verifier, the raw judge, the decoding-mitigated judge, and the corrected score. Two outcomes would falsify the proposal. One is that the raw judge already matches humans as well as the corrected score does. The other is that face-ablated acceptance rates are indistinguishable from unrelated-identity rates. Either result would itself be worth publishing, as the first evidence that Ho et al.'s judges are grounded.

A final transfer test applies the probe suite to defended models, using Crafter's average-face collapse. Defenses that push reconstructions toward a generic prior are exactly where an attribute-driven judge could misreport residual identity leakage.