Matched Identity Twins: Testing Whether Face-Editing Scores Reflect Memorized Faces

Instruction-based face editors are judged by how well they keep a person recognizable after an edit such as "make her smile" or "age him twenty years." The usual measure is identity similarity (ID-Sim): both faces go through a frozen face-recognition network, and the score is the cosine similarity of the two embeddings. Public portrait benchmarks are mostly photos of public figures scraped from the web, which is also where the editors and the face-recognition networks get their training data. A high ID-Sim may therefore show that the editor keeps identity in general. It may also show that the editor, or the network scoring it, has already seen this person many times. Contamination research has shown how large this gap can be for language and vision-question models. Nobody has measured it for identity preservation.

We propose a controlled probe. For each benchmark portrait, an image generator produces two images under the same procedure. The first is a re-rendering that keeps the original person. The second is a "twin" that keeps the pose, expression, lighting and scene but has a new identity, held far from the source and from any known public face by an identity-preserving loss used in reverse. Every editor is run on both images and scored with the same evaluator. Generation artifacts are now matched across the pair, so any ID-Sim advantage for the real person is evidence of memorized identity rather than general skill. We call this difference the identity memorization gap. The probe can also be pointed at the evaluator, which tests whether the face-recognition scorer is itself inflated on benchmark identities.

Core Claim

The face-recognition loss that editors use to keep identity fixed can be repurposed to vary identity alone while holding everything else constant. That turns it into a contamination instrument for generative face models. We claim that a generation-matched contrast between real and twin identities detects identity memorization reliably. Other perturbations change identity by accident, and the contrasts available today change image statistics along with identity, so neither can make this measurement. We also claim that once the gap is measured, the reported ID-Sim rankings of current editors may compress or reorder.

Where This Sits

Two papers motivate the question. Optimizing ID Consistency in Multimodal Large Models introduces EditedID, a training-free repair that aligns the sampling trajectories of source and edit and gates attention between them. It reports an ID-Sim of 0.73 against a prior best of about 0.65 on real-person portraits. Its "cross-source feature contamination" refers to identity and edit features bleeding into each other during denoising, not to train–test leakage. It never checks whether the benchmark people were in the training data of GPT-4o, Flux.1, or the scoring network. LaTo conditions a diffusion transformer on landmark coordinate tokens and reports a 7.8% identity-preservation gain on HFL-150K over Bagel, FLUX.1-Kontext and Step1X-Edit. HFL-150K was itself partly synthesized by GPT-4o and Step1X-Edit. That is the kind of teacher-to-student leakage that Rethinking Benchmark and Contamination with Rephrased Samples found in text, where 12.8% of CodeAlpaca turned out to be disguised HumanEval items.

Two papers supply the method, and one of them also constrains the design. Contamination Detection for VLMs Using Multi-Modal Semantic Perturbations regenerates benchmark images with Flux conditioned on edge maps (ControlNet with Canny edges) so that the correct answer changes. Clean models improve on the regenerated set, by +31.5% on MMStar, while contaminated models fall by as much as −45%. We reuse its central move: regenerate the benchmark and read memorization from a shift in performance. We differ in two ways.

The first difference concerns what is held fixed. Edge maps throw away fine facial geometry, so applied to faces their pipeline would shift identity and blur the very variable we want to control. The second difference is a constraint that their own numbers impose. The clean-model gain of +31.5% shows that generated images are systematically easier. A naive comparison of original against twin would therefore confuse "generated" with "unfamiliar." We remove that confound by passing both arms through the same generator.

Both Text and Images Leaked! (MM-DETECT) contributes the method of validating a detector by injecting contamination at controlled doses. It also contributes a warning: leaking a benchmark's training split inflates scores more than leaking its test split. That finding matters for face recognition, because the training identity lists of these networks overlap heavily with benchmark celebrities.

The identity losses we invert come from the editing literature. Continuous Face Aging GANs uses cosine similarity under a frozen recognition network as a training penalty. Text and Image Guided 3D Avatar Generation pairs that penalty with a semantic guidance loss acting as a separate constraint. Both papers use the loss only to hold identity fixed. We also use it to move identity in a controlled way.

Research Program

Twin generation is the main construction. For a source portrait $x$, we produce a re-rendering $\hat{x}$ and a twin $\tilde{x}$ with the same generator, a Flux-family editor conditioned on landmark tokens in the style of LaTo. We use landmark tokens rather than edge maps so that pose and expression are fixed without copying pixels. Both images are optimized with shared constraints on landmarks, a face-parsing mask, and a background match. They differ only in the identity term:

$$\cos(f(\hat{x}), f(x)) \ge \tau_{\text{keep}}, \qquad \max\big(\cos(f(\tilde{x}), f(x)),\ \max_{g \in G}\cos(f(\tilde{x}), g)\big) \le \tau_{\text{new}}.$$

Here $f$ is a frozen recognition network used only during generation, $G$ is a gallery of embeddings for public identities (celebrity sets and the benchmark itself), and the two thresholds come from the recognition network's verification curve.

To avoid making the probe a property of one recognizer, generation uses a different network from the one used in evaluation. We take recognition backbones from TFace and TransFace. We discard any pair in which the two images differ by more than a set margin in image quality, measured by a no-reference quality score, CLIP aesthetic score, and pose error. The point of this filter is to keep difficulty matched.

The quantity we measure is the identity memorization gap of editor $E$ under evaluator $h$:

$$\text{IMG}(E,h) = \mathbb{E}_{x,\,c}\big[\cos(h(E(\hat{x},c)), h(\hat{x})) - \cos(h(E(\tilde{x},c)), h(\tilde{x}))\big],$$

where $c$ is an editing instruction drawn from the benchmark. A clean editor should show a gap near zero, because both arms are equally generated and differ only in whether the person is familiar.

The probe is then applied to the evaluator itself. We train a small recognition network with TFace on a public training corpus from which every benchmark identity has been removed. We then compare its verdicts with those of standard ArcFace-style networks on the same pairs. If ArcFace gives higher absolute ID-Sim, or a larger gap, on benchmark people than the held-out network does, the scorer is contaminated as well as the editor. This tests Gap 5 of the file directly.

Baselines

The naive reference point is reported ID-Sim on the original benchmark images with no probe. For EditedID this is the 0.73 reported in its paper. For LaTo on HFL-150K it is the reported 7.8% identity-preservation gain over FLUX.1-Kontext, Step1X-Edit and Bagel. We re-run these editors with their released settings on our subset so that our numbers can be compared with theirs. Any ranking we report beside these scores must reproduce them within noise before we interpret our gap.

The strongest existing contamination detector is the semantic-perturbation pipeline of Contamination Detection for VLMs. It is the closest method, and it has the strongest reported separation: +31.5% for clean models against −45% for contaminated ones on MMStar, and in the dosage study a clean LLaVA-v1.5-7B moves from about 38% to about 69%, while after three contamination epochs the scores become about 54% and about 38%. There is no reported result on Lacuna for this method on any face-editing or face-identity benchmark. We port it faithfully: Flux with Canny-edge ControlNet regenerates each portrait, and memorization is read from the drop in ID-Sim between original and regenerated images. It gets the same images, the same editors, the same evaluators, and the same number of generation seeds as our method. We beat it if, on the controlled-dosage models, our gap reaches a higher area under the ROC curve for separating contaminated from clean editors than its delta does. We also beat it if its delta is significantly non-zero on clean controls because of identity drift while ours is not.

A second competitor is a fresh-photo held-out set: newly photographed, consenting, never-posted subjects, edited with the same instructions and roughly matched for pose and lighting. This is the most direct answer to "are these faces memorized." We collect a few hundred such portraits, both as a competitor and as ground truth for whether the twin arm behaves like a truly novel person.

Two ablations isolate the key ingredient. The first drops the generation-matched re-rendering and compares the original image directly with the twin. If our reasoning about the +31.5% generator effect is right, this ablation should report a spurious gap even for clean editors. The second drops the gallery repulsion term and only pushes the twin away from its source. This shows whether twins otherwise collapse onto other memorized faces.

Why not just…?

Why not just collect fresh, never-posted portraits and compare ID-Sim on them? This is the real threat, and we include it rather than argue against it. Fresh photos answer the question for whoever is in them. However, fresh photos differ from benchmark portraits in capture conditions, demographics and image quality. A lower ID-Sim on them cannot be cleanly attributed to unfamiliarity rather than difficulty. Fresh photos also need consent and new collection for every benchmark, and they cannot be matched scene by scene. Our experiments settle the comparison in two steps. If twins and fresh photos give the same gap on the controlled-dosage models, fresh photos validate the twins. The twins then add per-image matching and scale. If fresh photos detect contamination that twins miss, the twin generator is leaking identity, and we report that result.

Why not just apply the edge-map perturbation pipeline as published? Edge maps preserve layout but not the fine geometry that recognition networks rely on. Regenerated faces therefore drift in identity, and a lower ID-Sim then mixes memorization with drift. The second ablation above, and the ported baseline on clean controls, measure how large this drift is. The published clean-model improvement of +31.5% also shows that generated images are not difficulty-neutral, which is why both of our arms pass through the generator.

Why not just run membership inference on the editor, for instance by checking whether its denoising loss is unusually low on benchmark images? We found no reported result on Lacuna for diffusion membership inference on face-editing benchmarks, so we include a denoising-loss score as an additional baseline on the open editors. It cannot be computed for API models such as GPT-4o, which our behavioral probe can audit. It also detects memorization of a particular image, not of a person. Familiarity with a person seen in thousands of different photos is the case that matters for ID-Sim, and it is the case a twin targets.

Why not just swap in a different face-recognition evaluator? Changing the evaluator addresses only half the problem, the scorer's own contamination, and does nothing about the editor's. It is also only meaningful if the replacement network has never seen the benchmark identities. That is why the held-out-identity evaluator appears as one component of the program rather than as the whole program.

Proposed Experiments

The first experiment validates the detector with controlled contamination. We fine-tune an open editor, FLUX.1-Kontext or In-Context Edit, using low-rank adapters on portraits of benchmark identities. The adapter data contain no benchmark test images. We use doses of 0%, 10%, 50% and 100% of identities, following the design of MM-DETECT, and keep a disjoint set of identities as a within-model negative control. The detector passes if the gap is near zero at dose 0, rises with dose, and stays near zero on the held-out identities. The edge-map baseline and the original-versus-twin ablation are expected to fail the zero-dose check.

The second experiment audits deployed editors. We apply the probe to the EditedID real-person benchmark and to HFL-150K test faces for FLUX.1-Kontext, Step1X-Edit, Bagel, In-Context Edit with and without EditedID, LaTo where weights are available, and GPT-4o through its API. We report the gap with 95% bootstrap intervals over identities, broken down by edit type, because edits that preserve pose and edits that move it plausibly rely on memory to different degrees. The headline question is whether rankings by gap-corrected ID-Sim, meaning twin-arm ID-Sim, differ from the published rankings.

The third experiment audits the evaluator. We compare ArcFace-style scorers against the held-out-identity TFace network on identical image pairs. A larger benchmark-identity advantage under the standard scorers would show that the identity loss itself is contaminated.

The decisive check is agreement with ground truth. On the fresh-photo set, the twin-arm ID-Sim for each editor should match fresh-photo ID-Sim more closely than original-benchmark ID-Sim does. If it does not, the twins are not a valid stand-in for novel people, and the right conclusion is that only the fresh-photo audit should be used.