IDDM: Identity-Decoupled Personalized Diffusion Models with a Tunable Privacy-Utility Trade-off

Linyan Dai, Xinwei Zhang, Haoyang Li, Qingqing Ye, Haibo HuarXiv 2026 · arXiv

TL;DR

IDDM introduces a "model-side output immunization" defense for personalized text-to-image diffusion models, allowing users to safely create custom avatars without risking biometric tracking. By embedding a two-stage identity-decoupling optimization directly into the training pipeline, it produces high-fidelity images that look great to humans but successfully evade downstream face recognition systems.

What Problem Does This Paper Address?

Personalized text-to-image generation tools (like DreamBooth and LoRA) allow users to fine-tune a diffusion model on a few personal photos to generate high-fidelity avatars in diverse contexts. However, when users share these generated portraits on social media, they expose themselves to privacy risks: automated Face Recognition Systems (FRS) can scrape these images to track, profile, or link them back to the user's real-world identity.

Existing defenses rely on "anti-personalization," which injects adversarial noise into reference photos to intentionally break model training. While effective at preventing unauthorized model training, this approach fails when a user wants to use a legitimate personalization service (e.g., an AI headshot generator) but wants to ensure the resulting public outputs don't leak their biometric identity.

To bridge this gap, the authors propose a new paradigm called model-side output immunization. The goal is to allow authorized personalization while ensuring that any generated public outputs are decoupled from the user's real biometric signature, complete with a tunable privacy-utility trade-off.

Figure 1: Comparison of protection paradigms under differ- ent threat models in personalized text-to-image generation: prior anti-personalization training (left) versus our model- side output immunization (right).

Methods

The authors introduce Identity-Decoupled personalized Diffusion Models (IDDM). Instead of applying a one-time perturbation to images or post-processing outputs at inference time, IDDM integrates identity decoupling directly into the personalization training loop via an alternating procedure:

  1. Short Fine-tuning: The framework runs a few standard personalization steps (DreamBooth/LoRA) on a small, clean reference set to capture the current personalization trajectory.
  2. Identity-Decoupled Data Optimization: Using the temporary model state from the previous step, a "protection set" of images is optimized under an $\ell_\infty$ budget using a two-stage schedule:
  3. Stage I (Fidelity-Preserving): Minimizes a denoising-consistency loss ($\mathcal{L}_{rec}$) to ensure the modified images remain compatible with diffusion training, preventing facial distortions.
  4. Stage II (Identity-Decoupling): Minimizes the cosine similarity ($\mathcal{L}_{id}$) between the denoised image predictions and the user's identity prototype across an ensemble of surrogate face recognizers (e.g., ArcFace, FaceNet).
  5. Fine-tuning on Protected Data: The personalized model is then fine-tuned on these optimized images, baking the identity-decoupling effect directly into the model weights.

A key hyperparameter, the step-allocation ratio $\rho \in (0, 1)$, dictates how many steps are spent on Stage I versus Stage II, providing an explicit knob to tune the privacy-utility trade-off.

Datasets

The method was evaluated using two standard facial datasets:

  • CelebA-HQ: Used for evaluating high-quality face generation and visual fidelity.
  • VGGFace2: A large-scale, in-the-wild dataset characterized by substantial intra-identity variations in pose and appearance, used to test robustness.

Key Insights and Ideas

  • Moving Protection to the Model Side: By embedding the defense into the model weights rather than relying on fragile image-level perturbations, the protection becomes immune to inference-time image processing (like denoising or purification pipelines).
  • Avoiding the Gradient Tug-of-War: Simultaneously optimizing for visual fidelity and identity suppression creates conflicting gradients that degrade image quality. IDDM’s sequential, two-stage schedule stabilizes training by establishing denoising consistency before pushing the identity features away.
  • Adaptive Weighting for Ensembles: To defend against multiple face recognizers, IDDM uses a Softmin-based adaptive weighting mechanism that concentrates optimization gradients on the most vulnerable recognizer in the ensemble.

Key Results

  • Biometric Evasion: On CelebA-HQ, IDDM significantly reduced the True Accept Rate (TAR) at a False Accept Rate (FAR) of $1e-4$ across seven black-box face recognition models. For instance, under a "dslr portrait" prompt, the average TAR plummeted from 69.46% (No Defense) to 23.13% (IDDM).
  • Tunable Trade-offs: Lower values of $\rho$ (e.g., 0.1) yielded aggressive identity suppression, while higher values (e.g., 0.5) favored portrait usability and visual quality (improving FID and Face Detection Success Rates).
  • Robustness to Transformations: Unlike post-processing baselines (such as AdvDM or LowKey), IDDM's protection remained robust when outputs underwent common image operations like JPEG compression, resizing, blurring, and cropping.

Limitations and Future Work

  • No Absolute Anonymity: IDDM acts as a defense against standard automated profiling, but highly adaptive or future face recognition systems with entirely different feature spaces might still extract residual identity cues.
  • Future Directions: The authors suggest extending model-side defenses to more generalized generation pipelines beyond subject-driven personalization and finding more principled mathematical formulations to optimize the privacy-utility balance.

Summary updated

Abstract

Personalized text-to-image diffusion models (e.g., DreamBooth, LoRA) enable users to synthesize high-fidelity avatars from a few reference photos for social expression. However, once these generations are shared on social media platforms (e.g., Instagram, Facebook), they can be linked to the real user via face recognition systems, enabling identity tracking and profiling. Existing defenses mainly follow an anti-personalization strategy that protects publicly released reference photos by disrupting model fine-tuning. While effective against unauthorized personalization, they do not address another practical setting in which personalization is authorized, but the resulting public outputs still leak identity information. To address this problem, we introduce a new defense setting, termed model-side output immunization, whose goal is to produce a personalized model that supports authorized personalization while reducing the identity linkability of public generations, with tunable control over the privacy-utility trade-off to accommodate diverse privacy needs. To this end, we propose Identity-Decoupled personalized Diffusion Models (IDDM), a model-side defense that integrates identity decoupling into the personalization pipeline. Concretely, IDDM follows an alternating procedure that interleaves short personalization updates with identity-decoupled data optimization, using a two-stage schedule to balance identity linkability suppression and generation utility. Extensive experiments across multiple datasets, diverse prompts, and state-of-the-art face recognition systems show that IDDM consistently reduces identity linkability while preserving high-quality personalized generation.
BibTeX
@misc{dai2026iddm,
  author = {Linyan Dai and Xinwei Zhang and Haoyang Li and Qingqing Ye and Haibo Hu},
  title = {{IDDM: Identity-Decoupled Personalized Diffusion Models with a Tunable Privacy-Utility Trade-off}},
  year = {2026},
  eprint = {2604.00903},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.00903},
}
Figure