Anonymization Prompt Learning for Facial Privacy-Preserving Text-to-Image Generation

Liang Shi, Jie Zhang, Shiguang Shan

Introduction

The rapid advancement of text-to-image (T2I) diffusion models, such as Stable Diffusion and DALL-E 3, has introduced significant privacy concerns regarding the generation of high-fidelity deepfakes. By simply prompting a model with a public figure's name, users can generate indistinguishable likenesses, facilitating potential harassment or misinformation.

Existing mitigation strategies generally fall into two categories: content moderation (e.g., blacklisting names) and concept editing (erasing identities from model weights). However, blacklists are easily bypassed through descriptive prompts, and concept editing is computationally expensive, often leading to "catastrophic forgetting" where the model loses general generation capabilities. To address these issues, researchers have introduced Anonymization Prompt Learning (APL), a lightweight, scalable "plug-and-play" solution.

Methods: Anonymization Prompt Learning (APL)

APL employs a soft prompt prefix—a sequence of 10 learnable vectors—that is prepended to the text embedding of any user prompt. Unlike traditional methods that modify the model's weights, APL keeps the diffusion model and text encoder frozen, updating only the prefix vectors.

The APL training framework utilizes a dual-objective approach to balance identity anonymization with general image quality preservation.

The training process involves two primary objectives:

  1. Anonymization Loss ($L_{ID}$): When the input specifies a known identity (e.g., "Albert Einstein"), the prefix nudges the diffusion process away from the specific facial identity toward a generic attribute description (e.g., "a male German physicist with a bushy mustache").
  2. Regularization Loss ($L_{Reg}$): To maintain the model's general utility, the prefix is trained to remain "inactive" for non-identity prompts, ensuring that generic requests like "apples on a table" produce the same high-quality output as the original model.

Key Results and Insights

The primary goal of APL is to achieve a "look-alike" balance: generating images that retain the semantic context of the prompt (e.g., occupation, gender, specific features) while failing automated face recognition tests.

Visual comparison of original identities (top row) versus anonymized "look-alikes" generated using the APL prefix (bottom row).

Performance Metrics

  • Identity Anonymization: On Stable Diffusion v1.5, the average identity accuracy (ID-ACC) dropped from 0.31 to 0.039, rendering the generated faces unrecognizable to AI systems.
  • Attribute Preservation: Despite the drop in identity recognition, Attribute Accuracy (Attr-ACC) remained high (~0.60), confirming that the model still generates semantically relevant "look-alikes."
  • Generalization: Remarkably, training on only 100 identities allowed the APL prefix to generalize to unseen individuals, learning a general "direction" in the latent space to move from specific identities to generic attributes.
  • Quality Maintenance: General image quality, measured by FID scores on the MS COCO dataset, remained virtually identical to the base model, ensuring no "quality tax" for privacy protection.

Conclusion

Anonymization Prompt Learning offers a robust, portable, and efficient method for protecting facial privacy in T2I generation. By focusing on a "look-alike" strategy rather than total erasure, APL preserves the creative utility of diffusion models while mitigating the risks of deepfake generation. Future work will explore scaling the training data to further harden the system against sophisticated bypass attempts.

Summary updated

BibTeX
@article{shi2026anonymization,
  author = {Liang Shi and Jie Zhang and Shiguang Shan},
  title = {{Anonymization Prompt Learning for Facial Privacy-Preserving Text-to-Image Generation}},
  journal = {Int. J. Comput. Vis.},
  year = {2026},
  url = {https://openreview.net/forum?id=1SUW5b75Hk},
}
Figure