Zero-shot Face Editing via ID-Attribute Decoupled Inversion

Yang Hou, Minggu Wang, Jianjun Zhao

Abstract

Recent advancements in text-guided diffusion models have shown promise for general image editing via inversion techniques, but often struggle to maintain ID and structural consistency in real face editing tasks. To address this limitation, we propose a zero-shot face editing method based on ID-Attribute Decoupled Inversion. Specifically, we decompose the face representation into ID and attribute features, using them as joint conditions to guide both the inversion and the reverse diffusion processes. This allows independent control over ID and attributes, ensuring strong ID preservation and structural consistency while enabling precise facial attribute manipulation. Our method supports a wide range of complex multi-attribute face editing tasks using only text prompts, without requiring region-specific input, and operates at a speed comparable to DDIM inversion. Comprehensive experiments demonstrate its practicality and effectiveness.

Introduction

Editing human faces presents a unique challenge in computer vision compared to general object editing. In facial imagery, attributes such as expression, age, and gender are deeply intertwined with a person's unique identity (ID). Standard diffusion-based editing methods often struggle with this entanglement; for instance, a prompt to "make a person older" might inadvertently alter their bone structure or eye shape, resulting in a loss of identity consistency.

A demonstration of expression manipulation where the subject's identity remains consistent despite the change in emotion.

Existing approaches typically fall into two categories: GAN-based methods, which are efficient but limited in handling complex text prompts, and diffusion-based methods like DDIM inversion, which often fail to capture the fine-grained "essence" of a specific face due to the descriptive limitations of text prompts. This paper introduces a "zero-shot" method that decouples identity from attributes to maintain structural integrity without requiring manual masks or per-image optimization.

Methodology: ID-Attribute Decoupled Inversion

The core innovation of this work is the decomposition of facial representation into two distinct streams during the inversion process:

  • ID Features (The "Who"): A pre-trained CLIP vision model extracts a high-dimensional embedding of the face, acting as a structural anchor to preserve identity.
  • Attribute Features (The "What"): Standard CLIP text embeddings represent the target traits to be modified (e.g., "smiling" or "blue hair").

The Workflow

  1. Joint Conditioning: A Stable Diffusion model is fine-tuned using Low-Rank Adaptation (LoRA) and augmented with new cross-attention layers, allowing it to process image and text embeddings simultaneously.
  2. Precise Inversion: During the mapping of the image back to noise, the model utilizes both the original image and its text description. This creates a more accurate starting noise ($z^*_T$) than text-only methods.
  3. Reverse Diffusion (Editing): To perform an edit, the original image embedding is retained to preserve identity, while the text prompt is swapped for the target attribute.
  4. Refined Guidance: The original text prompt is used as a "negative prompt" during the editing phase, helping the model mathematically subtract old attributes while adding new ones.

Experimental Setup

The model was trained on a custom dataset of 69,900 face image-text pairs derived from the FFHQ (Flickr-Faces-HQ) dataset. Detailed captions describing expressions, hair, ethnicity, and accessories were generated using GPT-4o. For evaluation, the method was tested on 100 images each from FFHQ and CelebA-HQ to verify generalization to unseen real-world faces.

Results and Discussion

Our method demonstrates superior performance across various editing tasks, ranging from simple expression changes to complex multi-attribute transformations.

An overview of the diverse editing capabilities of the proposed method, including age, gender, and multi-attribute transformations.

Quantitative Performance

Compared to state-of-the-art tools like StyleClip and Null-text inversion, our method achieved significant gains:

  • Reconstruction Quality: A PSNR of 34.064, indicating the un-edited reconstruction is nearly identical to the source.
  • ID Preservation: An ID similarity score of 0.884 in single-attribute edits, outperforming DiffAE (0.851) and Collaborative Diffusion (0.301).
  • Editing Accuracy: The model successfully executed complex multi-attribute edits with a 75.32% accuracy rate while maintaining structural integrity.

Crucially, the method is "zero-shot," operating at speeds comparable to standard DDIM inversion without the need for iterative optimization for every new image.

Conclusion

By decoupling identity and attribute features, this method provides a robust solution for high-fidelity face editing. While the current implementation inherits some minor smoothing effects from the underlying Stable Diffusion model, future work will focus on integrating higher-resolution base models and expanding control mechanisms to include sketches or pose maps for even more granular editing.

Summary updated

Abstract

Recent advancements in text-guided diffusion models have shown promise for general image editing via inversion techniques, but often struggle to maintain ID and structural consistency in real face editing tasks. To address this limitation, we propose a zero-shot face editing method based on ID-Attribute Decoupled Inversion. Specifically, we decompose the face representation into ID and attribute features, using them as joint conditions to guide both the inversion and the reverse diffusion processes. This allows independent control over ID and attributes, ensuring strong ID preservation and structural consistency while enabling precise facial attribute manipulation. Our method supports a wide range of complex multi-attribute face editing tasks using only text prompts, without requiring region-specific input, and operates at a speed comparable to DDIM inversion. Comprehensive experiments demonstrate its practicality and effectiveness.
BibTeX
@misc{hou2025zero,
  author = {Yang Hou and Minggu Wang and Jianjun Zhao},
  title = {{Zero-shot Face Editing via ID-Attribute Decoupled Inversion}},
  year = {2025},
  eprint = {2510.11050},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.11050},
  howpublished = {arXiv},
}
Figure