When StyleGAN Meets Stable Diffusion: a $\mathcal{W}_{+}$ Adapter for Personalized Image Generation

Xiaoming Li, Xinyu Hou, Chen Change Loy

Abstract

Personalized text-to-image (T2I) generation aims to synthesize specific individuals in diverse scenarios. However, existing methods often struggle with attribute entanglement, where the reference image's expression or lighting is inadvertently baked into the generated output. We propose the $\mathcal{W}_+$ Adapter, a novel framework that bridges StyleGAN’s highly disentangled facial latent space with Stable Diffusion’s versatile generative capabilities. By leveraging the semantic properties of the $\mathcal{W}_+$ space, our method enables precise, fine-grained control over facial attributes while maintaining high identity fidelity and background consistency.

Introduction

The field of personalized T2I generation has seen significant progress, yet achieving a perfect balance between identity preservation and prompt adherence remains challenging. A primary issue is entanglement: if a reference photo depicts a subject with a specific expression, models often replicate that expression regardless of the text prompt. This occurs because current inversion techniques struggle to separate identity-defining features from transient attributes like pose or lighting. Existing solutions often face a trade-off, either losing the subject's likeness or failing to generate diverse, contextually accurate images.

Methodology

Our approach utilizes a two-stage adapter strategy to harness the disentangled $\mathcal{W}_+$ latent space of StyleGAN, which is known for its ability to control specific facial features through predictable trajectories.

Stage I: Alignment

We first train a mapping network to translate $\mathcal{W}_+$ vectors—extracted using a pre-trained StyleGAN encoder—into tokens compatible with Stable Diffusion (SD). To integrate this identity information, we employ a Residual Cross-Attention module within the SD UNet. Unlike traditional parallel attention mechanisms, this residual connection allows the model to selectively attend to facial features, preventing identity information from corrupting the background or other prompt-specified elements.

Stage II: In-the-Wild Refinement

To ensure robustness in real-world scenarios, the model is fine-tuned on "in-the-wild" datasets. We introduce a disentanglement loss that utilizes data augmentation to enforce background consistency during attribute editing. Additionally, a regularization loss is applied to prevent the identity condition from overpowering the textual instructions, ensuring the model remains responsive to complex prompts.

Experimental Results

The $\mathcal{W}_+$ Adapter was evaluated against state-of-the-art baselines, including IP-Adapter and CelebBasis, using datasets such as FFHQ, SHHQ, and CelebA-HQ.

Quantitative Performance

Our model demonstrates superior performance in both identity preservation and text alignment. Using the ArcFace ID distance metric, the $\mathcal{W}_+$ Adapter showed higher fidelity to reference subjects than general-purpose adapters. Furthermore, it achieved a superior CLIP Score (0.267), indicating that generated images more accurately reflect textual descriptions, such as specific clothing or environments.

Qualitative Attribute Editing

A key advantage of our method is the ability to transfer StyleGAN's semantic directions ($\Delta w$ trajectories) to the Stable Diffusion context. This allows for "surgical" edits to facial attributes—such as age, smile intensity, or eye closure—without altering the surrounding scene or the subject's core identity.

Comparison of the W+ Adapter against baselines, demonstrating fine-grained attribute editing for smile, age, and eye closure while maintaining background consistency.

Discussion and Limitations

While the $\mathcal{W}_+$ Adapter represents a significant step forward, certain limitations remain. The final image quality is inherently bounded by the performance of the StyleGAN encoder; any details lost during the initial inversion to $\mathcal{W}_+$ cannot be recovered. Currently, the framework is optimized for single-subject images, and handling multiple faces or extremely complex attributes (like specific eyewear or radical pose changes) remains a challenge for future research. Potential extensions include localized injection for multi-subject scenes and adapting the framework for other object categories with well-defined latent spaces.

Summary updated

Abstract

Text-to-image diffusion models have remarkably excelled in producing diverse, high-quality, and photo-realistic images. This advancement has spurred a growing interest in incorporating specific identities into generated content. Most current methods employ an inversion approach to em-bed a target visual concept into the text embedding space using a single reference image. However, the newly synthe-sized faces either closely resemble the reference image in terms of facial attributes, such as expression, or exhibit a reduced capacity for identity preservation. Text descriptions intended to guide the facial attributes of the synthesized face may fall short, owing to the intricate entanglement of identity information with identity-irrelevant facial attributes derived from the reference image. To address these issues, we present the novel use of the extended StyleGAN embed-ding space $\mathcal{W}+$, to achieve enhanced identity preservation and disentanglement for diffusion models. By aligning this semantically meaningful human face latent space with text-to-image diffusion models, we succeed in maintaining high fidelity in identity preservation, coupled with the capacity for semantic editing. Additionally, we propose new training objectives to balance the influences of both prompt and identity conditions, ensuring that the identity-irrelevant back-ground remains negligibly affected during facial attribute modifications. Extensive experiments reveal that our method adeptly generates personalized text-to-image outputs that are not only compatible with prompt descriptions but also amenable to common StyleGAN editing directions in diverse settings. Our code and model are available at https://github.com/csxmli2016/w-plus-adapter.
BibTeX
@inproceedings{li2024when,
  author = {Xiaoming Li and Xinyu Hou and Chen Change Loy},
  title = {{When StyleGAN Meets Stable Diffusion: a $\textbackslash\{\}mathcal\{W\}\_\{+\}$ Adapter for Personalized Image Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00213},
}
Figure