Abstract
Personalized text-to-image (T2I) generation aims to synthesize specific individuals in diverse scenarios. However, existing methods often struggle with attribute entanglement, where the reference image's expression or lighting is inadvertently baked into the generated output. We propose the $\mathcal{W}_+$ Adapter, a novel framework that bridges StyleGAN’s highly disentangled facial latent space with Stable Diffusion’s versatile generative capabilities. By leveraging the semantic properties of the $\mathcal{W}_+$ space, our method enables precise, fine-grained control over facial attributes while maintaining high identity fidelity and background consistency.
Introduction
The field of personalized T2I generation has seen significant progress, yet achieving a perfect balance between identity preservation and prompt adherence remains challenging. A primary issue is entanglement: if a reference photo depicts a subject with a specific expression, models often replicate that expression regardless of the text prompt. This occurs because current inversion techniques struggle to separate identity-defining features from transient attributes like pose or lighting. Existing solutions often face a trade-off, either losing the subject's likeness or failing to generate diverse, contextually accurate images.
Methodology
Our approach utilizes a two-stage adapter strategy to harness the disentangled $\mathcal{W}_+$ latent space of StyleGAN, which is known for its ability to control specific facial features through predictable trajectories.
Stage I: Alignment
We first train a mapping network to translate $\mathcal{W}_+$ vectors—extracted using a pre-trained StyleGAN encoder—into tokens compatible with Stable Diffusion (SD). To integrate this identity information, we employ a Residual Cross-Attention module within the SD UNet. Unlike traditional parallel attention mechanisms, this residual connection allows the model to selectively attend to facial features, preventing identity information from corrupting the background or other prompt-specified elements.
Stage II: In-the-Wild Refinement
To ensure robustness in real-world scenarios, the model is fine-tuned on "in-the-wild" datasets. We introduce a disentanglement loss that utilizes data augmentation to enforce background consistency during attribute editing. Additionally, a regularization loss is applied to prevent the identity condition from overpowering the textual instructions, ensuring the model remains responsive to complex prompts.
Experimental Results
The $\mathcal{W}_+$ Adapter was evaluated against state-of-the-art baselines, including IP-Adapter and CelebBasis, using datasets such as FFHQ, SHHQ, and CelebA-HQ.
Quantitative Performance
Our model demonstrates superior performance in both identity preservation and text alignment. Using the ArcFace ID distance metric, the $\mathcal{W}_+$ Adapter showed higher fidelity to reference subjects than general-purpose adapters. Furthermore, it achieved a superior CLIP Score (0.267), indicating that generated images more accurately reflect textual descriptions, such as specific clothing or environments.
Qualitative Attribute Editing
A key advantage of our method is the ability to transfer StyleGAN's semantic directions ($\Delta w$ trajectories) to the Stable Diffusion context. This allows for "surgical" edits to facial attributes—such as age, smile intensity, or eye closure—without altering the surrounding scene or the subject's core identity.
Discussion and Limitations
While the $\mathcal{W}_+$ Adapter represents a significant step forward, certain limitations remain. The final image quality is inherently bounded by the performance of the StyleGAN encoder; any details lost during the initial inversion to $\mathcal{W}_+$ cannot be recovered. Currently, the framework is optimized for single-subject images, and handling multiple faces or extremely complex attributes (like specific eyewear or radical pose changes) remains a challenge for future research. Potential extensions include localized injection for multi-subject scenes and adapting the framework for other object categories with well-defined latent spaces.
Summary updated