TL;DR
This paper introduces LOHO, a novel optimization-based approach for hairstyle transfer that leverages GAN inversion to manipulate hair attributes like structure, appearance, and style independently. By employing a two-stage optimization strategy and a unique gradient orthogonalization technique, LOHO achieves state-of-the-art results in photorealistic hairstyle synthesis while preserving identity, addressing a key challenge in fine-grained image editing.
What Problem Does This Paper Address?
Transferring hairstyles between images is a visually complex task. Existing methods often struggle with misaligned hair structures between the source and target images, leading to artifacts or unnatural results. Furthermore, achieving fine-grained control over specific hair attributes (like shape vs. color) without affecting others has been a significant challenge. This paper aims to solve these problems by developing a method that can disentangle and manipulate these attributes effectively, even when the input images have significant structural differences.
Methods
LOHO is an optimization-based approach that works within the latent space of a pre-trained StyleGANv2 generator. The core idea is to invert a target image into the StyleGANv2 latent space and then optimize this latent representation to achieve the desired hairstyle transfer.
Key techniques include:
- Attribute Decomposition: Hair is broken down into three distinct attributes: perceptual structure, appearance, and style. Tailored loss functions are designed for each attribute.
- GAN Inversion: The method uses GAN inversion to find a latent code that reconstructs the input image. This allows for manipulation in a semantically meaningful latent space.
- Two-Stage Optimization: To prevent attribute conflicts and improve convergence, optimization is performed in two stages. Stage 1 focuses on identity and hair structure, while Stage 2 incorporates appearance and style.
- Gradient Orthogonalization: This is a crucial novel contribution. It ensures that the gradients for different hair attributes (specifically structure vs. appearance/style) are orthogonal, preventing one attribute's optimization from negatively interfering with another's. This is achieved by projecting the structure gradient to remove components parallel to the appearance and style gradients.
- Infilling with StyleGANv2 Prior: For regions where hair is misaligned, LOHO leverages StyleGANv2's generative prior to "infill" missing details in the latent space, rather than relying on traditional pixel-space inpainting networks which can lead to blurriness.
Datasets
The primary dataset used for training and evaluation is Flickr-Faces-HQ (FFHQ). This dataset is chosen for its high quality and significant variation in ethnicity, age, and hair styles. The authors also categorize image tuples into "easy," "medium," and "difficult" based on IoU (Intersection over Union) and pose distance (PD) metrics derived from face masks and landmarks, creating a more nuanced evaluation benchmark.
Key Insights and Ideas
- Latent Space is Key for Disentanglement: Manipulating attributes in the latent space of a powerful generative model like StyleGANv2 offers a more robust way to achieve disentangled control compared to pixel-space operations.
- Orthogonal Gradients for Attribute Independence: The core insight is that by making the gradients of different attributes orthogonal, their optimization processes can be made independent. This is a generalizable technique for disentangled manipulation in latent spaces.
- Two-Stage Optimization for Stability: Complex optimization objectives can lead to divergence. A staged approach, prioritizing identity and structure before appearance and style, provides a more stable and effective training process.
- GAN Prior for Robust Inpainting: Using the generative prior of a GAN for inpainting misaligned regions is more effective than traditional methods, leading to sharper and more coherent results.
Key Results
- State-of-the-Art FID: LOHO achieves a superior Fréchet Inception Distance (FID) score of 8.419 compared to MichiGAN's 10.697 for hairstyle transfer on the FFHQ dataset. A masked FID (LOHO-HF) of 4.847 further highlights its ability to synthesize realistic hair and face regions.
- Identity Preservation: LOHO demonstrates comparable or better performance than existing image embedding methods (I2S, I2S++) in terms of PSNR and SSIM for identity reconstruction, indicating it preserves the subject's likeness well.
- Qualitative Improvements: LOHO produces more realistic and seamlessly blended hairstyles, especially in cases of significant misalignment, where other methods struggle. It also handles transferring varied hair colors even when the source has uniform color.
- Disentangled Editing: The method successfully demonstrates individual editing of hair attributes (appearance, style, structure, shape) and joint editing, without interference between attributes.
Limitations and Future Work
- Extreme Misalignment: LOHO can still struggle with extreme cases of hair and face misalignment, potentially leading to unnatural hair shapes. The authors suggest GAN-based alignment networks could help mitigate this.
- Segmentation Imperfections: Imperfect hair segmentation from tools like Graphonomy can lead to residual hair details from the identity image being carried over. More sophisticated segmentation networks are proposed as a solution.
- Future Work: The authors suggest introducing a deformation objective to enforce alignment across a wider range of head poses and hair shapes, and improving convergence by predicting an initial optimization point.
Summary updated