LOHO: Latent Optimization of Hairstyles via Orthogonalization

Rohit Saha, Brendan Duke, Florian Shkurti, Graham W. Taylor, Parham Aarabi

Abstract

Hairstyle transfer is challenging due to hair structure differences in the source and target hair. Therefore, we propose Latent Optimization of Hairstyles via Orthogonalization (LOHO), an optimization-based approach using GAN inversion to infill missing hair structure details in latent space during hairstyle transfer. Our approach decomposes hair into three attributes: perceptual structure, appearance, and style, and includes tailored losses to model each of these attributes independently. Furthermore, we propose two-stage optimization and gradient orthogonalization to enable disentangled latent space optimization of our hair attributes. Using LOHO for latent space manipulation, users can synthesize novel photorealistic images by manipulating hair attributes either individually or jointly, transferring the desired attributes from reference hairstyles. LOHO achieves a superior FID compared with the current state-of-the-art (SOTA) for hairstyle transfer. Additionally, LOHO preserves the subject's identity comparably well according to PSNR and SSIM when compared to SOTA image embedding pipelines.

Introduction

Hairstyle transfer is a notoriously difficult task in computer vision. Unlike changing eye color or skin tone, hair has complex geometry, varied textures, and often doesn't "fit" perfectly when moved from one head to another. Existing methods often struggle with "misalignment"—when the source hair shape doesn't match the target face's pose—leading to blurry artifacts or lighting inconsistencies. The core challenge is achieving fine-grained control over hair attributes (color vs. shape) without them interfering with each other or destroying the subject's original identity.

Methodology

LOHO shifts the strategy from training a new model to GAN Inversion. It utilizes a pretrained StyleGANv2 and optimizes the latent code ($W^+$) and noise space ($N$) to find the image that best satisfies the user's hair requirements.

Attribute Decomposition

The framework decomposes "hair" into three mathematically defined components:

  • Perceptual Structure: The coarse shape and layout, modeled using LPIPS loss on eroded masks.
  • Appearance: The global color, captured by averaging feature maps from shallow VGG layers.
  • Style: Fine-grained textures and wisps, captured via Gram matrices.

Two-Stage Optimization and Gradient Orthogonalization

To prevent attribute interference, LOHO employs a two-stage optimization process. Stage 1 focuses on identity and hair structure. Stage 2 introduces appearance and style losses. The "secret sauce" is Gradient Orthogonalization: in Stage 2, the structure gradient is projected onto a vector subspace orthogonal to the appearance and style gradients. This ensures that updating the hair's color does not inadvertently warp its shape.

Experimental Results

The researchers evaluated LOHO using the Flickr-Faces-HQ (FFHQ) dataset. They curated a subset of image tuples categorized into Easy, Medium, and Difficult based on head pose alignment.

LOHO demonstrates high-fidelity hairstyle transfer across various identities and attributes, handling complex styles and pose misalignments.

Key Findings

  • Superior Realism: LOHO achieved a Frechet Inception Distance (FID) of 8.419, significantly outperforming the previous SOTA, MichiGAN (10.697).
  • Identity Preservation: Quantitative metrics (PSNR: 32.2, SSIM: 0.93) confirmed that LOHO excels at maintaining the target person's recognizable features.
  • Latent Infilling: By optimizing in latent space, the model "hallucinates" realistic hair details that naturally fit the distribution of human faces, avoiding the harsh edges of traditional pixel-based inpainting.

Discussion and Limitations

The power of LOHO lies in its use of orthogonality to solve the geometric problem of attribute interference. By forcing gradients to be orthogonal, it achieves disentanglement usually requiring massive labeled datasets. However, the method has limitations:

  • Extreme Misalignment: Optimization can fail if the source hair and target face angles are drastically different.
  • Segmentation Dependency: Errors in the hair mask can lead to "hair trails" or artifacts.
  • Computational Cost: As an optimization-based approach (2,000 iterations), it is slower than feed-forward networks.

Future work may involve predicting an initial latent starting point to accelerate convergence.

Summary updated

Abstract

Hairstyle transfer is challenging due to hair structure differences in the source and target hair. Therefore, we propose Latent Optimization of Hairstyles via Orthogonalization (LOHO), an optimization-based approach using GAN inversion to infill missing hair structure details in latent space during hairstyle transfer. Our approach decomposes hair into three attributes: perceptual structure, appearance, and style, and includes tailored losses to model each of these attributes independently. Furthermore, we propose two-stage optimization and gradient orthogonalization to enable disentangled latent space optimization of our hair attributes. Using LOHO for latent space manipulation, users can synthesize novel photorealistic images by manipulating hair attributes either individually or jointly, transferring the desired attributes from reference hairstyles. LOHO achieves a superior FID compared with the current state-of-the-art (SOTA) for hairstyle transfer. Additionally, LOHO preserves the subject's identity comparably well according to PSNR and SSIM when compared to SOTA image embedding pipelines.
BibTeX
@inproceedings{saha2021loho,
  author = {Rohit Saha and Brendan Duke and Florian Shkurti and Graham W. Taylor and Parham Aarabi},
  title = {{LOHO: Latent Optimization of Hairstyles via Orthogonalization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=-JNOwtd_mTY},
}
Figure