Retrieve in Style: Unsupervised Facial Feature Transfer and Retrieval

Min Jin Chong, Wen-Sheng Chu, Abhishek Kumar, David Forsyth

TL;DR

Retrieve in Style (RIS) is an unsupervised framework that leverages StyleGAN's latent space to perform precise, fine-grained facial feature transfer and retrieval on real images. By discovering that feature-controlling style channels are image-specific ("submemberships"), RIS eliminates the need for tedious per-image hyperparameter tuning and enables the first unsupervised, feature-level image retrieval system (e.g., finding faces with similar eyes).

What Problem Does This Paper Address?

While deep generative models like StyleGAN can synthesize incredibly realistic faces, controlling specific local features (like the shape of a nose or style of hair) remains a challenge. Existing face-editing methods typically rely on supervised data, such as manual attribute labels or segmentation masks, which are expensive to produce and lack fine-grained detail.

Recent unsupervised methods like Editing in Style (EIS) attempted to solve this by identifying feature-specific channels in StyleGAN's latent space. However, EIS suffers from major limitations: it requires computing a semantic catalog over a large batch of images, demands manual hyperparameter tuning for every single image, struggles with complex features like hair and pose, and only works well on synthetic images. RIS addresses these scalability and precision gaps, extending local feature manipulation to real-world images and introducing fine-grained facial feature retrieval.

Figure 1: We propose an unsupervised method to transfer local facial appearance from real reference images to a real source image, e.g., (a) eyes, nose, and mouth. Compared to the state-of-the-art [10], our method enables photo-realistic transfers for (b) hair and (c) pose, and can be naturally extended for (d) semantic retrieval according to different facial features.

Methods

RIS builds upon the architecture of StyleGAN2 and improves local feature manipulation through several key innovations:

  1. Submembership Discovery: The authors discovered that the style channels driving a specific facial feature are not universal across a dataset; they vary from image to image. Instead of averaging channel contributions over a large dataset ($N$), RIS computes contribution scores ($M_{k,c}$) pairwise—using only the specific source and reference images ($N=2$).
  2. Softmax Channel Selection: To avoid the greedy optimization and manual thresholding used in prior work, RIS applies a Softmax function over the contribution scores of all facial features. This generates a clean, soft interpolation mask ($q_k$) for each channel automatically.
  3. Latent Direction Traversal: Rather than simple style interpolation, RIS formulates feature transfer as traversing along a latent direction vector. By introducing a scalar step size ($\alpha$), the framework allows for extrapolation (e.g., exaggerating or minimizing a feature smoothly).
  4. Feature-Specific Retrieval: By multiplying an image's style coefficients ($\sigma$) with the feature mask ($q_k$), RIS creates a distinct embedding ($v_k$) for individual features (like eyes or hair). Retrieval is performed by calculating the cosine distance between these feature embeddings across a dataset.
  5. Real Image Inversion: To operate on real images, RIS integrates the pre-trained pSp (pixel2style2pixel) encoder to map real photographs into StyleGAN’s latent space.

Datasets

The primary dataset used for evaluation on real images is CelebA-HQ, containing 30,000 high-resolution facial images. The underlying StyleGAN2 model was trained on the FFHQ dataset.

Key Insights and Ideas

  • The "Submembership" Reality: The most profound insight is that a facial feature like "hair" does not map to a static set of channels across all human faces in a GAN's latent space. Different hairstyles or hair colors activate unique subsets of channels. Dataset-wide averaging blurs these distinctions, which is why prior methods failed at complex transfers.
  • Pairwise is Better Than Batchwise: Computing channel importance purely between the source and reference images preserves fine-grained details that are otherwise lost when generalizing across a large batch.
  • Disentanglement Enables Retrieval: Once you can cleanly isolate the channels responsible for a specific feature on a single image, those masked style coefficients naturally function as an unsupervised, highly granular embedding for image retrieval.

Key Results

Facial Feature Transfer

RIS achieves highly localized feature transfers on real images without altering the rest of the face. For example, it successfully transfers hair and pose—tasks where the baseline EIS failed due to feature entanglement. RIS also maintains consistent skin tones and can even handle out-of-distribution generations, such as faithfully transferring a beard from a male reference image onto a female source image.

Quantitatively, RIS achieves an image fidelity score ($\text{FID}_\infty$) of 3.73 on CelebA-HQ, which is comparable to EIS (3.47) and close to the baseline StyleGAN2 (2.44), with the slight increase reflecting its unique ability to generate diverse, out-of-distribution feature combinations.

Fine-Grained Retrieval

Evaluated using a pre-trained attribute classifier, RIS outperforms EIS across all tested facial categories on the Attribute Matching Score (AMS):

  • Eyes: 96.3% (RIS) vs. 95.4% (EIS)
  • Mouth: 81.1% (RIS) vs. 75.8% (EIS)
  • Hair: 97.5% (RIS) vs. 97.1% (EIS)

Furthermore, RIS demonstrates significantly lower identity overlap ($\text{TRSI-IoU}$) when querying different features of the same face, proving that its retrieval mechanism is truly driven by the target feature rather than the overall identity of the person.

Limitations and Future Work

  • Dataset Biases: The evaluation relies heavily on CelebA-HQ, which lacks diversity in certain attributes (e.g., head coverings, specific hair textures), limiting the thorough evaluation of RIS on highly diverse populations.
  • Inversion Dependency: Because RIS operates on real images via GAN inversion, its performance is inherently bounded by the quality and reconstruction fidelity of the inversion encoder (pSp).
  • Societal Risks: As with all high-fidelity facial editing techniques, the authors note the potential risk of misuse for creating misleading altered imagery (deepfakes).

Summary updated

Abstract

We present Retrieve in Style (RIS), an unsupervised framework for facial feature transfer and retrieval on real images. Recent work shows capabilities of transferring local facial features by capitalizing on the disentanglement property of the StyleGAN latent space. RIS improves existing art on the following: 1) Introducing more effective feature disentanglement to allow for challenging transfers (ie, hair, pose) that were not shown possible in SoTA methods. 2) Eliminating the need for per-image hyperparameter tuning, and for computing a catalog over a large batch of images. 3) Enabling fine-grained face retrieval using disentangled facial features (eg, eyes). To our best knowledge, this is the first work to retrieve face images at this fine level. 4) Demonstrating robust, natural editing on real images. Our qualitative and quantitative analyses show RIS achieves both high-fidelity feature transfers and accurate fine-grained retrievals on real images. We also discuss the responsible applications of RIS.
BibTeX
@inproceedings{chong2021retrieve,
  author = {Min Jin Chong and Wen-Sheng Chu and Abhishek Kumar and David Forsyth},
  title = {{Retrieve in Style: Unsupervised Facial Feature Transfer and Retrieval}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00386},
}
Figure