TL;DR
Retrieve in Style (RIS) is an unsupervised framework that leverages StyleGAN's latent space to perform precise, fine-grained facial feature transfer and retrieval on real images. By discovering that feature-controlling style channels are image-specific ("submemberships"), RIS eliminates the need for tedious per-image hyperparameter tuning and enables the first unsupervised, feature-level image retrieval system (e.g., finding faces with similar eyes).
What Problem Does This Paper Address?
While deep generative models like StyleGAN can synthesize incredibly realistic faces, controlling specific local features (like the shape of a nose or style of hair) remains a challenge. Existing face-editing methods typically rely on supervised data, such as manual attribute labels or segmentation masks, which are expensive to produce and lack fine-grained detail.
Recent unsupervised methods like Editing in Style (EIS) attempted to solve this by identifying feature-specific channels in StyleGAN's latent space. However, EIS suffers from major limitations: it requires computing a semantic catalog over a large batch of images, demands manual hyperparameter tuning for every single image, struggles with complex features like hair and pose, and only works well on synthetic images. RIS addresses these scalability and precision gaps, extending local feature manipulation to real-world images and introducing fine-grained facial feature retrieval.
Methods
RIS builds upon the architecture of StyleGAN2 and improves local feature manipulation through several key innovations:
- Submembership Discovery: The authors discovered that the style channels driving a specific facial feature are not universal across a dataset; they vary from image to image. Instead of averaging channel contributions over a large dataset ($N$), RIS computes contribution scores ($M_{k,c}$) pairwise—using only the specific source and reference images ($N=2$).
- Softmax Channel Selection: To avoid the greedy optimization and manual thresholding used in prior work, RIS applies a Softmax function over the contribution scores of all facial features. This generates a clean, soft interpolation mask ($q_k$) for each channel automatically.
- Latent Direction Traversal: Rather than simple style interpolation, RIS formulates feature transfer as traversing along a latent direction vector. By introducing a scalar step size ($\alpha$), the framework allows for extrapolation (e.g., exaggerating or minimizing a feature smoothly).
- Feature-Specific Retrieval: By multiplying an image's style coefficients ($\sigma$) with the feature mask ($q_k$), RIS creates a distinct embedding ($v_k$) for individual features (like eyes or hair). Retrieval is performed by calculating the cosine distance between these feature embeddings across a dataset.
- Real Image Inversion: To operate on real images, RIS integrates the pre-trained pSp (pixel2style2pixel) encoder to map real photographs into StyleGAN’s latent space.
Datasets
The primary dataset used for evaluation on real images is CelebA-HQ, containing 30,000 high-resolution facial images. The underlying StyleGAN2 model was trained on the FFHQ dataset.
Key Insights and Ideas
- The "Submembership" Reality: The most profound insight is that a facial feature like "hair" does not map to a static set of channels across all human faces in a GAN's latent space. Different hairstyles or hair colors activate unique subsets of channels. Dataset-wide averaging blurs these distinctions, which is why prior methods failed at complex transfers.
- Pairwise is Better Than Batchwise: Computing channel importance purely between the source and reference images preserves fine-grained details that are otherwise lost when generalizing across a large batch.
- Disentanglement Enables Retrieval: Once you can cleanly isolate the channels responsible for a specific feature on a single image, those masked style coefficients naturally function as an unsupervised, highly granular embedding for image retrieval.
Key Results
Facial Feature Transfer
RIS achieves highly localized feature transfers on real images without altering the rest of the face. For example, it successfully transfers hair and pose—tasks where the baseline EIS failed due to feature entanglement. RIS also maintains consistent skin tones and can even handle out-of-distribution generations, such as faithfully transferring a beard from a male reference image onto a female source image.
Quantitatively, RIS achieves an image fidelity score ($\text{FID}_\infty$) of 3.73 on CelebA-HQ, which is comparable to EIS (3.47) and close to the baseline StyleGAN2 (2.44), with the slight increase reflecting its unique ability to generate diverse, out-of-distribution feature combinations.
Fine-Grained Retrieval
Evaluated using a pre-trained attribute classifier, RIS outperforms EIS across all tested facial categories on the Attribute Matching Score (AMS):
- Eyes: 96.3% (RIS) vs. 95.4% (EIS)
- Mouth: 81.1% (RIS) vs. 75.8% (EIS)
- Hair: 97.5% (RIS) vs. 97.1% (EIS)
Furthermore, RIS demonstrates significantly lower identity overlap ($\text{TRSI-IoU}$) when querying different features of the same face, proving that its retrieval mechanism is truly driven by the target feature rather than the overall identity of the person.
Limitations and Future Work
- Dataset Biases: The evaluation relies heavily on CelebA-HQ, which lacks diversity in certain attributes (e.g., head coverings, specific hair textures), limiting the thorough evaluation of RIS on highly diverse populations.
- Inversion Dependency: Because RIS operates on real images via GAN inversion, its performance is inherently bounded by the quality and reconstruction fidelity of the inversion encoder (pSp).
- Societal Risks: As with all high-fidelity facial editing techniques, the authors note the potential risk of misuse for creating misleading altered imagery (deepfakes).
Summary updated