Abstract
Recently, inversion methods have focused on additional high-rate information in the generator (e.g., weights or intermediate features) to refine inversion and editing results from embedded latent codes. Although these techniques gain reasonable improvement in reconstruction, they decrease editing capability, especially on complex images (e.g., containing occlusions, detailed backgrounds, and artifacts). A vital crux is refining inversion results, avoiding editing capability degradation. To tackle this problem, we introduce Domain-Specific Hybrid Refinement (DHR), which draws on the advantages and disadvantages of two mainstream refinement techniques to maintain editing ability with fidelity improvement. Specifically, we first propose Domain-Specific Segmentation to segment images into two parts: in-domain and out-of-domain parts. The refinement process aims to maintain the editability for in-domain areas and improve two domains' fidelity. We refine these two parts by weight modulation and feature modulation, which we call Hybrid Modulation Refinement. Our proposed method is compatible with all latent code embedding methods. Extension experiments demonstrate that our approach achieves state-of-the-art in real image inversion and editing.
Introduction
Generative Adversarial Networks (GANs) like StyleGAN excel at creating photorealistic faces, but editing real photos requires "GAN Inversion"—mapping a real image into the GAN’s latent space. This process faces a persistent trade-off: low-rate methods (latent code only) are highly editable but lose fine details, while high-rate refinement methods (tuning weights or features) reconstruct details perfectly but often "break" the latent space. When the generator is overfitted to a complex image, the smooth manifold of the latent space becomes jagged, causing semantic edits like "smiling" to produce artifacts.
Methodology
The proposed Domain-Specific Hybrid Refinement (DHR) framework employs a "divide and conquer" strategy to resolve this trade-off through two primary stages.
1. Domain-Specific Segmentation
DHR automatically segments an image into two distinct domains without requiring manual labels:
- In-domain: Areas the GAN understands well (e.g., face, skin, hair) that are easy to invert and edit.
- Out-of-domain: Complex areas the GAN struggles with, such as occlusions (microphones, hands), intricate backgrounds, or unique artifacts.
Segmentation is achieved via superpixel partitioning and a face parsing model. Areas with high reconstruction error after a coarse initial inversion are flagged as out-of-domain.
2. Hybrid Modulation Refinement
Once segmented, DHR applies a hybrid approach to optimize the generator:
- Weight Modulation for In-domain: The generator's weights are slightly tuned to match the face. Since the error is low, minimal weight deviation is required, preserving the GAN's ability to perform semantic edits like aging.
- Feature Modulation for Out-of-domain: For complex spatial details, the model injects information directly into intermediate feature maps. This "patches" the image without warping the generator's global weights.
Experiments and Results
The researchers evaluated DHR using the FFHQ dataset for training and the CelebA-HQ test set for benchmarking inversion and editing performance.
Key Performance Metrics
- Superior Fidelity: DHR achieved a Mean Squared Error (MSE) of 0.0035 on CelebA-HQ, significantly outperforming competitors like SAM (0.0117) and HFGI (0.0210).
- Identity Preservation: The method maintained a high identity similarity score of 0.8780, ensuring that edited versions of a person remain recognizable.
- Efficiency: DHR reached state-of-the-art performance in just 7.5 seconds, making it approximately 8x faster than Pivotal Tuning Inversion (PTI).
Conclusion
Domain-Specific Hybrid Refinement (DHR) addresses the fundamental conflict between reconstruction fidelity and editing capability in GAN inversion. By limiting weight tuning to in-domain areas and using feature modulation for complex details, DHR maintains a smooth latent manifold while achieving high-fidelity results. Future work may involve extending this "divide and conquer" approach to other generative architectures, such as Diffusion models.
Summary updated