TL;DR
This paper introduces "generalized one-shot GAN adaptation," a new task that goes beyond style transfer to also incorporate specific entities (like hats or masks) from a single reference image. Their novel framework effectively disentangles style and entity transfer, achieving impressive results in adapting pre-trained GANs to new domains with minimal data.
What Problem Does This Paper Address?
Generative Adversarial Networks (GANs) are powerful tools for creating realistic images, but training them from scratch requires massive datasets. GAN adaptation aims to leverage pre-trained GANs to generate images for a new domain with limited data. While previous work has focused on adapting GANs to new styles using a few examples (few-shot) or even just one (one-shot), these methods often neglect the entities present in the target domain. For instance, adapting a GAN to generate artistic portraits might fail to capture the hat or glasses that are crucial to the target style. This paper addresses this gap by proposing a "generalized one-shot GAN adaptation" task, which explicitly handles both style and entity transfer from a single reference image and its corresponding binary mask. This is crucial for more complex and creative applications where specific objects are integral to the desired output.
Methods
The authors propose a framework built upon StyleGAN that disentangles style and entity transfer. Here's a breakdown of their key techniques:
- Disentangled Generator Architecture: The target generator ($G_t$) is composed of two parts: a modified generator ($\hat{G}_t$) inherited from the pre-trained source GAN ($G_s$) for stylizing clear faces, and a new auxiliary network (aux) trained from scratch to handle entities. This separation allows $\hat{G}_t$ to focus on global style while aux generates entity-specific features.
- Style Fixation: To quickly capture the exemplary style of the reference image, they employ a "style fixation" technique. This involves using a pre-trained GAN inversion encoder to obtain the latent code of the reference image and then replacing the style components of other latent codes with those from the reference. This provides a strong initial style transfer.
- Internal Distribution Learning: Instead of relying on traditional GAN losses, which can be unstable with limited data, they use Sliced Wasserstein Distance (SWD) to minimize the divergence between internal patch distributions of synthesized images and the reference. This is applied separately for style ($L_{style}$) and entity ($L_{ent}$) losses, allowing for more efficient and stable learning.
- Variational Laplacian Regularization ($L_{VLapR}$): To ensure cross-domain correspondence and prevent content distortion, they introduce a novel regularization term. This term encourages the adapted generator ($G_t$) to maintain a smooth semantic difference with the source generator ($G_s$) across the latent space, preserving the geometric structure of the source manifold.
Datasets
The paper primarily uses pre-trained StyleGAN models on large-scale datasets for their source domains. For evaluation and demonstration, they utilize:
- FFHQ: A high-quality dataset of human faces, used as a primary source domain.
- Target Domains: They demonstrate adaptation to various target domains, including:
- Sketch, Disney, Arcane: Portraits without specific entities.
- Hat, Zelda decorations, Mask: Portraits with entities, often sourced from AAHQ or manually annotated.
- AFHQ (dog, cat): Animal datasets.
- LSUN (church): Architectural dataset.
They do not introduce new datasets but rather adapt existing pre-trained models to new domains, often requiring manual annotation or semantic segmentation for entity masks.
Key Insights and Ideas
- Decoupling Style and Entity is Key: The most significant insight is that treating style and entity transfer separately within the GAN architecture leads to much better results, especially in the challenging one-shot scenario.
- SWD for Stable Adaptation: Using Sliced Wasserstein Distance for internal distribution learning proves to be a more stable and efficient alternative to adversarial losses for GAN adaptation with limited data.
- Manifold Regularization for Content Preservation: The variational Laplacian regularization is crucial for maintaining the structural integrity of the generated images, preventing the content from being distorted by aggressive style transfer.
- Generalized One-Shot Adaptation is a More Realistic Task: By incorporating entities, the proposed task better reflects real-world creative applications where specific objects are integral to the desired output.
Key Results
- Qualitative Success: The paper demonstrates impressive qualitative results, generating realistic images that accurately capture both the style and entities of the reference images. Their method produces sharper and more detailed images compared to other state-of-the-art methods.
- Quantitative Performance:
- They achieve the best scores in terms of Normalized Mean Error (NME) for face shape preservation and Identity Similarity (IS) for content preservation, indicating strong cross-domain correspondence.
- User studies show their method is significantly more popular, garnering about 60% of user votes.
- Efficiency: The method is relatively efficient, with training times around 12 minutes for cases with entities and 3 minutes without, on a single NVIDIA RTX 3090.
- Versatility: The framework is shown to be applicable to various domains beyond faces, including animals and architecture.
- Image Manipulation Capabilities: The adapted models can perform semantic editing (smile, age, rotation) and even entity removal, offering powerful tools for artistic creation.
Limitations and Future Work
The authors acknowledge a few limitations:
- Precise Entity Positioning: The framework may struggle to precisely control the position of entities, especially when the pose changes significantly.
- Entity Complexity: Very complex entities might be difficult to learn effectively using their patch-based approach.
For future work, they suggest exploring these limitations further and believe their work on manifold regularization is a significant contribution.
Summary updated