Generalized One-shot Domain Adaptation of Generative Adversarial Networks

Zicheng Zhang, Yinglu Liu, Congying Han, Tiande Guo, Ting Yao, Tao Mei

TL;DR

This paper introduces "generalized one-shot GAN adaptation," a new task that goes beyond style transfer to also incorporate specific entities (like hats or masks) from a single reference image. Their novel framework effectively disentangles style and entity transfer, achieving impressive results in adapting pre-trained GANs to new domains with minimal data.

What Problem Does This Paper Address?

Generative Adversarial Networks (GANs) are powerful tools for creating realistic images, but training them from scratch requires massive datasets. GAN adaptation aims to leverage pre-trained GANs to generate images for a new domain with limited data. While previous work has focused on adapting GANs to new styles using a few examples (few-shot) or even just one (one-shot), these methods often neglect the entities present in the target domain. For instance, adapting a GAN to generate artistic portraits might fail to capture the hat or glasses that are crucial to the target style. This paper addresses this gap by proposing a "generalized one-shot GAN adaptation" task, which explicitly handles both style and entity transfer from a single reference image and its corresponding binary mask. This is crucial for more complex and creative applications where specific objects are integral to the desired output.

Methods

The authors propose a framework built upon StyleGAN that disentangles style and entity transfer. Here's a breakdown of their key techniques:

  • Disentangled Generator Architecture: The target generator ($G_t$) is composed of two parts: a modified generator ($\hat{G}_t$) inherited from the pre-trained source GAN ($G_s$) for stylizing clear faces, and a new auxiliary network (aux) trained from scratch to handle entities. This separation allows $\hat{G}_t$ to focus on global style while aux generates entity-specific features.
  • Style Fixation: To quickly capture the exemplary style of the reference image, they employ a "style fixation" technique. This involves using a pre-trained GAN inversion encoder to obtain the latent code of the reference image and then replacing the style components of other latent codes with those from the reference. This provides a strong initial style transfer.
  • Internal Distribution Learning: Instead of relying on traditional GAN losses, which can be unstable with limited data, they use Sliced Wasserstein Distance (SWD) to minimize the divergence between internal patch distributions of synthesized images and the reference. This is applied separately for style ($L_{style}$) and entity ($L_{ent}$) losses, allowing for more efficient and stable learning.
  • Variational Laplacian Regularization ($L_{VLapR}$): To ensure cross-domain correspondence and prevent content distortion, they introduce a novel regularization term. This term encourages the adapted generator ($G_t$) to maintain a smooth semantic difference with the source generator ($G_s$) across the latent space, preserving the geometric structure of the source manifold.

Datasets

The paper primarily uses pre-trained StyleGAN models on large-scale datasets for their source domains. For evaluation and demonstration, they utilize:

  • FFHQ: A high-quality dataset of human faces, used as a primary source domain.
  • Target Domains: They demonstrate adaptation to various target domains, including:
  • Sketch, Disney, Arcane: Portraits without specific entities.
  • Hat, Zelda decorations, Mask: Portraits with entities, often sourced from AAHQ or manually annotated.
  • AFHQ (dog, cat): Animal datasets.
  • LSUN (church): Architectural dataset.

They do not introduce new datasets but rather adapt existing pre-trained models to new domains, often requiring manual annotation or semantic segmentation for entity masks.

Key Insights and Ideas

  • Decoupling Style and Entity is Key: The most significant insight is that treating style and entity transfer separately within the GAN architecture leads to much better results, especially in the challenging one-shot scenario.
  • SWD for Stable Adaptation: Using Sliced Wasserstein Distance for internal distribution learning proves to be a more stable and efficient alternative to adversarial losses for GAN adaptation with limited data.
  • Manifold Regularization for Content Preservation: The variational Laplacian regularization is crucial for maintaining the structural integrity of the generated images, preventing the content from being distorted by aggressive style transfer.
  • Generalized One-Shot Adaptation is a More Realistic Task: By incorporating entities, the proposed task better reflects real-world creative applications where specific objects are integral to the desired output.

Key Results

  • Qualitative Success: The paper demonstrates impressive qualitative results, generating realistic images that accurately capture both the style and entities of the reference images. Their method produces sharper and more detailed images compared to other state-of-the-art methods.
  • Quantitative Performance:
    • They achieve the best scores in terms of Normalized Mean Error (NME) for face shape preservation and Identity Similarity (IS) for content preservation, indicating strong cross-domain correspondence.
  • User studies show their method is significantly more popular, garnering about 60% of user votes.
  • Efficiency: The method is relatively efficient, with training times around 12 minutes for cases with entities and 3 minutes without, on a single NVIDIA RTX 3090.
  • Versatility: The framework is shown to be applicable to various domains beyond faces, including animals and architecture.
  • Image Manipulation Capabilities: The adapted models can perform semantic editing (smile, age, rotation) and even entity removal, offering powerful tools for artistic creation.

Limitations and Future Work

The authors acknowledge a few limitations:

  • Precise Entity Positioning: The framework may struggle to precisely control the position of entities, especially when the pose changes significantly.
  • Entity Complexity: Very complex entities might be difficult to learn effectively using their patch-based approach.

For future work, they suggest exploring these limitations further and believe their work on manifold regularization is a significant contribution.

Summary updated

Abstract

The adaptation of a Generative Adversarial Network (GAN) aims to transfer a pre-trained GAN to a target domain with limited training data. In this paper, we focus on the one-shot case, which is more challenging and rarely explored in previous works. We consider that the adaptation from a source domain to a target domain can be decoupled into two parts: the transfer of global style like texture and color, and the emergence of new entities that do not belong to the source domain. While previous works mainly focus on style transfer, we propose a novel and concise framework to address the \textit{generalized one-shot adaptation} task for both style and entity transfer, in which a reference image and its binary entity mask are provided. Our core idea is to constrain the gap between the internal distributions of the reference and syntheses by sliced Wasserstein distance. To better achieve it, style fixation is used at first to roughly obtain the exemplary style, and an auxiliary network is introduced to the generator to disentangle entity and style transfer. Besides, to realize cross-domain correspondence, we propose the variational Laplacian regularization to constrain the smoothness of the adapted generator. Both quantitative and qualitative experiments demonstrate the effectiveness of our method in various scenarios. Code is available at \url{https://github.com/zhangzc21/Generalized-One-shot-GAN-adaptation}.
BibTeX
@inproceedings{zhang2022generalized,
  author = {Zicheng Zhang and Yinglu Liu and Congying Han and Tiande Guo and Ting Yao and Tao Mei},
  title = {{Generalized One-shot Domain Adaptation of Generative Adversarial Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
}
Figure