Progressive Few-Shot Adaptation of Generative Model with Align-Free Spatial Correlation

Jongbo Moon, Hyunjun Kim, Jae-Pil HeoAAAI 2023 · OpenReview

TL;DR

This paper introduces a novel method for adapting generative models to new domains with very limited data (few-shot learning). It tackles the problem of visual artifacts caused by misaligned spatial structures between source and target domains by proposing an "align-free" spatial correlation mechanism that leverages the progressive nature of model adaptation.

What Problem Does This Paper Address?

Generative Adversarial Networks (GANs) are powerful for creating realistic images, but training them from scratch requires vast amounts of data. When we want to adapt a GAN trained on a large dataset (source domain) to a new domain with very little data (target domain, known as few-shot domain adaptation), existing methods often struggle. A key challenge is that the spatial relationships between objects (like eyes in faces) might differ significantly between the source and target domains. Previous methods that tried to match spatial correlations between image patches often assumed these structures were aligned, leading to visual artifacts when they weren't. This paper aims to overcome these limitations by developing a few-shot adaptation method that doesn't rely on strict spatial alignment between source and target domains.

Methods

The core of the proposed method, "Progressive Few-Shot Adaptation with Align-Free Spatial Correlation," lies in two main ideas:

  1. Progressive Adaptation and Align-Free Spatial Correlation: Instead of directly comparing the source GAN and the final target GAN, the authors consider the adaptation process as a series of small steps. They observe that GANs at nearby training iterations have a smaller domain gap. This allows them to identify semantically similar image regions (e.g., eyes) between these intermediate models without assuming they are in the exact same pixel location. They use optical flow to find these corresponding regions and then enforce a consistency in their spatial correlations. This "align-free" approach avoids the pitfalls of rigid spatial matching.
  2. Importance-Based Patch Selection: To make the spatial correlation matching more efficient and robust, they introduce an importance sampling strategy. This strategy prioritizes image regions with significant intensity changes (edges) and excludes regions likely affected by optical flow estimation errors. This reduces computational complexity and focuses on more meaningful image content.

The overall objective function combines this new progressive spatial correlation loss with a standard adversarial loss.

Datasets

The authors used the following datasets:

  • Source Domain: FFHQ (a large-scale dataset of human faces).
  • Target Domains: Amedeo, Sketches, FFHQ-babies, FFHQ-sunglasses, and Animation Faces.

They adapted pre-trained StyleGAN2 models to these target domains using either 5-shot or 10-shot examples.

Key Insights and Ideas

  • Leveraging the Adaptation Process: The most significant insight is that the process of adaptation itself provides a valuable signal. By looking at models at intermediate stages, we can find semantically similar regions that are not spatially aligned, which is crucial for domains with large visual differences.
  • "Align-Free" is Key: The assumption of strict spatial alignment is a major bottleneck for many domain adaptation tasks. This paper demonstrates that by relaxing this assumption and focusing on semantic coherence across progressive steps, better results can be achieved.
  • Edges as Important Features: Focusing on image edges for spatial correlation matching is a smart heuristic. Edges often delineate important semantic structures, making them more reliable for cross-domain comparison than uniform textures or backgrounds.

Key Results

The proposed method achieved state-of-the-art performance in few-shot generative model adaptation.

  • Qualitative Results: The method generated more diverse and higher-quality images compared to baselines like IDC and RSSA, with significantly fewer visual artifacts and better preservation of identity. Figure 1 and Figure 4 visually demonstrate these improvements.
  • Quantitative Results:
    • Diversity (IS-mean): The method consistently achieved higher Inception Scores across various target domains, indicating more diverse generated images.
  • Fidelity (IC-LPIPS, FID): The method also showed strong performance in preserving the identity and style of the source domain (lower IC-LPIPS) and matching the target domain distribution (lower FID). For example, in Table 1, the proposed method achieved the highest IS-mean and competitive IC-LPIPS and FID scores across multiple datasets.
  • User Study: A user study confirmed that the proposed method was overwhelmingly preferred for preserving identity, diversity, style, and fidelity. Over 77% of participants found it best for identity and diversity, and over 64% for style and fidelity.

Limitations and Future Work

  • Limitations: While the method is effective, the paper acknowledges that the "align-free" approach still relies on identifying semantically coherent regions. If the domain gap is extremely large or the semantic content is vastly different, this might still be challenging. The importance sampling, while beneficial, is based on heuristics (edge detection, flow estimation errors) which might not be universally optimal.
  • Future Work: The paper suggests that further improvements could come from exploring more sophisticated methods for identifying semantically coherent regions. Additionally, investigating more adaptive importance sampling strategies or exploring other forms of progressive adaptation could be promising directions. The authors also mention that their method's effectiveness in preserving identity and diversity suggests potential for applications beyond image generation, such as style transfer or domain generalization.

Summary updated

Abstract

In few-shot generative model adaptation, the model for target domain is prone to the mode-collapse. Recent studies attempted to mitigate the problem by matching the relationship among samples generated from the same latent codes in source and target domains. The objective is further extended to image patch-level to transfer the spatial correlation within an instance. However, the patch-level approach assumes the consistency of spatial structure between source and target domains. For example, the positions of eyes in two domains are almost identical. Thus, it can bring visual artifacts if source and target domain images are not nicely aligned. In this paper, we propose a few-shot generative model adaptation method free from such assumption, based on a motivation that generative models are progressively adapting from the source domain to the target domain. Such progressive changes allow us to identify semantically coherent image regions between instances generated by models at a neighboring training iteration to consider the spatial correlation. We also propose an importance-based patch selection strategy to reduce the complexity of patch-level correlation matching. Our method shows the state-of-the-art few-shot domain adaptation performance in the qualitative and quantitative evaluations.
BibTeX
@inproceedings{moon2023progressive,
  author = {Jongbo Moon and Hyunjun Kim and Jae-Pil Heo},
  title = {{Progressive Few-Shot Adaptation of Generative Model with Align-Free Spatial Correlation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=yI7Puk5SOs},
}
Figure