CoachGAN

Mike BrodieWACV 2020

TL;DR

CoachGAN is a novel post-training algorithm that enhances the realism of images generated by GANs without retraining the generator or discriminator. By cleverly using the discriminator's feedback at inference time, CoachGAN guides the generator's input to produce more convincing outputs, leading to improved image quality metrics.

What Problem Does This Paper Address?

Generative Adversarial Networks (GANs) are powerful tools for creating realistic synthetic data, but their outputs can sometimes suffer from artifacts, blurriness, or a lack of fine detail. Traditionally, once a GAN is trained, the generator is used in isolation to produce images. While this is efficient, it doesn't allow for real-time refinement of the generated images based on the discriminator's learned understanding of "realness." The challenge is to improve the quality of GAN outputs after training, without the computationally expensive process of retraining or fine-tuning the entire GAN.

Methods

The core of CoachGAN is an inference-time optimization process. Instead of altering the generator's weights (which would require retraining), CoachGAN modifies the input to the generator. Here's how it works:

  1. Leveraging the Discriminator: CoachGAN uses a pre-trained, frozen discriminator model. This discriminator acts as an "advisor," providing feedback on the quality of the generated image.
  2. Gradient Descent on Input: CoachGAN takes the output of the generator, $G(z)$, and passes it through the discriminator, $D$. It then computes a loss function, $L_G$, which is typically the same loss used during the generator's training (e.g., a Wasserstein GAN loss). Crucially, CoachGAN backpropagates this loss through both the discriminator and the generator, but only updates the input latent vector, z.
  3. Iterative Refinement: This process is repeated for a specified number of iterations ($\kappa$) with a chosen learning rate ($\eta$). Each iteration nudges the latent vector $z$ in a direction that the discriminator deems more realistic, gradually improving the generated image.
  4. Architecture Agnostic: A key advantage is that CoachGAN is architecture-agnostic. It works with any GAN architecture that has differentiable components and loss functions, requiring no modifications to the original generator or discriminator.

Datasets

The paper evaluates CoachGAN on several standard datasets:

  • CelebA: A dataset of celebrity faces.
  • LSUN Bedroom: A dataset of interior bedroom scenes.
  • CIFAR-10: A dataset of small, diverse images across 10 classes.
  • ImageNet: Used for evaluating metrics like Inception Score (IS) and Fréchet Inception Distance (FID) due to its large scale and class diversity.
  • FFHQ: A high-quality dataset of human faces, used for evaluating models like PGGAN and StyleGAN.

Key Insights and Ideas

  • Inference-time Refinement is Powerful: The most significant insight is that valuable improvements can be made to GAN outputs after training, simply by optimizing the input latent vector using discriminator feedback. This bypasses the need for costly retraining.
  • Input Optimization vs. Output Optimization: The authors found that optimizing the input latent vector ($z$) yields much better results than directly optimizing the output image. Direct output optimization (as shown in Figure 6) tends to introduce adversarial noise rather than genuine realism.
  • CoachGAN Doesn't "Memorize" Training Data: Through nearest neighbor searches in feature space, the paper demonstrates that CoachGAN doesn't simply push generated images towards existing training samples. Instead, it guides the generator to produce outputs that are more aligned with the discriminator's learned distribution of real data.
  • Adaptability is Key: CoachGAN's modular design and architecture-agnostic nature make it highly versatile, easily integrating with various GAN architectures (DCGAN, PGGAN, StyleGAN, BigGAN) and tasks.

Key Results

CoachGAN consistently improved the quality of generated images across various GAN architectures and datasets:

  • Qualitative Improvements: Figures and examples throughout the paper show striking visual improvements, such as clarifying blurry details, removing artifacts, enhancing textures, and generating more complete or realistic features (e.g., a clear set of glasses in Figure 5).
  • Quantitative Metrics:
    • Inception Score (IS): Improved IS scores were observed for CIFAR-10 and BigGAN, indicating better image quality and diversity. For example, CIFAR-10 IS increased from 4.79 to 5.00.
  • Fréchet Inception Distance (FID): CoachGAN generally produced lower (better) FID scores for CelebA, CIFAR-10, LSUN, PGGAN, StyleGAN, and BigGAN, signifying that the generated images are statistically more similar to real images. For CelebA, FID decreased from 17.95 to 16.43.
  • Runtime: While CoachGAN adds computational overhead, the runtime is still manageable. For complex models like BigGAN, it roughly triples inference time, but the total time per image remains under 0.2 seconds on a Tesla P100 GPU.

Limitations and Future Work

  • Limitations:
    • Not a Guarantee: CoachGAN does not provably guarantee improved output realism, similar to other heuristic methods.
  • Magnifies Noise: In cases of poorly trained models or blurry training images, CoachGAN can sometimes magnify existing noise.
  • D Favoritism: If the initial output is already realistic, CoachGAN might adjust image characteristics to match the discriminator's biases rather than introducing entirely new realism.
  • Future Work:
    • Task-Specific Constraints: The authors suggest incorporating task-specific constraints to preserve desired features (e.g., gender, hair color, pose) when refining images, which would be useful for applications like video generation.
  • User Control: CoachGAN's ability to refine outputs and allow users to select precise results from SGD steps offers potential for enhanced user control in image editing and synthesis.

Summary updated

BibTeX
@inproceedings{brodie2020coachgan,
  author = {Mike Brodie},
  title = {{CoachGAN}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure