Evolving GAN Formulations for Higher Quality Image Synthesis

Santiago Gonzalez, Mohak Kant, Risto MiikkulainenarXiv 2021 · arXiv

Abstract

Generative Adversarial Networks (GANs) are highly effective for image synthesis but suffer from training instabilities and mode collapse. Traditionally, researchers have addressed these issues by manually designing loss functions. This paper introduces TaylorGAN, an automated approach that discovers optimized loss functions through evolutionary metalearning. By parameterizing losses as third-degree Taylor polynomials and evolving them using the Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES), TaylorGAN identifies formulations that surpass hand-designed standards like Wasserstein loss. On the CMP Facade dataset, TaylorGAN achieves significant improvements in both structural similarity (SSIM) and perceptual distance, producing images with enhanced textural realism and stability.

Introduction

Generative Adversarial Networks (GANs) have revolutionized the field of image synthesis, yet they remain notoriously difficult to train. The adversarial competition between a generator and a discriminator often leads to "brittle" dynamics, where small changes in hyperparameters or architecture can result in training failure or mode collapse. To stabilize this process, the research community has historically relied on human intuition to derive new loss functions, such as Minimax, Non-Saturating, and Wasserstein losses.

However, manual derivation is slow and may not capture the optimal mathematical dynamics for specific datasets. TaylorGAN addresses this by automating the design of loss functions, treating the search for an optimal loss as a metalearning problem.

Methodology: Taylor-Parameterized Genetic Loss Optimization

The core of TaylorGAN is the TaylorGLO (Taylor-parameterized Genetic Loss Optimization) technique. This method moves away from fixed logarithmic or linear formulas, instead representing loss functions as flexible third-degree multivariate Taylor polynomials.

The Evolutionary Search Process

The system evolves three distinct components of the GAN loss structure simultaneously:

  1. The discriminator’s loss for real samples.
  2. The discriminator’s loss for synthetic (fake) samples.
  3. The generator’s loss for synthetic samples.

To navigate this 12-parameter search space, the researchers employed the Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES). This derivative-free genetic algorithm iteratively samples candidate loss functions, evaluates them through partial model training, and updates its internal mean vector and covariance matrix to converge on the most effective parameters.

The TaylorGLO optimization pipeline using CMA-ES to evolve loss function parameters.

Composite Objectives

To ensure the evolved losses produce high-quality images, the evolution is guided by a "Composite Objective"—a weighted sum of the Structural Similarity Index (SSIM) and Perceptual Distance (based on VGG-16 features). This multiobjective approach prevents the evolutionary process from over-optimizing a single metric at the expense of visual quality.

Standard notation and symbols used in the GAN framework.

Experimental Setup

The approach was rigorously evaluated on the CMP Facade dataset, a benchmark for conditional image-to-image translation. The task involves generating photorealistic building images from semantic segmentation maps. The dataset consists of 606 perspective-corrected images, split into training (80%), validation (10%), and testing (10%) sets. This relatively small dataset size also served as a test for the robustness and regularization capabilities of the evolved loss functions.

Results and Discussion

TaylorGAN was compared against a state-of-the-art baseline: pix2pix-HD trained with Wasserstein loss and spectral normalization.

Quantitative Performance

The evolved loss functions demonstrated clear statistical superiority over the baseline after only 49 generations of evolution:

  • Structural Similarity Index (SSIM): Improved from a baseline average of 9.4359 to 11.6615.
  • Perceptual Distance: Reduced from 2129.5069 to 2040.2561, indicating higher visual fidelity.

Qualitative Analysis

Beyond the numbers, TaylorGAN produced visually distinct improvements. While baseline models often exhibited an unnatural "orange tint" or "plastic" textures, TaylorGAN images featured sharper details and more realistic material representations. There was also a more pronounced separation between foreground elements (like buildings) and the background (the sky), suggesting that the evolved losses better guide the discriminator in identifying fine-grained structural features.

Conclusion

TaylorGAN demonstrates that automated loss function discovery can significantly enhance GAN performance without requiring manual mathematical derivation. By treating loss design as an evolutionary optimization problem, the framework finds "mathematical sweet spots" that improve training stability and image quality. Future work will explore the generalizability of these evolved losses to unconditioned generation tasks and higher-resolution datasets.

Summary updated

Abstract

Generative Adversarial Networks (GANs) have extended deep learning to complex generation and translation tasks across different data modalities. However, GANs are notoriously difficult to train: Mode collapse and other instabilities in the training process often degrade the quality of the generated results, such as images. This paper presents a new technique called TaylorGAN for improving GANs by discovering customized loss functions for each of its two networks. The loss functions are parameterized as Taylor expansions and optimized through multiobjective evolution. On an image-to-image translation benchmark task, this approach qualitatively improves generated image quality and quantitatively improves two independent GAN performance metrics. It therefore forms a promising approach for applying GANs to more challenging tasks in the future.
BibTeX
@misc{gonzalez2021evolving,
  author = {Santiago Gonzalez and Mohak Kant and Risto Miikkulainen},
  title = {{Evolving GAN Formulations for Higher Quality Image Synthesis}},
  year = {2021},
  eprint = {2102.08578},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.08578},
}
Figure