The Hidden Tasks of Generative Adversarial Networks: An Alternative Perspective on GAN Training

Romann M. Weber

Abstract

We present an alternative perspective on the training of generative adversarial networks (GANs), showing that the training step for a GAN generator decomposes into two implicit subproblems. In the first, the discriminator provides new target data to the generator in the form of "inverse examples" produced by approximately inverting classifier labels. In the second, these examples are used as targets to update the generator via least-squares regression, regardless of the main loss specified to train the network. We experimentally validate our main theoretical result and demonstrate significant improvements over standard GAN training made possible by making these subproblems explicit.

Introduction

Generative Adversarial Networks (GANs) are renowned for producing high-fidelity synthetic images, yet they are notoriously difficult to train. Traditionally, GAN training is conceptualized as a "minimax" game—a zero-sum competition between a generator and a discriminator. While this adversarial metaphor is intuitive, it often obscures the underlying mathematical mechanics of how the training signal actually guides the generator's output toward the real data distribution. This paper seeks to demystify the generator update by decomposing it into two well-understood optimization subproblems, revealing that GANs are, in a sense, secretly cooperative.

Decomposing the Generator Update

The core contribution of this work is a mathematical proof showing that a single standard GAN generator update is identical to a two-stage process. Rather than viewing the gradient as a monolithic adversarial signal, we can interpret the update as follows:

  1. Target Generation (Network Inversion): The discriminator is essentially "inverted." It takes the generator's current output and calculates a slightly modified version—an inverse example ($x'$)—that would be more likely to be classified as "real."
  2. Regression: The generator then treats this inverse example as a ground-truth target. It performs a least-squares regression to update its weights so that, given the same noise input $z$, it produces an output closer to that new target.
Comparison between the standard GAN update and the proposed subproblem interpretation.

This decomposition is valid for any standard GAN loss function. By separating these steps, we can introduce explicit control over the number of inversion steps ($N_1$), the number of regression steps ($N_2$), and the specific loss functions used for each (e.g., switching from $\ell_2$ to $\ell_1$ regression).

Experimental Validation

We validated this theory using the CelebA face dataset and a simplified toy linear model. The results demonstrate that "unlinking" these implicit subproblems allows for significant improvements in image quality, as measured by the Fréchet Inception Distance (FID).

A key insight is the sensitivity of training to the "inversion learning rate" ($\lambda_1$). In standard GAN training, this rate is implicitly fixed. However, by treating it as an explicit hyperparameter, we can find an optimal balance that significantly outperforms the baseline.

Impact of the inversion learning rate (λ1) on FID scores for CelebA.

As shown in the figure above, a standard DCGAN baseline achieved an FID of 72.42 (represented by the dashed blue line). By adjusting the inversion learning rate to approximately 0.25 and making the subproblems explicit, we achieved a significantly improved FID of 63.15. Furthermore, experiments showed that while $\ell_2$ regression is the implicit default in GANs, explicitly choosing $\ell_2$ consistently outperformed $\ell_1$ in this framework. Interestingly, adding more inversion steps ($N_1 > 1$) tended to stabilize or improve results, whereas adding more regression steps ($N_2 > 1$) often hurt performance, suggesting that finding a "good target" is more critical than overfitting to a single intermediate one.

Discussion and Limitations

This alternative perspective suggests that the "adversarial" label may be a distraction; the discriminator's primary role is to act as a teacher providing better targets. While this method provides new "knobs" for stabilization and quality, it does increase computational overhead per iteration. Future work may explore non-adversarial ways to generate these targets, potentially bypassing the instabilities of the minimax game entirely while maintaining the generator's core regression task.

Summary updated

Abstract

We present an alternative perspective on the training of generative adversarial networks (GANs), showing that the training step for a GAN generator decomposes into two implicit subproblems. In the first, the discriminator provides new target data to the generator in the form of "inverse examples" produced by approximately inverting classifier labels. In the second, these examples are used as targets to update the generator via least-squares regression, regardless of the main loss specified to train the network. We experimentally validate our main theoretical result and demonstrate significant improvements over standard GAN training made possible by making these subproblems explicit.
BibTeX
@misc{weber2021hidden,
  author = {Romann M. Weber},
  title = {{The Hidden Tasks of Generative Adversarial Networks: An Alternative Perspective on GAN Training}},
  year = {2021},
  eprint = {2101.11863},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.11863},
}
Figure