Exploiting the Hidden Tasks of GANs: Making Implicit Subproblems Explicit

Romann M. Weber

Abstract

We present an alternative perspective on the training of generative adversarial networks (GANs), showing that the training step for a GAN generator decomposes into two implicit subproblems. In the first, the discriminator provides new target data to the generator in the form of "inverse examples" produced by approximately inverting classifier labels. In the second, these examples are used as targets to update the generator via least-squares regression, regardless of the main loss specified to train the network. We experimentally validate our main theoretical result and demonstrate significant improvements over standard GAN training made possible by making these subproblems explicit.

Introduction

Generative Adversarial Networks (GANs) have become a cornerstone of modern generative modeling, capable of producing high-fidelity images that are often indistinguishable from real data. However, despite their success, GANs remain notoriously difficult to train, often suffering from instabilities like mode collapse or vanishing gradients. The traditional "minimax" framework, often described through the analogy of a forger competing against an art expert, provides an intuitive high-level view but lacks the granularity needed to understand the precise mechanics of the generator's update.

This paper addresses this gap by demonstrating that the generator's update step is not a monolithic adversarial operation. Instead, it mathematically decomposes into two distinct, hidden subproblems: target generation and regression. By making these implicit tasks explicit, we gain new "control knobs" to stabilize and improve GAN training.

The Hidden Subproblems of GAN Training

The central theoretical contribution of this work is the proof that a standard GAN generator update is equivalent to a two-step sequential process.

Comparison of the standard GAN update (left) versus the proposed subproblem decomposition (right).

1. Target Generation (Network Inversion)

In the first step, the discriminator acts as a guide rather than just an adversary. It identifies how a currently generated image $x$ should be modified to appear more "real" according to its current classification boundary. This process is essentially a form of network inversion, creating an "inverse example" $x'$ in the data domain. This $x'$ serves as a constructive target for the generator.

2. Regression

Once the target $x'$ is established, the generator is updated to map its latent noise input $z$ to this new target. Mathematically, this update is equivalent to performing least-squares regression (squared $\ell_2$ loss), regardless of the original loss function (e.g., binary cross-entropy or Wasserstein loss) used to define the GAN objective.

This decomposition reveals that the generator's task is fundamentally a regression problem on a moving target provided by the discriminator. The generator's ability to succeed is constrained by its inductive bias ($K_\theta$), which limits the types of transformations it can represent.

Experimental Results

By decoupling these two steps, we can optimize them independently. For instance, we can use different learning rates for the inversion step ($\lambda_1$) and the regression step ($\lambda_2$), or even use different loss functions (like $\ell_1$ or SSIM) for the regression task.

We validated this approach on the CelebA dataset at $64 \times 64$ resolution. Using a baseline DCGAN, we observed an initial Fréchet Inception Distance (FID) of 72.42. By explicitly tuning the subproblem learning rates—specifically setting $\lambda_1 = 0.25$—we were able to significantly improve the image quality, reducing the FID to 63.15.

Qualitative comparison of samples from a baseline DCGAN (left) and the proposed subproblem formulation (right) after 10 epochs.

Ablation studies further revealed that while increasing the number of inversion steps ($N_1$) can lead to better targets, increasing the number of regression steps ($N_2$) per update often degrades performance. This suggests that the "moving target" nature of GAN training requires a delicate balance between finding good targets and over-fitting to any single set of them.

Conclusion

Reframing GAN training as a combination of inversion and regression provides a more controllable and interpretable framework. This perspective shifts the focus from pure competition to a cooperative mechanism where the discriminator provides actionable feedback. Future work may explore replacing the discriminator-based target generation with other methods, potentially leading to even more stable generative models that do not rely on adversarial training at all.

Summary updated

Abstract

We present an alternative perspective on the training of generative adversarial networks (GANs), showing that the training step for a GAN generator decomposes into two implicit subproblems. In the first, the discriminator provides new target data to the generator in the form of "inverse examples" produced by approximately inverting classifier labels. In the second, these examples are used as targets to update the generator via least-squares regression, regardless of the main loss specified to train the network. We experimentally validate our main theoretical result and demonstrate significant improvements over standard GAN training made possible by making these subproblems explicit.
BibTeX
@misc{weber2021exploiting,
  author = {Romann M. Weber},
  title = {{Exploiting the Hidden Tasks of GANs: Making Implicit Subproblems Explicit}},
  year = {2021},
  eprint = {2101.11863},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.11863},
}

Loading related content...

Figure