Kernel-Based Training of Generative Networks

Kalliopi Basioti, George V. Moustakides, Emmanouil Z. PsarakisarXiv 2018 · OpenReview

Abstract

Generative adversarial networks (GANs) are designed with the help of min-max optimization problems that are solved with stochastic gradient-type algorithms which are known to be non-robust. In this work we revisit a non-adversarial method based on kernels which relies on a pure minimization problem and propose a simple stochastic gradient algorithm for the computation of its solution. Using simplified tools from Stochastic Approximation theory we demonstrate that batch versions of the algorithm or smoothing of the gradient do not improve convergence. These observations allow for the development of a training algorithm that enjoys reduced computational complexity and increased robustness while exhibiting similar synthesis characteristics as classical GANs.

Introduction

Generative Adversarial Networks (GANs) have become a cornerstone of modern generative modeling, yet they remain notoriously difficult to train. The core of the difficulty lies in the adversarial "min-max" optimization framework, where a Generator and a Discriminator compete. This setup often results in unstable training dynamics, slow convergence, and phenomena such as mode collapse. This paper proposes a shift away from the adversarial paradigm, reframing generative training as a pure minimization problem using kernel-based distance metrics.

Methodology

The proposed approach replaces the unstable adversarial game with the direct minimization of a distance function $J(\theta)$, representing the mismatch between the real data distribution and the model distribution. This formulation allows for the application of standard stochastic gradient descent (SGD) tools, providing a more stable foundation for training.

Stochastic Approximation and the "Delayed" Gradient

A key contribution of this work is the theoretical analysis of the training algorithm using Stochastic Approximation theory. The authors investigate whether common techniques like batch processing or gradient smoothing improve the convergence rate. Their findings suggest that for this specific minimization problem, these methods do not fundamentally accelerate convergence, although batching remains useful for hardware acceleration.

To optimize computational efficiency, the authors introduce a "Delayed" gradient trick. Calculating kernel distances typically requires two independent samples from the generator. By utilizing the current sample and the sample from the previous iteration, the algorithm maintains statistical independence while halving the number of required generator passes per step.

Figure 3: Comparison of different gradient computation schemes, demonstrating that smoothing or batching does not fundamentally alter the steady-state convergence behavior compared to the efficient delayed gradient approach.

Results and Discussion

The algorithm was evaluated on the MNIST and CelebA datasets. In terms of computational efficiency, the method demonstrated a significant advantage; on CelebA, it required five times less physical time to achieve results superior to a standard GAN. On MNIST, the kernel-based approach achieved higher Inception Scores, indicating better image variety and clarity.

Training Dynamics and Robustness

Unlike the oscillatory behavior often seen in GANs, the proposed method exhibits a predictable training curve. As shown in the figure below, the training process consists of a rapid "transient" phase followed by a "steady-state" phase. This steady-state represents a saturation phenomenon where residual gradient noise prevents the model from reaching a zero-distance fit, though the resulting synthesis quality remains high.

Figure 2: Training dynamics showing the rapid error reduction during the transient phase followed by a steady-state saturation phase.

Conclusion

By moving from an adversarial to a kernel-based minimization framework, generative training becomes more robust and computationally efficient. While this work focused on fully connected architectures, the results provide a strong theoretical and empirical basis for future extensions. Ongoing research aims to apply this non-adversarial framework to convolutional neural networks (CNNs) to further enhance high-resolution image synthesis.

Summary updated

Abstract

Generative adversarial networks (GANs) are designed with the help of min-max optimization problems that are solved with stochastic gradient-type algorithms which are known to be non-robust. In this work we revisit a non-adversarial method based on kernels which relies on a pure minimization problem and propose a simple stochastic gradient algorithm for the computation of its solution. Using simplified tools from Stochastic Approximation theory we demonstrate that batch versions of the algorithm or smoothing of the gradient do not improve convergence. These observations allow for the development of a training algorithm that enjoys reduced computational complexity and increased robustness while exhibiting similar synthesis characteristics as classical GANs.
BibTeX
@misc{basioti2018kernel,
  author = {Kalliopi Basioti and George V. Moustakides and Emmanouil Z. Psarakis},
  title = {{Kernel-Based Training of Generative Networks}},
  year = {2018},
  eprint = {1811.09568},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.09568},
}
Figure