Mind the (optimality) Gap: A Gap-Aware Learning Rate Scheduler for Adversarial Nets

Hussein Hazimeh, Natalia Ponomareva

Abstract

Training adversarial networks such as Generative Adversarial Networks (GANs) and Domain Adversarial Neural Networks (DANNs) is notoriously difficult due to the requirement of maintaining a delicate equilibrium between competing models. This paper introduces a novel "Gap-Aware Learning Rate Scheduler" that dynamically adjusts the adversary's learning rate based on its distance from a theoretical "ideal" loss. By monitoring this optimality gap, the scheduler significantly improves training stability, enhances sample quality, and reduces the hyperparameter tuning budget by up to 10x across multiple benchmarks.

Introduction

Adversarial training has revolutionized generative modeling and domain adaptation, yet it remains one of the most unstable paradigms in deep learning. The core of the challenge lies in the competitive nature of the architecture: two networks, typically a Generator and a Discriminator, engage in a zero-sum game. If the Discriminator becomes too dominant, the Generator receives vanishing gradients and ceases to learn; conversely, if the Discriminator is too weak, the Generator lacks the necessary signal to improve.

Traditionally, researchers have managed this balance through exhaustive manual tuning of fixed learning rates or simple decay schedules. However, these "blind" schedules do not react to the real-time dynamics of training, leading to a "tuning nightmare" where thousands of experiments are often required to find a stable configuration. This paper proposes a self-correcting mechanism that uses the theoretical equilibrium of the adversarial game to guide learning rate adjustments automatically.

Methodology: The Optimality Gap

The authors leverage a fundamental property of adversarial loss functions: for many standard formulations, the optimal value of the loss at equilibrium ($V^*$) is a known constant. For example, in the original GAN formulation, when the generator perfectly matches the data distribution, the discriminator's loss is exactly $\log(4)$.

The paper defines the Optimality Gap as the absolute difference between the current discriminator loss ($V_d$) and this theoretical ideal ($V^*$). The proposed scheduler operates at the batch level through a three-step process:

  1. Monitoring: The discriminator's loss is tracked using an Exponential Moving Average (EMA) to filter out stochastic noise and provide a smooth signal of model performance.
  2. Reaction: The scheduler compares the current EMA loss to $V^*$. If $V_d \geq V^*$, the discriminator is underperforming, and the scheduler increases its learning rate. If $V_d < V^*$, the discriminator is overperforming, and the scheduler decreases its learning rate to allow the generator to catch up.
  3. Interpolation: To prevent wild oscillations, adjustments are constrained using exponential interpolation functions ($f(x)$ and $h(x)$) that bound the learning rate within a safe operating range.

This real-time feedback loop allows the model to self-correct during training, maintaining the competitive balance without manual intervention.

Experimental Setup

To validate the scheduler, the authors conducted an extensive empirical study involving over 25,000 trained models. The experiments spanned several standard benchmarks, including MNIST, Fashion MNIST, CIFAR-10, and CelebA for image generation. Additionally, the researchers tested the scheduler on Domain Adversarial Neural Networks (DANN) using the MNIST-M dataset to evaluate its utility in domain adaptation tasks.

Key Results and Insights

1. Drastic Reduction in Tuning Budget

One of the most significant findings is the scheduler's impact on efficiency. On the CelebA dataset, the Gap-Aware Scheduler allowed a GAN to reach peak performance while requiring only one-tenth of the hyperparameter tuning budget compared to standard fixed-rate methods. The scheduler makes the model significantly less sensitive to the initial learning rate choice, which is often the most difficult parameter to tune.

2. Improved Image Quality and Stability

The scheduler consistently outperformed baselines in terms of sample quality. On the CelebA benchmark, the Frechet Inception Distance (FID)—a standard metric for GAN quality—improved by up to 27%. Furthermore, the scheduler proved highly effective at suppressing outliers; in LSGAN experiments, it almost entirely eliminated "failed" runs that typically plague adversarial training.

3. Enhanced Domain Adaptation

In DANN experiments, the scheduler led to a 3% increase in test accuracy on MNIST-M. More importantly, it provided significantly more stable training trajectories, reducing the variance across different random seeds and making the results more reproducible.

Discussion and Limitations

While the Gap-Aware Scheduler offers a powerful tool for practitioners, it does introduce a small set of its own hyperparameters (specifically $x_{min}, x_{max}, h_{min}, f_{max}$). However, the authors demonstrate that a single set of "default" values performs robustly across various datasets and architectures, suggesting these are far easier to set than the learning rates themselves.

Currently, the method relies on knowing the theoretical $V^*$. While this is straightforward for most popular GAN variants (Standard, LSGAN, etc.), it may be more challenging to apply to custom loss functions where the equilibrium point is not mathematically trivial. Additionally, while the study focused on the DCGAN architecture, future work is needed to explore the scheduler's interaction with more modern architectures like StyleGAN.

Conclusion

The Gap-Aware Learning Rate Scheduler represents a significant step toward making adversarial training a "plug-and-play" technology. By using the theoretical optimality gap as a North Star, the scheduler replaces manual tuning with a dynamic, principled adjustment mechanism. This not only improves the final quality of the models but also democratizes adversarial training by drastically reducing the computational resources required to find a stable configuration.

Summary updated

BibTeX
@inproceedings{hazimeh2023mind,
  author = {Hussein Hazimeh and Natalia Ponomareva},
  title = {{Mind the (optimality) Gap: A Gap-Aware Learning Rate Scheduler for Adversarial Nets}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2023},
  url = {https://openreview.net/forum?id=gL1A7mxavz},
}
Figure