Abstract
Adversarial nets have proved to be powerful in various domains including generative modeling (GANs), transfer learning, and fairness. However, successfully training adversarial nets using first-order methods remains a major challenge. Typically, careful choices of the learning rates are needed to maintain the delicate balance between the competing networks. In this paper, we design a novel learning rate scheduler that dynamically adapts the learning rate of the adversary to maintain the right balance. The scheduler is driven by the fact that the loss of an ideal adversarial net is a constant known a priori. The scheduler is thus designed to keep the loss of the optimized adversarial net close to that of an ideal network. Our experiments indicate that adversarial nets trained with the scheduler are less likely to diverge and require significantly less tuning. For example, on CelebA, a GAN with the scheduler requires only one-tenth of the tuning budget needed without a scheduler. Moreover, the scheduler leads to statistically significant improvements in model quality, reaching up to 27% in Frechet Inception Distance for image generation and 3% in test accuracy for domain adaptation.
Introduction
Training adversarial networks, such as Generative Adversarial Networks (GANs), is notoriously unstable. This instability stems from the "min-max" game where two networks—typically a Generator and a Discriminator—compete. If one network gains an upper hand too quickly, the other may stop learning, leading to training collapse or poor local minima. Traditionally, researchers manage this balance through exhaustive hyperparameter tuning of fixed learning rates. However, these static rates fail to account for the dynamic state of the competition during training.
The Optimality Gap
The core insight of this work is the relationship between the "optimality gap" and model performance. For many popular adversarial objectives, the loss of an "ideal" network (where the data and model distributions perfectly match) is a known constant, denoted as $V^*$. For instance, in a standard GAN, the ideal discriminator loss is $\log(4) \approx 1.386$.
The authors discovered a strong positive correlation between the distance of the current loss from this ideal ($|V_d - V^*|$) and the Frechet Inception Distance (FID), a standard metric for image quality.
As shown in the figure above, models that maintain a smaller optimality gap consistently achieve better FID scores. This suggests that keeping the discriminator near its theoretical ideal is a viable proxy for optimizing generative quality.
Gap-Aware Learning Rate Scheduler
Building on this insight, the authors propose a Gap-Aware Learning Rate Scheduler. The scheduler monitors the current discriminator loss ($V_d$) at the batch level and adjusts the learning rate accordingly:
- If $V_d \geq V^*$: The discriminator is underperforming. The scheduler increases its learning rate to help it reach the ideal state faster.
- If $V_d < V^*$: The discriminator is overperforming relative to the generator. The scheduler decreases its learning rate, allowing the generator to improve against a less dominant adversary.
The adjustments use exponential interpolation functions to ensure smooth transitions and are bounded to prevent extreme learning rate fluctuations. This mechanism acts as a self-correcting loop, automatically stabilizing the training process without requiring manual intervention.
Experimental Results
The scheduler was evaluated on standard benchmarks including MNIST, Fashion MNIST, CIFAR-10, and CelebA for GANs, as well as MNIST to MNIST-M for domain adaptation.
Tuning Efficiency and Model Quality
A primary advantage of the scheduler is the drastic reduction in hyperparameter tuning. On the CelebA dataset, a GAN using the scheduler reached the same quality as a standard GAN with only one-tenth of the tuning budget. Furthermore, the scheduler consistently improved the final model quality. For WGAN on CelebA, the FID score improved by 27%.
As illustrated in the multi-dataset comparison, the "With Scheduler" configuration (green lines) consistently achieves lower FID scores across various random search trials compared to the "No Scheduler" baseline (red lines). This demonstrates both superior performance and increased robustness to initial learning rate choices.
Domain Adaptation
In Domain Adversarial Neural Nets (DANN), the scheduler improved test accuracy by up to 3% and significantly tightened the accuracy distribution, indicating more stable and predictable training outcomes.
Conclusion and Limitations
The Gap-Aware Learning Rate Scheduler provides a principled, "North Star" approach to balancing adversarial training by leveraging a priori knowledge of the loss function. While the study focused on DCGAN architectures and standard loss functions where $V^*$ is known, the results demonstrate a significant step toward making adversarial training more robust and accessible. Future work may explore the application of this scheduler to more complex architectures like StyleGAN or cases where the ideal loss is harder to estimate.
Summary updated