Accelerated WGAN Update Strategy With Loss Change Rate Balancing

Xu Ouyang, Ying Chen, Gady AgamWACV 2021

TL;DR

This paper introduces an adaptive update strategy for Wasserstein GANs (WGANs) that dynamically balances the training of the generator and discriminator. By monitoring the rate of change in their respective losses, the strategy intelligently decides which network to update, leading to faster convergence and improved accuracy compared to traditional fixed update ratios.

What Problem Does This Paper Address?

Training Generative Adversarial Networks (GANs), particularly WGANs, is notoriously difficult due to stability and convergence issues. A common practice is to update the discriminator multiple times for each generator update (e.g., 5:1 ratio). However, this fixed ratio is often suboptimal, leading to inefficient training and potentially hindering convergence. The paper argues that this fixed strategy is not ideal and can even prevent convergence in certain scenarios (like the Dirac-GAN problem). The core challenge is finding an optimal way to balance the training of the two competing networks, especially since their loss functions operate on different scales and their optimal update frequencies can change dynamically during training.

Methods

The paper proposes an "adaptive update strategy" that replaces the fixed update ratio with a dynamic decision-making process. The key idea is to monitor the rate of change of the generator's loss ($L_g$) and the discriminator's loss ($L_d$). Specifically, they calculate relative loss change ratios:

  • $r_g = |(L_g^{current} - L_g^{previous}) / L_g^{previous}|$
  • $r_d = |(L_d^{current} - L_d^{previous}) / L_d^{previous}|$

These ratios indicate how much each network's loss is changing. The strategy then uses a hyperparameter $\lambda$ to weigh the generator's loss change ratio relative to the discriminator's. If $r_d > \lambda \cdot r_g$, the discriminator is updated; otherwise, the generator is updated. This allows the system to adaptively prioritize the network that appears to be lagging or requires more attention. The authors also demonstrate that this adaptive strategy can achieve local convergence for the Dirac-GAN problem, which fixed strategies fail to do.

Datasets

The paper evaluates its proposed strategy on several standard datasets:

  • CIFAR-10: Used for image synthesis tasks with WGAN and WGAN-GP.
  • LSUN Conference Room: Used for image synthesis with WGAN.
  • LFW (Labeled Faces in the Wild): Used for image synthesis with WGAN.
  • Caltech-UCSD Birds-200-2011: Used for image deblurring with DeblurGAN.
  • DIV2K: Used for super-resolution tasks with SR-WGAN.

No new datasets were introduced; existing datasets were used to evaluate the performance of the adaptive strategy against established GAN models.

Key Insights and Ideas

  • Loss Change Rate as a Dynamic Indicator: The core insight is that the rate of change of losses, rather than just their absolute values or a fixed ratio, provides a more robust signal for balancing generator and discriminator training.
  • Adaptive Balancing Outperforms Fixed Ratios: The paper strongly suggests that a fixed update ratio is inherently suboptimal. The dynamic nature of GAN training means that the optimal balance between G and D updates can shift, making an adaptive approach crucial for efficiency and effectiveness.
  • Improved Convergence for Difficult Cases: The adaptive strategy's ability to achieve local convergence for the Dirac-GAN problem highlights its theoretical advantage over fixed strategies, which can get stuck or fail to converge.
  • The Role of $\lambda$: While $\lambda$ is a hyperparameter, the paper shows that even a default value of $\lambda=1$ (giving equal weight to both loss change rates) often outperforms fixed strategies. Higher values of $\lambda$ can be beneficial for more complex tasks, suggesting a tunable knob for performance.

Key Results

  • Faster Convergence: Across various GAN models (WGAN, WGAN-GP, DeblurGAN, SR-WGAN) and datasets, the proposed adaptive strategy consistently converged faster than their fixed-update counterparts. For example, in Table 2, adaptive WGAN with $\lambda=1$ reached target FID values in fewer epochs than fixed WGAN.
  • Better Accuracy/Quality: The adaptive strategy often achieved better results, indicated by lower FID scores (for image synthesis) and higher PSNR/SSIM (for deblurring/super-resolution). For instance, adaptive WGAN-GP with $\lambda=1$ achieved a lower FID than fixed WGAN-GP on CIFAR-10.
  • More Efficient Updates: The adaptive strategy often resulted in a different G/D update ratio than the commonly suggested fixed ratios, demonstrating that the optimal ratio is not static and can be learned. For example, adaptive strategies often ended up training the generator more than the discriminator, contrary to the typical 5:1 D:G ratio.
  • Qualitative Improvements: Visualizations (Figures 3, 4, 5) show that the adaptive strategy produces comparable or better quality images with faster training.

Limitations and Future Work

  • Hyperparameter $\lambda$: While the strategy is less sensitive to $\lambda$ than fixed ratios are to their parameters, $\lambda$ still needs to be selected. The paper suggests that $\lambda=1$ is a good default and higher values can be tuned for complex tasks, but a systematic way to select $\lambda$ is not fully explored.
  • Generalizability: The paper focuses on WGAN-related loss functions. Future work could investigate extending this adaptive strategy to other GAN architectures and loss terms.
  • Empirical Estimation: The authors acknowledge that empirically estimating the optimal G/D update ratio is difficult and can change during training. Their adaptive strategy alleviates this, but the selection of $\lambda$ still requires some consideration. They suggest investigating additional update strategies suitable for various GAN structures and loss terms.

Summary updated

BibTeX
@inproceedings{ouyang2021accelerated,
  author = {Xu Ouyang and Ying Chen and Gady Agam},
  title = {{Accelerated WGAN Update Strategy With Loss Change Rate Balancing}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure