Improving GAN Training with Probability Ratio Clipping and Sample Reweighting

Yue Wu, Pan Zhou, Andrew Gordon Wilson, Eric P. Xing, Zhiting Hu

Abstract

Generative Adversarial Networks (GANs) are powerful generative models but are notoriously difficult to train due to issues like destructive updates and vanishing gradients. This paper proposes a variational framework that stabilizes GAN training by incorporating techniques from Reinforcement Learning (RL). We introduce Probability Ratio Clipping to prevent large, erratic generator updates and Sample Reweighting to focus the discriminator on high-quality samples. Our approach, evaluated on image synthesis and text generation tasks, demonstrates significant improvements in both stability and final performance metrics.

Introduction

Training GANs involves a delicate minimax game between a generator and a discriminator. However, this process is often plagued by instability. If the discriminator becomes too proficient, the generator receives no useful gradient (the vanishing gradient problem). Conversely, a single noisy update can lead to a "destructive update," causing the model to collapse. This work addresses these challenges by framing GAN training as an RL-as-inference problem, allowing the application of robust optimization techniques like those used in Proximal Policy Optimization (PPO).

Methodology

The authors propose two primary stabilization mechanisms derived from the RL perspective:

1. Probability Ratio Clipping

Drawing inspiration from PPO, the generator's objective is modified to include a clipped probability ratio. This ratio tracks the change between the current generator and its previous iteration. By clipping this ratio, the model ensures that the generator does not deviate too far from a "trust region," effectively preventing destructive updates caused by extreme discriminator signals or temporary noise.

2. Sample Reweighting

Standard GAN training treats all generated samples with equal importance. In contrast, this paper employs importance sampling to reweight samples based on their discriminator scores. By assigning higher weights to "harder" samples—those that are nearly indistinguishable from real data—the discriminator provides more informative gradients to the generator throughout the training process.

Stabilization of Generator Loss through sample reweighting compared to standard WGAN-GP.

As shown in the figure above, the introduction of sample reweighting significantly reduces the variance of the generator loss compared to standard WGAN-GP, leading to a much smoother and more predictable training trajectory.

Experimental Results

The proposed methods were tested across several benchmarks, including CIFAR-10 for images and WMT News for text generation.

Image Synthesis (CIFAR-10)

The full model achieved an Inception Score (IS) of 8.69 and a Fréchet Inception Distance (FID) of 10.70, outperforming established baselines like SN-GANs (8.22 IS). Beyond raw performance, the method proved more robust; in random hyperparameter trials, the proposed approach had a 30.2% collapse rate compared to 52.4% for standard DCGANs.

Ablation study showing the impact of clipping and reweighting on Inception Score performance on CIFAR-10.

The ablation study illustrated in the figure above confirms that while both clipping and reweighting independently improve performance over the baseline, their combination yields the highest Inception Scores and the most stable growth throughout the training process.

Text Generation and Style Transfer

On the WMT News dataset, the model improved BLEU scores and sample diversity. For text style transfer on the Yelp dataset, it achieved a state-of-the-art BLEU score of 33.45, demonstrating superior content preservation during style conversion.

Conclusion

By bridging the gap between GANs and Reinforcement Learning, this work provides a robust framework for stabilizing adversarial training. The combination of probability ratio clipping and sample reweighting effectively mitigates common failure modes, resulting in higher quality generative models across multiple domains. Future work may explore this variational perspective to unify other machine learning paradigms beyond GANs and RL.

Summary updated

BibTeX
@inproceedings{wu2020improving,
  author = {Yue Wu and Pan Zhou and Andrew Gordon Wilson and Eric P. Xing and Zhiting Hu},
  title = {{Improving GAN Training with Probability Ratio Clipping and Sample Reweighting}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=vdfA9aW1ca9},
}
Figure