Abstract
Relativistic GANs (RGANs) have significantly improved the stability and output quality of generative models, yet their theoretical underpinnings remained largely unexplored. This paper provides the first formal proof that Relativistic GAN objectives are valid statistical divergences. We challenge the conventional wisdom that "weaker" metrics (like Wasserstein distance) are inherently better for GAN training, demonstrating that RGANs utilize "stronger" metrics to achieve superior results. Furthermore, we investigate the role of estimator bias, suggesting that the "noise" in standard RGAN estimators may serve as a beneficial regularizer during training.
Introduction
In the landscape of Generative Adversarial Networks (GANs), empirical breakthroughs often precede theoretical justification. Relativistic GANs (RGANs) introduced a paradigm shift by tasking the discriminator with estimating the relative realism of data pairs rather than making absolute classifications. While highly effective, the mathematical validity of these loss functions as true statistical divergences was not formally established until now.
Theoretical Foundation: RGANs as f-Divergences
The core contribution of this work is the rigorous proof that the objective functions for Relativistic paired (RpGAN), Relativistic average (RaGAN), and the newly introduced Relativistic centered (RcGAN) are indeed statistical divergences. This means they are non-negative and reach their global minimum of zero if and only if the generated distribution perfectly matches the real data distribution.
As shown in the figure above, different relativistic variants employ different functional forms ($f$) for the difference in critic scores $C(x) - C(y)$. These functions define how the model penalizes the relative distance between real and fake samples.
The Strength Paradox
A common belief in GAN research, popularized by the Wasserstein GAN (WGAN), is that "weaker" metrics—those that are easier to satisfy for converging distributions—are superior for training stability. This paper proves a counter-intuitive result: RGANs actually utilize metrics that are stronger than standard $f$-divergences. The empirical success of RGANs over WGAN suggests that metric weakness is not the sole, or even primary, driver of GAN performance.
The Role of Bias and Noisy Estimators
To further refine the theory, we introduced the Minimum-Variance Unbiased Estimator (MVUE) for RpGANs and a closed-form bias correction for the Least-Squares version of RaGAN (RaLSGAN). This allowed us to test whether the inherent bias in standard RGAN estimators was a flaw or a feature.
Our analysis of relative bias (visualized above) indicates that while RaGAN estimators are technically only asymptotically unbiased, the actual bias in practice is negligible. Surprisingly, our ablation studies showed that using "perfect" unbiased estimators actually degraded performance. The original, slightly biased and noisier estimators appear to act as a form of natural regularization, preventing the discriminator from overpowering the generator too early and causing vanishing gradients.
Empirical Results
We evaluated these insights across CIFAR-10, CelebA, and CAT benchmarks using the Fréchet Inception Distance (FID). Key findings include:
- RaGAN Dominance: The original RaGAN consistently outperformed both standard GANs and the newer RcGAN across all datasets.
- The Benefit of Noise: On CIFAR-10, the original (biased) RaLSGAN achieved an FID of 29.2, outperforming its unbiased counterpart (30.3). Similarly, on CelebA, the standard RpGAN (FID 16.7) significantly beat the MVUE version (FID 21.9).
Conclusion
This work solidifies the mathematical foundation of Relativistic GANs, proving they are sound statistical divergences. By demonstrating that stronger metrics and slightly biased estimators can lead to better generative performance, we open new avenues for understanding GAN stability. Future research will focus on scaling these theoretical insights to high-resolution image synthesis and further isolating the regularizing mechanisms of noisy estimators.
Summary updated