On Minimax Optimality of GANs for Robust Mean Estimation

Kaiwen Wu, Gavin Weiguang Ding, Ruitong Huang, Yaoliang YuAISTATS 2020 · OpenReview

Abstract

Generative adversarial networks (GANs) have emerged as powerful tools for generative modeling. This work investigates their statistical and robust properties in the context of mean estimation for Gaussian distributions under Huber's contamination model. We demonstrate that while $f$-GANs can achieve minimax optimal error rates in high-dimensional settings, other popular variants like MMD-GAN and Wasserstein GAN (W-GAN) are fundamentally sub-optimal.

Introduction

Estimating the mean of a dataset is a fundamental task, typically solved by the sample mean. However, real-world data is often subject to contamination. Under Huber’s contamination model, a fraction $\epsilon$ of the data is replaced by arbitrary outliers. In high-dimensional spaces (dimension $p$), standard estimators fail significantly even with minimal contamination. While robust estimators like Tukey’s median offer theoretical guarantees, they are computationally intractable for large $p$. Recent research has highlighted the inherent robustness of the adversarial training objective in GANs. This paper rigorously evaluates which GAN architectures truly achieve minimax optimality in these challenging scenarios.

GANs as Minimum Variational Discrepancy Estimators

The paper frames GANs as estimators that minimize a variational discrepancy between a generated distribution $P_\theta$ and the empirical data distribution $P_n$. The objective is to find $\theta$ that minimizes $d(P_\theta, P_n) = \sup_{D \in \mathcal{D}} \mathbb{E}_{x \sim P_n}[D(x)] - \mathbb{E}_{x \sim P_\theta}[D(x)]$, where $\mathcal{D}$ is a class of discriminator functions.

The study focuses on three primary GAN families:

  1. $f$-GANs: Utilizing divergences like KL or Jensen-Shannon, with a two-layer neural network discriminator.
  2. MMD-GAN: Employing Maximum Mean Discrepancy with a Gaussian kernel.
  3. Wasserstein GAN (W-GAN): Using Earth Mover’s distance with Lipschitz-constrained discriminators.

Statistical Optimality and the Role of the Discriminator

A central finding is that the discriminator's function class $\mathcal{D}$ is the primary determinant of robustness. For $f$-GANs, a sufficiently expressive discriminator (e.g., a two-layer neural network) allows the estimator to effectively ignore outliers, achieving the minimax optimal error rate of $O(\sqrt{p/n} \vee \epsilon)$.

In contrast, MMD-GANs utilize a "weaker" discriminator class. While effective for clean data, this class is insufficient to filter out certain types of high-dimensional contamination. Theoretically and empirically, MMD-GAN's error scales as $O(\sqrt{p}\epsilon)$, which is a factor of $\sqrt{p}$ worse than the optimal rate.

Figure 3: Empirical evidence of sub-optimality. (a) Error vs contamination distance for different scales $\sigma$. (b) Error growing linearly with $\sqrt{p}$, characteristic of sub-optimal estimators like MMD-GAN and W-GAN.

Similarly, W-GAN faces a "robustness paradox." Although robust in one dimension, its reliance on unbounded Lipschitz functions in high dimensions makes it vulnerable to distant outliers, which can arbitrarily inflate the Wasserstein distance.

Empirical Validation of $f$-GAN Optimality

Numerical simulations on $p$-dimensional standard Gaussians $\mathcal{N}(\theta, I_p)$ with $\epsilon$-contamination confirm these theoretical insights. Variants like Squared Hellinger (SH-GAN) and Reverse KL (RKL-GAN) consistently reach the minimax rate, performing comparably to state-of-the-art robust filtering algorithms.

Figure 4: Empirical evidence of $f$-GAN optimality. (a) Error vs $\sqrt{p}$ for $f$-GAN variants, showing a plateau at the optimal error level determined by $\epsilon$. (b) Performance across different regularization parameters $\lambda$.

The $f$-GAN framework also demonstrates remarkable flexibility, successfully adapting to sparse mean estimation tasks by incorporating $\ell_0$ constraints on the generator without prior knowledge of the sparsity pattern.

Conclusion

This work establishes that the choice of GAN architecture is critical for robust statistical estimation. While $f$-GANs with neural network discriminators are minimax optimal, MMD and Wasserstein variants fall short in high dimensions. Future research should focus on identifying discriminator classes that balance this statistical optimality with computational efficiency and optimization stability.

Summary updated

Abstract

Generative adversarial networks (GANs) have become one of the most popular generative modeling techniques in machine learning. In this work, we study the statistical and robust properties of GANs f...
BibTeX
@inproceedings{wu2020minimax,
  author = {Kaiwen Wu and Gavin Weiguang Ding and Ruitong Huang and Yaoliang Yu},
  title = {{On Minimax Optimality of GANs for Robust Mean Estimation}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2020},
  url = {https://openreview.net/forum?id=BHSVCs2O-W5},
}
Figure