Leverage Score Sampling for Complete Mode Coverage in Generative Adversarial Networks

Joachim Schreurs, Hannes De Meulemeester, Michaël Fanuel, Bart De Moor, Johan A. K. SuykensLOD 2021 · DOI

Introduction

Generative Adversarial Networks (GANs) have achieved remarkable success in synthesizing high-fidelity data. However, they suffer from a fundamental limitation: a tendency to prioritize majority modes while neglecting the "long tail" of a distribution. This "majority-rule" learning leads to incomplete mode coverage, where rare but significant subpopulations—such as minority classes in a dataset or infrequent medical conditions—are entirely omitted from the generated output. This paper introduces a novel sampling strategy based on Ridge Leverage Scores (RLS) to ensure that GANs learn from the entire data distribution, regardless of local density.

Methodology: Ridge Leverage Score Sampling

The core innovation lies in replacing uniform data sampling with a strategy that biases training toward geometrically unique samples. In linear algebra, leverage scores identify rows in a matrix that are most influential or distinct. By applying Ridge Leverage Scores to data features, the researchers can quantify the "uniqueness" of each training example.

Ridge Leverage Scores (blue) effectively highlight rare regions where the data density (orange) is low, providing a signal to the generator to focus on these underrepresented areas.

To implement this, the authors propose sampling training images with a probability proportional to their RLS. This "boosts" the visibility of minority modes during the optimization process. The scores are calculated using various feature maps:

  1. Implicit Gaussian Kernels for low-dimensional synthetic data.
  2. Pre-trained Networks (e.g., Inception) for high-dimensional images.
  3. Discriminator-based Features to dynamically identify unique features as training progresses.

To handle large-scale datasets, the authors employ computational shortcuts such as Gaussian sketching and UMAP dimensionality reduction, making the scoring process feasible for millions of images.

Experimental Results

The effectiveness of RLS sampling was evaluated on several unbalanced benchmarks designed to challenge standard GANs.

Synthetic Benchmarks: RING and GRID

On 2D synthetic datasets where certain clusters were 20 times less frequent than others, the RLS-boosted models demonstrated superior coverage. While a standard GAN might only capture a fraction of the available modes, the proposed method consistently recovers nearly all clusters.

Comparison of mode coverage on synthetic RING and GRID datasets. The blue points represent generated samples, while green points indicate ground truth modes. RLS sampling ensures the generator covers even the most sparsely populated modes.

Real-World Unbalanced Data

The method was further tested on "Unbalanced MNIST" and "Unbalanced CIFAR-10," where specific classes were intentionally depleted.

  • Unbalanced MNIST: In a scenario where the digit '2' was rare, an RLS-enhanced BuresGAN generated over 1,500 instances of the minority digit, compared to only 129 from the baseline.
  • Unbalanced CIFAR-10: The method significantly improved image quality and diversity. For a task where classes like "airplanes" were reduced to 5% frequency, the RLS approach achieved a Fréchet Inception Distance (FID) of 27.2, a substantial improvement over the baseline's 48.8.

Conclusion

Leverage score sampling provides a robust, unsupervised mechanism for rebalancing GAN training. By intentionally biasing the training process toward unique data points, the resulting model becomes less biased toward the majority, achieving more complete mode coverage. This "plug-and-play" data-loading strategy can be integrated into existing GAN architectures to improve performance on diverse and unbalanced real-world datasets. Future work will focus on refining feature extractors to capture even more subtle data subpopulations.

Summary updated

BibTeX
@inproceedings{schreurs2021leverage,
  author = {Joachim Schreurs and Hannes De Meulemeester and Michaël Fanuel and Bart De Moor and Johan A. K. Suykens},
  title = {{Leverage Score Sampling for Complete Mode Coverage in Generative Adversarial Networks}},
  booktitle = {LOD},
  year = {2021},
  doi = {10.1007/978-3-030-95470-3\_35},
}
Figure