Introduction
Generative Adversarial Networks (GANs) have achieved remarkable success in synthesizing high-fidelity data. However, they suffer from a fundamental limitation: a tendency to prioritize majority modes while neglecting the "long tail" of a distribution. This "majority-rule" learning leads to incomplete mode coverage, where rare but significant subpopulations—such as minority classes in a dataset or infrequent medical conditions—are entirely omitted from the generated output. This paper introduces a novel sampling strategy based on Ridge Leverage Scores (RLS) to ensure that GANs learn from the entire data distribution, regardless of local density.
Methodology: Ridge Leverage Score Sampling
The core innovation lies in replacing uniform data sampling with a strategy that biases training toward geometrically unique samples. In linear algebra, leverage scores identify rows in a matrix that are most influential or distinct. By applying Ridge Leverage Scores to data features, the researchers can quantify the "uniqueness" of each training example.
To implement this, the authors propose sampling training images with a probability proportional to their RLS. This "boosts" the visibility of minority modes during the optimization process. The scores are calculated using various feature maps:
- Implicit Gaussian Kernels for low-dimensional synthetic data.
- Pre-trained Networks (e.g., Inception) for high-dimensional images.
- Discriminator-based Features to dynamically identify unique features as training progresses.
To handle large-scale datasets, the authors employ computational shortcuts such as Gaussian sketching and UMAP dimensionality reduction, making the scoring process feasible for millions of images.
Experimental Results
The effectiveness of RLS sampling was evaluated on several unbalanced benchmarks designed to challenge standard GANs.
Synthetic Benchmarks: RING and GRID
On 2D synthetic datasets where certain clusters were 20 times less frequent than others, the RLS-boosted models demonstrated superior coverage. While a standard GAN might only capture a fraction of the available modes, the proposed method consistently recovers nearly all clusters.
Real-World Unbalanced Data
The method was further tested on "Unbalanced MNIST" and "Unbalanced CIFAR-10," where specific classes were intentionally depleted.
- Unbalanced MNIST: In a scenario where the digit '2' was rare, an RLS-enhanced BuresGAN generated over 1,500 instances of the minority digit, compared to only 129 from the baseline.
- Unbalanced CIFAR-10: The method significantly improved image quality and diversity. For a task where classes like "airplanes" were reduced to 5% frequency, the RLS approach achieved a Fréchet Inception Distance (FID) of 27.2, a substantial improvement over the baseline's 48.8.
Conclusion
Leverage score sampling provides a robust, unsupervised mechanism for rebalancing GAN training. By intentionally biasing the training process toward unique data points, the resulting model becomes less biased toward the majority, achieving more complete mode coverage. This "plug-and-play" data-loading strategy can be integrated into existing GAN architectures to improve performance on diverse and unbalanced real-world datasets. Future work will focus on refining feature extractors to capture even more subtle data subpopulations.
Summary updated