Abstract
Generative adversarial networks (GANs) have shown remarkable success in generation of data from natural data manifolds such as images. In several scenarios, it is desirable that generated data is well-clustered, especially when there is severe class imbalance. In this paper, we focus on the problem of clustering in generated space of GANs and uncover its relationship with the characteristics of the latent space. We derive from first principles, the necessary and sufficient conditions needed to achieve faithful clustering in the GAN framework: (i) presence of a multimodal latent space with adjustable priors, (ii) existence of a latent space inversion mechanism and (iii) imposition of the desired cluster priors on the latent space. We also identify the GAN models in the literature that partially satisfy these conditions and demonstrate the importance of all the components required, through ablative studies on multiple real world image datasets. Additionally, we describe a procedure to construct a multimodal latent space which facilitates learning of cluster priors with sparse supervision.
Introduction
Standard Generative Adversarial Networks (GANs) typically map a simple, unimodal latent distribution, such as a Gaussian, to a complex data manifold. While this approach is effective for generating high-quality images, it often fails to capture the inherent grouped structure of real-world data, such as distinct digit classes in MNIST or object categories in CIFAR-10. This failure manifests in two primary ways: mode collapse or mixing, where the generator merges distinct clusters, and poor handling of class imbalance, where minority classes are ignored or misrepresented.
Methodology: The Three Conditions for Faithful Clustering
To address these challenges, we identify three "first-principle" conditions necessary and sufficient for a GAN to achieve faithful clustering:
- $C_1$: Multimodal Latent Space. The input noise must be structured and clustered. We implement this using an additive mixture of a discrete variable (representing the cluster index) and a continuous variable (representing intra-cluster variation).
- $C_2$: Latent Inversion. A separate inverter network ($h$) is required to map generated images back to their latent cluster assignments. This mechanism prevents the generator from collapsing multiple latent modes into a single output mode.
- $C_3$: Prior Matching. The probability mass of each cluster in the latent space must align with the actual proportions in the real data.
A key innovation is the learnable multimodal latent space. Using a reparameterization trick involving a unit step function and a learnable vector $\alpha$, the model can adjust its internal cluster priors. This allows the framework to learn correct data proportions even with sparse supervision (e.g., labeling only 1% of the data).
Experiments and Results
The framework was evaluated on several benchmarks, including MNIST, Fashion-MNIST (FMNIST), CelebA, and CIFAR-10. To test robustness against imbalance, "skewed" versions of these datasets (e.g., 90:10 class ratios) were utilized.
Qualitative Analysis
On synthetic datasets like the "two moons" data, our model demonstrates superior cluster separation compared to existing methods like WGAN, DeliGAN, and InfoGAN, which often fail to distinguish the underlying modes or produce overlapping distributions.
Quantitative Performance
The proposed model consistently outperformed baselines, particularly in imbalanced scenarios:
- Accuracy: In a 90:10 imbalanced MNIST task, our method maintained 98% accuracy, whereas competing models dropped to 60-70%.
- Image Quality: Our approach achieved significantly lower Fréchet Inception Distance (FID) scores (e.g., 1.66 vs. 6.08 for the next best model on imbalanced MNIST), indicating higher diversity and fidelity.
- Prior Learning: With just 1% labeled data, the model precisely estimated true class priors (e.g., estimating a 0.7/0.3 split as 0.709/0.291).
Conclusion
Faithful clustering in GANs requires more than just a structured latent space; it necessitates an inversion mechanism and accurate prior matching. While our framework handles disjoint clusters effectively, future work will explore highly entangled manifolds and fully unsupervised prior estimation.
Summary updated