Abstract
Expectation maximization (EM) algorithm is to find maximum likelihood solution for models having latent variables. A typical example is Gaussian Mixture Model (GMM) which requires Gaussian assumption, however, natural images are highly non-Gaussian so that GMM cannot be applied to perform clustering task on pixel space. To overcome such limitation, we propose a GAN based EM learning framework that can maximize the likelihood of images and estimate the latent variables with only the constraint of L-Lipschitz continuity. We call this model GAN-EM, which is a framework for image clustering, semi-supervised classification and dimensionality reduction. In M-step, we design a novel loss function for discriminator of GAN to perform maximum likelihood estimation (MLE) on data with soft class label assignments. Specifically, a conditional generator captures data distribution for $K$ classes, and a discriminator tells whether a sample is real or fake for each class. Since our model is unsupervised, the class label of real data is regarded as latent variable, which is estimated by an additional network (E-net) in E-step. The proposed GAN-EM achieves state-of-the-art clustering and semi-supervised classification results on MNIST, SVHN and CelebA, as well as comparable quality of generated images to other recently developed generative models.
Introduction
The Expectation-Maximization (EM) algorithm is a cornerstone of machine learning, particularly for models with hidden (latent) variables. A classic application is the Gaussian Mixture Model (GMM), which clusters data by assuming it is composed of several Gaussian distributions. However, natural images are fundamentally non-Gaussian; pixel intensities exhibit highly non-linear behaviors that simple bell curves cannot capture. Consequently, GMMs perform poorly when clustering images directly in pixel space.
While deep learning has offered alternatives, many existing GAN-based clustering methods rely on "hard" label assignments (similar to K-means) or require training a separate GAN for every single cluster—a computationally expensive approach. GAN-EM addresses these limitations by embedding Generative Adversarial Networks into the classic EM framework, replacing rigid Gaussian assumptions with the flexible distribution-learning power of GANs.
The GAN-EM Framework
GAN-EM performs the EM process using a single conditional GAN and an inference network, referred to as the E-net. The framework iterates between two primary steps:
- The M-step (Maximization): The objective is to update the model to maximize the likelihood of the data given current cluster assignments. The discriminator loss function is modified to be a weighted binary cross-entropy, where the weights are "soft" class assignments passed from the E-step. This design allows the GAN to optimize the "Q-function," the core objective of the EM algorithm.
- The E-step (Expectation): This step estimates the latent variables (cluster labels). Since GANs are not naturally reversible, the E-net is trained using fake images generated by the GAN to learn a mapping back to their known class labels. Once trained, the E-net predicts soft labels for real images.
To ensure meaningful clusters, the authors enforce L-Lipschitz continuity on the generator (using weight clipping). This encourages "intra-cluster continuity" and "inter-cluster discontinuity," pushing the model to find clear gaps between different data groups.
Experimental Results
The framework was evaluated on three benchmarks: MNIST, SVHN (Street View House Numbers), and CelebA.
Clustering and Classification
- MNIST: GAN-EM achieved a clustering error rate of 3.97% (with $K=30$ clusters), significantly outperforming traditional GMMs (32.61%) and other generative models like InfoGAN (5.00%).
- SVHN: In semi-supervised settings with 1000 labels, GAN-EM reached a 6.05% error rate, surpassing Improved GAN (8.11%) and PixelGAN (6.96%).
- CelebA: The model excelled at identifying abstract features like "hats" and "glasses" in an unsupervised manner, achieving state-of-the-art results for attribute clustering.
Dimensionality Reduction
The internal layers of the E-net can also be utilized for dimensionality reduction. When compressing MNIST data to just 2 dimensions, the model maintained clear, separable clusters for different digits, demonstrating its ability to capture the underlying manifold of the data.
Conclusion
GAN-EM successfully bridges the gap between GAN loss and Maximum Likelihood Estimation. By using a single conditional generator and a multi-output discriminator, the model is parameter-efficient and avoids the need for $K$ separate GANs. While the current implementation uses a vanilla GAN, the framework is modular and can be extended with more advanced architectures to handle higher-resolution images and more complex datasets.
Summary updated