GAN-EM: GAN Based EM Learning Framework

Wentian Zhao, Shaojie Wang, Zhihuai Xie, Jing Shi, Chenliang Xu

Abstract

Expectation maximization (EM) algorithm is designed to find maximum likelihood solutions for models with latent variables. A typical example is the Gaussian Mixture Model (GMM), which requires a Gaussian assumption. However, natural images are highly non-Gaussian, meaning GMMs cannot be effectively applied to image clustering in pixel space. To overcome this limitation, we propose GAN-EM, a GAN-based EM learning framework that maximizes the likelihood of images and estimates latent variables. GAN-EM serves as a framework for image clustering, semi-supervised classification, and dimensionality reduction. In the M-step, we design a novel loss function for the GAN discriminator to perform maximum likelihood estimation (MLE) on data with soft class label assignments. In the E-step, an additional network (E-net) estimates the latent class labels of real data. The proposed GAN-EM achieves state-of-the-art clustering and semi-supervised classification results on MNIST, SVHN, and CelebA.

Introduction

Clustering images at the pixel level is a significant challenge in computer vision. While the Gaussian Mixture Model (GMM) and the EM algorithm are the "gold standard" for unsupervised clustering, they rely on the assumption that data in each cluster follows a Gaussian distribution. Natural images violate this assumption, as their pixel intensities follow complex, multi-modal distributions.

Generative Adversarial Networks (GANs) excel at modeling these complex distributions but lack a formal mechanism for "soft" cluster assignment and the monotonic convergence properties of the EM algorithm. GAN-EM bridges this gap by reformulating the EM algorithm to work within a deep generative framework.

Methodology

GAN-EM updates the two steps of the EM algorithm to function with deep neural networks, replacing rigid statistical assumptions with flexible generative modeling.

The M-step (Maximization)

In the M-step, the model updates its parameters to maximize the likelihood of the data given current cluster assignments. This is achieved using a Conditional GAN architecture:

  • Generator ($G$): Produces "fake" images for specific clusters based on a noise vector and a class label.
  • Discriminator ($D$): Features $K$ outputs, one for each cluster, rather than a single binary output.
  • Weighted Loss: The authors utilize a weighted binary cross-entropy loss for the discriminator. These weights are the "soft" class assignments (probabilities) derived from the E-step. This configuration allows the GAN to optimize the Q-function, the core objective of the EM algorithm.
The GAN-EM framework illustrating the iterative M-step and E-step.

The E-step (Expectation)

This step estimates the latent variables—the cluster labels for real images. Because GANs are not inherently invertible, the authors introduce an E-net:

  • The E-net is trained using fake images generated by $G$, where labels are known.
  • Once trained, the E-net acts as an inverse generator, taking a real image and outputting a soft probability distribution over the $K$ classes. These probabilities are then fed back into the M-step for the next iteration.

Experimental Results

The framework was evaluated on three standard benchmarks: MNIST (handwritten digits), SVHN (Street View House Numbers), and CelebA (face attributes).

Unsupervised Clustering and Semi-supervised Learning

On the MNIST dataset, GAN-EM achieved a state-of-the-art error rate of 3.97% (with $K=30$ clusters), significantly outperforming standard GMMs, which yield error rates over 32%. In semi-supervised tasks on SVHN, using only 1,000 labeled images, GAN-EM reached an error rate of 6.05%, surpassing other generative models like AAE (17.70%) and Improved GAN (8.11%).

Clustered MNIST digits generated by the GAN-EM model, demonstrating clear separation of digit classes.

Abstract Feature Clustering

Testing on the CelebA dataset demonstrated the model's ability to cluster faces based on high-level semantic features, such as eyewear, without explicit supervision. This highlights the model's capacity to handle complex, abstract attributes beyond simple pixel-level similarities.

Conclusion and Key Insights

GAN-EM successfully formalizes GANs within an EM framework, moving GAN-based clustering from heuristic methods to a theory-based approach. By using the E-net as a proxy for the inverse generator, the model effectively assigns soft labels to real data points. Furthermore, the use of a single conditional generator and multi-output discriminator ensures the framework is computationally efficient and scalable.

While the model is subject to common deep learning challenges such as local minima and dependency on generator quality (to avoid mode collapse), it demonstrates quick convergence—typically within 20 epochs—and provides a robust foundation for future integration with advanced GAN variants like WGAN or SAGAN.

Summary updated

Abstract

Expectation maximization (EM) algorithm is to find maximum likelihood solution for models having latent variables. A typical example is Gaussian Mixture Model (GMM) which requires Gaussian assumption, however, natural images are highly non-Gaussian so that GMM cannot be applied to perform image clustering task on pixel space. To overcome such limitation, we propose a GAN based EM learning framework that can maximize the likelihood of images and estimate the latent variables. We call this model GAN-EM, which is a framework for image clustering, semi-supervised classification and dimensionality reduction. In M-step, we design a novel loss function for discriminator of GAN to perform maximum likelihood estimation (MLE) on data with soft class label assignments. Specifically, a conditional generator captures data distribution for K classes, and a discriminator tells whether a sample is real or fake for each class. Since our model is unsupervised, the class label of real data is regarded as latent variable, which is estimated by an additional network (E-net) in E-step. The proposed GAN-EM achieves state-of-the-art clustering and semi-supervised classification results on MNIST, SVHN and CelebA, as well as comparable quality of generated images to other recently developed generative models.
BibTeX
@inproceedings{zhao2019gan,
  author = {Wentian Zhao and Shaojie Wang and Zhihuai Xie and Jing Shi and Chenliang Xu},
  title = {{GAN-EM: GAN Based EM Learning Framework}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=CtmY5MWCHMe9},
}
Figure