Variational Clustering: Leveraging Variational Autoencoders for Image Clustering

Vignesh Prasad, Dipanjan Das, Brojeshwar Bhowmick

TL;DR

This paper introduces a "true" Gaussian Mixture Variational Autoencoder (VAE) for image clustering that learns to group data and extract features simultaneously from scratch. By using a deterministic prior and a novel augmentation loss, the model achieves state-of-the-art unsupervised clustering performance on complex datasets like CIFAR-10 and STL-10 without requiring any pre-trained models.

What Problem Does This Paper Address?

Clustering images is a "chicken and egg" problem: to cluster effectively, you need robust feature representations, but to learn those features in an unsupervised way, you often need some structural guidance (like clusters).

Traditional methods like K-means often fail on raw pixels because they don't capture the semantic complexity of images. While Deep Learning has improved feature extraction, many existing "Deep Clustering" methods treat feature learning and clustering as separate steps or rely heavily on pre-trained models (like ResNet trained on ImageNet). Existing generative models for clustering, such as GMVAE and VaDE, suffer from either overly complex stochastic priors or a reliance on "warm-up" pre-training phases that can lead to sub-optimal convergence.

Methods

The authors propose a Variational Clustering framework that integrates a Gaussian Mixture Model (GMM) directly into the latent space of a VAE.

Key technical components include:

  • Deterministic GMM Prior: Unlike previous models where the prior might depend on a noise variable, this model uses a deterministic prior where each cluster is represented by its own learned mean and variance.
  • Direct Inference: The model directly predicts cluster assignment probabilities $q(k|z)$ from the input data, making the inference process more straightforward and principled.
  • Joint Learning: The parameters for the prior (the cluster shapes) and the posterior (the data encoding) are learned jointly in an end-to-end fashion. No pre-training is required.
  • Augmentation Loss: To make the model robust, the authors introduce a loss function that compares an image with its augmented version (e.g., rotated or brightened). They minimize the $L2$ distance between cluster predictions and the 2-Wasserstein distance between their posterior distributions. This ensures that a "cat" and a "rotated cat" end up in the same cluster with similar latent representations.

Datasets

The model was evaluated on a wide range of benchmarks:

  • Toy Data: A 2D dataset of 5 nested circles to test latent space separation.
  • MNIST & Fashion-MNIST: Standard benchmarks for handwritten digits and clothing items.
  • STL-10 & CIFAR-10/100: Challenging datasets containing real-world objects (animals, vehicles) at various resolutions.
  • FRGCv2: A face image dataset used to test clustering with a higher number of classes ($k=20$).

Key Insights and Ideas

  • Simplicity is Strength: By making the GMM prior deterministic rather than stochastic (as in GMVAE), the model becomes more intuitive and easier to train. The latent space naturally organizes into compact, well-separated "blobs."
  • Self-Supervision via Augmentation: The inclusion of the augmentation loss is a "game changer." It forces the network to ignore "noise" (like slight rotations) and focus on the semantic content of the image, leading to much stronger intra-cluster relations.
  • End-to-End beats Pre-training: The authors show that learning everything at once is superior to the "initialize then refine" approach used by models like VaDE. This prevents the model from getting stuck in poor local minima created during a separate pre-training phase.

Key Results

  • State-of-the-Art Performance: Among purely unsupervised methods (those not using pre-trained features), this model consistently outperformed others. For example, on CIFAR-10, it achieved 44.5% accuracy, significantly higher than GMVAE (24.74%).
  • Robustness to Imbalance: In tests where some clusters were much smaller than others, the model maintained high accuracy (e.g., 89.6% on MNIST with a 10:1 imbalance ratio), whereas K-means dropped to 47.14%.
  • Transfer Learning: The model showed impressive "generalizability." A model trained on CIFAR-10 could cluster the 9 overlapping classes in STL-10 with 41.03% accuracy without seeing the STL-10 images during training.
  • Generative Quality: Because it is a VAE, the model can "imagine" new images. The paper demonstrates that the model generates distinct, recognizable images for each cluster, proving it has truly learned the underlying data distribution.

Limitations and Future Work

While the results are impressive for unsupervised learning, there is still a significant gap compared to supervised models or those using massive pre-trained backbones (like ResNet-50).

The authors suggest two main paths forward:

  1. VAE-GAN Hybrid: Replacing the standard VAE generator with a GAN-based generator to produce even more realistic and high-resolution image samples.
  2. Explicit Constraints: Adding more explicit inference constraints on the prior to further sharpen the boundaries between clusters in the latent space.

Summary updated

Abstract

Recent advances in deep learning have shown their ability to learn strong feature representations for images. The task of image clustering naturally requires good feature representations to capture the distribution of the data and subsequently differentiate data points from one another. Often these two aspects are dealt with independently and thus traditional feature learning alone does not suffice in partitioning the data meaningfully. Variational Autoencoders (VAEs) naturally lend themselves to learning data distributions in a latent space. Since we wish to efficiently discriminate between different clusters in the data, we propose a method based on VAEs where we use a Gaussian Mixture prior to help cluster the images accurately. We jointly learn the parameters of both the prior and the posterior distributions. Our method represents a true Gaussian Mixture VAE. This way, our method simultaneously learns a prior that captures the latent distribution of the images and a posterior to help discriminate well between data points. We also propose a novel reparametrization of the latent space consisting of a mixture of discrete and continuous variables. One key takeaway is that our method generalizes better across different datasets without using any pre-training or learnt models, unlike existing methods, allowing it to be trained from scratch in an end-to-end manner. We verify our efficacy and generalizability experimentally by achieving state-of-the-art results among unsupervised methods on a variety of datasets. To the best of our knowledge, we are the first to pursue image clustering using VAEs in a purely unsupervised manner on real image datasets.
BibTeX
@inproceedings{prasad2020variational,
  author = {Vignesh Prasad and Dipanjan Das and Brojeshwar Bhowmick},
  title = {{Variational Clustering: Leveraging Variational Autoencoders for Image Clustering}},
  booktitle = {IJCNN},
  year = {2020},
  url = {https://openreview.net/forum?id=GRjfuBZK4b},
}

Loading related content...

Figure