Semi-Unsupervised Learning with Deep Generative Models: Clustering and Classifying using Ultra-Sparse Labels

Matthew Willetts, Stephen J. Roberts, Christopher C. Holmes

Introduction

In the landscape of modern machine learning, semi-supervised learning (SSL) has long been a cornerstone for leveraging large amounts of unlabeled data alongside a small set of labeled examples. However, SSL typically operates under a significant assumption: that the labeled set contains at least a few examples of every class present in the unlabeled pool.

This paper introduces and formalizes a more realistic but challenging regime termed Semi-Unsupervised Learning (SUS). In this scenario, some classes are sparsely labeled, while other "hidden" classes are entirely unlabeled. This mirrors real-world data collection biases, such as medical datasets where rare pathologies may exist in a massive unlabeled corpus but never appear in the limited labeled subset provided by a single clinic.

The Challenge of "Hidden" Classes

The SUS problem presents a twofold challenge. First, the model must accurately classify the sparsely labeled classes. Second, it must cluster the entirely unlabeled classes into distinct, meaningful groups. Standard semi-supervised models often fail here because they are designed to force all incoming data into the known, labeled categories. When confronted with a new, unlabeled class, these models either "hallucinate" them into known categories or hide their unique identity within the "style" parameters of the latent space, effectively leaving the classifier blind to their existence.

Methodology: GM-DGM vs. SSVAE

The researchers investigated two primary architectures based on Variational Autoencoders (VAEs):

  1. SSVAE (Semi-Supervised VAE): A standard model that utilizes a discrete latent variable ($y$) for class labels and a continuous latent variable ($z$) for "style."
  2. GM-DGM (Gaussian Mixture Deep Generative Model): This model treats the latent space as a mixture of Gaussians, where each component corresponds to a potential class.

The core of the paper's argument rests on Inductive Bias. For SUS to be successful, a model must naturally favor clustering data by identity even in the absence of labels. The authors demonstrate that SSVAEs possess a "bad" inductive bias for this task; they might cluster MNIST digits by stroke thickness (style) rather than digit identity (class) if labels are missing. In contrast, the GM-DGM forces the model to treat different clusters as distinct modes in the data distribution, providing a stronger "identity-first" bias.

Experimental Results

The authors evaluated their approach using three benchmarks: MNIST, Fashion-MNIST (F-MNIST), and Human Activity Recognition (HAR). In each case, labels were masked for half the classes entirely, while the remaining half received only 20% labeling.

Qualitative Failure of Standard Models

When a standard SSVAE is applied to the SUS regime (the "SUS accident" scenario), it fails to discover the unlabeled classes. Instead of generating new categories, the model produces blurry approximations or copies of the labeled classes.

Qualitative comparison of generated samples from standard semi-supervised models (SS) and those applied to the semi-unsupervised regime (SUS accident).

As seen in the figure above, the "SUS accident" models (b and d) struggle to represent the full diversity of the dataset compared to the standard SS models (a and c) that had access to all labels. The GM-DGM (not shown in this specific failure-case figure) was able to successfully generate distinct, recognizable images for classes it had never seen a label for.

Quantitative Performance and Entropy

The performance gap was stark across all datasets:

  • MNIST: GM-DGM reached 92.5% accuracy/purity, compared to the SSVAE's 54.3%.
  • Fashion-MNIST: GM-DGM achieved 78.2%, while SSVAE lagged at 38.7%.
  • HAR: GM-DGM nearly doubled the SSVAE's performance, reaching 87.1% vs 49.7%.

Analysis of the predictive entropy further illustrates why standard models fail. In the "SUS accident" regime, the model exhibits high uncertainty (entropy) when encountering unlabeled classes, as it lacks the structural bias to categorize them separately.

Kernel Density Estimation (KDE) of test-set predictive entropy for MNIST and Fashion-MNIST, illustrating model uncertainty across different training regimes.

Conclusion

The paper concludes that semi-unsupervised learning is a critical frontier for making deep generative models robust to real-world data biases. By employing models like the GM-DGM that prioritize an identity-first inductive bias, researchers can move beyond simple classification and toward true discovery of hidden categories within unlabeled data. Future work remains in scaling these methods to high-resolution natural images and addressing potential class imbalances in the "discovery" set.

Summary updated

Abstract

In semi-supervised learning for classification, it is assumed that every ground truth class of data is present in the small labelled dataset. Many real-world sparsely-labelled datasets are plausibly not of this type. It could easily be the case that some classes of data are found only in the unlabelled dataset -- perhaps the labelling process was biased -- so we do not have any labelled examples to train on for some classes. We call this learning regime $\textit{semi-unsupervised learning}$, an extreme case of semi-supervised learning, where some classes have no labelled exemplars in the training set. First, we outline the pitfalls associated with trying to apply deep generative model (DGM)-based semi-supervised learning algorithms to datasets of this type. We then show how a combination of clustering and semi-supervised learning, using DGMs, can be brought to bear on this problem. We study several different datasets, showing how one can still learn effectively when half of the ground truth classes are entirely unlabelled and the other half are sparsely labelled.
BibTeX
@misc{willetts2019semi,
  author = {Matthew Willetts and Stephen J. Roberts and Christopher C. Holmes},
  title = {{Semi-Unsupervised Learning with Deep Generative Models: Clustering and Classifying using Ultra-Sparse Labels}},
  year = {2019},
  eprint = {1901.08560},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1901.08560},
}
Figure