Introduction
In the landscape of modern machine learning, semi-supervised learning (SSL) has long been a cornerstone for leveraging large amounts of unlabeled data alongside a small set of labeled examples. However, SSL typically operates under a significant assumption: that the labeled set contains at least a few examples of every class present in the unlabeled pool.
This paper introduces and formalizes a more realistic but challenging regime termed Semi-Unsupervised Learning (SUS). In this scenario, some classes are sparsely labeled, while other "hidden" classes are entirely unlabeled. This mirrors real-world data collection biases, such as medical datasets where rare pathologies may exist in a massive unlabeled corpus but never appear in the limited labeled subset provided by a single clinic.
The Challenge of "Hidden" Classes
The SUS problem presents a twofold challenge. First, the model must accurately classify the sparsely labeled classes. Second, it must cluster the entirely unlabeled classes into distinct, meaningful groups. Standard semi-supervised models often fail here because they are designed to force all incoming data into the known, labeled categories. When confronted with a new, unlabeled class, these models either "hallucinate" them into known categories or hide their unique identity within the "style" parameters of the latent space, effectively leaving the classifier blind to their existence.
Methodology: GM-DGM vs. SSVAE
The researchers investigated two primary architectures based on Variational Autoencoders (VAEs):
- SSVAE (Semi-Supervised VAE): A standard model that utilizes a discrete latent variable ($y$) for class labels and a continuous latent variable ($z$) for "style."
- GM-DGM (Gaussian Mixture Deep Generative Model): This model treats the latent space as a mixture of Gaussians, where each component corresponds to a potential class.
The core of the paper's argument rests on Inductive Bias. For SUS to be successful, a model must naturally favor clustering data by identity even in the absence of labels. The authors demonstrate that SSVAEs possess a "bad" inductive bias for this task; they might cluster MNIST digits by stroke thickness (style) rather than digit identity (class) if labels are missing. In contrast, the GM-DGM forces the model to treat different clusters as distinct modes in the data distribution, providing a stronger "identity-first" bias.
Experimental Results
The authors evaluated their approach using three benchmarks: MNIST, Fashion-MNIST (F-MNIST), and Human Activity Recognition (HAR). In each case, labels were masked for half the classes entirely, while the remaining half received only 20% labeling.
Qualitative Failure of Standard Models
When a standard SSVAE is applied to the SUS regime (the "SUS accident" scenario), it fails to discover the unlabeled classes. Instead of generating new categories, the model produces blurry approximations or copies of the labeled classes.
As seen in the figure above, the "SUS accident" models (b and d) struggle to represent the full diversity of the dataset compared to the standard SS models (a and c) that had access to all labels. The GM-DGM (not shown in this specific failure-case figure) was able to successfully generate distinct, recognizable images for classes it had never seen a label for.
Quantitative Performance and Entropy
The performance gap was stark across all datasets:
- MNIST: GM-DGM reached 92.5% accuracy/purity, compared to the SSVAE's 54.3%.
- Fashion-MNIST: GM-DGM achieved 78.2%, while SSVAE lagged at 38.7%.
- HAR: GM-DGM nearly doubled the SSVAE's performance, reaching 87.1% vs 49.7%.
Analysis of the predictive entropy further illustrates why standard models fail. In the "SUS accident" regime, the model exhibits high uncertainty (entropy) when encountering unlabeled classes, as it lacks the structural bias to categorize them separately.
Conclusion
The paper concludes that semi-unsupervised learning is a critical frontier for making deep generative models robust to real-world data biases. By employing models like the GM-DGM that prioritize an identity-first inductive bias, researchers can move beyond simple classification and toward true discovery of hidden categories within unlabeled data. Future work remains in scaling these methods to high-resolution natural images and addressing potential class imbalances in the "discovery" set.
Summary updated