Abstract
In this paper, we investigate the algorithmic stability of unsupervised representation learning with deep generative models, as a function of repeated re-training on the same input data. Algorithms for learning low dimensional linear representations -- for example principal components analysis (PCA), or linear independent components analysis (ICA) -- come with guarantees that they will always reveal the same latent representations (perhaps up to an arbitrary rotation or permutation). Unfortunately, for non-linear representation learning, such as in a variational auto-encoder (VAE) model trained by stochastic gradient descent, we have no such guarantees. Recent work on identifiability in non-linear ICA have introduced a family of deep generative models that have identifiable latent representations, achieved by conditioning on side information (e.g. informative labels). We empirically evaluate the stability of these models under repeated re-estimation of parameters, and compare them to both standard VAEs and deep generative models which learn to cluster in their latent space. Surprisingly, we discover side information is not necessary for algorithmic stability: using standard quantitative measures of identifiability, we find deep generative models with latent clusterings are empirically identifiable to the same degree as models which rely on auxiliary labels. We relate these results to the possibility of identifiable non-linear ICA.
Introduction: The Challenge of Identifiability
In representation learning, "identifiability" refers to a model's ability to consistently recover the same underlying latent factors across different training runs. For linear methods like PCA, this is mathematically guaranteed. However, non-linear models like Variational Autoencoders (VAEs) struggle with this. Because deep neural networks are universal function approximators, they can "unmix" data in an infinite number of ways, leading to latent spaces that vary significantly between runs.
Recent advancements, such as the Identifiable VAE (iVAE), addressed this by incorporating auxiliary "side information" ($u$)—such as class labels or timestamps—to constrain the model toward a consistent solution. This paper investigates whether such side information is a strict requirement or if unsupervised models can achieve similar stability through internal structuring.
Methods and Experimental Setup
The study compares three distinct model architectures across various complexities (MLPs, ConvNets, and ResNets):
- iVAE (Supervised): Utilizes ground-truth labels ($u$) to structure the latent space, serving as the theoretical benchmark.
- VaDE (Unsupervised Clustering): A "Variational Deep Embedding" model that learns its own latent clustering ($u$) rather than relying on external labels.
- Vanilla VAE (Unsupervised): A standard control model with no explicit clustering or side information.
Identifiability is quantified using the Mean Correlation Coefficient (MCC), measuring the correlation between latent representations across multiple training runs with different random seeds. Evaluations were performed on synthetic data with known ground-truth latents and standard image benchmarks (MNIST, CIFAR10, SVHN).
Key Insights: Breaking Symmetry Without Labels
The core finding is that a model's own inductive biases—architecture, optimizer, and the mathematical form of the Evidence Lower Bound (ELBO)—can act as natural constraints. By allowing a model to learn a mixture of Gaussians in its latent space (clustering), it effectively generates its own internal "side information."
As shown in the figure above, latent factors can be partitioned by different attributes. While iVAE uses external labels to define these partitions, VaDE discovers them autonomously. The authors found that these models remain empirically stable even when theoretical conditions (like the number of latent dimensions relative to classes) are violated.
Results
- Performance Parity: Statistical analysis (Wilcoxon signed-rank tests) revealed no significant difference in identifiability between the unsupervised VaDE and the supervised iVAE. VaDE consistently matched or exceeded the stability of the theoretical gold standard.
- Baseline Stability: Standard VAEs exhibited a surprising degree of empirical identifiability, though they were consistently less stable than clustering-based models.
- Architectural Robustness: These results were consistent across different network architectures, from simple MLPs to complex ResNets, suggesting that stability is a fundamental property of the learning objective.
Discussion and Future Work
While the empirical results are robust, a theoretical gap remains: current mathematical proofs suggest unsupervised non-linear ICA should not be identifiable, yet these models clearly are. Future research is needed to bridge this gap and understand which specific inductive biases contribute most to stability. Additionally, scaling these unsupervised clustering approaches to massive, uncurated datasets remains an open challenge.
Summary updated