Abstract
In variational autoencoders, the prior on the latent codes $z$ is often treated as an afterthought, but the prior shapes the kind of latent representation that the model learns. If the goal is to learn a representation that is interpretable and useful, then the prior should reflect the ways in which the high-level factors that describe the data vary. The “default” prior is a standard normal, but if the natural factors of variation in the dataset exhibit discrete structure or are not independent, then the isotropic-normal prior will actually encourage learning representations that mask this structure. To alleviate this problem, we propose using a flexible Bayesian nonparametric hierarchical clustering prior based on the time-marginalized coalescent (TMC). To scale learning to large datasets, we develop a new inducing-point approximation and inference algorithm. We then apply the method without supervision to several datasets and examine the interpretability and practical performance of the inferred hierarchies and learned latent space.
The Problem: When Priors Mask Structure
In a Variational Autoencoder (VAE), the prior distribution over the latent space dictates how the model organizes the data it encodes. The "default" choice is an isotropic normal distribution ($\mathcal{N}(0, I)$). While mathematically convenient, this prior assumes that the underlying factors of variation in the data are completely independent and smoothly varying.
However, real-world data rarely behaves this way. Natural factors often exhibit discrete, non-smooth structures (like distinct clusters of handwritten digits) or dependent relationships (like the correlation between gender and facial hair length). If the prior forces a smooth, independent geometry, the VAE will actively mask these natural structures. While highly flexible priors (like normalizing flows) exist, they focus on density estimation rather than providing an interpretable organizational bias. The challenge lies in injecting a rich, interpretable hierarchical clustering prior into a VAE while keeping inference computationally tractable for large datasets.
Methods: Scaling the Time-Marginalized Coalescent
The authors propose using the Time-Marginalized Coalescent (TMC), a powerful Bayesian nonparametric model for hierarchical clustering (phylogenies). However, a standard TMC requires a tree with $N$ leaves for $N$ data points. This makes inference computationally disastrous for large datasets, as message-passing scales linearly with dataset size and tree-sampling algorithms mix very slowly.
To solve this, the authors introduce the LORACs (Latent ORganization of Arboreal Clusters) prior. The key innovation is an inducing-point approximation:
- Inducing Points: Instead of building a tree over the entire dataset, LORACs learns a fixed number of $M$ "inducing points" ($s_{1:M}$, where $M \ll N$) that act as the leaves of the TMC tree in the latent space.
- Decoupling Complexity: The actual data points ($x_n$) are modeled as conditionally independent draws from the predictive distribution of this smaller, induced tree.
- Scalable Inference: Because the tree size is capped at $M$, the model decouples the tree complexity from the dataset size. This allows the VAE to be trained using standard minibatch-based stochastic gradient descent, while the tree structure over the inducing points is sampled using Markov chain Monte Carlo (MCMC) techniques.
Key Insights: Trees Speak for the Data
By forcing the latent space to organize hierarchically, the model naturally discovers discrete classes and sub-features without supervision.
- Controllable Granularity: The TMC prior assigns "times" to the branches of the tree. Slicing the learned tree at different times provides a controllable granularity of the data. Nodes that coalesce "recently" are highly similar, while older nodes represent broader dataset averages.
- Inducing Points as Prototypes: The learned inducing points act as distinct, representative summaries of the dataset. When passed through the VAE's decoder, they produce much cleaner and more unique images than the pseudo-inputs learned by competing methods like VampPrior.
- Density vs. Utility: A highly flexible prior like Masked Autoregressive Flow (MAF) achieves better held-out log-likelihoods (better pure density estimation). However, LORACs produces a much more useful latent space for downstream tasks, reinforcing the idea that simply fitting a distribution perfectly doesn't always yield a meaningful representation.
Key Results
The model was evaluated on MNIST, Omniglot, and CelebA. The authors compared LORACs against several baselines: a standard Normal prior, VampPrior, DVAE#, and MAF.
- Few-Shot Classification: LORACs consistently outperformed baselines when training a classifier on a small number of labeled latent vectors. The advantage was particularly massive on the highly multi-class Omniglot dataset.
- Information Retrieval: To test if Euclidean distances in the latent space were actually meaningful, they measured precision-recall AUC for nearest-neighbor queries. LORACs achieved the highest AUC by a wide margin (e.g., 0.626 on MNIST vs. 0.317 for the Normal prior).
- Log-Likelihood Trade-off: LORACs achieved middling held-out log-likelihoods (e.g., -83.40 on MNIST), beating the standard Normal prior (-83.78) but falling short of the pure density-estimating MAF (-80.12).
Conclusion
LORACs demonstrates that imposing a strong structural inductive bias trades pure density estimation performance for interpretability and downstream utility. By learning discrete, hierarchical structures, the model opens the door for interactive deep unsupervised learning, where users could potentially provide instance-level constraints to guide the representation learning process.
Summary updated