The LORACs prior for VAEs: Letting the Trees Speak for the Data

Sharad Vikram, Matthew D. Hoffman, Matthew J. Johnson

Abstract

In variational autoencoders, the prior on the latent codes $z$ is often treated as an afterthought, but the prior shapes the kind of latent representation that the model learns. If the goal is to learn a representation that is interpretable and useful, then the prior should reflect the ways in which the high-level factors that describe the data vary. The "default" prior is an isotropic normal, but if the natural factors of variation in the dataset exhibit discrete structure or are not independent, then the isotropic-normal prior will actually encourage learning representations that mask this structure. To alleviate this problem, we propose using a flexible Bayesian nonparametric hierarchical clustering prior based on the time-marginalized coalescent (TMC). To scale learning to large datasets, we develop a new inducing-point approximation and inference algorithm. We then apply the method without supervision to several datasets and examine the interpretability and practical performance of the inferred hierarchies and learned latent space.

Introduction

In standard Variational Autoencoders (VAEs), the latent variables $z$ are typically assumed to follow an isotropic Normal distribution. This "blob" prior acts as a featureless cloud in the center of the latent space. While mathematically convenient, it imposes a strong inductive bias: it assumes that all factors of variation are independent and smoothly varying.

However, real-world data often possesses inherent hierarchical or discrete structures. Handwritten digits naturally cluster into ten classes; human faces exhibit correlated features like pose and expression; and linguistic data follows complex evolutionary paths. When a VAE is forced to map such structured data into a simple Gaussian prior, it must "mask" these natural relationships, often resulting in less interpretable and less useful latent representations. The LORACs (Latent ORganization of Arboreal Clusters) framework addresses this by introducing a prior that can explicitly represent hierarchy while remaining computationally scalable.

The LORACs Model

The core of the LORACs approach is the replacement of the Gaussian prior with the Time-Marginalized Coalescent (TMC). The TMC is a Bayesian nonparametric model that defines a distribution over rooted binary trees (phylogenies) with associated branch lengths. This allows the model to learn a hierarchical organization of the latent space.

Comparison of the standard VAE graphical model (a) with the proposed LORACs VAE (b) and its variational factorization (c).

To integrate this complex prior with deep learning architectures, the authors introduce two primary innovations:

  1. Inducing Point Approximation: Standard TMC inference scales poorly with the number of data points $N$. Borrowing from Gaussian Process literature, LORACs uses a small set of $M$ "inducing points" ($M \ll N$) that serve as the leaves of the hierarchical tree. Each data point $x_n$ is then treated as an independent sample from the tree's posterior predictive distribution, allowing the model to scale to large datasets.
  2. Hybrid Inference Scheme: The model optimizes an Evidence Lower Bound (ELBO) using a combination of stochastic gradient descent (for neural network weights and inducing point locations) and Metropolis-Hastings moves (specifically Subtree-Prune-and-Regraft) to explore the discrete space of tree topologies.

Experimental Setup

The model was evaluated on three benchmark image datasets:

  • MNIST: Handwritten digits, testing basic clustering.
  • Omniglot: Characters from many alphabets with few examples per class, ideal for testing hierarchical structure.
  • CelebA: Large-scale celebrity face images with diverse attributes.

Results and Discussion

LORACs demonstrates that an "opinionated" hierarchical prior leads to significantly more structured latent spaces. The learned inducing points act as "prototypes" or archetypal versions of the data. When decoded, these points represent clean summaries of the dataset, such as specific digit styles or facial orientations.

Visualization of the latent space organized by the LORACs prior. The contours represent the density defined by the hierarchical tree structure, showing how data points are grouped into arboreal clusters.

Quantitative Performance

The benefits of this hierarchical organization are evident in downstream tasks:

  • Few-Shot Classification: On the Omniglot dataset with only 5 labels per class, LORACs achieved 33.0% accuracy, a substantial improvement over the 18.7% achieved by a standard Normal prior.
  • Information Retrieval: The model's ability to preserve semantic relationships was measured by how well Euclidean distance in the latent space corresponds to class labels. LORACs achieved an AUC of 0.626 on MNIST, nearly doubling the performance of the standard VAE (0.317).

Interpretability

A unique feature of the TMC prior is the ability to "slice" the hierarchy at different time levels. Slicing near the root provides a coarse, high-level view of the data, while slicing near the leaves reveals fine-grained clusters. This allows for unsupervised discovery of meaningful hierarchies entirely without label information.

Conclusion

While LORACs significantly improves representation quality and interpretability, it involves a trade-off. The model achieves "middling" held-out log-likelihood scores compared to some density-focused priors, suggesting that optimal density estimation does not always coincide with the most useful latent representations. Future work may explore interactive unsupervised learning, where users can provide high-level constraints to guide the formation of the hierarchy.

Summary updated

Abstract

In variational autoencoders, the prior on the latent codes $z$ is often treated as an afterthought, but the prior shapes the kind of latent representation that the model learns. If the goal is to learn a representation that is interpretable and useful, then the prior should reflect the ways in which the high-level factors that describe the data vary. The "default" prior is an isotropic normal, but if the natural factors of variation in the dataset exhibit discrete structure or are not independent, then the isotropic-normal prior will actually encourage learning representations that mask this structure. To alleviate this problem, we propose using a flexible Bayesian nonparametric hierarchical clustering prior based on the time-marginalized coalescent (TMC). To scale learning to large datasets, we develop a new inducing-point approximation and inference algorithm. We then apply the method without supervision to several datasets and examine the interpretability and practical performance of the inferred hierarchies and learned latent space.
BibTeX
@inproceedings{vikram2019loracs,
  author = {Sharad Vikram and Matthew D. Hoffman and Matthew J. Johnson},
  title = {{The LORACs prior for VAEs: Letting the Trees Speak for the Data}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2019},
  url = {https://openreview.net/forum?id=IoU451tLms},
}
Figure