Abstract
A mixture of Gaussians fit to a single curved or heavy-tailed cluster will report that the data contains many clusters. To produce more appropriate clusterings, we introduce a model which warps a latent mixture of Gaussians to produce nonparametric cluster shapes. The possibly low-dimensional latent mixture model allows us to summarize the properties of the high-dimensional clusters (or density manifolds) describing the data. The number of manifolds, as well as the shape and dimension of each manifold is automatically inferred. We derive a simple inference scheme for this model which analytically integrates out both the mixture parameters and the warping function. We show that our model is effective for density estimation, performs better than infinite Gaussian mixture models at recovering the true number of clusters, and produces interpretable summaries of high-dimensional datasets.
Introduction
Traditional unsupervised learning often relies on Gaussian Mixture Models (GMMs) for clustering. However, GMMs assume that clusters follow rigid parametric densities, typically ellipsoidal. When faced with real-world data containing curved or complex structures—such as spirals or "banana" shapes—standard GMMs tend to over-partition a single coherent structure into multiple smaller components.
To address this, we propose the Infinite Warped Mixture Model (iWMM). This model bridges the gap between manifold learning and probabilistic clustering by learning a nonlinear transformation that maps simple latent densities into complex observed manifolds. By doing so, it can automatically infer the number, shape, and intrinsic dimensionality of non-Gaussian clusters.
Methodology
The iWMM operates by performing clustering in a low-dimensional "simple" latent space and mapping those results into the "complex" observed space via a warping function.
- Latent Space: The model assumes data points originate in a latent space where they follow an Infinite Gaussian Mixture Model (iGMM). This allows the model to represent clusters as simple ellipsoids initially.
- Warping Function: A nonlinear mapping function $f$, sampled from a Gaussian Process (GP), transforms the latent coordinates into the observed space. This warping allows the model to smoothly bend and stretch the latent Gaussians into arbitrary nonparametric shapes.
- Inference: We employ a Markov chain Monte Carlo (MCMC) approach, specifically using Collapsed Gibbs Sampling for cluster assignments and Hybrid Monte Carlo (HMC) for latent coordinates and kernel hyperparameters. A key technical contribution is the analytical integration of both the mixture parameters and the warping function, which significantly enhances inference efficiency.
This framework can be viewed as a generalization of the Gaussian Process Latent Variable Model (GPLVM), extending it from a single warped Gaussian to a warped mixture.
Experimental Results
Synthetic Benchmarks
The model was evaluated on several synthetic datasets designed to challenge standard clustering algorithms: "2-curve", "3-semi", "2-circle", and "Pinwheel".
On the "2-curve" dataset, the iWMM achieved an average Rand Index of 0.86, a substantial improvement over the 0.52 achieved by standard infinite GMMs. The model successfully recognized that complex 10D structures often lie on low-dimensional curved manifolds.
Real-World Performance
- Density Estimation: The iWMM consistently outperformed Kernel Density Estimation (KDE) and standard iGMMs in test log-likelihood across UCI benchmarks, including the Iris, Glass, Wine, and Vowel datasets. For example, on the Glass dataset, iWMM achieved a score of 5.76 compared to KDE's 1.26.
- Face Manifolds: Applied to the UMIST face dataset, the model recovered 1D manifolds representing head-turning motions. It effectively grouped images of different individuals based on pose in the latent space, demonstrating its ability to produce interpretable summaries of high-dimensional data.
Discussion and Future Work
While the iWMM offers significant flexibility, it inherits the $O(N^3)$ computational complexity of Gaussian Processes due to covariance matrix inversion. This currently limits its application to datasets of a few thousand points. Furthermore, the model can struggle with certain complex topologies, such as perfectly closed loops.
Future research will focus on integrating sparse GP approximations to scale the model to larger datasets and exploring more complex latent structures using Hierarchical Dirichlet Processes or Dirichlet Diffusion Trees.
Summary updated