Hierarchical Bayesian nonparametric mixture models for clustering with variable relevance determination

Christopher Yau, Chris HolmesBayesian 2011 · DOI

Abstract

We propose a hierarchical Bayesian nonparametric mixture model for clustering when some of the covariates are assumed to be of varying relevance to the clustering problem. This can be thought of as an issue in variable selection for unsupervised learning. We demonstrate that by defining a hierarchical population based nonparametric prior on the cluster locations scaled by the inverse covariance matrices of the likelihood we arrive at a 'sparsity prior' representation which admits a conditionally conjugate prior. This allows us to perform full Gibbs sampling to obtain posterior distributions over parameters of interest including an explicit measure of each covariate's relevance and a distribution over the number of potential clusters present in the data. This also allows for individual cluster specific variable selection. We demonstrate improved inference on a number of canonical problems.

Introduction

In unsupervised learning, identifying meaningful clusters in high-dimensional datasets remains a significant challenge. As the number of covariates increases, the "curse of dimensionality" often obscures the underlying structure, as irrelevant variables (noise) can drown out the signal from relevant ones. While variable selection is well-established in supervised learning, it is inherently more complex in unsupervised settings where group labels are unknown. This paper, originally published in Bayesian Analysis by Christopher Yau and Chris Holmes, introduces a unified probabilistic framework to simultaneously determine the number of clusters and identify the specific variables that define them.

Methodology

The authors extend the standard Gaussian Mixture Model (GMM) through two primary Bayesian innovations:

1. Nonparametric Clustering via Dirichlet Processes

Traditional mixture models require the user to specify the number of clusters ($K$) a priori. To circumvent this, the authors employ a Dirichlet Process (DP) prior. This nonparametric approach allows the model to treat the number of clusters as a random variable, effectively "discovering" the optimal number of groups based on the data's complexity.

2. Variable Relevance and the Sparsity Prior

The core contribution of the work is the mechanism for Variable Relevance Determination (VRD). The authors place a hierarchical prior on the cluster locations (means). Specifically, they utilize a double-exponential (Laplace) prior on the standardized distances between cluster centers. This creates a "sparsity prior" that mimics the behavior of the LASSO penalty in frequentist statistics.

If a variable does not contribute to the clustering structure, the prior shrinks the differences between cluster means for that variable toward zero. Conversely, for relevant variables, the model allows the means to remain distinct. A key advantage of this formulation is its conditional conjugacy, which facilitates efficient posterior inference via Gibbs sampling. Furthermore, the model supports cluster-specific variable selection, acknowledging that a feature might be relevant for identifying one specific sub-population but irrelevant for others.

Experimental Evaluation

The model's performance was validated across synthetic and empirical datasets:

  • Simulated Data: In scenarios involving 80 synthetic datasets with varying noise levels and cluster overlaps, the model consistently identified the correct number of clusters and accurately distinguished between signal and noise variables.
  • Benchmark Datasets: Using the classic Iris and Leptograpsus crabs datasets, the authors added artificial noise variables. The model successfully ignored these "fake" covariates while correctly identifying the known biological classes.
  • Cancer Genomics: The model was applied to a high-dimensional SNP-CGH dataset from 80 patients with B-cell chronic lymphocytic leukaemia (B-CLL). It identified 10 clusters, three of which aligned with known genetic subtypes. The variable relevance output automatically pinpointed specific genomic regions on chromosomes 11, 12, and 17, which are established sites for mutations associated with the disease.

Discussion and Limitations

The hierarchical Bayesian approach provides a robust, scale-invariant method for unsupervised feature selection. By providing a continuous measure of relevance rather than a binary inclusion/exclusion, it allows researchers to rank variables by their importance to the clustering structure.

However, the reliance on Markov Chain Monte Carlo (MCMC) methods like Gibbs sampling presents a computational bottleneck. For datasets with thousands of points, run-times can reach an hour, suggesting that scaling to larger datasets may require variational approximations. Additionally, the model is primarily "mean-centric"; while it excels at finding clusters that differ in location, it may be less sensitive to clusters that differ only in variance or shape.

Conclusion

Yau and Holmes demonstrate that by integrating nonparametric priors with sparsity-inducing hierarchical structures, it is possible to solve the dual problem of clustering and variable selection within a single, coherent Bayesian framework. This approach offers a powerful tool for exploratory data analysis in fields where high-dimensional noise is prevalent.

Summary updated

Abstract

We propose a hierarchical Bayesian nonparametric mixture model for clustering when some of the covariates are assumed to be of varying relevance to the clustering problem. This can be thought of as an issue in variable selection for unsupervised learning. We demonstrate that by defining a hierarchical population based nonparametric prior on the cluster locations scaled by the inverse covariance matrices of the likelihood we arrive at a 'sparsity prior' representation which admits a conditionally conjugate prior. This allows us to perform full Gibbs sampling to obtain posterior distributions over parameters of interest including an explicit measure of each covariate's relevance and a distribution over the number of potential clusters present in the data. This also allows for individual cluster specific variable selection. We demonstrate improved inference on a number of canonical problems.
BibTeX
@inproceedings{yau2011hierarchical,
  author = {Christopher Yau and Chris Holmes},
  title = {{Hierarchical Bayesian nonparametric mixture models for clustering with variable relevance determination}},
  booktitle = {Bayesian},
  year = {2011},
  doi = {10.1214/11-ba612},
}
Figure