Abstract
Most clustering algorithms produce a single clustering solution. This is inadequate for many data sets that are multi-faceted and can be grouped and interpreted in many different ways. Moreover, for high-dimensional data, different features may be relevant or irrelevant to each clustering solution, suggesting the need for feature selection in clustering. Features relevant to one clustering interpretation may be different from the ones relevant for an alternative interpretation or view of the data. In this paper, we introduce a probabilistic nonparametric Bayesian model that can discover multiple clustering solutions from data and the feature subsets that are relevant for the clusters in each view. In our model, the features in different views may be shared and therefore the sets of relevant features are allowed to overlap. We model feature relevance to each view using an Indian Buffet Process and the cluster membership in each view using a Chinese Restaurant Process. We provide an inference approach to learn the latent parameters corresponding to this multiple partitioning problem. Our model not only learns the features and clusters in each view but also automatically learns the number of clusters, number of views and number of features in each view.
Introduction
Standard clustering algorithms typically aim to find a single "best" partition of a dataset. However, real-world data is often multi-faceted. For instance, a collection of face images can be meaningfully grouped by the subject's identity or by their pose. Traditional "multiple clustering" approaches often assume that the features relevant to one view must be entirely distinct from those relevant to another. This "disjoint feature" assumption is often unrealistic; many features (e.g., specific pixels in an image) may contain information pertinent to multiple interpretations. Forcing these feature sets to be disjoint can degrade performance and lead to unnatural groupings.
The NBMC-OFV Model
To address these limitations, we propose NBMC-OFV (Nonparametric Bayesian Model for Multiple Clustering with Overlapping Feature Views). The core innovation lies in allowing different clustering "views" to share overlapping sets of features, more accurately reflecting the complexity of real-world data.
The model leverages two nonparametric stochastic processes to automate complexity control:
- Indian Buffet Process (IBP): This determines which features belong to which view. Because the IBP allows a single feature to be associated with multiple views, it naturally supports overlapping feature sets and automatically determines the total number of views.
- Chinese Restaurant Process (CRP): Within each discovered view, a CRP groups data instances into clusters. This allows the model to learn the optimal number of clusters for each specific view without prior specification.
The model treats the data as a multiplicative mixture of views, where the likelihood of an observation is the product of its likelihoods across all discovered views. Inference is performed using Gibbs sampling to iteratively assign features to views and instances to clusters.
Experimental Results
The model was evaluated against several state-of-the-art baselines, including CRP-CRP (which assumes disjoint features) and D-Kmeans.
Synthetic and Real-World Performance
On synthetic data with heavy feature overlap, NBMC-OFV achieved a Normalized Mutual Information (NMI) of 0.59, significantly outperforming the disjoint-feature CRP-CRP model (0.34). This confirms that modeling feature overlap is crucial when the underlying data structure is complex.
In real-world applications, NBMC-OFV consistently demonstrated superior performance across diverse domains, including computer vision, web mining, and industrial acoustics.
As shown in the table above, the model excels at identifying multiple valid partitions. For the Face dataset, it successfully recovered both identity (NMI 0.89) and pose (NMI 0.57). In the Machine Sound domain, it accurately separated overlapping sound sources like pumps, motors, and fans.
Text Modeling
For large-scale text datasets like NSF Abstracts and NYTimes, the model achieved lower perplexity scores than standard models. This indicates that NBMC-OFV is better at capturing the multi-faceted nature of documents—for example, recognizing that a research abstract has both a primary "topic" and a distinct "methodological type."
Discussion
The primary advantage of NBMC-OFV is its ability to handle feature overlap, which significantly improves clustering accuracy. Furthermore, the use of nonparametric priors (IBP and CRP) allows the model to scale its complexity to the data, automatically discovering the number of views, clusters, and the size of feature subsets. The Bayesian framework also provides an implicit redundancy penalty, preventing the creation of unnecessary views unless they provide a significantly better fit.
While the current implementation using Gibbs sampling is effective for datasets of moderate size, future work will focus on scaling the model to larger data points using variational inference techniques. Additionally, the framework could be extended to handle non-Gaussian data distributions, such as categorical or graph-based data.
Summary updated