Crowdclustering with Partition Labels

Junxiang Chen, Yale Chang, Peter J. Castaldi, Michael H. Cho, Brian Hobbs, Jennifer G. DyAISTATS 2018 · OpenReview

Abstract

Crowdclustering is a practical way to incorporate domain knowledge into clustering by combining opinions from multiple domain experts. Existing crowdclustering methods often rely on binary pairwise similarity labels, which can be inefficient and lose context. This paper introduces a novel crowdclustering model that aggregates expert opinions provided as partition labels (e.g., "Sample A belongs to Group 1"). By using a modified multinomial logistic regression with a "rich-get-richer" property, the model automatically determines the optimal number of clusters and provides a transparent mapping of how different experts agree or disagree.

What Problem Does This Paper Address?

Clustering is often subjective. In complex fields like medicine, different experts might look at the same data and see different patterns based on their specific goals. Crowdclustering aims to combine these diverse expert opinions into a single consensus.

Most existing methods rely on pairwise similarity labels (asking experts: "Do these two samples belong together?"). However, in the real world, experts often provide partition labels (assigning samples to specific categories). While you can convert partitions into pairs, you lose vital information: you can no longer easily see why experts disagree or how their specific categories map to the final consensus. Furthermore, most clustering algorithms require you to guess the number of clusters ($K$) beforehand, which is difficult when experts don't even agree on how many groups exist.

Methods

The authors propose a probabilistic, discriminative clustering model designed to handle partition labels directly.

Mapping expert partition labels to a consensus clustering.

The architecture consists of two main parts:

  1. Modified Multinomial Logistic Regression (MLR): Unlike standard MLR, the authors add a specific penalty term ($\lambda w_k^T w_k$) to the loss function. This creates a "rich-get-richer" effect inspired by the Dirichlet Process. The model starts with a large number of potential clusters and, during training, "rewards" clusters that already have many members. This allows the model to automatically prune empty clusters and converge on the "natural" number of groups in the data.
  2. Expert Label Mapping (BCC Variant): To handle expert opinions, the model uses a variant of the Bayesian Classifier Combination (BCC). Instead of just looking at error rates, it learns a mapping between the latent (discovered) clusters and the labels provided by each expert. This reveals the "logic" of each expert—for example, showing that Expert A combined two of the model's clusters into one category.

The model is trained using Variational Expectation Maximization (VEM), an efficient way to approximate complex probability distributions.

Key Insights and Ideas

  • Discriminative over Generative: Most "automatic $K$" models use generative approaches (like Dirichlet Process Mixture Models), which make heavy assumptions about data distribution. By using a discriminative (regression-based) approach, this model is more robust to "messy" real-world data.
  • The "Aha" Moment on Disagreement: The model doesn't just treat expert disagreement as "noise." By learning the mapping, it can explicitly show that Expert A is a "splitter" (creating many small groups) while Expert B is a "lumper" (creating few large groups), or even identify if an expert simply flipped their label names.
  • Hyper-plane Partitioning: Because the model is based on logistic regression, it effectively learns linear hyper-planes to separate clusters, making the resulting boundaries easier to visualize and interpret than "black-box" clustering methods.

Key Results

The model was evaluated on UCI benchmarks, face recognition datasets, and a real-world COPD dataset.

Validation NMI performance on the warpAR10P dataset across different percentages of observed labels.
  • Automatic Cluster Discovery: In benchmark tests, the model consistently recovered the ground-truth number of clusters (e.g., finding exactly 3 clusters for the iris and seeds datasets) without being told the answer in advance.
  • Superior Accuracy: Measured by Normalized Mutual Information (NMI), the proposed method outperformed or matched state-of-the-art competitors like KCC and MECC, especially when expert labels were sparse (only available for 10-30% of the data).
  • Biological Relevance: In the COPD study, the model discovered 4 clusters. These clusters were shown to be statistically significant across four different genetic risk scores, suggesting the model found biologically meaningful subtypes of the disease that correlate with underlying DNA patterns.

Limitations and Future Work

  • Linear Boundaries: The current model uses linear logistic regression, meaning it assumes clusters can be separated by straight lines (hyper-planes). It might struggle with complex, non-linear "manifold" data structures unless the features are pre-transformed.
  • Sensitivity to Noise: The authors noted that on certain noisy datasets, the model tended to overestimate the number of clusters, mistaking noise for small, distinct groups.
  • Future Directions: A natural extension would be incorporating non-linear kernels or deep learning architectures to handle more complex feature spaces while maintaining the "rich-get-richer" cluster pruning logic.

Summary updated

Abstract

Crowdclustering is a practical way to incorporate domain knowledge into clustering, by combining opinions from multiple domain experts. Existing crowdclustering methods analyze binary pairwise simi...
BibTeX
@inproceedings{chen2018crowdclustering,
  author = {Junxiang Chen and Yale Chang and Peter J. Castaldi and Michael H. Cho and Brian Hobbs and Jennifer G. Dy},
  title = {{Crowdclustering with Partition Labels}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2018},
  url = {https://openreview.net/forum?id=PMk3hcn3Tsp},
}
Figure