Dependency Clustering of Mixed Data with Gaussian Mixture Copulas.

Vaibhav Rajan, Sakyajit BhattacharyaIJCAI 2016 · OpenReview

Abstract

Clustering mixed-type data containing both continuous and discrete variables is a fundamental challenge in machine learning. Traditional methods often struggle to capture complex, non-linear dependencies or face computational bottlenecks. This paper presents the Extended Gaussian Mixture Copula Model (EGMCM), a semiparametric approach that leverages Gaussian Mixture Copulas and Extended Rank Likelihood to model dependencies flexibly across diverse data types. EGMCM decouples marginal distributions from dependency structures, allowing for robust clustering that scales linearly with the number of observations. Experimental results on synthetic and real-world datasets demonstrate that EGMCM consistently outperforms state-of-the-art competitors in terms of accuracy and efficiency.

Introduction

Real-world datasets are frequently "mixed," comprising a combination of continuous numerical features and discrete categorical or ordinal variables. Clustering such data is difficult because standard algorithms typically specialize in one data type. Distance-based methods often fail to account for feature correlations, while parametric models like Gaussian Mixture Models (GMMs) may impose restrictive distributional assumptions. The core challenge lies in modeling the underlying dependency structure—how features interact—without becoming computationally prohibitive when dealing with discrete data.

The EGMCM Framework

The authors introduce EGMCM to address these limitations. The model is built on two primary components:

  1. Gaussian Mixture Copulas (GMCM): Unlike meta-Gaussian models restricted to linear correlations, GMCMs can capture asymmetric relationships and "tail dependencies" (where variables correlate strongly only at extreme values). This allows the model to identify clusters based on complex interaction patterns.
  2. Extended Rank Likelihood (ERL): To handle mixed data, EGMCM employs ERL, which focuses on the relative ranks of observations rather than their exact values. By mapping data into a latent space based on these ranks, the algorithm creates a unified framework for continuous, binary, and ordinal data.

The EGMCM algorithm utilizes an Expectation-Maximization (EM) procedure in the latent space, coupled with a resampling step to handle rank constraints. This approach maintains a computational complexity of $O(tGnp^2)$, ensuring scalability to large datasets where previous methods like CoupledMC might fail due to cubic complexity relative to data dimensions.

Experimental Evaluation

The performance of EGMCM was evaluated against several baselines, including SCENIC, CoupledMC, and standard GMMs. The evaluation spanned 10 synthetic simulation settings and 10 UCI benchmark datasets, including Heart Disease, Breast Cancer Wisconsin, and Wine Quality.

Figure 1: Comparison of Adjusted Rand Index (ARI) across ten synthetic simulation settings. EGMCM (blue diamonds) consistently achieves higher accuracy than competing methods.

Results and Discussion

EGMCM demonstrated superior accuracy across both synthetic and real-world scenarios. In synthetic tests, it consistently led in Adjusted Mutual Information (AMI) and Adjusted Rand Index (ARI). On the UCI benchmarks, EGMCM achieved the highest scores in 7 out of 10 datasets, often by a significant margin.

A key strength of EGMCM is its robustness to the number of clusters ($G$). As shown in the sensitivity analysis, EGMCM maintains a performance lead even as the complexity of the clustering task increases.

Figure 6: Performance metrics (AMI and ARI) as a function of the number of clusters (G). EGMCM exhibits stable and superior performance compared to baselines across different cluster counts.

The efficiency of the rank-based approach is particularly evident when the ratio of observations to dimensions ($n/p$) is high, where the rank approximation becomes increasingly precise. Furthermore, the linear scaling with respect to the number of data points makes EGMCM a practical choice for large-scale mixed-data clustering compared to distance-based metrics that are slow to compute.

Conclusion and Future Work

EGMCM provides a powerful and efficient solution for dependency-based clustering of mixed data. By separating marginal distributions from dependency structures through copulas and utilizing rank-based likelihoods, it achieves high accuracy without the computational burden of traditional mixed-data models.

However, limitations remain: the current model requires features to be at least ordinal, struggling with nominal categories (e.g., "Color"). Future work may extend this framework to handle nominal data and explore parsimonious covariance structures to further improve performance in high-dimensional, sparse settings.

Summary updated

BibTeX
@inproceedings{rajan2016dependency,
  author = {Vaibhav Rajan and Sakyajit Bhattacharya},
  title = {{Dependency Clustering of Mixed Data with Gaussian Mixture Copulas.}},
  booktitle = {IJCAI},
  year = {2016},
  url = {https://openreview.net/forum?id=HjocKvGpVD1C},
}
Figure