The DBCV index is more informative than DCSI, CDbw, and VIASCKDE indices for unsupervised clustering internal assessment of concave-shaped and density-based clusters

Davide Chicco, Giuseppe Sabino, Luca Oneto, Giuseppe JurmanPeerJ Comput. Sci 2025 · DOI

TL;DR

This paper introduces and validates the DBCV index as a superior metric for evaluating unsupervised clustering, particularly for complex, non-convex shapes. By testing DBCV against other density-based clustering metrics on both artificial and real-world medical datasets, the authors demonstrate its robustness and reliability in scenarios where traditional metrics falter.

What Problem Does This Paper Address?

Unsupervised clustering is a powerful tool for discovering hidden patterns in data, but evaluating the quality of the resulting clusters is a significant challenge. Unlike supervised learning, there's no "ground truth" to compare against. While various internal clustering metrics exist (like Silhouette, Davies-Bouldin, etc.), they primarily work well for well-separated, convex-shaped clusters. They often fail or provide misleading results when dealing with concave-shaped, nested, or density-based clusters, which are common in real-world data. This paper addresses this gap by focusing on density-based clustering validation and comparing the effectiveness of the DBCV index against other density-aware metrics (DCSI, CDbw, VIASCKDE) in these challenging scenarios.

Methods

The core of this paper is a comparative evaluation of clustering validation metrics. The authors introduce and mathematically define the DBCV index, highlighting its density-based approach. They then propose a novel validation methodology:

  1. Artificial Data Generation: They create synthetic datasets with varying shapes (half-moons, shifting circles, sparse circles, tulip shapes) and progressively increase noise levels. The hypothesis is that as noise increases, clustering quality deteriorates, and reliable metrics should reflect this decline.
  2. Real-World Data Validation: They use five diverse electronic health record (EHR) datasets. Here, the validation approach involves perturbing clustering algorithm hyperparameters (e.g., epsilon, minimum points for DBSCAN/HDBSCAN) from default settings to random ones. The trend of each metric is then compared against the trend of the Adjusted Rand Index (ARI), which serves as a proxy for ground truth when comparing different clustering outputs. A metric is considered consistent if its trend aligns with the ARI's trend.

The paper also lists available open-source software packages in R and Python for implementing these metrics.

Datasets

The study utilizes a combination of artificial and real-world datasets:

  • Artificial Datasets:
    • Half Moons: Datasets with two interlocking crescent shapes, varying noise levels.
  • Shifting Circles: Datasets with an outer and a progressively shifted inner circle, varying overlap and noise.
  • Sparse Circles: Datasets with two concentric circles, increasing noise.
  • Tulip Shapes: A more complex dataset built on crescent shapes with added noise.
  • Real-World Datasets (Electronic Health Records - EHRs):
    • Neuroblastoma: Data on patients with a malignant tumor of the sympathetic nervous system.
  • Diabetes type one: Data on patients with type 1 diabetes.
  • Sepsis & SIRS: Data on patients with sepsis and Systemic Inflammatory Response Syndrome.
  • Heart Failure and Depression: Data on patients with heart failure and depression.
  • Cardiac Arrest: Data on patients who experienced cardiac arrest.

Key Insights and Ideas

  • DBCV's Superiority for Non-Convex Clusters: The central finding is that DBCV consistently outperforms DCSI, CDbw, and VIASCKDE when assessing concave, nested, or density-based clusters, especially under increasing noise or parameter variations.
  • Limitations of Traditional Metrics: The paper reinforces that standard metrics like Silhouette and Davies-Bouldin are ill-suited for complex cluster shapes, and even some density-aware metrics struggle.
  • Robustness to Noise: DBCV shows a more gradual and consistent decline as noise increases in artificial datasets, indicating its reliability in reflecting deteriorating clustering quality. Other metrics can be less sensitive or exhibit counterintuitive behavior.
  • Consistency with ARI: On real-world EHR data, DBCV demonstrates the highest consistency with the Adjusted Rand Index trend across various clustering algorithms and parameter settings, suggesting it captures meaningful structural changes in complex data.
  • Counterintuitive Behavior of CDbw: The CDbw index often shows unexpected trends, sometimes increasing with noise or parameter changes, making it less reliable for evaluating clustering quality.
  • Focus on Trend, Not Just Absolute Value: The validation approach emphasizes observing how metrics change as data or parameters degrade, rather than just their absolute scores, which is crucial for assessing metric reliability.

Key Results

  • Artificial Data: DBCV consistently showed a downward trend as noise increased across various shapes, reflecting the degradation of clustering quality. VIASCKDE and DCSI also showed declines but sometimes exhibited less stable or counterintuitive behavior at higher noise levels. CDbw often showed an increasing trend with noise, which is contrary to expectations.
  • Real-World Data:
    • Neuroblastoma: DBCV was consistent with ARI trends in 100% of tests.
  • Diabetes: DBCV was consistent in 33% of tests, while DCSI was consistent in 67%.
  • Sepsis & SIRS: DBCV was consistent in 67% of tests.
  • Heart Failure & Depression: DBCV was consistent in 33% of tests.
  • Cardiac Arrest: DBCV was consistent in 67% of tests.
  • Overall Consistency: Across all datasets and tests, DBCV showed the highest overall consistency with ARI trends (9 out of 15 tests), followed by DCSI (7 out of 15) and VIASCKDE (4 out of 15). CDbw showed no consistent trends.

Limitations and Future Work

  • Artificial Data Limitations: While artificial data was designed to represent concave clusters, the authors acknowledge that real-world N-dimensional data might not perfectly map to these shapes.
  • Future Metrics Comparison: The authors plan to compare DBCV with other popular metrics like Within-Cluster Sum of Squares (WCSS), Xie-Beni index, and Hartigan index.
  • Data Types: The study focused solely on EHR data. Future work could extend the approach to other data types like medical images or physiological signals.
  • Focus on Trend: The study deliberately avoided the debate on finding the "best" number of clusters, focusing instead on how metrics behave as clustering results worsen or improve. This approach proved effective but might not cover all aspects of metric evaluation.

Summary updated

BibTeX
@article{chicco2025dbcv,
  author = {Davide Chicco and Giuseppe Sabino and Luca Oneto and Giuseppe Jurman},
  title = {{The DBCV index is more informative than DCSI, CDbw, and VIASCKDE indices for unsupervised clustering internal assessment of concave-shaped and density-based clusters}},
  journal = {PeerJ Comput. Sci.},
  year = {2025},
  doi = {10.7717/peerj-cs.3095},
}
Figure