From horizontal to vertical collaborative clustering using generative topographic maps

Jérémie Sublime, Nistor Grozavu, Guénaël Cabanès, Younès Bennani, Antoine Cornuéjols

TL;DR

This paper introduces a novel method for "vertical collaborative clustering," enabling multiple clustering algorithms to learn from different datasets that share similar underlying structures. By leveraging the Generative Topographic Mapping (GTM) algorithm, the approach effectively transfers knowledge between these independent learning processes, leading to improved clustering performance and a more efficient overall learning pipeline.

What Problem Does This Paper Address?

The paper tackles a significant challenge in machine learning: the difficulty of achieving high-performance clustering on large datasets and the limitations of traditional transfer learning. While "collaborative clustering" has shown promise by having multiple algorithms work on the same data to improve each other's results, a gap exists in scenarios where algorithms operate on different datasets but are expected to find similar cluster structures. This "vertical collaboration" is crucial for effective transfer learning, where knowledge gained from one data source can be applied to another, but existing methods struggle to bridge this divide. The authors aim to bridge this gap by enabling algorithms to learn from distinct datasets while still benefiting from shared insights.

Methods

The core of the proposed method lies in adapting an existing "horizontal collaborative clustering" framework to a "vertical collaboration" scenario. This is achieved by integrating the Generative Topographic Mapping (GTM) algorithm.

Here's a breakdown of the key techniques:

  • Generative Topographic Mapping (GTM): GTM is an unsupervised neural network that maps a low-dimensional latent space to a high-dimensional data space, creating a topographic map of prototypes. The paper utilizes GTM's ability to generate similar topographic maps when initialized similarly and applied to datasets with similar distributions.
  • Horizontal Collaborative Clustering Framework: The authors adapt a previously developed framework that allows multiple clustering algorithms to work independently on their data (local step) and then exchange information to improve their results (collaborative step).
  • Vertical Collaboration Adaptation: The novelty lies in how they apply the horizontal framework to vertical collaboration. Instead of having algorithms work on the same data, they first run GTM on different datasets. The resulting topographic maps (prototypes) from each GTM instance are then treated as "split data sets" for the collaborative clustering framework. This allows the GTM instances to collaborate and refine their maps based on the insights from other GTMs trained on different, but structurally similar, data.
  • EM Algorithm for GTM: The paper details the use of the Expectation-Maximization (EM) algorithm for both the initial GTM training and for the collaborative clustering step applied to the GTM prototypes.

Datasets

The authors evaluated their proposed method on several standard datasets:

  • Waveform: A dataset with 5000 instances and 3 classes, featuring noise attributes.
  • Wisconsin Diagnostic Breast Cancer (WDBC): A dataset with 569 instances and 32 variables, used for classifying breast masses as benign or malignant.
  • Spam Base: A dataset of 4601 observations with 57 variables, used for classifying emails as spam or not-spam.
  • Madelon: An artificial dataset designed for feature selection challenges, involving continuous input variables and two classes.

The paper mentions that these datasets were randomly shuffled and split into 5 subsets with equivalent data distributions to simulate the collaborative scenario.

Key Insights and Ideas

  • Bridging the Gap Between Horizontal and Vertical Collaboration: The most significant insight is the clever repurposing of a horizontal collaboration framework for vertical transfer learning by using GTM's ability to produce comparable topographic maps from different datasets.
  • GTM as a Knowledge Transfer Enabler: The paper demonstrates that GTM's topographic maps, when generated under similar conditions, can serve as a common ground for collaborative learning, even when the underlying data is distinct.
  • Self-Regulation in Collaboration: A key advantage highlighted is that the proposed method doesn't require a fixed "confidence parameter" for collaboration. Instead, it benefits from self-regulation by diminishing the influence of collaborators whose solutions are highly diverse, leading to more robust outcomes.
  • Potential for Faster and Better Clustering: By enabling collaborative learning across datasets, the method promises to accelerate the clustering process and potentially achieve better results than individual, isolated clustering efforts.

Key Results

The experimental results, presented in Table 1, indicate that the proposed Vertical Collaborative Clustering using GTM (V2C-GTM) method generally leads to improvements in clustering quality across various datasets.

  • Improved Purity and Adjusted Rand Index: The V2C-GTM method showed positive average gains in purity and Adjusted Rand Index (ARI) across most datasets, indicating better cluster quality and agreement with ground truth. For instance, on the SpamBase dataset, the purity gain was +1.43%.
  • Contrasted Performance on Davies-Bouldin Index: While the method showed improvements in purity and ARI, the Davies-Bouldin Index (DB index), which measures cluster separation and compactness, showed more varied results, with improvements in some cases and stable or slightly worse performance in others.
  • Comparison with Other Methods: Table 2 shows that V2C-GTM generally outperforms other collaborative methods (GTMCol and SOMCol) in terms of average purity gain, often showing positive gains where others exhibit negative ones. The paper attributes this to the probabilistic approach of V2C-GTM compared to the derivative approach of other methods.

Limitations and Future Work

  • Constraint on Data Distribution: The current method relies on the assumption that the different datasets have similar underlying distributions and are in the same feature spaces for the GTM maps to be comparable.
  • Unsupervised Nature and Potential Deterioration: While unsupervised, the method can potentially lead to a deterioration of quality indexes if datasets have significantly mismatched distributions or if the initial maps are of poor quality.
  • Future Work - Removing Constraints: The authors express a strong desire to remove the constraints on identical data distributions and feature spaces. The goal is to enable transfer learning between datasets that are very different but still share similar cluster structures. This would involve finding ways to train topographic maps (SOM or GTM) such that despite different feature spaces, the resulting parallel maps learn from all data sets and maintain similar features.

Summary updated

Abstract

Collaborative clustering is a recent field of Machine Learning that shows similarities with both ensemble learning and transfer learning. Using a two-step approach where different clustering algorithms first process data individually and then exchange their information and results with a goal of mu tual improvement, collaborative clustering has shown promising performances when trying to have several algorithms working on the same data. However the field is still lagging behind when it comes to transfer learning where several algorithms are working on different data with similar clusters and the same features. In this article, we propose an original method where we combine the topological structure of the Generative Topographic Mapping (GTM) algorithm and take advantage of it to transfer information between collaborating algorithms working on different data sets featuring similar distributions. The proposed approach has been validated on several data sets, and the experimental results have shown very promising performances.
BibTeX
@article{sublime2016from,
  author = {Jérémie Sublime and Nistor Grozavu and Guénaël Cabanès and Younès Bennani and Antoine Cornuéjols},
  title = {{From horizontal to vertical collaborative clustering using generative topographic maps}},
  journal = {International Journal of Hybrid Intelligent Systems},
  year = {2016},
  doi = {10.3233/his-160219},
}
Figure