A Semi-Supervised Self-Organizing Map for Clustering and Classification

Pedro H. M. Braga, Hansenclever de F. Bassani

TL;DR

This paper introduces the Semi-Supervised Self-Organizing Map (SS-SOM), a novel hybrid algorithm that dynamically adapts between supervised and unsupervised learning based on the availability of labeled data. SS-SOM demonstrates superior performance in semi-supervised classification and clustering, particularly when labeled data is scarce, achieving accuracy comparable to fully supervised methods.

What Problem Does This Paper Address?

In many real-world machine learning scenarios, datasets are abundant but largely unlabeled, making traditional supervised learning impractical due to the high cost and effort of manual labeling. This gap necessitates methods that can effectively leverage both labeled and unlabeled data. Semi-supervised learning (SSL) aims to bridge this gap, but existing methods may struggle with high-dimensional data or specific learning tasks. This paper addresses the challenge of developing a robust SSL method that can handle high-dimensional data and adapt its learning strategy based on the proportion of labeled samples available, excelling particularly in low-label regimes.

Methods

The core innovation is the Semi-Supervised Self-Organizing Map (SS-SOM), an extension of the LARFDSSOM (Local Adaptive Receptive Field Dimension Selective Self-Organizing Map). SS-SOM builds upon the prototype-based learning of Self-Organizing Maps (SOMs) and incorporates elements of Learning Vector Quantization (LVQ) for supervised learning.

Key methodological aspects include:

  • Hybrid Learning: SS-SOM dynamically switches between supervised and unsupervised learning modes during training. If an input pattern has a class label, it uses a supervised mode; otherwise, it defaults to an unsupervised mode.
  • Adaptive Receptive Fields and Relevance: Similar to its predecessor, SS-SOM uses adaptive receptive fields and relevance vectors for input dimensions, allowing it to handle high-dimensional data by assigning different importance to different features.
  • Node Structure and Activation: Each node in the map has a center vector (prototype), a relevance vector (feature importance), and a distance vector (moving average of distances). Node activation is calculated using a radial basis function that considers both distance and feature relevance.
  • Dynamic Node Management: The algorithm includes mechanisms for node insertion (when a pattern is too far from existing nodes) and node removal (when nodes fail to win a sufficient number of competitions), preventing map collapse and adapting to data density.
  • Supervised Mode Adaptation: When a labeled sample is processed, SS-SOM attempts to find a winner node that matches the sample's class. If the initial winner is incorrect, it searches for an alternative and can "push" the incorrect winner away from the input pattern.

Datasets

The paper evaluates SS-SOM on seven real-world datasets from the OpenSubspace framework. These datasets are described as being adapted from the UCI machine learning repository and include a variety of synthetic datasets. The specific datasets used for evaluation are:

  • Breast
  • Diabetes
  • Glass
  • Liver
  • Pendigits
  • Shape
  • Vowel

Key Insights and Ideas

  • Robustness in Low-Label Regimes: The most significant insight is SS-SOM's exceptional performance when only a small percentage of labeled data is available (as low as 1%). This highlights its potential for practical applications where labeling is a bottleneck.
  • Adaptability is Key: The ability to dynamically switch between supervised and unsupervised learning is crucial. This allows SS-SOM to leverage available labels effectively while still benefiting from the structure inherent in unlabeled data.
  • High-Dimensional Data Handling: By incorporating dimension relevance and adaptive receptive fields, SS-SOM effectively tackles the "curse of dimensionality," a common challenge in many real-world datasets.
  • Comparable to Supervised Methods: Even when 100% of the data is labeled, SS-SOM performs comparably to, and sometimes better than, traditional supervised methods like MLP, SVM, and GRLVQ, suggesting its underlying structure is highly effective.
  • Parameter Sensitivity: While SS-SOM has 11 parameters, the study indicates that only a few (activation threshold at, lowest cluster percentage lp, and winner learning rate eb) significantly impact results, simplifying tuning in practice.

Key Results

  • Superior Semi-Supervised Performance: SS-SOM consistently outperformed Label Propagation and Label Spreading on real-world datasets for supervision rates between 1% and 75%.
  • Competitive at 100% Supervision: When using 100% of the labeled data, SS-SOM achieved comparable or better results than other semi-supervised methods and was competitive with supervised methods like MLP and SVM across most datasets.
  • Best Performance on "Shape" Dataset: SS-SOM achieved the best result on the "Shape" dataset, outperforming all other methods, including supervised ones.
  • Low Accuracy for SVM on "Vowel": Notably, the SVM showed low accuracy on the "Vowel" dataset, while SS-SOM and other semi-supervised methods performed better.

Limitations and Future Work

  • Parameter Tuning: Although only a few parameters are highly sensitive, SS-SOM still has 11 parameters to set up, which can be a challenge.
  • Stop Criterion: The current implementation uses a fixed number of epochs, which may be more than necessary for convergence, leading to longer training times. The authors suggest that an adequate stop criterion is an area for future work to reduce training time.
  • Reinforcement Learning Integration: The paper notes that SS-SOM could be trivially modified to incorporate reinforcement learning, enabling it to switch between three learning approaches (supervised, unsupervised, and reinforcement). This is proposed as a direction for future work.
  • Subspace Clustering: While SS-SOM inherits LARFDSSOM's subspace clustering capabilities when no labels are available, its primary focus in this paper is on projected clustering and classification. Further exploration of its subspace clustering performance with labels could be a future avenue.

Summary updated

Abstract

There has been an increasing interest in semi-supervised learning in the recent years because of the great number of datasets with a large number of unlabeled data but only a few labeled samples. Semi-supervised learning algorithms can work with both types of data, combining them to obtain better performance for both clustering and classification. Also, these datasets commonly have a high number of dimensions. This article presents a new semi-supervised method based on self-organizing maps (SOMs) for clustering and classification, called Semi-Supervised Self-Organizing Map (SS-SOM). The method can dynamically switch between supervised and unsupervised learning during the training according to the availability of the class labels for each pattern. Our results show that the SS-SOM outperforms other semi-supervised methods in conditions in which there is a low amount of labeled samples, also achieving good results when all samples are labeled.
BibTeX
@inproceedings{braga2018semi,
  author = {Pedro H. M. Braga and Hansenclever de F. Bassani},
  title = {{A Semi-Supervised Self-Organizing Map for Clustering and Classification}},
  booktitle = {IJCNN},
  year = {2018},
  doi = {10.1109/ijcnn.2018.8489675},
}
Figure