Clustering versus Incremental Learning Multi-Codebook Fuzzy Neural Network for Multi-Modal Data Classification

Muhammad Anwar Ma'sum, Hadaiq Rolis Sanabila, Petrus Mursanto, Wisnu JatmikoComput 2020 · DOI

TL;DR

This paper introduces a novel multi-codebook fuzzy neural network (MC-FNGLVQ) designed to tackle the challenge of classifying multi-modal data, where data features are spread across multiple, distinct areas. By employing either clustering or incremental learning to create multiple "codebooks" per class, the proposed method significantly outperforms traditional single-codebook approaches and existing neural networks on both synthetic and benchmark datasets.

What Problem Does This Paper Address?

Classification is a fundamental task in machine learning, but it becomes challenging when dealing with "multi-modal data." This type of data is characterized by features that don't follow a single, continuous distribution but rather appear in multiple, separate clusters or "peaks." Imagine predicting election results: one candidate might receive votes from young adults (e.g., 18-30) and older adults (e.g., 50-65), creating two distinct peaks in the age distribution for that candidate's voters. Traditional single-codebook classifiers, which try to represent each class with a single set of features, struggle to accurately capture these dispersed patterns, leading to misclassifications. This paper aims to develop a more robust classification method specifically for such multi-modal scenarios.

Methods

The core innovation lies in the Multi-Codebook Fuzzy-Neuro Generalized Learning Vector Quantization (MC-FNGLVQ). Unlike standard approaches that use a single "codebook" (a representative set of features) per class, MC-FNGLVQ employs multiple codebooks. This allows for a more nuanced representation of multi-modal data. The paper proposes two primary strategies for generating these multiple codebooks:

  • Clustering Approach: Before training, the data for each class is first clustered using either K-Means or Gaussian Mixture Model (GMM) clustering. Each cluster then serves as the basis for generating a codebook for that class. This approach aims to capture distinct modes of the data distribution.
  • Incremental Learning Approach: In this method, the network starts with a single codebook per class. During the training process, the algorithm dynamically decides when it's necessary to create a new codebook. This decision is triggered when the input data is too distant from existing codebooks, suggesting a new mode needs representation.

The underlying classifier is based on the Fuzzy-Neuro Generalized Learning Vector Quantization (FNGLVQ), which itself is an extension of Learning Vector Quantization (LVQ) incorporating fuzzy logic for improved handling of overlapping data.

Datasets

The research utilizes two types of datasets:

  • Synthetic Datasets: Nine synthetic datasets were created to specifically mimic multi-modal distributions. These datasets feature varying numbers of peaks (2, 3, and 5) and classes (2, 3, and 5), allowing for controlled evaluation of the proposed methods.
  • Benchmark Datasets: Several real-world datasets were used for evaluation, including those from the UCI database (e.g., Ionosphere, Glass, Fertility, Pima, SPECTF, EEG) and others from previous research (e.g., Pinwheel). These datasets were chosen for their potential multi-modal characteristics.

Key Insights and Ideas

  • Multi-Codebook is Crucial for Multi-Modal Data: The fundamental insight is that single-codebook models inherently struggle with multi-modal data due to overlapping feature spaces. The multi-codebook approach, by design, can better approximate these dispersed distributions.
  • Clustering vs. Incremental Learning Trade-offs: Both clustering and incremental learning proved effective in generating multi-codebooks. Clustering offers a more structured, pre-training approach, while incremental learning adapts dynamically during training, potentially offering more flexibility.
  • GMM Outperforms K-Means on Synthetic Data: For the synthetic datasets, which were generated with Gaussian distributions, the GMM clustering approach generally yielded better accuracy than K-Means. This suggests GMM's ability to model more complex distributions is advantageous in such scenarios.
  • Pruning Can Be Beneficial, But Not Always: The study explored pruning small clusters. While a small pruning threshold (2.5%) showed slight improvements, larger thresholds led to decreased accuracy, indicating that aggressive pruning can remove valuable information.
  • Incremental Learning Benefits from Higher Thresholds and Random Initialization: For the incremental learning approach, a higher threshold (0.5) for creating new codebooks and random data initialization for the initial codebook generally led to better performance on synthetic datasets.

Key Results

  • Synthetic Dataset Performance: The proposed MC-FNGLVQ achieved a highest accuracy of 84.76% on synthetic datasets (using GMM clustering). This represents a significant improvement of 24.9% over the original FNGLVQ.
  • Benchmark Dataset Performance: On benchmark datasets, MC-FNGLVQ using K-Means clustering achieved the highest accuracy of 79.94%. This was an improvement of 4.7% over the original FNGLVQ.
  • Comparison to Other Networks: The proposed MC-FNGLVQ (K-Means) showed a 10% margin in accuracy over a popular neural network (MLP) on synthetic data and a 4.7% margin on benchmark data.
  • Scoring: Across all datasets, MC-FNGLVQ using K-Means and GMM clustering received the highest overall scores, indicating consistent strong performance.

Limitations and Future Work

  • Not Recommended for Uni-modal Data: The paper explicitly states that the proposed multi-codebook methods are not recommended for uni-modal datasets, as they offer only small improvements over the original FNGLVQ in such cases. The original FNGLVQ is preferred for uni-modal classification.
  • Potential for Further Improvement: While the proposed methods show significant gains, the authors acknowledge that the performance is still below 90% for some applications (like medical analysis) and that there is "room for improvements."
  • Future Work: The paper doesn't explicitly detail future work, but the findings suggest exploring hybrid approaches combining clustering and incremental learning, investigating different clustering algorithms, and applying the method to a wider range of real-world multi-modal problems.

Summary updated

BibTeX
@article{masum2020clustering,
  author = {Muhammad Anwar Ma'sum and Hadaiq Rolis Sanabila and Petrus Mursanto and Wisnu Jatmiko},
  title = {{Clustering versus Incremental Learning Multi-Codebook Fuzzy Neural Network for Multi-Modal Data Classification}},
  journal = {Comput.},
  year = {2020},
  doi = {10.3390/computation8010006},
}
Figure