Adaptive Resonance Theory-Based Topological Clustering With a Divisive Hierarchical Structure Capable of Continual Learning

Naoki Masuyama, Narito Amako, Yuna Yamada, Yusuke Nojima, Hisao Ishibuchi

TL;DR

This paper introduces HCAEA, a new hierarchical clustering algorithm that solves two major pain points in machine learning: the need for manual parameter tuning and the "catastrophic forgetting" that happens during continual learning. By combining Adaptive Resonance Theory (ART) with automated threshold estimation, the authors created a system that grows its own structure to match data complexity without losing what it previously learned.

What Problem Does This Paper Address?

Most clustering algorithms struggle with the plasticity-stability dilemma: how do you stay "plastic" enough to learn new patterns without overwriting the "stable" knowledge you already have? While Adaptive Resonance Theory (ART) was designed specifically for this, it has a notorious Achilles' heel: the vigilance parameter. This is a manually set similarity threshold that determines when to create a new cluster. If you set it too high, you get too many clusters; too low, and your clusters become a meaningless blur.

Furthermore, standard clustering often provides a "flat" view of data. In the real world, data is often hierarchical (e.g., "Golden Retriever" $\subset$ "Dog" $\subset$ "Mammal"). Existing hierarchical models like Growing Hierarchical Self-Organizing Maps (GHSOM) often fail at continual learning because they tend to forget old information as they adapt to new data streams.

Methods

The authors propose two nested solutions:

  1. CAEA (CIM-based ART with Edge and Age): This is the base "flat" clustering algorithm. It uses the Correntropy-Induced Metric (CIM) instead of standard Euclidean distance, which makes it much more robust against outliers. Crucially, it uses a Kernel Density Estimator (KDE) to automatically calculate the similarity threshold based on the distribution of the first few data points, removing the need for manual "vigilance" tuning.
  2. HCAEA (Hierarchical CAEA): This is a divisive hierarchical structure. It starts with one CAEA model. Once that model is trained, the data points assigned to each node are passed down to "child" CAEA models in a new layer. This process repeats, building a tree-like structure that captures information at multiple levels of granularity.

The system handles continual learning by using ART’s logic: if a new data point doesn't fit any existing cluster (based on the automated threshold), a new node is created. Old nodes are preserved, preventing the "catastrophic forgetting" seen in other neural networks.

Datasets

The researchers evaluated their approach using a comprehensive mix of benchmarks:

  • Synthetic Datasets: Aggregation, Compound, Hard Distribution, Jain, and Pathbased (used to test the ability to handle complex shapes).
  • Real-World Datasets: 9 datasets from the UCI repository, ranging from low-dimensional (Iris, Wine) to high-dimensional (COIL20 with 1,024 attributes and TOX171 with 5,748 attributes).

Key Insights and Ideas

  • Automation is Possible: By using KDE to estimate bandwidth ($\sigma$), the algorithm can "sense" the density of the data and set its own sensitivity. This makes the model much more "plug-and-play" across different domains.
  • Divisive is Better for Continual Learning: Unlike agglomerative clustering (which builds from the bottom up and usually requires the whole dataset at once), this divisive (top-down) approach can adaptively generate new partitions as the data distribution shifts over time.
  • Outlier Rejection: Using Correntropy (CIM) allows the model to ignore "noise" that would typically pull a standard K-means cluster center off-target.

Key Results

The authors compared HCAEA against state-of-the-art models like GH-EXIN and HFTCA.

  • Performance: In both stationary and non-stationary (changing) environments, HCAEA consistently ranked among the top performers. Statistical analysis (Friedman and Nemenyi tests) showed that HCAEA and GH-EXIN were the superior models overall.
  • Stability: In non-stationary tests (where classes are introduced one by one), HCAEA maintained high accuracy on early classes even after learning later ones, proving its capability for continual learning.
  • Efficiency: While GH-EXIN often generated a massive number of nodes (thousands in some cases), HCAEA maintained a much more compact and interpretable structure while achieving similar or better accuracy.

Limitations and Future Work

The authors identify a few areas for improvement:

  • Node Explosion: On specific datasets like "Iris" or "Pathbased," the algorithm occasionally generates more nodes than necessary. They suggest a future mechanism to prune or delete redundant nodes.
  • Initial Sample Dependency: The automated threshold relies on the distribution of the first $\lambda/2$ data points. If the very first batch of data isn't representative of the whole, the initial threshold might be suboptimal.
  • Concept Drift: While the model handles new classes well, the authors want to further explore how it handles "concept drift"—where the definition of an existing class changes over time.

Summary updated

Abstract

Adaptive Resonance Theory (ART) is considered as an effective approach for realizing continual learning thanks to its ability to handle the plasticity-stability dilemma. In general, however, the clustering performance of ART-based algorithms strongly depends on the specification of a similarity threshold, i.e., a vigilance parameter, which is data-dependent and specified by hand. This paper proposes an ART-based topological clustering algorithm with a mechanism that automatically estimates a similarity threshold from the distribution of data points. In addition, for improving information extraction performance, a divisive hierarchical clustering algorithm capable of continual learning is proposed by introducing a hierarchical structure to the proposed algorithm. Experimental results demonstrate that the proposed algorithm has high clustering performance comparable with recently-proposed state-of-the-art hierarchical clustering algorithms.
BibTeX
@article{masuyama2022adaptive,
  author = {Naoki Masuyama and Narito Amako and Yuna Yamada and Yusuke Nojima and Hisao Ishibuchi},
  title = {{Adaptive Resonance Theory-Based Topological Clustering With a Divisive Hierarchical Structure Capable of Continual Learning}},
  journal = {IEEE Access},
  year = {2022},
  doi = {10.1109/access.2022.3186479},
}

Loading related content...

Figure