Fuzzy Adaptive Resonance Theory, Diffusion Maps and their applications to Clustering and Biclustering

Steven B. Damelin, Yuantao Gu, Donald C. Wunsch, Rui XuarXiv 2014 · DOI · arXiv

Abstract

In this paper, we describe an algorithm FARDiff (Fuzzy Adaptive Resonance Diffusion) which combines Diffusion Maps and Fuzzy Adaptive Resonance Theory to do clustering on high dimensional data. We describe some applications of this method and some problems for future research.

Introduction

High-dimensional data—where each observation might have thousands of features—poses a significant challenge for traditional clustering. Standard methods often rely on Euclidean distance, which fails to capture the underlying non-linear geometric structure of complex data. Furthermore, many clustering algorithms struggle with the "stability-plasticity dilemma": the difficulty of learning new information without washing away previously learned patterns.

The FARDiff algorithm addresses these gaps by creating a pipeline that first "unfolds" the data’s geometry using a graph-based approach and then applies a biologically inspired neural network to group the results.

Methodology

The FARDiff algorithm operates in two primary stages: non-linear dimensionality reduction followed by robust categorization.

Non-linear Dimensionality Reduction (Diffusion Maps)

The algorithm constructs a finite graph where each data point is a node. Edges are weighted using a Gaussian kernel, representing local similarity. This is transformed into a Markov transition matrix, where entries represent the probability of "jumping" from one point to another. By calculating the eigenvectors of this matrix, the data is mapped into a lower-dimensional space ($R^L$). In this new space, the Euclidean distance between points approximates the "diffusion distance"—a measure of how well-connected points are through all possible paths in the graph, rather than just their straight-line proximity.

Clustering (Fuzzy Adaptive Resonance Theory)

The reduced data is fed into a Fuzzy ART neural network. Fuzzy ART uses a vigilance parameter ($\rho$) to control cluster granularity. If a new data point doesn't match an existing cluster prototype within the threshold set by $\rho$, the network dynamically creates a new category. It utilizes fuzzy logic operators (like the fuzzy AND) to handle real-valued input patterns, ensuring stable and scalable learning.

Architecture of the Fuzzy Adaptive Resonance Theory (ART) network showing layers F1 and F2, weights w, and the orienting subsystem with vigilance parameter ρ.

Datasets and Applications

The paper validates FARDiff using two distinct, high-dimensional application areas:

  • Cancer Detection: The Small Round Blue-Cell Tumor (SRBCT) dataset, which consists of gene expression profiles used to classify different types of childhood tumors.
  • Geological Surveying: Hyperspectral image data from core samples. These datasets contain hundreds of spectral bands per pixel, forming a "hypercube" of data used to identify mineral compositions.

Key Insights

  • Connectivity vs. Proximity: In complex datasets, two points might be "close" in Euclidean space but belong to different structures. Diffusion maps prioritize how points are connected through the data manifold, making the subsequent clustering much more accurate.
  • The Power of $t$: The diffusion process includes a time parameter ($t$). Increasing $t$ allows the algorithm to integrate more global geometric information, giving researchers a "knob" to control the scale of the analysis.
  • Dynamic Category Creation: Unlike K-means, which requires you to specify the number of clusters upfront, the Fuzzy ART component of FARDiff can "discover" the number of clusters based on the data and the vigilance parameter.

Results

FARDiff demonstrated effectiveness in extracting useful diagnostic information from the SRBCT dataset, performing competitively against other widely-used clustering algorithms for tumor classification. In the hyperspectral study, the algorithm successfully identified minerals in core samples, proving that FARDiff could address the "hypercube" complexity effectively.

Limitations and Future Work

The authors identify several frontiers for this research:

  • Biclustering: A major goal is extending FARDiff to perform simultaneous clustering of both dimensions (e.g., clustering genes and samples at the same time), which is vital for bioinformatics.
  • Computational Complexity: While effective, the graph-based nature of diffusion maps can be computationally expensive as the number of data points ($N$) grows.
  • Generalized Structures: The authors suggest applying FARDiff to more abstract data structures, such as trees and grammars, rather than just standard vector-based datasets.

Summary updated

Abstract

In this paper, we describe an algorithm FARDiff (Fuzzy Adaptive Resonance Dif- fusion) which combines Diffusion Maps and Fuzzy Adaptive Resonance Theory to do clustering on high dimensional data. We describe some applications of this method and some problems for future research.
BibTeX
@misc{damelin2014fuzzy,
  author = {Steven B. Damelin and Yuantao Gu and Donald C. Wunsch and Rui Xu},
  title = {{Fuzzy Adaptive Resonance Theory, Diffusion Maps and their applications to Clustering and Biclustering}},
  year = {2014},
  eprint = {1411.5737},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1411.5737},
}
Figure