Kernel-based topographic map formation achieved with an information-theoretic approach

Marc M. Van HulleNeural Networks 2002 · DOI

TL;DR

This paper introduces a novel information-theoretic learning algorithm for "kernel-based topographic maps" (KTMs). By maximizing the joint entropy of kernel outputs, the algorithm aims to create maps that effectively represent input data distributions while maintaining topological order, offering a principled way to learn these complex structures.

What Problem Does This Paper Address?

Traditional Self-Organizing Maps (SOMs) use a "winner-take-all" approach where only one neuron activates for a given input. While effective, this can limit their ability to accurately model complex data distributions and handle noise. Kernel-based topographic maps (KTMs) aim to improve upon SOMs by equipping neurons with local "kernel" functions (like Gaussians) instead of simple activation units. This allows neurons to have overlapping receptive fields, leading to better density estimation and noise tolerance. However, designing learning algorithms for KTMs that are both topologically ordered and information-theoretically optimal has been a challenge. Specifically, extending principles like Linsker's "infomax" (maximizing information transfer) to KTMs is difficult due to the overlapping nature of kernels, which introduces statistical dependencies between neuron outputs. This paper tackles the problem of forming KTMs that are not only topologically ordered but also optimize an information-theoretic criterion, specifically by maximizing the joint entropy of the kernel outputs.

Methods

The core of the paper is a new learning algorithm for KTMs based on information theory. The authors start by considering Linsker's infomax principle, which suggests maximizing differential entropy. However, they find this insufficient for KTMs due to kernel overlap. They then adopt Bell and Sejnowski's generalization, which involves maximizing differential entropy of individual kernel outputs and minimizing the mutual information between these outputs. This dual objective is framed as maximizing the joint entropy of the kernel outputs.

The learning algorithm has two main components:

  1. Kernel Adaptation: The algorithm updates the centers (w_i) and radii (σ_i) of the Gaussian kernels. The update rules are derived from maximizing the differential entropy of each kernel's output.
  2. Competitive Learning: To heuristically minimize the mutual information (and thus reduce output redundancy and ensure topology preservation), a competitive "winner-take-all" (WTA) stage is introduced. The neuron whose kernel output is highest for a given input "wins" the competition, and its parameters are updated. This is a key departure from some other KTM algorithms that use distance-based WTA rules.

The paper also defines the kernel function as the cumulative distribution function of the input's radial distance from its mean, based on a Gaussian assumption. This choice is motivated by information-theoretic principles for maximizing output entropy.

Datasets

The paper evaluates its algorithm on two types of data:

  1. Synthetic Data: A one-dimensional, unit-variance Gaussian distribution is used. This is a standard benchmark for density estimation and allows for analytical calculation of the theoretically optimal joint entropy and mutual information.
  2. Real-World Data: The "Old Faithful" geyser eruption lengths dataset is used. This is a classic benchmark for density estimation algorithms.

The paper does not introduce new datasets but uses established ones to compare its method against others.

Key Insights and Ideas

  • Joint Entropy Maximization for KTMs: The paper proposes maximizing joint entropy as a principled information-theoretic criterion for KTM formation. This elegantly balances the desire for high information transfer through individual kernels with the need to reduce redundancy between kernels.
  • Activity-Based WTA for Topology: Using an activity-based winner-take-all (where the neuron with the highest kernel output wins) is a novel heuristic for minimizing mutual information and preserving topology. This differs from distance-based WTA rules common in other SOMs.
  • Kernel as Cumulative Distribution: Defining the kernel as the cumulative distribution function of the input's radial distance is theoretically motivated for maximizing output entropy.
  • Trade-off between JE and MI: The simulations clearly demonstrate the inherent trade-off: increasing joint entropy often leads to an increase in mutual information, indicating higher kernel overlap and redundancy. The proposed algorithm aims to find a favorable balance.

Key Results

  • Synthetic Data: The algorithm achieves joint entropy values very close to the theoretical optimum for a 1D Gaussian input. Crucially, it achieves the highest joint entropy while simultaneously having the lowest mutual information among the compared algorithms, suggesting a more efficient representation with less redundancy.
  • Real-World Data (Old Faithful): The algorithm again performs best in terms of joint entropy. It also achieves the best mutual information, indicating a more compact and less redundant representation of the data.
  • Density Estimation: The algorithm produces good density estimates, visually comparable to or better than other methods, especially with more kernels.

Limitations and Future Work

  • Heuristic for MI Minimization: While the activity-based WTA is effective, it's acknowledged as a heuristic for minimizing mutual information. The paper suggests exploring more direct methods for mutual information minimization.
  • Gaussian Input Assumption: The kernel definition is based on the assumption of locally Gaussian input densities. While this is common, its performance on non-Gaussian data might be a limitation.
  • Future Work: The authors intend to explore how the density estimates generated by these maps can be used for clustering purposes. They also mention the potential for extending the framework to multidimensional function approximation.

Summary updated

BibTeX
@article{hulle2002kernel,
  author = {Marc M. Van Hulle},
  title = {{Kernel-based topographic map formation achieved with an information-theoretic approach}},
  journal = {Neural Networks},
  year = {2002},
  doi = {10.1016/s0893-6080(02)00077-1},
}
Figure