S-Map: A Network with a Simple Self-Organization Algorithm for Generative Topographic Mappings

Kimmo Kiviluoto, Erkki OjaNeurIPS 1997

Introduction

The S-Map introduces a hybrid algorithm that combines the robust self-organizing power of Kohonen’s Self-Organizing Map (SOM) with the rigorous probabilistic framework of the Generative Topographic Mapping (GTM). By crossbreeding these two approaches, the S-Map solves the GTM’s notorious sensitivity to initial conditions while providing the well-defined objective functions that the original SOM lacks.

The Clash of Paradigms

In the world of unsupervised learning and dimensionality reduction, two major paradigms have historically been at odds:

  1. The Self-Organizing Map (SOM): Highly successful in practice and biologically inspired, the SOM is excellent at unfolding from random initializations. However, it is theoretically "messy." It lacks a generative model—meaning it cannot estimate the probability density of your data—and it does not strictly minimize a single, well-defined objective function.
  2. The Generative Topographic Mapping (GTM): A principled probabilistic alternative that treats mapping as a density estimation problem. While mathematically elegant, GTM is notoriously "fussy." It often fails to "unfold" or self-organize properly, remaining tangled unless it is very carefully initialized.

The authors aim to create an algorithm that inherits the SOM’s ability to organize itself from a completely random start while maintaining the GTM’s probabilistic interpretability.

The S-Map Approach

The S-Map resolves this tension through a clever hybrid methodology:

  • Probabilistic Activation: It utilizes the GTM’s "softmax" activations, which represent the posterior probability of a latent unit given a specific data point.
  • SOM-like Learning Rule: Instead of the GTM’s standard update (where the entire manifold moves at once), the S-Map adopts a learning rule similar to the SOM, where individual weight vectors move directly toward the data points.
  • Rule Variations: The authors propose two versions of this update: a "full" rule where neighboring units influence the update, and a simplified "Hebbian" rule where a weight is updated based solely on its own activation.
  • Theoretical Grounding: A significant portion of the paper proves that, under a "dot-product metric" (assuming normalized vectors), the S-Map update rule actually minimizes the GTM’s negative log-likelihood error function.

Key Insights and Results

The algorithm was tested on a standard benchmark for self-organization: an artificial 2D dataset consisting of 500 points sampled from a uniform random distribution within a unit square. This setup is designed to visualize how well the grid "unfolds" from a random starting configuration to evenly cover the data space.

Comparison of map unfolding and error minimization

Qualitative Success

The core "aha moment" of the paper is that the GTM’s failure to self-organize is largely due to its weight update logic. By switching to a SOM-like update, the S-Map becomes remarkably resilient to random starts. In simulations starting from random initial weights, both the full and simplified S-Map successfully unfolded to cover the unit square. In contrast, the standard GTM remained "tangled" and failed to adapt well to the data unless its basis functions were artificially widened to pre-organize the map.

Error Minimization and Hebbian Learning

The S-Map successfully minimizes the GTM error function. As the inverse variance ($\beta$) is increased during an "annealing" process, the map spreads out, and the error eventually decreases after an initial unfolding phase. Furthermore, the authors demonstrate that the algorithm can be simplified to a pure Hebbian rule without losing its qualitative self-organizing behavior, proving that complex neighborhood updates aren't strictly necessary for unfolding.

Limitations and Future Work

While the S-Map offers a compelling bridge between SOM and GTM, the authors note a few limitations:

  • Euclidean Convergence: While convergence is proven for the dot-product metric, guaranteed convergence for the S-Map in a standard Euclidean metric remains an open theoretical question.
  • Practical Scaling: Tests were performed on simple 2D toy data. More experimentation is needed to determine if the S-Map can match the SOM’s massive track record in complex, high-dimensional engineering applications.
  • Parameter Tuning: Adjusting the $\beta$ parameter (the noise model) requires an annealing schedule, adding a layer of manual tuning compared to the simplest versions of the SOM.

Summary updated

BibTeX
@inproceedings{kiviluoto1997s,
  author = {Kimmo Kiviluoto and Erkki Oja},
  title = {{S-Map: A Network with a Simple Self-Organization Algorithm for Generative Topographic Mappings}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1997},
}
Figure