Relevance learning in generative topographic maps

Andrej Gisbrecht, Barbara HammerESANN 2010 · OpenReview

Introduction

Generative Topographic Maps (GTM) are powerful unsupervised learning tools used to visualize high-dimensional data by projecting it onto a 2D manifold. However, standard GTM implementations often suffer from a "garbage in, garbage out" problem: the model treats every dimension of the data as equally important. In datasets containing significant noise or irrelevant features, GTM may prioritize visualizing that noise, thereby obscuring the underlying patterns of interest. This paper introduces "relevance learning" to the GTM framework, bridging the gap between unsupervised visualization and supervised classification to create a more focused and discriminative tool.

The Metric Learning Framework

The core innovation of this research is the integration of metric learning into GTM. Instead of the standard Euclidean distance, the authors employ a weighted diagonal metric:

$$\|\mathbf{x} - \mathbf{w}\|^2_\lambda = \sum_{d=1}^{D} \lambda_d^2(x_d - w_d)^2$$

In this formulation, $\lambda_d$ represents the "relevance" of the $d$-th dimension. As the model trains, the relevance values for useless or noisy dimensions shrink toward zero, effectively allowing the GTM to ignore them. The model utilizes a hybrid training approach, alternating between standard Expectation-Maximization (EM) for GTM parameters and Stochastic Gradient Descent (SGD) to update the relevance weights.

Adaptation Strategies

The authors propose three distinct objective functions to optimize these relevance weights:

  1. GRGTM (Generalized Relevance GTM): Based on margin maximization, this strategy pushes prototypes of the same class closer to data points while repelling different-class prototypes.
  2. SRGTM (Soft Relevance GTM): Optimizes the ratio of the probability of correct classification to the overall probability.
  3. EGTM (Entropy GTM): Minimizes the entropy of the label distribution assigned to prototypes, aiming to make each prototype "pure" (representing a single class).

Relevance Profiles and Interpretability

One of the primary benefits of this approach is the creation of "relevance profiles." By examining the learned $\lambda$ values, researchers can identify exactly which features the model found most useful for the task. Interestingly, the three different cost functions often agree on which dimensions are most important, suggesting that feature relevance is a robust property of the data.

Relevance profiles for the Letter dataset (16 dimensions) across three different adaptation strategies, showing strong agreement on feature importance.

Performance Evaluation

The approach was evaluated on several benchmarks, including the Landsat Satellite, Letter Recognition, and Phoneme datasets. A key finding was that adding relevance learning significantly improved classification accuracy without compromising the quality of the visualization. For instance, on the Phoneme dataset, accuracy jumped from 40.15% (standard GTM) to 83.44% (EGTM).

Experimental results for the Landsat dataset, demonstrating improved accuracy and stable topographic products across the proposed methods.

As shown in the results for the Landsat dataset, the "topographic product"—a measure of how well the 2D map preserves high-dimensional structure—remained stable across all methods. This indicates that the model gains discriminative power without ruining the underlying topographic mapping.

Conclusion

By incorporating relevance learning, GTM becomes a "supervised visualization" tool that shows data through the lens of a specific classification task. This method is computationally efficient ($O(N)$ complexity) and provides interpretable feature selection. Future work may explore local relevance matrices to allow the model to prioritize different features in different regions of the data space.

Summary updated

BibTeX
@inproceedings{gisbrecht2010relevance,
  author = {Andrej Gisbrecht and Barbara Hammer},
  title = {{Relevance learning in generative topographic maps}},
  booktitle = {ESANN},
  year = {2010},
  url = {https://openreview.net/forum?id=2G7EYW6DTbd},
}
Figure