Self-organizing context learning

Marc Strickert, Barbara HammerESANN 2004 · OpenReview

Introduction

Processing sequential data, where temporal order is paramount, remains a significant challenge for standard neural networks. Traditional approaches often rely on "sliding windows"—concatenating a fixed-size history into a high-dimensional vector. However, this method suffers from the curse of dimensionality, immediate information loss for data outside the window, and a metric mismatch that ignores underlying temporal relationships. This paper, originally presented at ESANN 2004, revisits the Merging Self-Organizing Map (MSOM) to address these limitations through a recursive encoding process that represents history as a fractal mathematical structure.

The Merge SOM (MSOM) Framework

The core innovation of the MSOM is the Merge Context. Unlike standard SOMs, each neuron in this architecture stores two components: a weight vector ($w$) representing the current input pattern and a context vector ($c$) representing the sequence history.

When new data is processed, the network identifies a "winner" neuron based on a distance function that balances the current input and the current context. The "Merge" occurs during the update phase: the context for the subsequent step is calculated as a weighted linear combination of the previous winner’s weight and its context. The authors also extend this logic to Merging Neural Gas (MNG), allowing the network to adapt its topology more freely than a rigid 2D grid.

Experimental Evaluation

The model's performance was evaluated across three distinct benchmarks:

  1. Artificial Binary Sequences: Used to visualize the development of the context space.
  2. Reber Grammar: A complex synthetic benchmark used in linguistics to test a model's ability to distinguish "legal" from "illegal" sequences based on hidden transition rules.
  3. UCI Japanese Vowels: A real-world dataset of 12-dimensional frequency vectors from nine speakers, testing temporal speaker identification.
The Reber Grammar state transition diagram used for benchmarking temporal sequence learning.

Fractal Encoding and Self-Organization

A key finding of this research is that the MSOM’s recursive update is mathematically equivalent to a fractal encoding. By repeatedly merging past states with the present, the network naturally organizes sequences into a "Cantor set-like" structure. In this representation, the influence of an event decays over time but is never entirely deleted, allowing for a highly efficient representation of the past.

Visualization of the fractal-like context space generated by the merging process.

Furthermore, the MSOM’s capacity grows naturally with the number of neurons, avoiding the need to pre-define history length. The authors suggest maximizing "activation entropy" during training to ensure broad neural participation and prevent the network from becoming underutilized.

Results and Discussion

The MSOM demonstrated superior memory and efficiency compared to existing models:

  • Memory Capacity: On the Reber Grammar task, MSOM uniquely identified strings with an average length of 8.90, outperforming the SOMSD baseline (7.23).
  • Neural Utilization: The MSOM utilized 428 active neurons compared to 338 in the baseline, indicating more effective use of internal representational capacity.
  • Real-World Performance: In the Japanese Vowel speaker identification task, Merging Neural Gas (MNG) achieved a test error rate of 2.7%, significantly lower than the 5.9% reference error. Scaling the model to 1000 neurons further reduced the error to 1.6%.

Conclusion

While the MSOM provides a powerful framework for unsupervised sequence learning, future work could explore adaptive metrics beyond squared Euclidean distance to handle noisier data. Additionally, extending the "merge" logic to complex graph structures and utilizing these fractal representations for supervised fine-tuning remain promising avenues for research.

Summary updated

BibTeX
@inproceedings{strickert2004self,
  author = {Marc Strickert and Barbara Hammer},
  title = {{Self-organizing context learning}},
  booktitle = {ESANN},
  year = {2004},
  url = {https://openreview.net/forum?id=kXY-EW68HQ},
}
Figure