TL;DR
This paper introduces Hyperseed, a novel unsupervised learning algorithm that leverages Vector Symbolic Architectures (VSA) for efficient learning of topology-preserving feature maps. Hyperseed offers few-shot learning capabilities and a single-vector operation learning rule, making it suitable for neuromorphic hardware and reducing computational complexity.
What Problem Does This Paper Address?
The paper addresses the growing need for efficient unsupervised learning algorithms, particularly in the context of emerging neuromorphic hardware. While Vector Symbolic Architectures (VSA) show promise for energy efficiency and fast learning, existing approaches often focus on creating single hypervectors for specific classes. This paper proposes a new VSA-based algorithm, Hyperseed, that learns a topology-preserving map of unlabeled data, enabling unsupervised clustering and feature extraction with reduced computational demands. The authors aim to demonstrate that VSA can be used to implement the entire learning pipeline for unsupervised tasks, a novel approach compared to using VSA solely for data encoding.
Methods
Hyperseed is built upon the principles of Vector Symbolic Architectures (VSA), specifically utilizing the Fourier Holographic Reduced Representations (FHRR) model. The core operations of VSA, binding and bundling (superposition), are central to Hyperseed. The algorithm works by mapping input data hypervectors (seeds) onto a pre-defined, topologically arranged set of hypervectors (the HD-map).
Key methodological aspects include:
- HD-map Initialization: A 2D grid of hypervectors is generated using the Fractional Power Encoding (FPE) method, where the similarity between hypervectors reflects their topological proximity on the grid. This HD-map is stored in an associative memory.
- Seed Hypervector (s) Update: A "seed" hypervector,
s, is iteratively updated. In each iteration, a data hypervector is chosen, unbound froms, and then a "perfect mapping" hypervector is computed by binding the data hypervector to a target hypervector on the HD-map. This perfect mapping hypervector is then added tos. - Weakest Match Search (WMS): To select data hypervectors for updates, Hyperseed employs a heuristic based on the farthest-first traversal rule (FFTR). This involves finding the data hypervector that, when unbound from
s, results in the lowest similarity to its best matching vector (BMV) on the HD-map. - Single Vector Operation Learning: A key novelty is that the learning update for the seed hypervector
sinvolves a single vector operation (addition), significantly reducing computational cost. - Few-Shot Learning: The algorithm is designed to learn effectively from a small number of input vectors.
Datasets
The paper evaluates Hyperseed on several datasets:
- Synthetic Datasets: Six datasets from the Fundamental Clustering Problems Suite (FCPS) are used to evaluate clustering and non-linear classification. These datasets are chosen for their representative non-linearity.
- Iris Dataset: A standard benchmark for classification, used to evaluate Hyperseed's performance with labeled data and explore its few-shot learning capabilities.
- Language Identification Dataset: n-gram statistics from texts of 21 European languages are used to test Hyperseed on a larger-scale classification problem.
Key Insights and Ideas
- VSA for End-to-End Unsupervised Learning: The paper demonstrates that VSA can be used to implement the entire unsupervised learning pipeline, not just data encoding, which is a novel contribution.
- Few-Shot Learning with Single Vector Updates: Hyperseed's ability to learn effectively from a limited number of examples and its use of a single vector operation for learning updates are significant for efficiency.
- Topology Preservation: Hyperseed's HD-map projections are shown to be more representative of the original data topology compared to conventional Self-Organizing Maps (SOMs) in certain cases.
- Neuromorphic Hardware Suitability: The FHRR model and the algorithm's operations are well-suited for implementation on spiking neuromorphic hardware like Intel's Loihi, offering potential for significant speedups and energy savings.
- Trade-off in Hyperparameter Tuning: The classification accuracy is sensitive to the choice of FPE hyperparameters (bandwidth and quantization levels), with an inverse relationship observed for optimal performance.
Key Results
- Classification Accuracy: Hyperseed achieved average classification accuracy on par with conventional SOMs on the FCPS datasets (0.948 vs. 0.943). For the Iris dataset, Hyperseed achieved accuracies up to 0.95 with few updates. In the language identification task, Hyperseed achieved accuracies up to 0.91.
- Efficiency: Hyperseed requires substantially fewer iterations and uses a single vector operation for learning updates, leading to reduced time and computation complexity compared to SOMs.
- Neuromorphic Performance: On Intel's Loihi, Hyperseed's search procedure (BMV finding) demonstrated constant-time performance regardless of HD-map size, highlighting the computational benefits of neuromorphic implementation. The measured accuracy on Loihi matched the CPU implementation, indicating no significant error introduction.
- Topology Visualization: In some FCPS datasets, Hyperseed's projections provided a more accurate representation of the data's underlying topology than SOMs.
Limitations and Future Work
- Complex Manifold Structures: Similar to SOMs, Hyperseed's 2D HD-map implementation has limitations in describing complex manifold structures.
- Hyperparameter Sensitivity: The performance of Hyperseed is sensitive to the choice of hyperparameters, particularly for FPE encoding.
- Future Work: The authors suggest exploring higher-dimensional HD-maps and non-grid topologies for improved capability in describing complex structures. Further investigation into the performance comparison of different embeddings and the optimization of Hyperseed for sparse representations is also recommended. Adapting Hyperseed for neuromorphic hardware is an ongoing area of research.
Summary updated