SynergicLearning: Neural Network-Based Feature Extraction for Highly-Accurate Hyperdimensional Learning

Mahdi Nazemi, Amirhossein Esmaili, Arash Fayyazi, Massoud Pedram

Introduction

The pursuit of efficient "on-chip learning" often involves a trade-off between the high accuracy of Neural Networks (NNs) and the computational efficiency of Hyperdimensional (HD) computing. While NNs excel at automatic feature extraction, they are resource-intensive and prone to "catastrophic forgetting." Conversely, HD learning—inspired by the brain's use of massive, high-dimensional patterns—is fast and hardware-friendly but often suffers from lower accuracy due to its reliance on raw or handcrafted features. Furthermore, maintaining accuracy in traditional HD models typically requires massive vectors (e.g., $d=10,000$), which increases hardware latency.

SynergicLearning addresses these challenges by introducing a hybrid framework that combines an "encoder-aware" NN feature extractor with an HD classifier. This synergy allows for NN-level accuracy while significantly improving hardware latency by 2.13x and power efficiency by 1.60x compared to state-of-the-art HD implementations.

Methodology

The SynergicLearning framework is built upon three primary components: encoder-aware training, a two-step learning process, and specialized hardware.

The SynergicLearning framework architecture, highlighting the integration of the HD codec within the NN training loop and the subsequent HD classifier.

Encoder-Aware Training

The core innovation is the inclusion of an HD encoder-decoder (codec) directly within the NN's training loop. Because HD operations are non-differentiable, the authors utilize an identity function approximation during backpropagation. This "encoder-aware" approach forces the NN to learn features that are inherently robust to the quantization noise and errors introduced during the HD encoding process.

Two-Step Learning and Hardware Optimization

The learning process occurs in two stages. First, the NN is trained to become a high-quality feature extractor. Once trained, the NN's weights are fixed, and the HD classifier is trained on-chip. This enables rapid, one-pass fine-tuning as new data arrives, mitigating catastrophic forgetting. To support this, the authors developed a custom hardware architecture with dedicated NN (systolic array) and HD (parallel XOR and majority counters) processing modules, optimized by a specialized compiler.

Key Results

The framework was evaluated on the Human Activity Recognition (HAR) and ISOLET datasets. SynergicLearning achieved 96.44% accuracy on HAR and 96.67% on ISOLET, outperforming traditional HD models by over 10% and slightly exceeding standard NN performance.

Accuracy comparison of SynergicLearning (Encoder-aware NN + HDL) against standard NN+HDL and standalone HDL across varying hyperdimensional vector sizes.

A significant finding is the impact of feature quality on dimensionality. While traditional HD computing requires massive dimensions to overcome noise, SynergicLearning maintains high accuracy with vectors as small as $d=16$—a reduction of two to three orders of magnitude. This reduction directly translates to the observed 2.13x improvement in latency and 1.60x improvement in power efficiency. Furthermore, the system demonstrated superior scaling in incremental learning scenarios, effectively utilizing new data without the need for full retraining.

Conclusion

SynergicLearning successfully bridges the gap between NNs and HD computing by making the feature extraction process aware of the downstream HD constraints. While currently validated on sensor-based tasks, future work could explore scaling this hybrid approach to more complex computer vision tasks and further automating the initial NN training for fully autonomous edge operation.

Summary updated

BibTeX
@inproceedings{nazemi2020synergiclearning,
  author = {Mahdi Nazemi and Amirhossein Esmaili and Arash Fayyazi and Massoud Pedram},
  title = {{SynergicLearning: Neural Network-Based Feature Extraction for Highly-Accurate Hyperdimensional Learning}},
  booktitle = {ICCAD},
  year = {2020},
  doi = {10.1145/3400302.3415696},
}
Figure