Automated Architecture Search for Brain-inspired Hyperdimensional Computing

Junhuan Yang, Yi Sheng, Sizhe Zhang, Ruixuan Wang, K. Bo Foreman, Mikell Paige, Xun Jiao, Weiwen Jiang, Lei Yang

Abstract

This paper represents the first effort to explore an automated architecture search for hyperdimensional computing (HDC), a type of brain-inspired neural network. Currently, HDC design is largely carried out in an application-specific ad-hoc manner, which significantly limits its application. Furthermore, the approach leads to inferior accuracy and efficiency, which suggests that HDC cannot perform competitively against deep neural networks. Herein, we present a thorough study to formulate an HDC architecture search space. On top of the search space, we apply reinforcement-learning to automatically explore the HDC architectures. The searched HDC architectures show competitive performance on case studies involving a drug discovery dataset and a language recognition task. On the ClinTox dataset, which tries to learn features from developed drugs that passed/failed clinical trials for toxicity reasons, the searched HDC architecture obtains the state-of-the-art ROC-AUC scores, which are 0.80% higher than the manually designed HDC and 9.75% higher than conventional neural networks. Similar results are achieved on the language recognition task, with 1.27% higher performance than conventional methods.

Introduction: The Challenge of Manual HDC Design

Hyperdimensional Computing (HDC) is a brain-inspired paradigm that uses high-dimensional "hypervectors" (typically 10,000+ dimensions) to perform robust and energy-efficient computations. While HDC offers a promising alternative to traditional neural networks, its adoption has been hindered by an ad-hoc design process. Unlike deep learning, which has matured through Neural Architecture Search (NAS), HDC architectures are often manually tuned for specific applications.

This "hand-crafting" problem leads to sub-optimal performance and limits the generalizability of HDC. Researchers typically guess at the best dimensionality, sparsity, and mathematical operations to use, which often makes HDC seem less competitive than deep learning. The authors aim to bridge this gap by formalizing the HDC architecture and automating its discovery through a framework called AutoHDC.

AutoHDC: Automating the Search Space

To address the limitations of manual design, the authors identified ten critical parameters that define an HDC model, categorized into three distinct phases. This formalization allows for a systematic exploration of the design space.

The AutoHDC search space encompasses global properties, datatypes, and operations across three distinct phases of the HDC pipeline.

The search space includes:

  1. Global Properties: Hypervector dimension (ranging from 1k to 20k) and sparsity (10% to 90% of bits set to '1').
  2. Datatypes: Precision choices (binary, bipolar, or various integer precisions from int8 to int64) for base, encoded, and resultant vectors.
  3. Operations: Binding methods (XOR, AND, OR, Multiplication) and permutation strategies (rotations) for representing sequential data.

The Reinforcement Learning Search Engine

AutoHDC employs a Reinforcement Learning (RL) approach to navigate this complex space. An RNN-based controller proposes an architecture, which is then executed and evaluated by an HDC executor. The resulting performance metric (e.g., accuracy or ROC-AUC) is fed back to the controller as a reward signal, guiding it toward better designs in subsequent iterations.

The RL-based search engine uses an RNN controller to iteratively propose and refine HDC architectures based on validation rewards.

Experimental Evaluation

The framework was tested on two diverse sequential tasks:

  • Drug Discovery (ClinTox): Predicting clinical trial toxicity from SMILES strings (text-based chemical notation). This dataset contains 1,491 compounds.
  • Language Recognition: A 21-class classification task using the Wortschatz and Europarl corpora.

Key Results

On the ClinTox dataset, AutoHDC achieved a state-of-the-art ROC-AUC of 0.9949, outperforming manually designed HDC by 0.80% and conventional neural networks (like D-MPNN) by 9.75%. In language recognition, the automated model reached 99.07% accuracy, surpassing previous state-of-the-art methods (like Random Indexing) by 1.27% and other baselines by up to 8.07%.

The automated search revealed several non-intuitive findings that challenge traditional HDC assumptions:

  • Sparsity is a Critical Lever: While 50% sparsity is often assumed to be ideal, AutoHDC found that different levels are required for specific datasets like ClinTox to achieve peak performance.
  • Operations Matter: Standard XOR binding is not always optimal; element-wise AND or OR operations can yield better results depending on the nature of the data.
  • HDC vs. NNs in Low-Data Regimes: Well-tuned HDC models can outperform massive conventional neural networks (like Graph Convolutional Networks) in structured tasks, especially when data is limited.

Conclusion

AutoHDC demonstrates that brain-inspired computing can be systematically optimized. By moving away from ad-hoc design, HDC becomes a competitive and adaptable tool for diverse domains. While the search process itself requires initial computation, the resulting models maintain HDC’s inherent advantages of high parallelism and efficiency. Future work will focus on expanding the search space and mapping these designs onto specialized hardware to fully realize their energy-efficiency potential.

Summary updated

Abstract

This paper represents the first effort to explore an automated architecture search for hyperdimensional computing (HDC), a type of brain-inspired neural network. Currently, HDC design is largely carried out in an application-specific ad-hoc manner, which significantly limits its application. Furthermore, the approach leads to inferior accuracy and efficiency, which suggests that HDC cannot perform competitively against deep neural networks. Herein, we present a thorough study to formulate an HDC architecture search space. On top of the search space, we apply reinforcement-learning to automatically explore the HDC architectures. The searched HDC architectures show competitive performance on case studies involving a drug discovery dataset and a language recognition task. On the Clintox dataset, which tries to learn features from developed drugs that passed/failed clinical trials for toxicity reasons, the searched HDC architecture obtains the state-of-the-art ROC-AUC scores, which are 0.80% higher than the manually designed HDC and 9.75% higher than conventional neural networks. Similar results are achieved on the language recognition task, with 1.27% higher performance than conventional methods.
BibTeX
@misc{yang2022automated,
  author = {Junhuan Yang and Yi Sheng and Sizhe Zhang and Ruixuan Wang and K. Bo Foreman and Mikell Paige and Xun Jiao and Weiwen Jiang and Lei Yang},
  title = {{Automated Architecture Search for Brain-inspired Hyperdimensional Computing}},
  year = {2022},
  eprint = {2202.05827},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.05827},
}
Figure