hxtorch: PyTorch for BrainScaleS-2 - Perceptrons on Analog Neuromorphic Hardware

Philipp Spilger, Eric Müller, Arne Emmel, Aron Leibfried, Christian Mauch, Christian Pehle, Johannes Weis, Oliver Breitwieser, Sebastian Billaudelle, Sebastian Schmitt, Timo C. Wunderlich, Yannik Stradmann, Johannes Schemmel

Abstract

We present software facilitating the usage of the BrainScaleS-2 analog neuromorphic hardware system as an inference accelerator for artificial neural networks. The accelerator hardware is transparently integrated into the PyTorch machine learning framework using its extension interface. In particular, we provide accelerator support for vector-matrix multiplications and convolutions; corresponding software-based autograd functionality is provided for hardware-in-the-loop training. Automatic partitioning of neural networks onto one or multiple accelerator chips is supported. We analyze implementation runtime overhead during training as well as inference, provide measurements for existing setups and evaluate the results in terms of the accelerator hardware design limitations. As an application of the introduced framework, we present a model that classifies activities of daily living with smartphone sensor data.

Introduction

Neuromorphic hardware—chips designed to mimic the architecture of the brain—promises massive gains in energy efficiency and speed for artificial neural networks (ANNs). However, these chips are notoriously difficult to use. Unlike standard CPUs or GPUs, analog neuromorphic systems often require specialized knowledge of hardware-level configuration and suffer from "analog noise" (variations in hardware components) that can degrade model accuracy.

Most machine learning researchers use high-level frameworks like PyTorch or TensorFlow. There is a significant "usability gap" between these software tools and the low-level drivers required to run code on analog substrates. This paper introduces hxtorch to bridge that gap, making the BrainScaleS-2 (BSS-2) system behave like any other PyTorch-compatible backend.

The BrainScaleS-2 hardware setup and the analog neuromorphic chip.

The hxtorch Framework

The authors built hxtorch using PyTorch’s C++ extension interface. The core implementation focuses on transparently offloading heavy computations to the BSS-2 hardware while maintaining the standard PyTorch user experience.

Transparent Integration and Execution

Custom operators for matmul (matrix multiplication) and convNd (convolutions) are implemented. To a user, these look like standard PyTorch layers but are executed on the BSS-2 hardware. The software constructs a hardware-centric data flow graph and uses a Just-In-Time (JIT) execution model to manage the complex timing of sending data to the chip and receiving results.

Data flow graph for a matrix multiplication (matmul) operation within hxtorch, showing the transition from PyTorch to the BSS-2 hardware.

Automatic Partitioning

Since the physical synapse array on the chip is fixed (256 rows by 256 columns), hxtorch automatically splits larger neural network layers into "chunks" that fit on the hardware and sums the results digitally. This allows the framework to scale models beyond the physical constraints of a single chip.

Hardware-in-the-Loop Training

To combat analog noise, the authors use a hybrid approach. The "forward pass" (inference) runs on the actual hardware to capture its unique distortions, while the "backward pass" (gradient calculation) is performed in software using a standard mathematical model. This allows the network to "learn" to be robust against the specific noise of the chip. A model trained purely in software might fail on analog hardware due to component mismatch, but just one additional epoch of training with the hardware in the loop is enough to significantly recover accuracy.

Experimental Results

The framework was evaluated using the Human Activity Recognition (HAR) dataset, which contains smartphone sensor data for activity classification.

  • Performance: The system achieved a peak performance of 14.7 GOp/s (Giga-operations per second) for a single simulated accelerator.
  • Accuracy: On the HAR task, the software-only model achieved 92.7% accuracy. When moved to the BSS-2 chip without retraining, accuracy dropped significantly due to noise. However, after one epoch of hardware-in-the-loop training, the on-chip accuracy reached 82.3%.
  • Efficiency: The current implementation reaches within a factor of two of the hardware's theoretical design limits for large matrix multiplications, proving that the software overhead is manageable.

Limitations and Future Work

The current setup is limited by a 1 GBit/s Ethernet link between the host computer and the FPGA/chip, creating a communication bottleneck. Additionally, the first-generation hardware (v1) used in the study had a bug requiring the synapse array to be rewritten for every input, increasing data volume by 100x. Future work will utilize v2 hardware and expand hxtorch to support spiking neural networks (SNNs), which are even closer to biological brain function.

Summary updated

Abstract

We present software facilitating the usage of the BrainScaleS-2 analog neuromorphic hardware system as an inference accelerator for artificial neural networks. The accelerator hardware is transparently integrated into the PyTorch machine learning framework using its extension interface. In particular, we provide accelerator support for vector-matrix multiplications and convolutions; corresponding software-based autograd functionality is provided for hardware-in-the-loop training. Automatic partitioning of neural networks onto one or multiple accelerator chips is supported. We analyze implementation runtime overhead during training as well as inference, provide measurements for existing setups and evaluate the results in terms of the accelerator hardware design limitations. As an application of the introduced framework, we present a model that classifies activities of daily living with smartphone sensor data.
BibTeX
@inproceedings{spilger2020hxtorch,
  author = {Philipp Spilger and Eric Müller and Arne Emmel and Aron Leibfried and Christian Mauch and Christian Pehle and Johannes Weis and Oliver Breitwieser and Sebastian Billaudelle and Sebastian Schmitt and Timo C. Wunderlich and Yannik Stradmann and Johannes Schemmel},
  title = {{hxtorch: PyTorch for BrainScaleS-2 - Perceptrons on Analog Neuromorphic Hardware}},
  booktitle = {IoT Streams/ITEM@PKDD/ECML},
  year = {2020},
  url = {https://openreview.net/forum?id=uEOXdjSHAXi},
}
Figure