hxtorch: PyTorch for BrainScaleS-2 - Perceptrons on Analog Neuromorphic Hardware

Philipp Spilger, Eric Müller, Arne Emmel, Aron Leibfried, Christian Mauch, Christian Pehle, Johannes Weis, Oliver Breitwieser, Sebastian Billaudelle, Sebastian Schmitt, Timo C. Wunderlich, Yannik Stradmann, Johannes Schemmel

TL;DR

The researchers present hxtorch, a software framework that seamlessly integrates the BrainScaleS-2 (BSS-2) analog neuromorphic chip into the PyTorch ecosystem. By providing a transparent interface for vector-matrix multiplications and convolutions, hxtorch allows researchers to use specialized analog hardware as a high-efficiency inference accelerator without leaving their familiar deep learning environment.

What Problem Does This Paper Address?

Neuromorphic hardware—chips designed to mimic the architecture of the brain—promises massive gains in energy efficiency and speed for artificial neural networks (ANNs). However, these chips are notoriously difficult to use. Unlike standard CPUs or GPUs, analog neuromorphic systems often require specialized knowledge of hardware-level configuration, and they suffer from "analog noise" (variations in hardware components) that can degrade model accuracy.

Most machine learning researchers use high-level frameworks like PyTorch or TensorFlow. There is a significant "usability gap" between these software tools and the low-level drivers required to run code on analog substrates. This paper introduces hxtorch to bridge that gap, making the BrainScaleS-2 system behave like any other PyTorch-compatible backend.

Methods

The authors built hxtorch using PyTorch’s C++ extension interface. The core of the implementation involves:

  • Transparent Integration: Custom operators for matmul (matrix multiplication) and convNd (convolutions) are implemented. To a user, these look like standard PyTorch layers but are executed on the BSS-2 hardware.
  • Graph-Based Execution: The software constructs a hardware-centric data flow graph. It uses a Just-In-Time (JIT) execution model to manage the complex timing of sending data to the chip and receiving results.
  • Automatic Partitioning: Since the physical synapse array on the chip is fixed (256 rows by 256 columns), hxtorch automatically splits larger neural network layers into "chunks" that fit on the hardware and sums the results digitally.
  • Hardware-in-the-Loop Training: To combat analog noise, the authors use a hybrid approach. The "forward pass" (inference) runs on the actual hardware to capture its unique distortions, while the "backward pass" (gradient calculation) is performed in software using a standard mathematical model. This allows the network to "learn" to be robust against the specific noise of the chip.

Datasets

The framework was evaluated using the Human Activity Recognition (HAR) dataset. This dataset contains smartphone sensor data (accelerometer and gyroscope) from 30 subjects performing activities like walking, sitting, and standing. The goal is to classify the activity based on 9 channels of sensor data.

Key Insights and Ideas

  • Analog as an Accelerator: While BSS-2 is often used for spiking neural networks (SNNs), this paper highlights its power in "non-spiking mode" as a high-speed engine for the vector-matrix multiplications that power traditional ANNs.
  • Software-Defined Hardware: The authors emphasize that the success of custom accelerators depends on user-friendly interfaces. By adhering to the torch.nn.Module and torch.autograd.Function interfaces, they allow pre-trained weights from standard models to be loaded directly onto the neuromorphic chip.
  • Training Against Noise: A key "aha" moment is the effectiveness of hardware-in-the-loop training. A model trained purely in software might fail on analog hardware due to component mismatch, but just one additional epoch of training with the hardware in the loop is enough to significantly recover accuracy.
  • Parallelizing Convolutions: The paper describes a clever layout for 1D convolutions that maximizes the usage of the synapse array, reducing execution time and energy consumption by a factor proportional to the number of parallel executions.

Key Results

  • Performance: The system achieved a peak performance of 14.7 GOp/s (Giga-operations per second) for a single simulated accelerator.
  • Accuracy: On the Human Activity Recognition task, the software-only model achieved 92.7% accuracy. When moved to the BSS-2 chip without retraining, accuracy dropped significantly due to noise. However, after one epoch of hardware-in-the-loop training, the on-chip accuracy reached 82.3%.
  • Efficiency: The current implementation reaches within a factor of two of the hardware's theoretical design limits for large matrix multiplications, proving that the software overhead is manageable.

Limitations and Future Work

  • Communication Bottlenecks: The current setup is limited by a 1 GBit/s Ethernet link between the host computer and the FPGA/chip. This creates a bottleneck that prevents the system from reaching its full potential speed.
  • Hardware "Bugs": The first-generation hardware (v1) used in the study had a bug requiring the synapse array to be rewritten for every input, increasing data volume by 100x. The authors expect significantly better performance on the upcoming v2 hardware.
  • Precision: The hardware uses 5-bit inputs and 6-bit weights. While sufficient for many tasks, this reduced precision requires careful quantization during training.
  • Expanding to Spiking: While this paper focuses on standard ANNs (perceptrons), the authors plan to extend hxtorch to support spiking neural networks, which are even closer to biological brain function.

Summary updated

Abstract

We present software facilitating the usage of the BrainScaleS-2 analog neuromorphic hardware system as an inference accelerator for artificial neural networks. The hardware is transparently integrated into the PyTorch machine learning framework. In particular, we support vector-matrix multiplications and convolutions; corresponding software-based autograd functionality is provided for hardware-in-the-loop training. The software provides support for automatic partitioning and scheduling of neural networks onto one or multiple chips. We discuss the implementation including optimizations, analyze runtime overhead, measure performance and evaluate the results in terms of the hardware design limitations. As an application of the introduced framework, we present a model that classifies activities of daily living with smartphone sensor data.
BibTeX
@inproceedings{spilger2020hxtorch,
  author = {Philipp Spilger and Eric Müller and Arne Emmel and Aron Leibfried and Christian Mauch and Christian Pehle and Johannes Weis and Oliver Breitwieser and Sebastian Billaudelle and Sebastian Schmitt and Timo C. Wunderlich and Yannik Stradmann and Johannes Schemmel},
  title = {{hxtorch: PyTorch for BrainScaleS-2 - Perceptrons on Analog Neuromorphic Hardware}},
  booktitle = {IoT Streams/ITEM@PKDD/ECML},
  year = {2020},
  url = {https://openreview.net/forum?id=uEOXdjSHAXi},
}

Loading related content...

Figure