Abstract
We present software facilitating the usage of the BrainScaleS-2 analog neuromorphic hardware system as an inference accelerator for artificial neural networks. The accelerator hardware is transparently integrated into the PyTorch machine learning framework using its extension interface. In particular, we provide accelerator support for vector-matrix multiplications and convolutions; corresponding software-based autograd functionality is provided for hardware-in-the-loop training. Automatic partitioning of neural networks onto one or multiple accelerator chips is supported. We analyze implementation runtime overhead during training as well as inference, provide measurements for existing setups and evaluate the results in terms of the accelerator hardware design limitations. As an application of the introduced framework, we present a model that classifies activities of daily living with smartphone sensor data.
Introduction
Neuromorphic hardware—chips designed to mimic the architecture of the brain—promises massive gains in energy efficiency and speed for artificial neural networks (ANNs). However, these chips are notoriously difficult to use. Unlike standard CPUs or GPUs, analog neuromorphic systems often require specialized knowledge of hardware-level configuration and suffer from "analog noise" (variations in hardware components) that can degrade model accuracy.
Most machine learning researchers use high-level frameworks like PyTorch or TensorFlow. There is a significant "usability gap" between these software tools and the low-level drivers required to run code on analog substrates. This paper introduces hxtorch to bridge that gap, making the BrainScaleS-2 (BSS-2) system behave like any other PyTorch-compatible backend.
The hxtorch Framework
The authors built hxtorch using PyTorch’s C++ extension interface. The core implementation focuses on transparently offloading heavy computations to the BSS-2 hardware while maintaining the standard PyTorch user experience.
Transparent Integration and Execution
Custom operators for matmul (matrix multiplication) and convNd (convolutions) are implemented. To a user, these look like standard PyTorch layers but are executed on the BSS-2 hardware. The software constructs a hardware-centric data flow graph and uses a Just-In-Time (JIT) execution model to manage the complex timing of sending data to the chip and receiving results.
Automatic Partitioning
Since the physical synapse array on the chip is fixed (256 rows by 256 columns), hxtorch automatically splits larger neural network layers into "chunks" that fit on the hardware and sums the results digitally. This allows the framework to scale models beyond the physical constraints of a single chip.
Hardware-in-the-Loop Training
To combat analog noise, the authors use a hybrid approach. The "forward pass" (inference) runs on the actual hardware to capture its unique distortions, while the "backward pass" (gradient calculation) is performed in software using a standard mathematical model. This allows the network to "learn" to be robust against the specific noise of the chip. A model trained purely in software might fail on analog hardware due to component mismatch, but just one additional epoch of training with the hardware in the loop is enough to significantly recover accuracy.
Experimental Results
The framework was evaluated using the Human Activity Recognition (HAR) dataset, which contains smartphone sensor data for activity classification.
- Performance: The system achieved a peak performance of 14.7 GOp/s (Giga-operations per second) for a single simulated accelerator.
- Accuracy: On the HAR task, the software-only model achieved 92.7% accuracy. When moved to the BSS-2 chip without retraining, accuracy dropped significantly due to noise. However, after one epoch of hardware-in-the-loop training, the on-chip accuracy reached 82.3%.
- Efficiency: The current implementation reaches within a factor of two of the hardware's theoretical design limits for large matrix multiplications, proving that the software overhead is manageable.
Limitations and Future Work
The current setup is limited by a 1 GBit/s Ethernet link between the host computer and the FPGA/chip, creating a communication bottleneck. Additionally, the first-generation hardware (v1) used in the study had a bug requiring the synapse array to be rewritten for every input, increasing data volume by 100x. Future work will utilize v2 hardware and expand hxtorch to support spiking neural networks (SNNs), which are even closer to biological brain function.
Summary updated