Efficient Reward-Based Structural Plasticity on a SpiNNaker 2 Prototype

Yexin Yan, David Kappel, Felix Neumärker, Johannes Partzsch, Bernhard Vogginger, Sebastian Höppner, Steve B. Furber, Wolfgang Maass, Robert Legenstein, Christian Mayr

TL;DR

This paper demonstrates how to efficiently run a complex, biologically plausible learning algorithm—reward-based synaptic sampling—on the new SpiNNaker 2 neuromorphic hardware prototype. By combining custom hardware accelerators for random number generation and exponential math with clever software-level memory management, the authors achieved a 2x speedup and a 62% reduction in energy consumption compared to standard software implementations.

What Problem Does This Paper Address?

The brain is incredibly efficient at learning complex tasks with limited power, largely because its connections are sparse, noisy, and constantly rewiring (a process called structural plasticity). Neuroscientists have modeled this using algorithms like "synaptic sampling," where synapses undergo a continuous random walk, disconnecting and reconnecting to explore new network architectures.

However, when engineers try to run these brain-inspired algorithms on silicon, the biological efficiency disappears. These models require massive amounts of random number generation (RNG) and complex exponential functions. Standard CPUs and GPUs choke on this math, while most dedicated neuromorphic chips have rigid, hardcoded plasticity rules that lack the flexibility to run such complex stochastic models. The authors tackle this gap by co-designing the software algorithm and the hardware architecture to make simulating these noisy, dynamic brains computationally viable.

Methods

The researchers utilized a 4-core prototype of the SpiNNaker 2 system. Unlike its predecessor, SpiNNaker 2 features ARM Cortex-M4F cores (which support floating-point arithmetic) augmented with specific hardware accelerators for random number generation and exponential functions.

To map the synaptic sampling algorithm to this hardware, they employed several key techniques:

  • Hardware Acceleration: They replaced expensive software-based Gaussian noise generation with hardware-accelerated uniform noise, and offloaded exponential calculations to the custom fixed-point accelerator.
  • Local SRAM Computation: To avoid the massive energy and latency penalties of fetching data from off-chip DRAM, they compressed the model using 16-bit floating-point data types so all synapse parameters could live entirely in the chip's local SRAM.
  • Random Reallocation of Synapses: In the standard biological model, disconnected synapses are still tracked in memory until they randomly reconnect. The authors introduced a novel software trick: when a synapse's weight drops to zero, its memory slot is immediately recycled and reassigned to a new random target neuron. This keeps the fanout constant, saves memory, and stops the CPU from wasting cycles on "dead" connections.

Datasets

Not discussed in the paper. Instead of standard machine learning benchmarks (like MNIST), the authors evaluated the system using a custom, synthetic closed-loop reinforcement learning task. The network was fed two distinct random spike patterns and had to learn to route them to specific output populations based on a global reward signal.

Key Insights and Ideas

  • Hardware-Software Co-design is the Sweet Spot: You don't necessarily need fully custom analog circuits to achieve neuromorphic efficiency. Standard ARM cores paired with just a few highly specific math accelerators (RNG and Exp) provide a perfect blend of software flexibility and hardware efficiency.
  • Turning Bugs into Features: The True Random Number Generator (TRNG) on the chip actually harvests its entropy from the jitter of the chip's own clock generators. They took an unavoidable hardware flaw (clock noise) and turned it into a free computational resource to drive the network's structural exploration.
  • "Garbage Collection" Accelerates Learning: The random reallocation technique isn't just a memory-saving hack; it actually makes the AI learn faster. By instantly recycling dead synapses into new random connections, the network explores the parameter space much more aggressively than the strict biological model.

Key Results

  • Speedup: Using the hardware accelerators reduced the computation time for a single plasticity update by more than half (from 236 clock cycles in pure software to 110 cycles). The time spent calculating random numbers and exponentials plummeted from 62% to just 18% of the total workload.
  • Energy Efficiency: By keeping all data in local SRAM and utilizing the accelerators, energy consumption per time step was slashed by 62% (from 450.3 $\mu$J down to 171 $\mu$J).
  • Real-Time Capability: The system successfully ran the closed-loop reinforcement learning task in real-time. A 1ms biological time step took only 0.76ms to compute on the hardware.

Limitations and Future Work

Limitations: The current implementation was tested on a small prototype chip with only 4 cores. Because the math is still relatively heavy, the network size is currently compute-bound, limited to about 3,000 plastic synapses per core.

Future Work: The authors plan to scale this implementation up to the final full-scale SpiNNaker 2 system, which will feature 144 cores per chip and support massive multi-chip networks. Once scaled, they aim to apply this reward-based learning framework to high-dimensional, real-world inputs, such as processing data from event-based cameras (Dynamic Vision Sensors).

Summary updated

Abstract

Advances in neuroscience uncover the mechanisms employed by the brain to efficiently solve complex learning tasks with very limited resources. However, the efficiency is often lost when one tries to port these findings to a silicon substrate, since brain-inspired algorithms often make extensive use of complex functions, such as random number generators, that are expensive to compute on standard general purpose hardware. The prototype chip of the second generation SpiNNaker system is designed to overcome this problem. Low-power advanced RISC machine (ARM) processors equipped with a random number generator and an exponential function accelerator enable the efficient execution of brain-inspired algorithms. We implement the recently introduced reward-based synaptic sampling model that employs structural plasticity to learn a function or task. The numerical simulation of the model requires to update the synapse variables in each time step including an explorative random term. To the best of our knowledge, this is the most complex synapse model implemented so far on the SpiNNaker system. By making efficient use of the hardware accelerators and numerical optimizations, the computation time of one plasticity update is reduced by a factor of 2. This, combined with fitting the model into to the local static random access memory (SRAM), leads to 62% energy reduction compared to the case without accelerators and the use of external dynamic random access memory (DRAM). The model implementation is integrated into the SpiNNaker software framework allowing for scalability onto larger systems. The hardware-software system presented in this paper paves the way for power-efficient mobile and biomedical applications with biologically plausible brain-inspired algorithms.
BibTeX
@article{yan2019efficient,
  author = {Yexin Yan and David Kappel and Felix Neumärker and Johannes Partzsch and Bernhard Vogginger and Sebastian Höppner and Steve B. Furber and Wolfgang Maass and Robert Legenstein and Christian Mayr},
  title = {{Efficient Reward-Based Structural Plasticity on a SpiNNaker 2 Prototype}},
  journal = {IEEE Trans. Biomed. Circuits Syst.},
  year = {2019},
  url = {https://openreview.net/forum?id=iL6cv5dYjB},
}

Loading related content...

Figure