Reward-based learning under hardware constraints - Using a RISC\n processor embedded in a neuromorphic substrate

Simon Friedmann, Nicolas Frémaux, Johannes Schemmel, Wulfram Gerstner, Karlheinz Meier

Abstract

In this study, we propose and analyze in simulations a new, highly flexible method of implementing synaptic plasticity in a wafer-scale, accelerated neuromorphic hardware system. Flexibility is achieved by embedding a general-purpose processor dedicated to plasticity into the wafer. To evaluate the suitability of the proposed system, we use a reward modulated STDP rule in a spike train learning task. We find that probabilistic updates can increase the performance of low-resolution weights, a simple interface between analog synapses and processor is sufficient for learning, and performance is insensitive to mismatch. However, for highly accelerated systems, communication latency must be kept to a minimum.

Introduction

Synaptic plasticity is the cornerstone of learning in biological systems. For neuromorphic engineers, implementing this plasticity in silicon presents a significant challenge: balancing the efficiency of hard-wired circuits with the flexibility required to explore diverse learning rules. Hard-wiring specific rules like Spike-Timing Dependent Plasticity (STDP) into silicon makes chips inflexible, while offloading computation to external CPUs creates communication bottlenecks. This paper investigates a hybrid approach: embedding a small RISC processor directly into a wafer-scale neuromorphic substrate to manage synaptic updates.

System Architecture

The researchers utilized the BrainScaleS system, a wafer-scale hardware platform operating at $10^5$ times biological real-time. The core innovation is the integration of an Embedded Plasticity Processor (EPP) within the wafer building blocks.

In this architecture, analog circuits in each synapse track the timing between pre- and post-synaptic spikes to generate an "eligibility trace." The EPP, a digital RISC processor based on the PowerISA, reads these analog values, calculates weight changes based on a global reward signal, and writes the updated weights back to the synapses.

The Embedded Plasticity Processor (EPP)

The EPP is designed for area efficiency and specialized functionality. Occupying only 0.14 mm² in 65nm technology, it includes a dedicated "SYNAPSE" functional unit in its backend to facilitate high-speed interaction with the analog synaptic array.

Learning under Hardware Constraints

To evaluate the system, the authors implemented Reward-modulated STDP (R-STDP) in a spike train learning task. A single layer of neurons was trained to fire at specific target times using only a global reward signal. The study simulated several real-world hardware constraints:

  • Discretization: Weights are restricted to 4-bit or 8-bit integers.
  • Thresholded Readout: The processor uses a 1-bit comparison for analog trace values rather than high-precision ADCs.
  • Analog Mismatch: Variations in analog components due to manufacturing imperfections.
  • Latency: Delays in the reward signal arrival from the external environment.

Key Results and Insights

The simulations revealed several critical findings for neuromorphic design:

  • Probabilistic Rounding: Standard rounding in 4-bit hardware significantly degrades learning (reward drops from 0.54 to 0.37). However, using probabilistic updates—where the probability of a bit flip is proportional to the calculated update—restores performance to 0.46.
  • Robustness to Analog Imperfections: The system proved remarkably resilient to analog circuit mismatch. The learning algorithm naturally compensates for physical variations across synapses.
  • The Latency Bottleneck: Due to the $10^5$ acceleration factor, even a 1ms real-world delay in reward delivery equates to a 100-second delay for the neurons. This causes eligibility traces to decay, highlighting the need for low-latency environments in accelerated hardware.

Conclusion

This study demonstrates that embedding general-purpose RISC processors into neuromorphic wafers provides a viable and flexible path for implementing complex learning rules. While hardware constraints like low weight resolution and communication latency pose challenges, they can be mitigated through algorithmic techniques like probabilistic rounding and careful system design. Future work will explore the scalability of this architecture to deeper networks and more complex, closed-loop reinforcement learning tasks.

Summary updated

Abstract

In this study, we propose and analyze in simulations a new, highly flexible\nmethod of implementing synaptic plasticity in a wafer-scale, accelerated\nneuromorphic hardware system. The study focuses on globally modulated STDP, as\na special use-case of this method. Flexibility is achieved by embedding a\ngeneral-purpose processor dedicated to plasticity into the wafer. To evaluate\nthe suitability of the proposed system, we use a reward modulated STDP rule in\na spike train learning task. A single layer of neurons is trained to fire at\nspecific points in time with only the reward as feedback. This model is\nsimulated to measure its performance, i.e. the increase in received reward\nafter learning. Using this performance as baseline, we then simulate the model\nwith various constraints imposed by the proposed implementation and compare the\nperformance. The simulated constraints include discretized synaptic weights, a\nrestricted interface between analog synapses and embedded processor, and\nmismatch of analog circuits. We find that probabilistic updates can increase\nthe performance of low-resolution weights, a simple interface between analog\nsynapses and processor is sufficient for learning, and performance is\ninsensitive to mismatch. Further, we consider communication latency between\nwafer and the conventional control computer system that is simulating the\nenvironment. This latency increases the delay, with which the reward is sent to\nthe embedded processor. Because of the time continuous operation of the analog\nsynapses, delay can cause a deviation of the updates as compared to the not\ndelayed situation. We find that for highly accelerated systems latency has to\nbe kept to a minimum. This study demonstrates the suitability of the proposed\nimplementation to emulate the selected reward modulated STDP learning rule.\n
BibTeX
@inproceedings{friedmann2013reward,
  author = {Simon Friedmann and Nicolas Frémaux and Johannes Schemmel and Wulfram Gerstner and Karlheinz Meier},
  title = {{Reward-based learning under hardware constraints - Using a RISC\textbackslash\{\}n processor embedded in a neuromorphic substrate}},
  booktitle = {Frontiers in Neuroscience},
  year = {2013},
  doi = {10.3389/fnins.2013.00160},
}
Figure