Memory-Efficient Synaptic Connectivity for Spike-Timing- Dependent Plasticity

Bruno U. Pedroni, Siddharth Joshi, Stephen Deiss, Sadique Sheik, Georgios Detorakis, Somnath Paul, Charles Augustine, Emre Neftci, Gert CauwenberghsFrontiers in Neuroscience 2019 · DOI

Abstract

Spike-Timing-Dependent Plasticity (STDP) is a bio-inspired local incremental weight update rule commonly used for online learning in spike-based neuromorphic systems. In STDP, the intensity of long-term potentiation and depression in synaptic efficacy (weight) between neurons is expressed as a function of the relative timing between pre- and post-synaptic action potentials (spikes), while the polarity of change is dependent on the order (causality) of the spikes. Online STDP weight updates for causal and acausal relative spike times are activated at the onset of post- and pre-synaptic spike events, respectively, implying access to synaptic connectivity both in forward (pre-to-post) and reverse (post-to-pre) directions. Here we study the impact of different arrangements of synaptic connectivity tables on weight storage and STDP updates for large-scale neuromorphic systems. We analyze the memory efficiency for varying degrees of density in synaptic connectivity, ranging from crossbar arrays for full connectivity to pointer-based lookup for sparse connectivity. Finally, we present an alternative formulation of STDP via a delayed causal update mechanism that permits efficient weight access, requiring no more than forward connectivity lookup.

The Connectivity Bottleneck in Neuromorphic Hardware

In neuromorphic engineering, the goal is to build hardware that mimics the brain's efficiency. A cornerstone of this is STDP, a learning rule where synaptic weights change based on the relative timing of spikes from pre-synaptic and post-synaptic neurons. However, standard STDP requires bidirectional lookups:

  1. Acausal updates: Triggered by a pre-synaptic spike (requires "forward" access: which post-synaptic neurons do I connect to?).
  2. Causal updates: Triggered by a post-synaptic spike (requires "reverse" access: which pre-synaptic neurons connect to me?).
Synaptic connectivity architecture between pre-synaptic inputs and post-synaptic neurons.

Most hardware is optimized for forward access. To support reverse access, designers often have to double their memory to store a second "reverse" connectivity table or perform exhaustive, slow searches. For large-scale sparse networks, this bidirectional requirement is a massive bottleneck for both power and area.

Methods: Delayed Causal Updates and Sparse Data Structures

The authors analyze four synaptic weight data structures: Crossbar (Dense), PB-CSR (Pointer-Based Compressed Sparse Row), PB-RLE (Run-Length Encoding), and PB-BMP (Bitmap).

The core innovation is the Delayed Causal Update. Instead of performing a causal update the moment a post-synaptic neuron spikes, the system waits until the pre-synaptic neuron’s STDP timer expires. At that moment, the system performs a forward lookup to update all its connections at once. Because the update is mathematically equivalent to the original rule but only uses forward pointers, it bypasses the need for reverse connectivity tables entirely.

Overview of STDP: (A) Network architecture, (B) Timing-dependent weight change, and (C) Common window functions.

Key Results and Insights

  • Functional Equivalence: The delayed causal update achieved a Mean Squared Error (MSE) of zero compared to the original STDP formulation when using multiple timers to handle high firing rates.
  • Memory Savings: By eliminating reverse connectivity tables, the proposed method can reduce the memory footprint of sparse synaptic connectivity by nearly 50%.
  • Density Dictates Structure: Traditional crossbars only become more efficient than pointer-based sparse structures when connectivity density exceeds ~80%, which is rare in biological modeling. PB-CSR is the most efficient for forward access in sparse regimes.

Conclusion

This paper introduces a mechanism that eliminates the need for costly reverse connectivity lookups in neuromorphic hardware. By restructuring how and when synaptic weights are updated, the authors enable the use of memory-efficient compressed data structures, significantly reducing storage and access costs for large-scale, sparse neural networks. Future work includes extending this "forward-only" logic to more complex rules, such as triplet-based plasticity.

Summary updated

Abstract

Spike-Timing-Dependent Plasticity (STDP) is a bio-inspired local incremental weight update rule commonly used for online learning in spike-based neuromorphic systems. In STDP, the intensity of long-term potentiation and depression in synaptic efficacy (weight) between neurons is expressed as a function of the relative timing between pre- and post-synaptic action potentials (spikes), while the polarity of change is dependent on the order (causality) of the spikes. Online STDP weight updates for causal and acausal relative spike times are activated at the onset of post- and pre-synaptic spike events, respectively, implying access to synaptic connectivity both in forward (pre-to-post) and reverse (post-to-pre) directions. Here we study the impact of different arrangements of synaptic connectivity tables on weight storage and STDP updates for large-scale neuromorphic systems. We analyze the memory efficiency for varying degrees of density in synaptic connectivity, ranging from crossbar arrays for full connectivity to pointer-based lookup for sparse connectivity. The study includes comparison of storage and access costs and efficiencies for each memory arrangement, along with a trade-off analysis of the benefits of each data structure depending on application requirements and budget. Finally, we present an alternative formulation of STDP via a delayed causal update mechanism that permits efficient weight access, requiring no more than forward connectivity lookup. We show functional equivalence of the delayed causal updates to the original STDP formulation, with substantial savings in storage and access costs and efficiencies for networks with sparse synaptic connectivity as typically encountered in large-scale models in computational neuroscience.
BibTeX
@inproceedings{pedroni2019memory,
  author = {Bruno U. Pedroni and Siddharth Joshi and Stephen Deiss and Sadique Sheik and Georgios Detorakis and Somnath Paul and Charles Augustine and Emre Neftci and Gert Cauwenberghs},
  title = {{Memory-Efficient Synaptic Connectivity for Spike-Timing- Dependent Plasticity}},
  booktitle = {Frontiers in Neuroscience},
  year = {2019},
  doi = {10.3389/fnins.2019.00357},
}
Figure