Algorithm-hardware co-design of neuromorphic networks with dual memory pathways

Pengfei Sun, Zhe Su, Jascha Achterberg, Giacomo Indiveri, Marcus Ghosh, Danyal Akarca

TL;DR

This paper introduces a novel "dual memory pathway" (DMP) architecture for spiking neural networks (SNNs) that mimics the brain's fast and slow processing. By explicitly separating fast spiking activity from a compact, slow memory state, they achieve significant improvements in accuracy, parameter efficiency, and energy consumption on temporal tasks, paving the way for more efficient neuromorphic hardware.

What Problem Does This Paper Address?

Spiking neural networks (SNNs) are promising for event-driven sensing due to their energy efficiency. However, a major challenge is enabling them to maintain context over long temporal sequences without sacrificing their inherent sparsity and low power consumption. Traditional approaches like dense recurrence or long, learnable delays introduce significant computational and memory overhead, making them impractical for hardware. This paper tackles the problem of achieving long-term temporal memory in SNNs while remaining hardware-efficient.

Methods

The core innovation is the Dual Memory Pathway (DMP) architecture. This is a co-design approach at both the algorithmic and hardware levels:

  • Algorithmic Level: Inspired by the brain's fast-slow cortical organization, each layer in the SNN is augmented with an explicit slow memory pathway. This pathway maintains a compact, low-dimensional state that summarizes recent activity. This slow state then modulates the fast spiking activity of the neurons. This explicit memory allows the network to retain context over long timescales without needing dense recurrence or long delays.
  • Hardware Level: A near-memory-compute architecture is designed to leverage the DMP algorithm. It features heterogeneous data paths for sparse spike integration and dense memory operations, keeping the compact shared state on-chip and optimizing data flow to minimize off-chip memory traffic. Key optimizations include:
  • Dataflow-dependency optimization: Breaking temporal dependencies to allow parallel processing of fast and slow pathways.
  • Operator-fusion optimization: Combining multiple operations (leaky updates, spike integration, memory integration) into a single process to reduce memory accesses and maximize arithmetic intensity.
  • Operand-stationarity optimization: Using input-stationary for sparse spike integration and output-stationary for dense memory integration to further improve efficiency.

Datasets

The authors evaluate their approach on several temporally structured benchmarks:

  • Permuted Sequential MNIST (PS-MNIST): A variant of MNIST where images are presented as a pixel stream, requiring long-range temporal memory.
  • Sequential MNIST (S-MNIST): Similar to PS-MNIST, but without the permutation, testing integration across hundreds of steps.
  • Spiking Heidelberg Digits (SHD): Event-based auditory classification dataset encoding spoken digits with spike trains.
  • Spiking Speech Commands (SSC): Another event-based auditory dataset representing spoken word categories.

Key Insights and Ideas

  • Explicit, compact slow memory is key: The paper demonstrates that making temporal context explicit in a low-dimensional, shared state is more efficient than relying on dense recurrence or long, explicit delays.
  • Algorithmic-hardware co-design is crucial: The DMP architecture's efficiency is realized by tailoring the hardware specifically to its dual-pathway nature, rather than trying to fit a complex algorithm onto generic hardware.
  • Brain inspiration leads to practical gains: The "fast-slow" motif from neuroscience provides a powerful engineering principle that translates directly into improved performance and efficiency.
  • Long-range credit assignment is enabled: The slow memory pathway facilitates the propagation of gradients over much longer timescales compared to standard feedforward SNNs, enabling better learning on long sequences.

Key Results

  • Competitive Accuracy: The DMP-SNN achieves competitive accuracy on benchmarks like S-MNIST (99.3%) and PS-MNIST (97.3%), matching or exceeding state-of-the-art SNNs.
  • Parameter Efficiency: The DMP-SNN uses 40-60% fewer parameters than equivalent state-of-the-art SNNs.
  • Hardware Performance: The co-designed hardware achieves over 4x increase in throughput and over 5x improvement in energy efficiency compared to state-of-the-art implementations (e.g., Loihi2, DenRAM).
  • Area Efficiency: The DMP-SNN hardware offers 2x greater area efficiency than recurrent SNN architectures.
  • Learning Dynamics: The DMP-SNN converges faster and achieves higher accuracy than baseline models under identical training settings.

Limitations and Future Work

  • Hardware Implementation: While the paper presents post-layout simulations, experimental validation on actual silicon is the next step.
  • Co-tuning: The authors suggest that neuron count, memory size, and temporal horizon should be co-tuned for optimal performance, implying that simply maximizing these might not be the most efficient approach.
  • Scalability: The architecture is designed to be scalable, but further exploration of its performance on even larger and more complex tasks would be beneficial.
  • Open-sourcing: The authors plan to open-source the hardware architecture to encourage community adoption and further co-design exploration.

Summary updated

Abstract

Spiking neural networks excel at event-driven sensing. Yet, maintaining task-relevant context over long timescales both algorithmically and in hardware, while respecting both tight energy and memory budgets, remains a core challenge in the field. We address this challenge through novel algorithm-hardware co-design effort. At the algorithm level, inspired by the cortical fast-slow organization in the brain, we introduce a neural network with an explicit slow memory pathway that, combined with fast spiking activity, enables a dual memory pathway (DMP) architecture in which each layer maintains a compact low-dimensional state that summarizes recent activity and modulates spiking dynamics. This explicit memory stabilizes learning while preserving event-driven sparsity, achieving competitive accuracy on long-sequence benchmarks with 40-60% fewer parameters than equivalent state-of-the-art spiking neural networks. At the hardware level, we introduce a near-memory-compute architecture that fully leverages the advantages of the DMP architecture by retaining its compact shared state while optimizing dataflow, across heterogeneous sparse-spike and dense-memory pathways. We show experimental results that demonstrate more than a 4x increase in throughput and over a 5x improvement in energy efficiency compared with state-of-the-art implementations. Together, these contributions demonstrate that biological principles can guide functional abstractions that are both algorithmically effective and hardware-efficient, establishing a scalable co-design paradigm for real-time neuromorphic computation and learning.
BibTeX
@inproceedings{sun2025algorithm,
  author = {Pengfei Sun and Zhe Su and Jascha Achterberg and Giacomo Indiveri and Marcus Ghosh and Danyal Akarca},
  title = {{Algorithm-hardware co-design of neuromorphic networks with dual memory pathways}},
  booktitle = {Repository},
  year = {2025},
  doi = {10.3929/ethz-c-000794098},
}
Figure