GRSN: Gated Recurrent Spiking Neurons for POMDPs and MARL

Lang Qin, Ziming Wang, Runhao Jiang, Rui Yan, Huajin Tang

Abstract

Energy-efficient Spiking Neural Networks (SNNs) often struggle with complex reinforcement learning tasks due to temporal mismatches and limited memory capacity. We present Gated Recurrent Spiking Neurons (GRSN), an architecture that aligns SNN internal dynamics with environment decision steps. By incorporating gated units to manage input current, GRSN achieves performance comparable to traditional Recurrent Neural Networks (RNNs) in partially observable and multi-agent environments while reducing power consumption by approximately 50%.

Introduction

Spiking Neural Networks (SNNs) offer a brain-inspired, energy-efficient alternative to standard artificial neural networks by communicating via discrete spikes. However, their application in Deep Reinforcement Learning (DRL) has been hindered by "temporal mismatch"—where multiple internal SNN steps are required for a single environment action—and a lack of long-term memory, making them unsuitable for Partially Observable Markov Decision Processes (POMDPs) and Multi-Agent Reinforcement Learning (MARL).

Addressing Temporal Mismatch

The core issue in existing spiking RL algorithms is the disconnect between the SNN's internal "ticks" and the agent's decision steps. This inefficiency ignores the SNN’s natural capacity for temporal processing.

Temporal mismatch occurs when multiple SNN steps are used for a single decision, whereas the proposed approach aligns them 1-to-1.

To bridge this gap, the researchers introduced the Temporal Alignment Paradigm (TAP). TAP aligns the SNN updates 1-to-1 with state transitions in the environment, allowing the network to accumulate historical information naturally over time, much like a Recurrent Neural Network (RNN).

Gated Recurrent Spiking Neurons (GRSN)

To provide SNNs with the "long-term memory" necessary for complex tasks, the authors developed Gated Recurrent Spiking Neurons (GRSN).

The GRSN architecture replaces standard RNN cells with Spike Cells and utilizes gated current updates to preserve information.

Key innovations include:

  1. Gated Current Updates: Gradient analysis revealed that input current ($c$) is a more effective carrier of long-term information than membrane potential ($u$). GRSN uses learnable "forget" and "input" gates to manage this current flow.
  2. Soft Reset Mechanism: Unlike traditional "hard resets" that zero out voltage after a spike, GRSN employs a "soft reset" that subtracts the threshold value, preserving residual information for subsequent steps.

Experimental Evaluation

The GRSN model was evaluated in two primary domains:

  • Partially Observable Classic Control: Modified CartPole and Pendulum tasks where agents must infer hidden states (e.g., velocity) from past observations.
  • StarCraft Multi-Agent Challenge (SMAC): A benchmark for high-dimensional multi-agent coordination across various difficulty levels.

Key Results

  • Performance: GRSN achieved performance parity with state-of-the-art GRUs in POMDP tasks and outperformed them in several "Hard" and "Super Hard" SMAC scenarios.
  • Efficiency: By replacing floating-point multiplications with simple additions via binary spikes, GRSN reduced estimated energy consumption by 49.1% compared to RNN baselines. The spiking components alone were found to be up to 3,800 times more efficient.

Conclusion

GRSN demonstrates that SNNs can be both high-performing and energy-efficient in complex RL settings. By aligning temporal dynamics and introducing gated current management, this work paves the way for deploying sophisticated AI on power-constrained neuromorphic hardware. Future research will focus on real-world hardware deployment and scaling to complex robotic control tasks.

Summary updated

BibTeX
@inproceedings{qin2025grsn,
  author = {Lang Qin and Ziming Wang and Runhao Jiang and Rui Yan and Huajin Tang},
  title = {{GRSN: Gated Recurrent Spiking Neurons for POMDPs and MARL}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i2.32139},
}
Figure