Training Spiking Neural Networks for Reinforcement Learning Tasks With Temporal Coding Method

Guanlin Wu, Dongchen Liang, Shaotong Luan, Ji WangFrontiers in Neuroscience 2022 · DOI

This paper, published in Frontiers in Neuroscience, presents a novel framework for training Spiking Neural Networks (SNNs) using temporal coding specifically for reinforcement learning (RL) tasks. By introducing a "self-incremental variable" and a new input encoding scheme, the authors overcome the traditional hurdles of non-differentiability and information loss that have previously hindered SNNs in complex RL environments.

TL;DR

The authors propose Continuous-Valued Temporal Coding (CVTC), a method that makes Spiking Neural Networks fully differentiable and suitable for reinforcement learning. By forcing neurons to fire through a self-incremental variable and using a Gaussian-based input encoding, they achieve performance on par with traditional deep learning models while maintaining the asynchronous efficiency of SNNs.

What Problem Does This Paper Address?

Spiking Neural Networks are the "third generation" of neural networks, designed to mimic the energy-efficient, asynchronous nature of the biological brain. However, training them for Reinforcement Learning is notoriously difficult.

Most existing SNNs for RL use rate coding (where information is represented by the frequency of spikes). This approach essentially treats SNNs as synchronous systems, losing the speed and power advantages of asynchrony. Temporal coding (where information is represented by the exact timing of a single spike) is more efficient but faces two major problems:

  1. Non-differentiability: In RL, many neurons may never fire (high sparsity). In standard temporal coding, a non-firing neuron has an "infinite" spike time, which results in a zero gradient, effectively "killing" the learning process during backpropagation.
  2. Information Loss: Standard encoding methods often ignore input signals that arrive after the first few neurons have already fired, leading to a biased or incomplete representation of the environment.

Methods

The authors introduce Continuous-Valued Temporal Coding (CVTC) to bridge the gap between SNNs and RL. Their approach consists of two key innovations:

  • The Self-Incremental Variable: To solve the "dead neuron" problem, the authors modify the membrane potential dynamics by adding a term: $\beta \exp(t)$. This acts as a constant "nudge" that ensures every neuron will eventually reach its firing threshold within a finite time. This mathematical trick ensures the spike time function is continuous and differentiable everywhere, allowing standard backpropagation (like that used in ANNs) to work reliably.
  • Gaussian Population Encoding: To prevent information loss, they expand each input channel into a population of $K$ neurons. The spike times of these neurons are determined by a Normal (Gaussian) distribution centered around the input value. This ensures that the original continuous signal can be mathematically recovered from the spike times, preserving the precision needed for RL tasks.

Crucially, the authors provide a mathematical proof (Theorem 1) showing that if the hyperparameter $\beta$ is small enough during training, the self-incremental variable can be removed during the inference phase on real neuromorphic hardware without significant performance degradation.

Datasets

The researchers evaluated their CVTC method on three distinct benchmarks:

  1. MNIST: The standard handwritten digit dataset, used to compare convergence speed and gradient stability against traditional temporal coding.
  2. CartPole-v0 (OpenAI Gym): A classic RL task where an agent must balance a pole on a cart.
  3. MountainCar-v0 (OpenAI Gym): An RL task requiring the agent to build momentum to drive a car up a steep hill.

Key Insights and Ideas

  • Solving the Gradient Vanishing Problem: The "aha moment" is realizing that the lack of spikes isn't just a biological reality—it's a mathematical barrier to gradient descent. By forcing a "guaranteed fire," they turn a discrete, sparse system into a smooth, differentiable one.
  • Asynchrony as a Feature, Not a Bug: Unlike rate-coded SNNs that must wait for a specific time window to "count" spikes, CVTC allows for "first-spike" inference. This means the network can react as soon as the first relevant spikes propagate through the layers, drastically reducing latency.
  • Hardware Compatibility: The method is designed with deployment in mind. By proving the $\beta$ term can be dropped post-training, the authors ensure these models can run on asynchronous neuromorphic chips like Intel’s Loihi or SynSense’s DynapCNN.

Key Results

  • Comparable Performance: In the CartPole task, the CVTC-SNN achieved a score of 180.19 ± 2.73, nearing the performance of a standard ANN-based Deep Q-Network (DQN) at 195.95 ± 0.59.
  • Superiority over Standard TC: Standard temporal coding (DDQN-SNN-TC) failed the CartPole task entirely, scoring only 17.89 ± 0.3, proving that the differentiability provided by the CVTC method is essential for RL.
  • Faster Convergence: On the MNIST task, CVTC converged significantly faster than standard temporal coding because the gradients were more stable and present across more neurons.

Limitations and Future Work

  • Policy-Based RL Challenges: While the method works well for value-based RL (like DQN), the authors note challenges in applying it to policy-based methods (like PPO). Policy networks often require unbounded outputs, whereas SNN spike times are naturally bounded by the simulation time.
  • Input Dimensionality: The population encoding method increases the input size (e.g., expanding 4 inputs to 80 in CartPole). While this improves accuracy, it adds computational overhead during the encoding stage.
  • Real-World Hardware Testing: While the theory supports migration to neuromorphic chips, the paper focuses on simulated results. Future work would benefit from demonstrating these RL agents running on physical neuromorphic hardware in real-time.

Summary updated

Abstract

Recent years witness an increasing demand for using spiking neural networks (SNNs) to implement artificial intelligent systems. There is a demand of combining SNNs with reinforcement learning architectures to find an effective training method. Recently, temporal coding method has been proposed to train spiking neural networks while preserving the asynchronous nature of spiking neurons to preserve the asynchronous nature of SNNs. We propose a training method that enables temporal coding method in RL tasks. To tackle the problem of high sparsity of spikes, we introduce a self-incremental variable to push each spiking neuron to fire, which makes SNNs fully differentiable. In addition, an encoding method is proposed to solve the problem of information loss of temporal-coded inputs. The experimental results show that the SNNs trained by our proposed method can achieve comparable performance of the state-of-the-art artificial neural networks in benchmark tasks of reinforcement learning.
BibTeX
@inproceedings{wu2022training,
  author = {Guanlin Wu and Dongchen Liang and Shaotong Luan and Ji Wang},
  title = {{Training Spiking Neural Networks for Reinforcement Learning Tasks With Temporal Coding Method}},
  booktitle = {Frontiers in Neuroscience},
  year = {2022},
  doi = {10.3389/fnins.2022.877701},
}
Figure