Evolutionary and spike-timing-dependent reinforcement learning train spiking neuronal network motor control

Daniel Haşegan, Matt Deible, Christopher Earl, David Donofrio, Hananel Hazan, Haroon Anwar, Samuel A. NeymotinbioRxiv 2021 · DOI

TL;DR

This paper explores two biologically inspired learning algorithms, Spike-Timing Dependent Reinforcement Learning (STDP-RL) and Evolutionary Strategies (EVOL), for training Spiking Neural Networks (SNNs) to perform the classic CartPole control task. They found that EVOL significantly outperforms STDP-RL, achieving optimal performance and demonstrating a more robust and efficient approach for training SNNs in reinforcement learning scenarios.

What Problem Does This Paper Address?

Artificial Neural Networks (ANNs) have achieved remarkable success in various tasks, including complex motor control. However, more biologically realistic Spiking Neural Networks (SNNs), which mimic the temporal dynamics of biological neurons, have historically struggled to match ANN performance in these domains. This gap highlights a need for effective learning mechanisms that can harness the temporal processing capabilities of SNNs. The authors aim to bridge this gap by comparing two biologically inspired learning approaches: STDP-RL, which is grounded in known biological plasticity rules, and EVOL, a more general optimization strategy. They investigate how these different learning paradigms, operating at potentially different timescales, impact SNN performance and the resulting neural dynamics.

Methods

The researchers employed two primary learning strategies to train their Spiking Neural Network (SNN) model on the CartPole problem:

  • Spike-Timing Dependent Reinforcement Learning (STDP-RL): This method adapts synaptic weights based on the precise timing of pre- and post-synaptic spikes, modulated by a "critic" signal that provides reward or punishment feedback. The critic evaluates the state of the environment and provides a signal that influences synaptic weight changes. The authors explored various configurations of STDP-RL, including targeted reinforcement to specific motor neuron subpopulations.
  • Evolutionary Strategies (EVOL): This approach treats synaptic weights as the "genome" of an individual SNN. A population of SNNs is generated, their performance (fitness) is evaluated on the CartPole task, and then the population is updated by perturbing the weights of the fittest individuals. This process iteratively evolves the network towards better performance without relying on explicit biological plasticity rules for weight updates during the evolutionary steps.

The SNN architecture itself was a multi-layered network with sensory (ES), association (EA), and motor (EM) populations, inspired by previous work. They also incorporated biologically plausible mechanisms like inhibitory neurons to prevent hyperexcitability and synaptic weight normalization.

Datasets

The primary "dataset" used for training and evaluation was the CartPole-v1 environment from the OpenAI Gym platform. This environment simulates the task of balancing a pole on a cart. The authors used fixed initializations of the environment (game states) for validation and testing to ensure consistent evaluation of trained models. No new datasets were introduced.

Key Insights and Ideas

  • EVOL Outperforms STDP-RL: The most striking insight is that the Evolutionary Strategy (EVOL) approach significantly outperformed STDP-RL in training SNNs for the CartPole task, achieving optimal performance and demonstrating greater robustness.
  • Timescale Matters: The paper suggests that the longer timescales inherent in EVOL's population-based evaluation might be more effective for learning complex behaviors compared to the shorter, event-driven feedback of STDP-RL.
  • Broader Sensory-Motor Associations: EVOL-trained models learned to associate actions with a wider range of sensory input parameters (position, velocity, angle, and angular velocity), whereas STDP-RL models primarily relied on angular velocity, likely due to the critic's design.
  • Neuron Pruning in EVOL: The EVOL model exhibited a form of "neuron pruning" where some neurons became deactivated, which the authors suggest might be a beneficial strategy for optimizing performance, unlike in STDP-RL.
  • Initialization Sensitivity: STDP-RL performance was found to be highly sensitive to initial synaptic weights, leading to significant variability. EVOL, while also influenced by initialization, showed less pronounced performance differences.

Key Results

  • Optimal Performance with EVOL: EVOL-trained SNNs achieved optimal performance, consistently balancing the pole for an average of ~500 steps per episode.
  • STDP-RL Performance: STDP-RL models showed improved performance compared to untrained models, reaching an average of 118 steps per episode, but remained suboptimal.
  • Training Efficiency: EVOL models reached optimal performance in significantly fewer iterations (145 iterations, ~7250 episodes) compared to STDP-RL (3931 episodes on average).
  • Variability: STDP-RL models exhibited high variability in performance across episodes and initializations, while EVOL models showed more consistent and robust performance.
  • Sensory-Motor Mapping: EVOL models developed more distributed sensory-motor associations, utilizing multiple sensory inputs for decision-making, whereas STDP-RL models primarily learned associations based on angular velocity.

Limitations and Future Work

  • CartPole Simplicity: The CartPole task, while a standard benchmark, is relatively simple. The authors acknowledge that applying these methods to more complex tasks and environments will be crucial for future work.
  • STDP-RL Critic Design: The performance of STDP-RL was likely influenced by the specific design of the critic, which was limited to two sensory parameters. Exploring different critic functions could yield different results.
  • Computational Cost of EVOL: While effective, evolutionary algorithms can be computationally expensive, especially for high-dimensional search spaces.
  • Future Work: The authors suggest testing their algorithms on more complex models, tasks, and environments. They also propose that their work can serve as a testbed for neurobiologists to understand multi-timescale learning mechanisms and dynamics in neuronal circuits. The potential for EVOL to be applied to more complex SNN models and tasks is a key future direction.

Summary updated

Abstract

Abstract Despite being biologically unrealistic, artificial neural networks (ANNs) have been successfully trained to perform a wide range of sensory-motor behaviors. In contrast, the performance of more biologically realistic spiking neuronal network (SNN) models trained to perform similar behaviors remains relatively suboptimal. In this work, we aimed at pushing the field of SNNs forward by exploring the potential of different learning mechanisms to achieve optimal performance. Inspired by biological learning mechanisms operating at multiple timescales, we used spike-timing-dependent reinforcement learning (STDP-RL) and evolutionary strategy (EVOL) with SNNs to solve the CartPole reinforcement learning (RL) control problem. Though the role of STDP-RL in biological systems is well established, several other mechanisms, though not fully understood, work in concert during learning in vivo. Recreating accurate models that capture the interaction of STDP-RL with these diverse learning mechanisms is extremely difficult. EVOL is an alternative method, and has been successfully used in many studies to fit model neural responsiveness to electrophysiological recordings and in some cases for classification problems. One advantage of EVOL is that it may not need to capture all interacting components of synaptic plasticity, and thus provides a better alternative to STDP-RL. Here, we compared the performance of each algorithm after training, which revealed EVOL as a powerful method to training SNNs to perform sensory-motor behaviors. Our modeling opens up new capabilities for SNNs in RL and could serve as a testbed for neurobiologists aiming to understand multi-timescale learning mechanisms and dynamics in neuronal circuits.
BibTeX
@misc{hasegan2021evolutionary,
  author = {Daniel Haşegan and Matt Deible and Christopher Earl and David Donofrio and Hananel Hazan and Haroon Anwar and Samuel A. Neymotin},
  title = {{Evolutionary and spike-timing-dependent reinforcement learning train spiking neuronal network motor control}},
  year = {2021},
  doi = {10.1101/2021.11.20.469405},
  howpublished = {bioRxiv},
}
Figure