TL;DR
This paper explores two biologically inspired learning algorithms, Spike-Timing Dependent Reinforcement Learning (STDP-RL) and Evolutionary Strategies (EVOL), for training Spiking Neural Networks (SNNs) to perform the classic CartPole control task. They found that EVOL significantly outperforms STDP-RL, achieving optimal performance and demonstrating a more robust and efficient approach for training SNNs in reinforcement learning scenarios.
What Problem Does This Paper Address?
Artificial Neural Networks (ANNs) have achieved remarkable success in various tasks, including complex motor control. However, more biologically realistic Spiking Neural Networks (SNNs), which mimic the temporal dynamics of biological neurons, have historically struggled to match ANN performance in these domains. This gap highlights a need for effective learning mechanisms that can harness the temporal processing capabilities of SNNs. The authors aim to bridge this gap by comparing two biologically inspired learning approaches: STDP-RL, which is grounded in known biological plasticity rules, and EVOL, a more general optimization strategy. They investigate how these different learning paradigms, operating at potentially different timescales, impact SNN performance and the resulting neural dynamics.
Methods
The researchers employed two primary learning strategies to train their Spiking Neural Network (SNN) model on the CartPole problem:
- Spike-Timing Dependent Reinforcement Learning (STDP-RL): This method adapts synaptic weights based on the precise timing of pre- and post-synaptic spikes, modulated by a "critic" signal that provides reward or punishment feedback. The critic evaluates the state of the environment and provides a signal that influences synaptic weight changes. The authors explored various configurations of STDP-RL, including targeted reinforcement to specific motor neuron subpopulations.
- Evolutionary Strategies (EVOL): This approach treats synaptic weights as the "genome" of an individual SNN. A population of SNNs is generated, their performance (fitness) is evaluated on the CartPole task, and then the population is updated by perturbing the weights of the fittest individuals. This process iteratively evolves the network towards better performance without relying on explicit biological plasticity rules for weight updates during the evolutionary steps.
The SNN architecture itself was a multi-layered network with sensory (ES), association (EA), and motor (EM) populations, inspired by previous work. They also incorporated biologically plausible mechanisms like inhibitory neurons to prevent hyperexcitability and synaptic weight normalization.
Datasets
The primary "dataset" used for training and evaluation was the CartPole-v1 environment from the OpenAI Gym platform. This environment simulates the task of balancing a pole on a cart. The authors used fixed initializations of the environment (game states) for validation and testing to ensure consistent evaluation of trained models. No new datasets were introduced.
Key Insights and Ideas
- EVOL Outperforms STDP-RL: The most striking insight is that the Evolutionary Strategy (EVOL) approach significantly outperformed STDP-RL in training SNNs for the CartPole task, achieving optimal performance and demonstrating greater robustness.
- Timescale Matters: The paper suggests that the longer timescales inherent in EVOL's population-based evaluation might be more effective for learning complex behaviors compared to the shorter, event-driven feedback of STDP-RL.
- Broader Sensory-Motor Associations: EVOL-trained models learned to associate actions with a wider range of sensory input parameters (position, velocity, angle, and angular velocity), whereas STDP-RL models primarily relied on angular velocity, likely due to the critic's design.
- Neuron Pruning in EVOL: The EVOL model exhibited a form of "neuron pruning" where some neurons became deactivated, which the authors suggest might be a beneficial strategy for optimizing performance, unlike in STDP-RL.
- Initialization Sensitivity: STDP-RL performance was found to be highly sensitive to initial synaptic weights, leading to significant variability. EVOL, while also influenced by initialization, showed less pronounced performance differences.
Key Results
- Optimal Performance with EVOL: EVOL-trained SNNs achieved optimal performance, consistently balancing the pole for an average of ~500 steps per episode.
- STDP-RL Performance: STDP-RL models showed improved performance compared to untrained models, reaching an average of 118 steps per episode, but remained suboptimal.
- Training Efficiency: EVOL models reached optimal performance in significantly fewer iterations (145 iterations, ~7250 episodes) compared to STDP-RL (3931 episodes on average).
- Variability: STDP-RL models exhibited high variability in performance across episodes and initializations, while EVOL models showed more consistent and robust performance.
- Sensory-Motor Mapping: EVOL models developed more distributed sensory-motor associations, utilizing multiple sensory inputs for decision-making, whereas STDP-RL models primarily learned associations based on angular velocity.
Limitations and Future Work
- CartPole Simplicity: The CartPole task, while a standard benchmark, is relatively simple. The authors acknowledge that applying these methods to more complex tasks and environments will be crucial for future work.
- STDP-RL Critic Design: The performance of STDP-RL was likely influenced by the specific design of the critic, which was limited to two sensory parameters. Exploring different critic functions could yield different results.
- Computational Cost of EVOL: While effective, evolutionary algorithms can be computationally expensive, especially for high-dimensional search spaces.
- Future Work: The authors suggest testing their algorithms on more complex models, tasks, and environments. They also propose that their work can serve as a testbed for neurobiologists to understand multi-timescale learning mechanisms and dynamics in neuronal circuits. The potential for EVOL to be applied to more complex SNN models and tasks is a key future direction.
Summary updated