Learning predictive cognitive maps with spiking neurons during behaviour and replays

Jacopo Bono, Sara Zannone, Victor Pedrosa, Claudia ClopathbioRxiv 2021 · DOI

TL;DR

This paper presents a biologically plausible spiking neural network model that learns a "successor representation" (a type of cognitive map) using spike-timing dependent plasticity (STDP). This model demonstrates how STDP can implement the powerful TD(λ) reinforcement learning algorithm, unifying behavioral learning and experience replay for efficient environmental representation.

What Problem Does This Paper Address?

The hippocampus is believed to be crucial for forming cognitive maps, which are internal representations of the environment used for navigation and decision-making. While place cells in the hippocampus provide evidence for these maps, the exact nature of the learned representation and how it's updated remains an open question. Specifically, how can a system learn complex temporal relationships over long timescales (seconds) using neural mechanisms that operate on very short timescales (milliseconds), like STDP? Furthermore, how do different types of neural activity, such as ongoing behavior versus brief "replay" events, contribute to this learning process? This paper addresses these questions by proposing a spiking neural network model that bridges the gap between biological neural mechanisms and established reinforcement learning algorithms.

Methods

The authors propose a two-layer spiking neural network model, inspired by the hippocampus (CA3 and CA1 regions). The core of the model is a biologically plausible spike-timing dependent plasticity (STDP) rule that governs the synaptic weight changes between neurons. They analytically show that this STDP rule, when applied to the network's activity during simulated behavior and "replay" events, leads to the learning of the successor representation (SR).

Key aspects of their approach include:

  • Spiking Neural Network: Utilizes neurons that communicate via discrete spikes, mimicking biological neural activity.
  • STDP: A learning rule where synaptic strength is modified based on the precise timing of pre- and post-synaptic spikes.
  • Successor Representation (SR): A predictive representation of the environment that encodes the expected future occupancy of states. This is a powerful tool for efficient learning and flexible decision-making.
  • TD(λ) Equivalence: The paper demonstrates that the learning dynamics of their STDP-based SR learning are mathematically equivalent to the TD(λ) reinforcement learning algorithm. This is a significant finding, bridging a gap between biological plausibility and algorithmic theory.
  • Behavioral vs. Replay Learning: The model distinguishes between learning during ongoing behavior (modeled as rate-based activity) and learning during replay events (modeled as temporally precise spikes). They show these contribute differently to the learning process.
  • State-Dependent Discounting: The model incorporates a flexible discounting mechanism that can be modulated by neuronal firing rates and neuromodulation, allowing for state-dependent valuation of future rewards.

Datasets

Not discussed in the paper. The paper uses simulated environments (linear tracks) and generates its own data through simulations.

Key Insights and Ideas

  • STDP as a Reinforcement Learning Engine: The most striking insight is that a biologically realistic STDP rule can directly implement a sophisticated reinforcement learning algorithm like TD(λ). This suggests that the brain might be using such low-level plasticity rules to achieve high-level cognitive functions.
  • Replays as "Monte Carlo" Learning: The paper proposes that replay events, characterized by precise temporal sequences, are akin to Monte Carlo (MC) updates in reinforcement learning. MC learning is known for its low bias but high variance, making it good for initial learning in unfamiliar environments.
  • Behavioral Activity as "TD" Learning: Conversely, ongoing behavior, modeled with rate-based activity, is shown to be equivalent to Temporal Difference (TD) learning. TD learning has higher bias but lower variance, making it suitable for refining estimates in familiar environments.
  • Bridging Timescales: The model elegantly explains how millisecond-timescale STDP can learn relationships over behavioral timescales (seconds) through the bootstrapping property of TD(λ).
  • Flexible Discounting: The ability to modulate the discount factor (γ) based on neuronal activity provides a mechanism for state-dependent valuation, allowing the model to prioritize certain locations or events.

Key Results

  • The spiking neural network model successfully learns the successor representation, as evidenced by synaptic weights converging to the theoretical SR matrix.
  • The learning dynamics of the spiking model closely match the theoretical TD(λ) algorithm for both behavioral learning (λ ≈ 0) and replay learning (λ ≈ 1).
  • The model demonstrates that a mix of behavioral and replay learning can effectively control the bias-variance trade-off, leading to faster initial learning and lower asymptotic error compared to using either strategy exclusively.
  • Simulations show that replay activity can significantly influence learned policies and behavior, even when the replayed trajectories were not directly experienced. For example, replays helped an agent avoid a shock zone more effectively.

Limitations and Future Work

  • Simplified Environments: The current model is primarily tested on linear track environments. Future work could explore more complex, multi-dimensional environments.
  • Place Cell Input: The model assumes external place-tuned inputs to CA3 neurons. Investigating how these inputs are generated and how they interact with learning is a potential avenue.
  • Neuromodulation Details: While the paper discusses state-dependent discounting via neuromodulation, the specific mechanisms are not fully detailed. Further exploration of how neuromodulatory signals could implement these changes is suggested.
  • Experimental Validation: The authors suggest designing experiments to investigate the relationship between replays in different layers and their timing, which would be crucial for validating the model's predictions.
  • Generalization to Other Brain Areas: The framework could potentially be generalized to other brain areas beyond the hippocampus, such as the neocortex.

Summary updated

Abstract

Abstract We describe a framework where a biologically plausible spiking neural network mimicking hippocampal layers learns a cognitive map known as the successor representation. We show analytically how, on the algorithmic level, the learning follows the TD( λ ) algorithm, which emerges from the underlying spike-timing dependent plasticity rule. We then analyze the implications of this framework, uncovering how behavioural activity and experience replays can play complementary roles when learning the representation of the environment, how we can learn relations over behavioural timescales with synaptic plasticity acting on the range of milliseconds, and how the learned representation can be flexibly encoded by allowing state-dependent delay discounting through neuromodulation and altered firing rates.
BibTeX
@misc{bono2021learning,
  author = {Jacopo Bono and Sara Zannone and Victor Pedrosa and Claudia Clopath},
  title = {{Learning predictive cognitive maps with spiking neurons during behaviour and replays}},
  year = {2021},
  doi = {10.1101/2021.08.16.456545},
  howpublished = {bioRxiv},
}
Figure