Goal-Directed Decision Making with Spiking Neurons

Johannes Friedrich, Máté LengyelJournal of Neuroscience 2016 · DOI

TL;DR

This paper presents a novel spiking neural network model that can perform goal-directed decision-making by learning an internal model of the environment. The model not only reproduces complex behavioral and neural data from various decision-making tasks but also offers a biologically plausible mechanism for how the brain might solve computationally challenging planning problems.

What Problem Does This Paper Address?

The brain exhibits two main strategies for decision-making: habitual (fast, automatic) and goal-directed (flexible, planning-based). While the neural basis of habitual decision-making is relatively well-understood (e.g., through reward prediction error signals), the mechanisms underlying goal-directed decision-making, which requires complex online value estimation and planning, remain largely unknown. This paper aims to bridge this gap by proposing a biologically realistic spiking neural network that can perform this computationally demanding task.

Methods

The authors developed a spiking neural network model based on the stochastic spike-response model, a biologically plausible neuron model. The core innovation lies in how the network dynamics implicitly solve the Bellman optimality equation, which is fundamental to dynamic programming and optimal decision-making. This is achieved through:

  • Recurrent Connectivity: Neurons are interconnected to represent states and actions.
  • Lateral Inhibition: This mechanism implements the "max" operation in the Bellman equation, allowing neurons representing different actions within the same state to compete.
  • Local Plasticity Rules: Synaptic weights are learned using biologically plausible rules (akin to reward-prediction and state-prediction errors) that allow the network to build an internal model of the environment.
  • Function Approximation: The model is extended to handle continuous state spaces using basis functions, making it more generalizable.

Datasets

The paper validates its model on several established reinforcement learning benchmark tasks and uses data from existing neuroscience experiments.

  • Benchmark Tasks:
    • Blackjack
    • Maze task
    • Pendulum swing-up task
  • Experimental Data:
    • Data from Padoa-Schioppa (2013) on binary choice tasks (orbitofrontal cortex recordings).
  • Data from Roesch and Olson (2003) on binary choice tasks (frontal eye field recordings).
  • Data from Sohn and Lee (2007) on sequential movement tasks (pre-SMA recordings).

No new datasets or benchmarks were introduced.

Key Insights and Ideas

  • Spiking Neurons Can Plan: The paper demonstrates that a network of spiking neurons, through its intrinsic dynamics and local learning rules, can solve complex planning problems that were previously thought to require more abstract computational models.
  • Lateral Inhibition as a "Max" Operator: The use of lateral inhibition to implement the maximization step in the Bellman optimality equation is a key insight, providing a biologically plausible mechanism for selecting the best action.
  • Unified Framework for Learning and Planning: The model integrates learning an internal model of the environment (via synaptic plasticity) with planning (via network dynamics), offering a cohesive view of goal-directed behavior.
  • Dynamic Value Representation: The model shows that neural activity doesn't just represent static values but evolves dynamically, reflecting the process of planning and value computation. This dynamic aspect is crucial for explaining experimental observations.
  • Predictive Power for Neural and Behavioral Data: The model's ability to reproduce not only choice probabilities and reaction times but also the fine-grained temporal dynamics of neural activity in various tasks is a strong validation of its biological plausibility.

Key Results

  • The spiking neural network model successfully learned and executed optimal policies in benchmark tasks, achieving near-optimal performance.
  • The model reproduced behavioral data, including decision times and choice probabilities, in binary and sequential decision-making tasks.
  • The temporal dynamics of neural activity in the model closely matched experimental recordings from prefrontal cortex areas (OFC, FEF, pre-SMA) across different tasks and conditions.
  • The model demonstrated that synaptic plasticity rules can enable the network to learn an internal model of the environment, which is essential for goal-directed decision-making.
  • The model made novel predictions for tasks with multiple rewards and for the effects of motivational state changes (devaluation).

Limitations and Future Work

  • Discrete vs. Continuous States: While the paper extends to continuous states using function approximation, the initial formulation and some analyses focus on discrete states.
  • Metabolic Efficiency: The model's direct implementation of state-action neurons for each time step in sequential tasks can lead to large network sizes. Future work could explore more efficient representations.
  • Dale's Principle: The model uses neurons that can be both excitatory and inhibitory, violating Dale's principle. The authors suggest this could be addressed by explicitly modeling interneurons.
  • Generalization of Learning Rules: While the learning rules are local, their precise biological implementation and generalization to more complex scenarios warrant further investigation.
  • Exploration: The paper mentions using a simplified exploration strategy for learning. More sophisticated exploration mechanisms could be integrated.
  • Future Directions: The authors suggest extending the model to handle more complex reward structures, incorporating more sophisticated inference algorithms, and exploring its application to other cognitive functions beyond decision-making. They also propose further experimental tests to distinguish their model from alternative theories.

Summary updated

Abstract

Goal-directed actions requiring prospective planning pervade decision making, but their circuit-level mechanisms remain elusive. We show how a model circuit of biologically realistic spiking neurons can solve this computationally challenging problem in a novel way. The synaptic weights of our network can be learned using local plasticity rules such that its dynamics devise a near-optimal plan of action. By systematically comparing our model results to experimental data, we show that it reproduces behavioral decision times and choice probabilities as well as neural responses in a rich set of tasks. Our results thus offer the first biologically realistic account for complex goal-directed decision making at a computational, algorithmic, and implementational level.
BibTeX
@article{friedrich2016goal,
  author = {Johannes Friedrich and Máté Lengyel},
  title = {{Goal-Directed Decision Making with Spiking Neurons}},
  journal = {Journal of Neuroscience},
  year = {2016},
  doi = {10.1523/jneurosci.2854-15.2016},
}
Figure