Distributional reinforcement learning in prefrontal cortex

Timothy Müller, James L. Butler, Sebastijan Veselič, Bruno Miranda, Timothy E.j. Behrens, Zeb Kurth‐Nelson, Steven W. KennerleybioRxiv 2021 · DOI

TL;DR

This paper demonstrates that a more sophisticated model of reinforcement learning, called distributional RL, better explains how neurons in the prefrontal cortex (PFC) process reward information than traditional models. This suggests that the PFC, like dopamine neurons, uses distributional RL to guide learning and decision-making, highlighting its importance as a fundamental mechanism in the brain.

What Problem Does This Paper Address?

Traditional reinforcement learning (RL) models assume that the brain learns to predict the average outcome of an action or stimulus. While these models have been successful in explaining some neural activity, they fail to capture the full complexity of how animals make decisions, especially when risk or uncertainty is involved. Specifically, these "classic" RL models cannot account for the observed diversity in how individual neurons in the prefrontal cortex (PFC) respond to rewards. Distributional RL, a newer framework, proposes that the brain learns the entire distribution of possible rewards, not just the average. This richer representation is hypothesized to explain why some neurons might be "optimistic" (predicting higher rewards) and others "pessimistic" (predicting lower rewards), thus accounting for neuronal diversity. This paper investigates whether this distributional RL framework, which has already shown promise in explaining dopamine neuron activity, also provides a better account of neural responses in the primate PFC.

Methods

The researchers employed a combination of neurophysiological recordings from non-human primates and computational modeling. They analyzed neural activity from the anterior cingulate cortex (ACC), a region within the PFC known to be involved in learning and decision-making.

Key methodological approaches include:

  • Measuring "Optimism" at Choice: They quantified neuronal "optimism" by analyzing the non-linear relationship between stimulus value and neuronal firing rate. A "reversal point" was calculated, representing the value at which a neuron's response flips from above to below the average. This measure captures how individual neurons represent value distributions.
  • Asymmetric Scaling of Reward Prediction Errors (RPEs): They investigated whether neurons differentially weight positive versus negative RPEs. This was done by regressing firing rate against RPEs elicited during feedback, looking for asymmetries in how positive and negative RPEs influence neuronal responses.
  • Asymmetric Learning: They tested for differences in the rate at which neurons update their value estimates based on positive versus negative RPEs. This was modeled using temporal difference learning, allowing for distinct learning rates for positive and negative prediction errors.
  • Model Fitting: They compared the explanatory power of four different computational models: classic RL (symmetric scaling and learning), distributional RL with asymmetric scaling only, distributional RL with asymmetric learning only, and distributional RL with both asymmetric scaling and learning. Model fits were assessed using cross-validation.

Datasets

The study utilized two datasets:

  1. Re-analysis of Kennerley et al. 2011 data: This dataset involved two non-human primates performing a two-alternative forced-choice task where stimuli varied in reward probability. This data was used to test for distributional RL signatures related to value coding and RPE scaling.
  2. New data from Miranda et al. (in prep): This dataset involved two different non-human primates performing a "two-step task." This task featured dynamic reward structures where outcome levels changed over trials, making it suitable for testing asymmetric learning dynamics.

Key Insights and Ideas

  • PFC Neurons Encode Value Distributions, Not Just Averages: The study provides compelling evidence that individual neurons in the ACC don't just represent the average expected reward, but rather contribute to encoding the full distribution of possible rewards. This is demonstrated by the diverse "reversal points" observed across neurons, indicating varying levels of optimism.
  • Asymmetric Processing of Rewards: Neurons in the ACC exhibit asymmetries in how they process positive and negative reward prediction errors (RPEs). This includes differences in how strongly they scale their responses to positive versus negative RPEs, and crucially, differences in the rates at which they learn from positive versus negative outcomes.
  • Distributional RL as a Unified Framework: The findings suggest that distributional RL is not just a theoretical construct but a plausible computational mechanism employed by the primate PFC, mirroring findings in dopamine neurons. This points towards a more unified understanding of reward processing across different brain regions.
  • Anatomical Gradient of Optimism: The study found that optimism in ACC neurons is topographically organized along the anterior-posterior axis, with more anterior neurons being more optimistic. This suggests a structured neural architecture for representing value distributions.
  • Asymmetric Learning is Crucial for Dynamic Environments: The best model explaining ACC neural activity in a dynamic reward environment incorporated both asymmetric scaling and asymmetric learning, highlighting the importance of differential learning rates for positive and negative outcomes in adapting to changing reward structures.

Key Results

  • Diverse Value Predictions: In the ACC, 117 reward-sensitive neurons showed diverse "reversal points," indicating a spectrum of optimism and pessimism in value coding. This diversity was consistent and not explained by noise or stimulus features.
  • Asymmetric RPE Scaling: A subset of RPE-selective ACC neurons (41 neurons) showed diversity in the relative weighting of positive versus negative RPEs, with this asymmetry being stable across data partitions.
  • RPE Asymmetry Predicts Choice Optimism: The degree of asymmetric RPE scaling at feedback predicted the reversal points (optimism) observed at choice, linking these two aspects of distributional RL.
  • Asymmetric Learning and Scaling Best Explains ACC Activity: In the two-step task, a model incorporating both asymmetric learning and asymmetric scaling provided the best fit to ACC neuronal responses, outperforming models with only one or neither of these features.
  • Model Comparison: Pairwise model comparisons showed that models with asymmetric learning and/or scaling significantly outperformed simpler, classic RL models. The model with both asymmetric learning and scaling was the best overall fit.

Limitations and Future Work

  • Focus on ACC: While the study focused on the ACC, it acknowledges that other PFC regions might also exhibit distributional RL signatures. The initial analysis of OFC and LPFC showed less diversity, but further investigation might be warranted.
  • Model Complexity: Fitting complex models to neural data can be challenging. The study adapted some equations to simplify fitting procedures (e.g., single asymmetric scaling parameter).
  • Future Directions: The authors suggest that distributional RL might be a ubiquitous mechanism for reward-guided learning, potentially hidden in many existing RL datasets. They also raise intriguing questions about the interplay between dopaminergic and cortical distributional RL, proposing that cortical representations might implement the dopaminergic signals used for risk-sensitive decision-making. Further research could explore how these distributional representations are implemented and utilized in more complex decision-making scenarios.

Summary updated

Abstract

Abstract Prefrontal cortex is crucial for learning and decision-making. Classic reinforcement learning (RL) theories centre on learning the expectation of potential rewarding outcomes and explain a wealth of neural data in prefrontal cortex. Distributional RL, on the other hand, learns the full distribution of rewarding outcomes and better explains dopamine responses. Here we show distributional RL also better explains prefrontal cortical responses, suggesting it is a ubiquitous mechanism for reward-guided learning.
BibTeX
@misc{muller2021distributional,
  author = {Timothy Müller and James L. Butler and Sebastijan Veselič and Bruno Miranda and Timothy E.J. Behrens and Zeb Kurth‐Nelson and Steven W. Kennerley},
  title = {{Distributional reinforcement learning in prefrontal cortex}},
  year = {2021},
  doi = {10.1101/2021.06.14.448422},
  howpublished = {bioRxiv},
}
Figure