Control of entropy in neural models of environmental state

Timothy Müller, Rogier B. Mars, Timothy E.j. Behrens, Jill X. O’ReillyeLife 2019 · DOI

TL;DR

This study investigates how the brain controls uncertainty in its internal models of the environment, a crucial aspect for flexible decision-making. Using fMRI and pupillometry, the researchers found that changes in pupil diameter, a proxy for neuromodulatory states like noradrenaline, predict increases in neural model uncertainty in the medial orbitofrontal cortex (mOFC). This suggests a top-down, neuromodulatory mechanism for adjusting belief flexibility.

What Problem Does This Paper Address?

Humans and animals constantly build and update internal models of their environment to make informed decisions. A key feature of these models is representing uncertainty about the current state of the world. This uncertainty is vital for learning and adapting to changes. However, while theoretical models suggest a mechanism to increase uncertainty (to maintain flexibility), it's unclear how the brain implements this. Specifically, how does the brain control the "entropy" or uncertainty within these internal neural models? This paper aims to bridge this gap by investigating the neural and neuromodulatory mechanisms underlying the control of uncertainty in human decision-making.

Methods

The researchers employed a combination of behavioral tasks, functional Magnetic Resonance Imaging (fMRI), and pupillometry.

  • Behavioral Task: Participants engaged in a 4-arm bandit task where they had to infer which option provided a high reward probability. This task was designed to create uncertainty about the environment's state.
  • fMRI: Multivariate pattern analysis (searchlight approach) was used to decode the representation of the environment's state in different brain regions. Specifically, they looked for regions where the representation of the chosen option was stronger when uncertainty (model entropy) was lower.
  • Pupillometry: Pupil diameter was measured as a proxy for neuromodulatory states, particularly noradrenaline, which is hypothesized to play a role in regulating uncertainty.
  • Computational Modeling: A Bayesian learning model was used to estimate participants' belief uncertainty (entropy) on a trial-by-trial basis. This model's output served as a regressor for fMRI analyses.
  • Statistical Analysis: Linear regressions (GLMs) were used to link pupil diameter changes, brain activity (in mOFC and ACC), and model entropy.

Datasets

The study used data collected from 22 healthy human volunteers. No new benchmark datasets were introduced; the focus was on analyzing novel fMRI and pupillometry data collected during the described task.

Key Insights and Ideas

  • mOFC as a Probabilistic State Representation: The medial orbitofrontal cortex (mOFC) appears to encode a probabilistic model of the environment. The strength of representing a specific option in mOFC was higher when the model's uncertainty about the environment's state was lower. This suggests mOFC represents beliefs about which option is currently the best, not just a general value.
  • Pupil Dilation as a Neuromodulatory Signal for Uncertainty: Changes in pupil diameter, a known indicator of noradrenaline release, predicted changes in representation strength in mOFC. Crucially, this effect was specific to unrewarded trials, suggesting that pupil-linked neuromodulation might be involved in increasing uncertainty when evidence contradicts current beliefs.
  • Anterior Cingulate Cortex (ACC) as a Regulator: Activity in the anterior cingulate cortex (ACC) was associated with increases in model entropy. Furthermore, ACC activity predicted changes in pupil diameter, suggesting a top-down pathway where ACC signals uncertainty increases, which then influences pupil dilation and subsequently the representation of uncertainty in mOFC.
  • Top-Down Control of Uncertainty: The findings collectively point towards a neuromodulatory system, involving ACC and pupil-linked signals, that exerts top-down control over the flexibility (entropy) of internal neural models. This mechanism is particularly active when the environment's state is uncertain or changing.

Key Results

  • mOFC Decoding: A multivariate analysis revealed that the identity of the chosen option could be decoded from mOFC with higher confidence when model entropy was lower. This effect remained significant even after controlling for option value and recent reward history.
  • Pupil-mOFC Link: A linear regression showed that increases in pupil size predicted decreases in representation strength in mOFC on unrewarded exploit trials (t(15)=-2.94, p=0.01). This relationship was specific to unrewarded trials, suggesting it relates to increasing uncertainty.
  • ACC-Pupil Link: Whole-brain analysis identified ACC activity that predicted changes in baseline pupil size, even after accounting for model entropy and overall brain activity (p<0.05, corrected). This suggests ACC activity drives pupil dilation changes.
  • Behavioral Prediction: The Bayesian model, which simulated belief uncertainty, accurately predicted participants' choices, particularly during exploitation phases.

Limitations and Future Work

  • Pupillometry Specificity: The authors acknowledge that pupil dilation is influenced by multiple neuromodulators (noradrenaline and acetylcholine) and arousal factors, not solely noradrenaline. Therefore, the results reflect a complex interplay of neuromodulatory signals.
  • Indirect Measure: Pupillometry is an indirect measure of neuromodulation.
  • Exploit vs. Explore: While the study focused on "core exploit" trials to isolate belief uncertainty from behavioral changes, they also noted similar patterns in explore trials and during transitions between phases, suggesting broader implications.
  • Future Work: The authors suggest exploring the exact nature of the representation in mOFC and how it relates to option-reward contingencies. They also highlight the need to further investigate the role of neuromodulatory systems in implementing flexibility of internal representations during decision-making. The findings also open avenues for exploring how ACC might broadcast information to other regions via neuromodulatory systems.

Summary updated

Abstract

Humans and animals construct internal models of their environment in order to select appropriate courses of action. The representation of uncertainty about the current state of the environment is a key feature of these models that controls the rate of learning as well as directly affecting choice behaviour. To maintain flexibility, given that uncertainty naturally decreases over time, most theoretical inference models include a dedicated mechanism to drive up model uncertainty. Here we probe the long-standing hypothesis that noradrenaline is involved in determining the uncertainty, or entropy, and thus flexibility, of neural models. Pupil diameter, which indexes neuromodulatory state including noradrenaline release, predicted increases (but not decreases) in entropy in a neural state model encoded in human medial orbitofrontal cortex, as measured using multivariate functional MRI. Activity in anterior cingulate cortex predicted pupil diameter. These results provide evidence for top-down, neuromodulatory control of entropy in neural state models.
BibTeX
@inproceedings{muller2019control,
  author = {Timothy Müller and Rogier B. Mars and Timothy E.J. Behrens and Jill X. O’Reilly},
  title = {{Control of entropy in neural models of environmental state}},
  booktitle = {eLife},
  year = {2019},
  doi = {10.7554/elife.39404},
}
Figure