Rational metareasoning and the plasticity of cognitive control

Falk Lieder, Amitai Shenhav, Sebastian Musslick, Thomas L. Griffiths

TL;DR

This paper introduces the "Learned Value of Control" (LVOC) model, a computational framework that explains how humans learn to dynamically adjust their cognitive control based on experience and environmental cues. The LVOC model, grounded in rational metareasoning and reinforcement learning, successfully predicts and explains learning and transfer effects in various cognitive control tasks, outperforming simpler associative learning models.

What Problem Does This Paper Address?

The human brain's remarkable ability to adapt its information processing to achieve high-level goals relies on cognitive control. While it's known that these control skills are malleable and can be trained, the underlying learning mechanisms remain poorly understood. Specifically, how do people learn when to exert cognitive control, which control strategy to employ, and how much effort to invest? Existing theories often focus on the "what" (e.g., the value of control) but not the "how" (the learning process). This paper aims to bridge this gap by developing a computational model that explains the plasticity of cognitive control.

Methods

The authors propose the Learned Value of Control (LVOC) model, which extends the Expected Value of Control (EVC) theory. The core idea is that cognitive control is specified through a rational metareasoning process, framed as a meta-level Markov Decision Process (MDP). In essence, the brain learns to predict the "value of control" (EVOC) for different control signals based on features of the current situation.

Key methodological aspects include:

  • Reinforcement Learning: The LVOC model uses reinforcement learning to learn the EVOC. Specifically, it employs a Bayesian linear regression approach to learn a feature-based, probabilistic model of the EVOC.
  • Feature-Based Learning: The model assumes that control signals are selected based on features of the internal state and the current context. These features are used to predict the EVOC.
  • Probabilistic Control Signal Selection: The model selects control signals by sampling from the learned EVOC distribution, balancing exploration and exploitation.
  • Options Framework: The model incorporates the concept of "options" from reinforcement learning, allowing it to model not just elementary control signals but also more complex cognitive strategies.

Datasets

The paper evaluates the LVOC model against data from five previously published experiments:

  • Lin et al. (2016): Visual search task.
  • Krebs et al. (2010): Color-word Stroop task with rewarded and unrewarded colors.
  • Braem et al. (2012): Flanker task with rewarded and unrewarded trials.
  • Bugg et al. (2008): Color-word Stroop task with varying congruency probabilities.
  • Bugg et al. (2011): Picture-word Stroop task with novel stimuli.

No new datasets were introduced.

Key Insights and Ideas

  • Cognitive Control is Learned, Not Just Innate: The LVOC model provides a concrete mechanism for how people learn to adapt their cognitive control, moving beyond simply stating that it's plastic.
  • Features Drive Control Allocation: The model highlights that specific environmental features (like color, word, or congruency) are learned to predict the value of exerting control, leading to adaptive adjustments.
  • Transfer of Learning is Feature-Based: The model explains how learning in one context can transfer to novel situations if those situations share predictive features, a key aspect of generalization.
  • Rationality in Control: The framework suggests that even seemingly complex or "maladaptive" control behaviors might be rational given the learned predictive relationships between features and control outcomes.
  • LVOC Outperforms Simpler Models: The LVOC model's ability to capture nuanced learning and transfer effects significantly better than associative learning or Win-Stay Lose-Shift models underscores the importance of its feature-based, rational metareasoning approach.

Key Results

The LVOC model successfully predicted and explained findings across five experiments:

  • Visual Search (Lin et al., 2016): The model accurately predicted participants' faster search times during training with predictable target locations and slower times on unpredictable test trials, as well as the gradual unlearning of this bias. It outperformed Win-Stay Lose-Shift and Rescorla-Wagner models.
  • Interference Control (Krebs et al., 2010; Braem et al., 2012; Bugg et al., 2008): The model captured how reward associations influence control allocation. For instance, it predicted faster and more accurate responses to incongruent stimuli when their color predicted reward, and it explained how reward sensitivity modulates these effects. The model significantly outperformed associative learning and Win-Stay Lose-Shift models in these tasks.
  • Transfer of Learning (Bugg et al., 2011): The LVOC model correctly predicted positive and negative transfer effects to novel stimuli, demonstrating its ability to generalize learned feature-based associations.

Limitations and Future Work

  • Model Complexity: While more powerful, the LVOC model is more complex than simpler alternatives. The authors suggest that this complexity is currently warranted to capture the nuances of cognitive control plasticity.
  • Neural Implementation: The paper focuses on the computational principles and does not deeply detail the specific neural mechanisms. Future work could explore the neural implementation of the LVOC model.
  • Extensions: The authors suggest extending the model to include mechanisms for detecting changes in learned regularities and to model more complex phenomena like planning and strategy selection.
  • Self-Control Failure: The paper concludes by predicting circumstances under which these learning mechanisms might lead to self-control failure, suggesting this as an avenue for future research.
  • Broader Applications: The authors envision the rational metareasoning framework as a unifying approach for modeling various aspects of cognitive control, including attention, response inhibition, and cognitive flexibility, and for connecting AI/ML with neuroscience.

Summary updated

Abstract

The human brain has the impressive capacity to adapt how it processes information to high-level goals. While it is known that these cognitive control skills are malleable and can be improved through training, the underlying plasticity mechanisms are not well understood. Here, we develop and evaluate a model of how people learn when to exert cognitive control, which controlled process to use, and how much effort to exert. We derive this model from a general theory according to which the function of cognitive control is to select and configure neural pathways so as to make optimal use of finite time and limited computational resources. The central idea of our Learned Value of Control model is that people use reinforcement learning to predict the value of candidate control signals of different types and intensities based on stimulus features. This model correctly predicts the learning and transfer effects underlying the adaptive control-demanding behavior observed in an experiment on visual attention and four experiments on interference control in Stroop and Flanker paradigms. Moreover, our model explained these findings significantly better than an associative learning model and a Win-Stay Lose-Shift model. Our findings elucidate how learning and experience might shape people’s ability and propensity to adaptively control their minds and behavior. We conclude by predicting under which circumstances these learning mechanisms might lead to self-control failure.
BibTeX
@article{lieder2018rational,
  author = {Falk Lieder and Amitai Shenhav and Sebastian Musslick and Thomas L. Griffiths},
  title = {{Rational metareasoning and the plasticity of cognitive control}},
  journal = {PLOS Computational Biology},
  year = {2018},
  doi = {10.1371/journal.pcbi.1006043},
}
Figure