Controllability governs the balance between Pavlovian and instrumental action selection

Hayley M. Dorfman, Samuel J. Gershman

Abstract

A Pavlovian bias to approach reward-predictive cues and avoid punishment-predictive cues can conflict with instrumentally-optimal actions. Here, we propose that the brain arbitrates between Pavlovian and instrumental control by inferring which is a better predictor of reward. The instrumental predictor is more flexible; it can learn values that depend on both stimuli and actions, whereas the Pavlovian predictor learns values that depend only on stimuli. The arbitration theory predicts that the Pavlovian predictor will be favored when rewards are relatively uncontrollable, because the additional flexibility of the instrumental predictor is not useful. Consistent with this hypothesis, we find that the Pavlovian approach bias is stronger under low control compared to high control contexts.

Introduction

Psychology has long recognized a fundamental tension in human decision-making between two distinct learning systems. The Pavlovian system consists of hard-wired tendencies to approach rewards and avoid threats—think of a pigeon pecking at a light that predicts food, even if the act of pecking actually prevents the food from appearing. In contrast, the instrumental system is a flexible mechanism that learns which specific actions lead to desired outcomes in specific contexts.

The central challenge addressed in this research is the problem of arbitration: How does the brain decide how much weight to give each system? Specifically, why do "Pavlovian misbehaviors"—such as impulsively reaching for a reward even when it is counterproductive—persist? The authors propose that the brain performs a sophisticated Bayesian calculation to determine if an environment is "controllable" enough to justify the cognitive cost of complex instrumental learning.

Methods and Task Design

To test this theory, researchers combined behavioral experiments with computational modeling across two independent samples of human adults ($N=185$ and $N=129$).

The Go/No-Go Task

Participants performed a task where they viewed a shape (cue) and had to decide whether to press a button ("Go") or do nothing ("No-Go") to earn a reward. The critical manipulation was controllability. In "High Control" contexts, the participant's actions significantly influenced the probability of receiving a reward. In "Low Control" contexts, rewards were less dependent on the participant's choices, making the environment appear more random.

Experimental task design and reward probabilities across conditions.

Computational Modeling: Bayesian Arbitration

The authors compared a standard fixed-weight model against a novel Adaptive Model based on Bayesian Model Averaging. This adaptive model treats the Pavlovian and instrumental systems as competing "hypotheses" about the world. It updates the weight of each system trial-by-trial based on which one predicts rewards more accurately.

This approach utilizes "Bayesian Occam’s Razor": the brain prefers the simpler Pavlovian model (which only considers stimuli) unless the complex instrumental model (which considers both stimuli and actions) proves significantly more useful.

Key Insights: The Logic of Control

The study frames the two systems through the lens of the bias-variance trade-off in machine learning. The Pavlovian system is "high bias, low variance"—it makes consistent, simple assumptions that are often "good enough." The instrumental system is "low bias, high variance"—it is highly flexible but prone to "overfitting" or being misled by noise in the data.

Theoretical framework illustrating (a) Bayesian Occam's Razor and (b) the bias-variance trade-off in model complexity.

From this perspective, "Pavlovian misbehavior" is not necessarily irrational. If you cannot control the outcome of a situation, it is computationally efficient to default to simple instincts rather than wasting energy on complex action-mapping.

Key Results

  • Controllability Modulates Bias: As predicted, participants showed a significantly higher "Go bias" (the urge to act for a reward) in Low Control conditions compared to High Control conditions.
  • Superior Model Fit: The Adaptive Bayesian Model provided a vastly superior fit for the behavioral data (Protected Exceedance Probability > 0.99), suggesting the brain dynamically updates its reliance on these systems.
  • Temporal Dynamics: The Pavlovian bias was strongest at the start of the experiment and diminished as participants learned that their actions were effective, shifting the weight toward the instrumental system as more data was gathered.

Discussion and Future Directions

While the study provides a robust computational framework for arbitration, several questions remain. The current research focused on reward-seeking ("Go" bias); future studies should investigate if avoidance behaviors ("No-Go" bias for punishments) follow the same controllability rules. Additionally, while the computational logic is clear, the specific neural implementation—likely involving the prefrontal cortex and striatum—requires further neuroimaging validation.

These findings have significant clinical implications. For instance, the framework could help explain learned helplessness in depression, where a perceived lack of control might lead to a chronic "shutdown" of the instrumental system in favor of maladaptive Pavlovian responses.

Summary updated

Abstract

A Pavlovian bias to approach reward-predictive cues and avoid punishment-predictive cues can conflict with instrumentally-optimal actions. Here, we propose that the brain arbitrates between Pavlovian and instrumental control by inferring which is a better predictor of reward. The instrumental predictor is more flexible; it can learn values that depend on both stimuli and actions, whereas the Pavlovian predictor learns values that depend only on stimuli. The arbitration theory predicts that the Pavlovian predictor will be favored when rewards are relatively uncontrollable, because the additional flexibility of the instrumental predictor is not useful. Consistent with this hypothesis, we find that the Pavlovian approach bias is stronger under low control compared to high control contexts.
BibTeX
@inproceedings{dorfman2019controllability,
  author = {Hayley M. Dorfman and Samuel J. Gershman},
  title = {{Controllability governs the balance between Pavlovian and instrumental action selection}},
  booktitle = {Nature Communications},
  year = {2019},
  doi = {10.1038/s41467-019-13737-7},
}
Figure