Model-based learning protects against forming habits

Claire M. Gillan, A. Ross Otto, Elizabeth A. Phelps, Nathaniel D. DawCognitive 2015 · DOI

TL;DR

This study reveals that a "model-based" learning strategy, which involves planning and predicting outcomes, protects individuals from developing rigid, habitual behaviors. By analyzing how people learn in a sequential decision task and then testing their sensitivity to devalued rewards, the researchers found that those who relied more on model-based learning were less prone to forming habits.

What Problem Does This Paper Address?

The paper tackles a fundamental question in behavioral neuroscience and psychology: how do we learn and control our actions? We know that behavior can be either "goal-directed" (planned and flexible, driven by current desires) or "habitual" (automatic and inflexible, triggered by cues). This distinction is crucial for understanding conditions like addiction and obsessive-compulsive disorder, where habitual behaviors can become maladaptive. While we can measure the dominance of habits using "outcome devaluation" (reducing the value of a reward to see if behavior changes), we know surprisingly little about the learning mechanisms that lead to one type of behavior over the other. This paper aims to bridge that gap by investigating how different learning strategies during the acquisition phase influence the subsequent development of habits.

Methods

The researchers employed a computational framework that distinguishes between two primary reinforcement learning mechanisms:

  • Model-free learning: This system learns action values based on past reward history. It's like learning "if I do X, I get Y" through repeated experience, without necessarily understanding why.
  • Model-based learning: This system builds a "cognitive map" of the environment, allowing it to predict outcomes of actions and sequences of actions. It's more flexible, as it can use this map to plan and adapt to changes in outcome values.

They designed a two-step sequential decision-making task where participants chose between two options (represented by fractals). These choices led to different "second-stage" states, each with a probability of yielding a reward (coins). The task was structured to allow for the dissociation of model-based and model-free learning signals.

After participants learned the task through numerous trials, a devaluation procedure was implemented. One of the rewards (coins) was symbolically devalued (e.g., by indicating its associated container was "full"). The researchers then tested participants' behavior to see if they adjusted their choices to avoid the devalued outcome. This devaluation sensitivity was then correlated with individual differences in model-based and model-free learning signatures observed during the initial training phase.

Two experiments were conducted. Experiment 1 used two interleaved tasks, while Experiment 2 used a single task with a modified devaluation probe to ensure the learned associations were directly tested.

Datasets

Not discussed in the paper. The study used human participants recruited online (Amazon Mechanical Turk) and in laboratory settings.

Key Insights and Ideas

  • Model-based learning is a "habit-breaker": The most striking finding is that individuals who relied more on model-based learning during the acquisition phase were significantly less likely to develop habits. This suggests that actively planning and predicting outcomes acts as a protective mechanism against rigid, automatic behavior.
  • Habits emerge from a lack of model-based control: Conversely, individuals who showed less evidence of model-based learning were more prone to forming habits, meaning their behavior remained insensitive to changes in outcome value.
  • Devaluation sensitivity as a proxy for goal-directedness: The study validates the use of devaluation sensitivity as a measure of goal-directed behavior, directly linking it to underlying learning mechanisms.
  • Distinct neural substrates are implied: While not directly measured, the findings align with existing neuroimaging research suggesting that the caudate nucleus and ventromedial prefrontal cortex are involved in goal-directed control and model-based learning, while the putamen is associated with habit formation.
  • Model-free learning's role is less clear: Interestingly, the study found no significant relationship between model-free learning and devaluation sensitivity. This suggests that while model-free learning contributes to behavior, it might not be the primary driver of habit formation in the way model-based learning is of goal-directed behavior.

Key Results

  • Experiment 1: A significant three-way interaction between reward, transition, and devaluation sensitivity in a logistic regression model indicated that greater devaluation sensitivity was associated with a greater contribution of model-based learning during training. Specifically, a "model-based index" (derived from the Reward × Transition interaction) significantly predicted devaluation sensitivity.
  • Experiment 2: This finding was replicated. A significant three-way interaction confirmed that individuals with higher devaluation sensitivity showed more evidence of model-based learning. The model-based index again predicted devaluation sensitivity.
  • Consumption Test: In both experiments, a consumption test confirmed that the devaluation manipulation was effective in reducing the incentive value of the devalued coin.
  • No correlation with general comprehension: The relationship between model-based learning and devaluation sensitivity was not explained by general task comprehension.

Limitations and Future Work

  • Computational model complexity: While the study used a computational framework, the regression analyses focused on trial-by-trial dynamics. A full computational model was also fit, but the primary findings were derived from the regression approach.
  • Task design nuances: Differences in task design between Experiment 1 and previous studies (e.g., two tasks vs. one, presence of costs) might have influenced the strength of model-based learning effects. Future work could further explore how task parameters modulate these learning strategies.
  • Generalizability of findings: While the study used online and laboratory participants, further research could explore these dynamics in clinical populations known for compulsive behaviors.
  • Role of model-free learning: The lack of a clear link between model-free learning and habit formation warrants further investigation. It's possible that model-free learning is more about reinforcing existing associations rather than driving the formation of new, rigid habits, or that its influence is more subtle and context-dependent.
  • External promoters of habit: Future studies could investigate how factors like training duration, contingency strength, and implicit vs. explicit learning interact with model-based and model-free learning to influence habit formation.

The authors suggest that this framework opens new avenues for understanding the neural mechanisms underlying the interplay between learning, value, and habit formation, particularly in the context of psychiatric disorders.

Summary updated

Abstract

Studies in humans and rodents have suggested that behavior can at times be "goal-directed"-that is, planned, and purposeful-and at times "habitual"-that is, inflexible and automatically evoked by stimuli. This distinction is central to conceptions of pathological compulsion, as in drug abuse and obsessive-compulsive disorder. Evidence for the distinction has primarily come from outcome devaluation studies, in which the sensitivity of a previously learned behavior to motivational change is used to assay the dominance of habits versus goal-directed actions. However, little is known about how habits and goal-directed control arise. Specifically, in the present study we sought to reveal the trial-by-trial dynamics of instrumental learning that would promote, and protect against, developing habits. In two complementary experiments with independent samples, participants completed a sequential decision task that dissociated two computational-learning mechanisms, model-based and model-free. We then tested for habits by devaluing one of the rewards that had reinforced behavior. In each case, we found that individual differences in model-based learning predicted the participants' subsequent sensitivity to outcome devaluation, suggesting that an associative mechanism underlies a bias toward habit formation in healthy individuals.
BibTeX
@inproceedings{gillan2015model,
  author = {Claire M. Gillan and A. Ross Otto and Elizabeth A. Phelps and Nathaniel D. Daw},
  title = {{Model-based learning protects against forming habits}},
  booktitle = {Cognitive},
  year = {2015},
  doi = {10.3758/s13415-015-0347-6},
}
Figure