Confirmatory reinforcement learning changes with age during adolescence

Gabriele Chierchia, Magdaléna Soukupová, Emma J. Kilford, Cáit Griffin, Jovita T. Leung, Stefano Palminteri, Sarah‐Jayne Blakemore

TL;DR

This study reveals that as individuals transition from adolescence to adulthood, their reinforcement learning becomes more "confirmatory" – they learn more from outcomes that align with their expectations. This shift is primarily driven by a reduction in "noise" or exploration, rather than a change in the strength of the confirmation bias itself, offering insights into how learning adapts across the lifespan.

What Problem Does This Paper Address?

Developmental science has long been interested in how learning abilities change throughout human development. Recent computational models have shown that humans exhibit a "confirmatory learning" bias: they learn more from outcomes that confirm their prior expectations than from those that disconfirm them. However, it was unclear whether this bias is stable or if it changes during adolescence, a period characterized by increased exploration and a drive for independence. This paper investigates how confirmatory reinforcement learning (RL) evolves from adolescence to early adulthood, specifically asking if this bias is attenuated during a developmental stage where exploration is crucial.

Methods

The researchers employed a standard probabilistic instrumental RL task. Participants (aged 11-32) repeatedly chose between pairs of novel options with varying reward/punishment probabilities, aiming to maximize monetary rewards. The task included three conditions:

  • Asymmetric: One option consistently offered a higher win probability.
  • Reversal: Probabilities were stable for the first half of trials and then reversed.
  • Symmetric: Both options had equal win probabilities.

Behavioral analyses focused on accuracy, preferred choice rate, win-stay/lose-shift asymmetry, decision times, and outcome observation times. Computational modeling used a confirmation model, which allows for separate learning rates for confirmatory and disconfirmatory evidence, to capture age-related differences.

Datasets

The study used data from 77 participants (aged 11-32 years, 4 men, mean age = 16.26). The data from adult participants (N=20) had been previously published. No new benchmarks were introduced.

Key Insights and Ideas

  • Adolescence is a transition period for learning: The study highlights that adolescence isn't a static learning phase but a dynamic period where learning strategies shift towards greater confirmation.
  • Confirmatory learning is adaptive in stable environments: The findings suggest that learning to rely more on confirming evidence is beneficial when the environment's rules (reward probabilities) are stable.
  • Exploration remains important in volatile environments: The age-related advantage in learning is diminished when environmental rules change, indicating that flexibility and disconfirming evidence are still crucial.
  • Win-stay/lose-shift asymmetry grows with age: Older participants were more likely to repeat a winning choice and less likely to switch after a losing choice, suggesting a stronger tendency to stick with what has worked.
  • Computational models reveal the "why": The confirmation model provided a better fit to the data, showing that age-related differences in learning are best explained by changes in "inverse temperature" (a measure of noise/exploration) rather than the magnitude of the confirmation bias itself.

Key Results

  • Accuracy: Accuracy increased linearly with age in stable (asymmetric) learning environments. This age-related benefit was reduced in volatile (reversal) conditions.
  • Choice Conservatism: There was a non-significant trend for increased conservatism (sticking with a preferred option) with age in symmetric trials, but this did not reach statistical significance.
  • Win-Stay/Lose-Shift: A significant increase in win-stay/lose-shift asymmetry was observed with age. Participants were more likely to stay after a win and this tendency increased with age, while the tendency to switch after a loss did not change significantly with age.
  • Decision and Outcome Observation Times: Decision times showed no overall age effect, but increased with age in symmetric trials. Outcome observation times decreased linearly with age.
  • Computational Model: The confirmation model best explained the data, and its fit improved with age. Age-related differences were best captured by decreases in inverse temperature (noise/exploration), not by changes in the confirmation bias magnitude.

Limitations and Future Work

  • Sample Size: The study acknowledges that it might have been underpowered, particularly for detecting smaller age-related effects in older participants or for investigating choice conservatism in symmetric trials.
  • Gender Bias: The sample was predominantly female, limiting the generalizability of findings to males. Future research should investigate potential gender differences in RL.
  • Cross-sectional Design: The study used a cross-sectional design, which conflates age-related effects with inter-individual differences. Longitudinal studies are recommended to track developmental changes more directly.
  • Future Directions: The authors suggest that better-powered studies and longitudinal approaches are needed. Further investigation into how environmental factors (like option differentiability) interact with age-related learning changes is also warranted.

Summary updated

Abstract

Understanding how learning changes during human development has been one of the long-standing objectives of developmental science. Recently, advances in computational biology have demonstrated that humans display a bias when learning to navigate novel environments through rewards and punishments: they learn more from outcomes that confirm their expectations than from outcomes that disconfirm them. Here, we ask whether confirmatory learning is stable across development, or whether it might be attenuated in developmental stages in which exploration is beneficial, such as in adolescence. In a reinforcement learning (RL) task, 77 participants aged 11-32 years (four men, mean age = 16.26) attempted to maximize monetary rewards by repeatedly sampling different pairs of novel options, which varied in their reward/punishment probabilities. Mixed-effect models showed an age-related increase in accuracy as long as learning contingencies remained stable across trials, but less so when they reversed halfway through the trials. Age was also associated with a greater tendency to stay with an option that had just delivered a reward, more than to switch away from an option that had just delivered a punishment. At the computational level, a confirmation model provided increasingly better fit with age. This model showed that age differences are captured by decreases in noise or exploration, rather than in the magnitude of the confirmation bias. These findings provide new insights into how learning changes during development and could help better tailor learning environments to people of different ages. RESEARCH HIGHLIGHTS: Reinforcement learning shows age-related improvement during adolescence, but more in stable learning environments compared with volatile learning environments. People tend to stay with an option after a win more than they shift from an option after a loss, and this asymmetry increases with age during adolescence. Computationally, these changes are captured by a developing confirmatory learning style, in which people learn more from outcomes that confirm rather than disconfirm their choices. Age-related differences in confirmatory learning are explained by decreases in stochasticity, rather than changes in the magnitude of the confirmation bias.
BibTeX
@inproceedings{chierchia2022confirmatory,
  author = {Gabriele Chierchia and Magdaléna Soukupová and Emma J. Kilford and Cáit Griffin and Jovita T. Leung and Stefano Palminteri and Sarah‐Jayne Blakemore},
  title = {{Confirmatory reinforcement learning changes with age during adolescence}},
  booktitle = {Developmental},
  year = {2022},
  doi = {10.1111/desc.13330},
}
Figure