Do learning rates adapt to the distribution of rewards?

Samuel J. GershmanPsychonomic 2015 · DOI

Abstract

Recent normative theories suggest that human learning rates should dynamically adapt to the "richness" or average reward rate of an environment. This study tests this hypothesis using a series of multi-armed bandit experiments. While participants consistently exhibited asymmetric learning—specifically a "negativity bias" where they learned more rapidly from negative prediction errors than positive ones—this asymmetry remained remarkably stable across different reward conditions. Computational model comparison favored a fixed dual-learning rate model over more complex adaptive meta-learning models, suggesting that asymmetric learning may be a stable trait rather than a context-dependent state.

Introduction

In reinforcement learning (RL), agents update their beliefs based on Reward Prediction Errors (RPEs)—the difference between expected and actual outcomes. While standard models often employ a single, static learning rate ($\eta$), psychological evidence suggests that humans may process "good news" (positive RPEs) and "bad news" (negative RPEs) differently.

A prominent theoretical model (Cazé & van der Meer, 2013) posits that this asymmetry should be adaptive: in high-reward environments, agents should prioritize learning from negative errors to maintain high performance, whereas in low-reward environments, they should prioritize positive errors to identify rare opportunities. This paper empirically tests whether human decision-making reflects this normative ideal of "adaptive asymmetry."

Methods

The study involved 166 participants recruited via Amazon Mechanical Turk who performed a two-armed bandit task. Participants chose between two options yielding binary rewards (0 or 1). The core manipulation involved varying the average reward rate across blocks: "Low-reward" blocks featured reward probabilities below 0.5, while "High-reward" blocks featured probabilities above 0.5.

To analyze the behavioral data, five reinforcement learning models were compared:

  1. Model 1: A standard TD model with a single learning rate.
  2. Model 2: A fixed dual-learning rate model ($\eta^+$ and $\eta^-$).
  3. Model 3: A dual adaptive learning rate model (meta-learning).
  4. Model 4: An extended dual adaptive model.
  5. Model 5: A descriptive model with block-specific dual rates.

Model parameters were estimated using importance sampling, and hierarchical Bayesian model comparison was used to identify the best-fitting architecture.

Results

The primary finding was a persistent "negativity bias." Across experiments, the learning rate for negative prediction errors ($\eta^-$) was significantly higher than for positive ones ($\eta^+$). For instance, in Experiment 1, the mean $\eta^-$ was approximately 0.57 compared to 0.37 for $\eta^+$.

Crucially, this asymmetry did not adapt to the environment's reward richness. An ANOVA confirmed a significant effect of prediction error type ($p < 0.0001$) but found no significant interaction between reward rate and error type ($p = 0.12$). This lack of interaction directly contradicts the adaptive asymmetry hypothesis.

Figure 2: Comparison of choice probability data against predictions from four computational models. Model 2 (fixed dual rates) consistently provided the best fit to human behavior.

Model comparison further supported this conclusion. Model 2, which utilizes fixed dual learning rates, achieved the highest exceedance probability, outperforming more complex models that allowed for dynamic adaptation of the learning rates (Models 3 and 4).

Discussion

The stability of the negativity bias suggests it may serve a fundamental purpose, such as a built-in mechanism for risk aversion. By over-weighting negative outcomes, the brain may more aggressively steer the individual away from uncertain or declining options.

While these results challenge current normative theories of adaptive meta-learning, several limitations warrant further study. The use of binary rewards and relatively short experimental blocks (25 trials) might have limited the participants' ability to detect and adapt to environmental shifts. Future research could explore whether continuous reward distributions or longer exposure to stable environments might eventually trigger the predicted adaptive mechanisms.

Summary updated

BibTeX
@inproceedings{gershman2015do,
  author = {Samuel J. Gershman},
  title = {{Do learning rates adapt to the distribution of rewards?}},
  booktitle = {Psychonomic},
  year = {2015},
  doi = {10.3758/s13423-014-0790-3},
}
Figure