Evolving choice hysteresis in reinforcement learning: Comparing the adaptive value of positivity bias and gradual perseveration

Isabelle Hoxha, Léo Sperber, Stefano Palminteri

TL;DR

This paper uses evolutionary algorithms to show that common human cognitive biases—specifically positivity bias (learning more from gains than losses) and choice perseveration (the tendency to repeat past actions)—are not necessarily "errors." Instead, these biases are often evolutionarily adaptive strategies that outperform unbiased agents by acting as a form of "noise cancellation" in stable environments or by facilitating rapid adaptation in volatile ones.

What Problem Does This Paper Address?

In reinforcement learning (RL), standard models assume agents should be "unbiased," treating positive and negative outcomes with equal weight. However, empirical studies consistently show that humans exhibit choice hysteresis: we repeat past choices more often than an objective history of rewards would suggest.

Two main computational mechanisms explain this:

  1. Positivity Bias (Asymmetric Update): Learning more from positive prediction errors than negative ones.
  2. Gradual Perseveration: A choice-based process where the act of choosing an option increases the likelihood of choosing it again, regardless of the outcome.

While these are well-documented descriptively (they describe how humans behave), their normative status (whether they are actually "good" strategies) remained unclear. This paper investigates whether these biases provide a fitness advantage in different environmental niches.

Methods

The authors employed a "virtual evolution" approach:

  • The Task: Agents performed a two-armed bandit task where they chose between two options with different reward probabilities. Feedback was "partial" (agents only saw the outcome of their chosen action).
  • The Model: A modified Q-learning agent characterized by five "genes" (parameters):
  • $\alpha_+$ and $\alpha_-$: Separate learning rates for positive and negative outcomes.
  • $\beta$: Inverse temperature (exploration vs. exploitation).
  • $\tau$ and $\phi$: Parameters governing the "choice trace," which tracks choice history to allow for perseveration or alternation.
  • Evolutionary Algorithm: A population of 1,000 agents competed over 200 generations. Fitness was determined by the rate of reward-maximizing choices. The top 5% of performers were duplicated, while the bottom 5% were removed.
  • Environmental Manipulation: The authors tested agents across various "environments" by varying task difficulty, reward richness (average probability of reward), learning period length, and volatility (frequency and predictability of reward reversals).

Datasets

As a computational simulation study, the paper does not use external datasets. Instead, it generates a broad range of synthetic environments:

  • Stable Environments: 7 unique configurations varying difficulty (e.g., 95/05% vs 55/45% reward probabilities) and richness.
  • Volatile Environments: 9 unique configurations where reward contingencies reversed according to different distributions (Fixed, Gaussian, or Uniform).

Key Insights and Ideas

  • Positivity Bias as Noise Cancellation: In stable environments, a positivity bias ($\alpha_+ > \alpha_-$) was almost universally selected. By overestimating "good" options and underestimating "bad" ones, the agent creates a wider internal gap between options, making it more robust to the "noise" of occasional stochastic losses.
  • The "Rare Outcome" Rule: In "rich" environments (where both options pay out frequently), evolution actually favored a negativity bias. When success is common, a failure is a more informative signal; paying more attention to rare negative outcomes helps the agent stay on the optimal path.
  • Perseveration vs. Alternation: While humans often show perseveration, evolution only selected it in very easy or long-duration tasks. In volatile environments, evolution favored gradual alternation ($\phi < 0$)—an internal "urge" to switch options that helps the agent anticipate and react to reversals.
  • Environment-Specific Optimality: No single set of parameters was optimal everywhere. The "best" bias is highly dependent on the statistics of the environment, suggesting that human cognitive biases might be "tuned" to the specific types of volatility and richness found in our ancestral environments.

Key Results

  • Robustness of Positivity Bias: In the "stable macro-environment" (encompassing all stable tasks), positivity bias emerged in 100% of reboots.
  • Volatility and Negativity: As environment volatility increased (from 1 to 31 reversals), the selected bias shifted from positive to significantly negative ($\Delta\alpha < 0$), as agents needed to be hyper-sensitive to losses to detect contingency shifts.
  • Fitness Plateaus: Across all initializations, the evolutionary algorithm successfully converged to a fitness plateau (approx. 80% accuracy), demonstrating that the emergence of these biases is a stable outcome of the selection process.

Limitations and Future Work

  • Simplified Model: The agents used a relatively simple RL architecture. Future work could explore more complex models, such as those incorporating "meta-learning" or hierarchical RL.
  • Social Learning: The study focused on individual learning. The authors note that in social contexts, biases like positivity might have different (potentially polarizing) effects that weren't captured here.
  • Task Scope: The results are specific to two-armed bandits. Whether these findings generalize to more complex decision trees or continuous action spaces remains an open question.
  • Biological Plausibility: While the "pruning" of phenotypes is a standard evolutionary simulation, real-world biological evolution involves more complex genetic interactions and developmental constraints.

Summary updated

Abstract

The tendency to repeat past choices more often than expected from the history of outcomes has been repeatedly empirically observed in reinforcement learning experiments. It can be explained by at least two computational processes: asymmetric update and (gradual) choice perseveration. A recent meta-analysis showed that both mechanisms are detectable in human reinforcement learning. However, while their descriptive value seems to be well established, they have not been compared regarding their possible adaptive value. In this study, we address this gap by simulating reinforcement learning agents in a variety of environments with a variant of an evolutionary algorithm. Our results show that positivity bias (in the form of asymmetric update) is evolutionary stable in many situations, while the emergence of gradual perseveration is less systematic and robust. Overall, our results illustrate that biases can be adaptive and selected by evolution, in an environment-specific manner.
BibTeX
@inproceedings{hoxha2025evolving,
  author = {Isabelle Hoxha and Léo Sperber and Stefano Palminteri},
  title = {{Evolving choice hysteresis in reinforcement learning: Comparing the adaptive value of positivity bias and gradual perseveration}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2025},
  doi = {10.1073/pnas.2422144122},
}

Loading related content...

Figure