Evolving choice hysteresis in reinforcement learning: comparing the adaptive value of positivity bias and gradual perseveration

Isabelle Hoxha, Leo Sperber, Stefano Palminteri

Abstract

The tendency of repeating past choices more often than expected from the history of outcomes has been repeatedly empirically observed in reinforcement learning experiments. It can be explained by at least two computational processes: asymmetric update and (gradual) choice perseveration. A recent meta-analysis showed that both mechanisms are detectable in human reinforcement learning. However, while their descriptive value seems to be well established, they have not been compared regarding their possible adaptive value. In this study, we address this gap by simulating reinforcement learning agents in a variety of environments with a new variant of an evolutionary algorithm. Our results show that positivity bias (in the form of asymmetric update) is evolutionary stable in many situations, while the emergence of gradual perseveration is less systematic and robust. Overall, our results illustrate that biases can be adaptive and selected by evolution, in an environment-specific manner.

Introduction

Standard reinforcement learning (RL) models typically assume that agents should process positive and negative outcomes with equal weight. However, empirical research consistently demonstrates that humans exhibit "choice hysteresis"—a tendency to repeat previous actions more frequently than an objective reward history would predict. This phenomenon is often attributed to two distinct computational mechanisms: positivity bias (asymmetric updates where gains are weighted more heavily than losses) and gradual perseveration (a choice-based process where the act of selection itself increases future selection probability).

While these biases are well-documented descriptively, their normative status remains a subject of debate. Are these behaviors merely cognitive "errors," or do they represent evolutionarily adaptive strategies? This study employs a "virtual evolution" approach to investigate whether these biases provide a fitness advantage across diverse environmental niches.

Methods

To test the adaptive value of these biases, we simulated populations of 1,000 agents performing a two-armed bandit task. In this task, agents chose between two options with varying reward probabilities and received partial feedback (only seeing the outcome of the chosen action).

Experimental task and computational architecture. Panel A illustrates the trial flow (presentation, choice, and feedback). Panel B details the modified Q-learning model, highlighting the integration of asymmetric updates (PEc and Q) and perseveration (choice trace τ and C).

The agents were characterized by five evolvable parameters: separate learning rates for positive and negative outcomes ($\alpha_+$, $\alpha_-$), an exploration/exploitation parameter ($\beta$), and parameters governing the choice trace ($\tau$, $\phi$). Over 200 generations, the top 5% of performers were duplicated while the bottom 5% were removed, allowing the population to converge toward optimal strategies for specific environments.

Results

Our simulations revealed that the "optimality" of a bias is highly dependent on the environmental context. In stable environments, evolution almost universally selected for a positivity bias ($\alpha_+ > \alpha_-$). This bias acts as a form of "noise cancellation": by overestimating successful options and underestimating failures, the agent creates a wider internal gap between the decision values of the two options. This makes the agent more robust to the "noise" of occasional stochastic losses in an otherwise reliable environment.

Conceptual mechanism and environmental volatility. Panel C shows how bias increases the decision value gap between high- and low-reward options. Panel D illustrates the various environmental configurations tested, including different learning periods and reversal probability distributions (Fixed, Gaussian, Uniform).

However, this rule was not universal. In "rich" environments where rewards are frequent for both options, evolution favored a negativity bias. In such contexts, a failure is a rare and highly informative signal; prioritizing negative outcomes helps the agent maintain the optimal path. Furthermore, in volatile environments characterized by frequent reward reversals, evolution often favored gradual alternation (an internal urge to switch) rather than perseveration, facilitating rapid adaptation to shifting contingencies.

Discussion

These results suggest that common human cognitive biases are not necessarily flaws in reasoning but may be finely tuned adaptations to specific environmental statistics. Positivity bias emerged as a remarkably robust strategy for stable environments, appearing in 100% of evolutionary reboots.

While this study provides a normative foundation for understanding choice hysteresis, it is limited by the use of a simplified RL architecture and a focus on individual learning. Future research should explore whether these findings generalize to social learning contexts or more complex, hierarchical decision-making tasks. Ultimately, our work illustrates that what we perceive as cognitive bias may be the result of evolutionary selection for robustness and efficiency in ancestral environments.

Summary updated

Abstract

The tendency of repeating past choices more often than expected from the history of outcomes has been repeatedly empirically observed in reinforcement learning experiments. It can be explained by at least two computational processes: asymmetric update and (gradual) choice perseveration. A recent meta-analysis showed that both mechanisms are detectable in human reinforcement learning. However, while their descriptive value seems to be well established, they have not been compared regarding their possible adaptive value. In this study, we address this gap by simulating reinforcement learning agents in a variety of environments with a new variant of an evolutionary algorithm. Our results show that positivity bias (in the form of asymmetric update) is evolutionary stable in many situations, while the emergence of gradual perseveration is less systematic and robust. Overall, our results illustrate that biases can be adaptive and selected by evolution, in an environment-specific manner.
BibTeX
@inproceedings{hoxha2025evolving,
  author = {Isabelle Hoxha and Leo Sperber and Stefano Palminteri},
  title = {{Evolving choice hysteresis in reinforcement learning: comparing the adaptive value of positivity bias and gradual perseveration}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2025},
  doi = {10.1073/pnas.2422144122},
}
Figure