TL;DR
This paper uses evolutionary algorithms to show that common human cognitive biases—specifically positivity bias (learning more from gains than losses) and choice perseveration (the tendency to repeat past actions)—are not necessarily "errors." Instead, these biases are often evolutionarily adaptive strategies that outperform unbiased agents by acting as a form of "noise cancellation" in stable environments or by facilitating rapid adaptation in volatile ones.
What Problem Does This Paper Address?
In reinforcement learning (RL), standard models assume agents should be "unbiased," treating positive and negative outcomes with equal weight. However, empirical studies consistently show that humans exhibit choice hysteresis: we repeat past choices more often than an objective history of rewards would suggest.
Two main computational mechanisms explain this:
- Positivity Bias (Asymmetric Update): Learning more from positive prediction errors than negative ones.
- Gradual Perseveration: A choice-based process where the act of choosing an option increases the likelihood of choosing it again, regardless of the outcome.
While these are well-documented descriptively (they describe how humans behave), their normative status (whether they are actually "good" strategies) remained unclear. This paper investigates whether these biases provide a fitness advantage in different environmental niches.
Methods
The authors employed a "virtual evolution" approach:
- The Task: Agents performed a two-armed bandit task where they chose between two options with different reward probabilities. Feedback was "partial" (agents only saw the outcome of their chosen action).
- The Model: A modified Q-learning agent characterized by five "genes" (parameters):
- $\alpha_+$ and $\alpha_-$: Separate learning rates for positive and negative outcomes.
- $\beta$: Inverse temperature (exploration vs. exploitation).
- $\tau$ and $\phi$: Parameters governing the "choice trace," which tracks choice history to allow for perseveration or alternation.
- Evolutionary Algorithm: A population of 1,000 agents competed over 200 generations. Fitness was determined by the rate of reward-maximizing choices. The top 5% of performers were duplicated, while the bottom 5% were removed.
- Environmental Manipulation: The authors tested agents across various "environments" by varying task difficulty, reward richness (average probability of reward), learning period length, and volatility (frequency and predictability of reward reversals).
Datasets
As a computational simulation study, the paper does not use external datasets. Instead, it generates a broad range of synthetic environments:
- Stable Environments: 7 unique configurations varying difficulty (e.g., 95/05% vs 55/45% reward probabilities) and richness.
- Volatile Environments: 9 unique configurations where reward contingencies reversed according to different distributions (Fixed, Gaussian, or Uniform).
Key Insights and Ideas
- Positivity Bias as Noise Cancellation: In stable environments, a positivity bias ($\alpha_+ > \alpha_-$) was almost universally selected. By overestimating "good" options and underestimating "bad" ones, the agent creates a wider internal gap between options, making it more robust to the "noise" of occasional stochastic losses.
- The "Rare Outcome" Rule: In "rich" environments (where both options pay out frequently), evolution actually favored a negativity bias. When success is common, a failure is a more informative signal; paying more attention to rare negative outcomes helps the agent stay on the optimal path.
- Perseveration vs. Alternation: While humans often show perseveration, evolution only selected it in very easy or long-duration tasks. In volatile environments, evolution favored gradual alternation ($\phi < 0$)—an internal "urge" to switch options that helps the agent anticipate and react to reversals.
- Environment-Specific Optimality: No single set of parameters was optimal everywhere. The "best" bias is highly dependent on the statistics of the environment, suggesting that human cognitive biases might be "tuned" to the specific types of volatility and richness found in our ancestral environments.
Key Results
- Robustness of Positivity Bias: In the "stable macro-environment" (encompassing all stable tasks), positivity bias emerged in 100% of reboots.
- Volatility and Negativity: As environment volatility increased (from 1 to 31 reversals), the selected bias shifted from positive to significantly negative ($\Delta\alpha < 0$), as agents needed to be hyper-sensitive to losses to detect contingency shifts.
- Fitness Plateaus: Across all initializations, the evolutionary algorithm successfully converged to a fitness plateau (approx. 80% accuracy), demonstrating that the emergence of these biases is a stable outcome of the selection process.
Limitations and Future Work
- Simplified Model: The agents used a relatively simple RL architecture. Future work could explore more complex models, such as those incorporating "meta-learning" or hierarchical RL.
- Social Learning: The study focused on individual learning. The authors note that in social contexts, biases like positivity might have different (potentially polarizing) effects that weren't captured here.
- Task Scope: The results are specific to two-armed bandits. Whether these findings generalize to more complex decision trees or continuous action spaces remains an open question.
- Biological Plausibility: While the "pruning" of phenotypes is a standard evolutionary simulation, real-world biological evolution involves more complex genetic interactions and developmental constraints.
Summary updated