In standard reinforcement learning (RL), value updates follow a simple rule: when an outcome deviates from expectations, the agent calculates a reward prediction error (RPE) and updates its value estimate by multiplying this error by a single learning rate ($\alpha$). In this symmetric view, a surprise is just a surprise; whether an outcome is better or worse than expected, the magnitude of the update depends solely on the size of the error.
However, a vast body of research across neuroscience, psychology, and machine learning reveals that both biological brains and artificial agents break this symmetry. By decoupling the learning rate into distinct parameters for positive and negative prediction errors ($\alpha^+$ and $\alpha^-$), agents can weight good news and bad news differently. This architectural shift—known as asymmetric reinforcement learning or valence-dependent belief updating—fundamentally alters how an agent navigates risk, represents uncertainty, and stabilizes its learning process.
Why Asymmetry Emerges: Risk and Distributional RL
Why would an agent want to learn at different speeds from positive and negative surprises? The primary intuition lies in risk sensitivity. As Yael Niv et al. demonstrated, individual differences in risk temperament are deeply rooted in this asymmetry rather than a static, non-linear utility of rewards. A risk-averse agent updates value estimates more aggressively following losses than gains ($\alpha^- > \alpha^+$), causing it to quickly abandon options with fluctuating outcomes. Conversely, a risk-seeking agent prioritizes unexpected wins ($\alpha^+ > \alpha^-$).
This mechanism becomes even more powerful when scaled across a population of neurons. Instead of tracking just a single "average" expected reward, the brain uses asymmetric learning rates to perform distributional reinforcement learning. In the striatum, an opponent circuit consisting of D1 and D2 dopamine-modulated neurons specializes in optimistic and pessimistic updates, respectively (Adam S. Lowet et al.). Similarly, cortical neurons in the anterior cingulate cortex exhibit diverse, asymmetric learning rates (Timothy Müller et al.). By varying the ratio of $\alpha^+$ to $\alpha^-$ across different units, the neural population can collectively map out different quantiles or expectiles of a reward distribution, allowing the brain to maintain a rich, probabilistic representation of environmental uncertainty.
The Cognitive Landscape: Biases as Adaptive Strategies
In human psychology, asymmetric updating manifests as familiar cognitive biases. For instance, humans frequently exhibit a confirmation bias, integrating information that supports their choices or signals rewards much faster than disconfirmatory or negative feedback (Confirmation bias in human reinforcement learning: Evidence from counterfactual feedback processing). While this might look like a flaw, evolutionary modeling by Isabelle Hoxha et al. suggests that a positivity bias can act as an effective noise-reduction strategy in stable environments, widening the internal value gap between options to prevent random, stochastic losses from triggering suboptimal behavioral shifts.
However, these asymmetries are highly context-dependent and can become maladaptive when disrupted. For example, Henri Vandendriessche et al. showed that individuals with depression experience a shift toward a negativity bias ($\alpha^- > \alpha^+$), causing them to over-react to omissions or punishments and struggle significantly in sparse reward environments. In contrast, during goal-directed instrumental learning, healthy individuals sometimes exhibit "loss neglect," largely discounting negative outcomes when updating rule-based strategies (Roshan Cools). Interestingly, even modern large language models (LLMs) performing in-context learning mirror these human tendencies, behaving as asymmetric belief updaters that prioritize positive prediction errors over negative ones (Johannes A. Schubert et al.).
Engineering Stability in Artificial Intelligence
Beyond explaining biological behavior, asymmetric learning has emerged as a crucial tool for stabilizing machine learning algorithms, particularly in off-policy reinforcement learning and large language model alignment.
Off-policy learning is highly efficient but notoriously unstable because the agent learns from trajectories generated by a different policy. When an algorithm tries to penalize the failures of an old policy, it often introduces misleading gradient signals that cause catastrophic policy collapse. To resolve this, Charles Arnal et al. introduced Asymmetric REINFORCE (AsymRE), which uses baseline manipulation to prioritize learning from successful trajectories over penalizing failures. Similarly, Nicolas Le Roux et al. developed Tapered Off-Policy REINFORCE (TOPR), applying supervised-style updates to positive outcomes while using truncated importance sampling for negative ones to prevent numerical instability in large-scale models.
Asymmetric reward weighting also plays a vital role in refining LLM reasoning. When training models on verifiable rewards (like math or logic problems), applying symmetric updates to uniform-outcome batches can degrade performance. Thanh-Long V. Le et al. demonstrated that effective learning from these groups requires an asymmetric approach, separating the magnitude of rewards for all-correct sets from penalties for all-incorrect sets. Furthermore, Xinyu Zhu et al. discovered that down-weighting positive reinforcement relative to negative feedback prevents LLMs from becoming overconfident in a narrow set of correct answers, preserving the creative diversity of their reasoning chains.