Abstract
We study the in-context learning dynamics of large language models (LLMs) using three instrumental learning tasks adapted from cognitive psychology. We find that LLMs update their beliefs in an asymmetric manner and learn more from better-than-expected outcomes than from worse-than-expected ones. Furthermore, we show that this effect reverses when learning about counterfactual feedback and disappears when no agency is implied. We corroborate these findings by investigating idealized in-context learning agents derived through meta-reinforcement learning, where we observe similar patterns. Taken together, our results contribute to our understanding of how in-context learning works by highlighting that the framing of a problem significantly influences how learning occurs, a phenomenon also observed in human cognition.
Introduction
While Large Language Models (LLMs) excel at in-context learning (ICL)—improving performance through examples provided in a prompt—the underlying dynamics of how they update their internal beliefs remain largely opaque. In cognitive science, humans are known as "asymmetric updaters," often exhibiting an optimism bias by learning more from positive surprises than negative ones. This research investigates whether LLMs share these human-like learning dynamics or if they follow a symmetric, strictly "rational" update rule.
Methodology
The study treats LLMs as participants in classic cognitive psychology experiments using Two-Alternative Forced Choice (2AFC) tasks. In these scenarios, agents choose between options with different reward probabilities. The researchers compared the behavior of Claude v1.3 against human behavioral data and Meta-Reinforcement Learning (Meta-RL) agents. To quantify learning, agent behavior was fitted to variants of the Rescorla-Wagner (RW) model, specifically comparing a symmetric model to an asymmetric RW± model that allows for distinct learning rates for positive ($\alpha^+$) and negative ($\alpha^-$) prediction errors.
Results
The experiments revealed that LLMs are indeed asymmetric updaters. In partial feedback tasks, Claude v1.3 demonstrated a significant optimism bias, with a learning rate for positive errors ($\alpha^+ \approx 0.87$) nearly nine times higher than for negative errors ($\alpha^- \approx 0.10$). As shown in the model comparison, the asymmetric RW± model provided a significantly better fit for both LLM and human behavior than the standard symmetric model.
Key findings include:
- Agency as a Trigger: The observed biases are context-dependent. If the prompt framing removes the implication of agency (e.g., "someone else played"), the asymmetry disappears.
- Bias as Rationality: Meta-RL agents, trained to be optimal reward-maximizers, developed the same biases. This suggests that "optimism" may be a mathematically sound strategy for handling exploration-exploitation trade-offs in certain environments.
- Performance: Claude v1.3 successfully reduced "regret" (missed potential reward) from 0.09 to 0.05 over 24 trials, mirroring human learning curves while often learning at a faster rate.
Discussion and Future Work
These results suggest that the framing of a problem significantly influences how LLMs learn in-context. However, the study was limited to controlled, synthetic environments. Future research is needed to determine if these biases persist in complex, real-world reasoning or open-ended dialogue. Additionally, the degree of bias may vary across different architectures and training regimes, such as those utilizing Reinforcement Learning from Human Feedback (RLHF). Understanding these dynamics allows practitioners to use problem framing to intentionally steer LLM learning toward desired outcomes.
Summary updated