Imagine choosing between a reward of \$1 and \$2. Now imagine choosing between \$101 and \$102. In standard, objective reinforcement learning models, the absolute difference in reward is identical ($\Delta R = 1$), so an agent should evaluate the choice with the exact same level of discriminative sensitivity.
Yet, human choices and neural responses treat these two scenarios completely differently. Biological brains do not process rewards in a vacuum. Instead, they constantly shift and scale their valuation metrics based on context. As reviewed by P. Read Montague et al., valuation signals in the brain are dynamically modulated by environmental and cognitive factors rather than remaining static.
Why does the brain do this? The answer lies in a fundamental hardware limitation: neurons have a bounded dynamic range. They cannot infinitely scale their firing rates to represent arbitrary reward magnitudes. To maintain high sensitivity across vastly different environments—from scarcity to abundance—the brain dynamically normalizes its internal value representations.
The Mechanics of Contextual Scaling
To map absolute real-world outcomes into an efficient, bounded neural code, reinforcement learning agents rely on two primary computational mechanisms: reference-point centering and range adaptation.
Reference-Point Centering
Instead of tracking absolute values, the brain tracks a running average of the environment's overall quality to establish a local baseline. Outcomes are then evaluated as gains or losses relative to this reference point.
Maël Lebreton et al. demonstrated that humans track this global context-value to scale both choice evaluations and metacognitive confidence. This reference-dependent approach allows an agent to exhibit remarkable flexibility. For instance, Emma J. Kilford et al. showed that through contextual relative encoding, an agent can perceive a completely neutral outcome (e.g., receiving nothing) as a positive, rewarding signal if it occurs within a highly punitive context.
Range Adaptation
Once a baseline is centered, the agent must determine the scale of its valuation function. How wide should the representation stretch?
One prominent theory is divisive normalization, where option values are scaled by dividing them by a metric representing the total available choice set, a mechanism observed in prefrontal cortex models by Marion Rouault et al.
However, a closer look at human reinforcement learning by Sophie Bavard suggests that value representation is often better explained by range normalization. Under this framework, values are explicitly scaled relative to the minimum and maximum bounds of the currently experienced options. This ensures that the brain's error-related negativity (ERN) signal remains highly sensitive to the specific spectrum of possibilities within the current task, as shown by Clay B. Holroyd et al.
Absolute Scale: 0 -------------------------------------------------> 100
Range Normalization:
Context A [0, 10]: Min(0) ======================= Max(10) -> Full Neural Range
Context B [90, 100]: Min(90) ======================= Max(100) -> Full Neural Range
Internal and External Dimensions of Context
Context is not merely the immediate menu of choices displayed on a screen; it encompasses both the external statistical structure of the environment and the internal physiological state of the agent.
External Context and Choice Sets
In multi-item environments, an item's perceived value changes depending on its competitors. Traditional reward models assume that an item's value is intrinsic and static. However, Jasmin Brandt et al. and Shuo Yang et al. highlight that when feedback is subset-dependent or pluralistic, an item's perceived utility shifts based on the accompanying alternatives. The brain handles this by tuning neural circuits in the ventral striatum and midbrain to match the prevailing environmental reward statistics, as documented by Francesco Rigoli et al.
Internal Context and Homeostasis
Context also operates from within. A resource like water has immense subjective value when an organism is dehydrated, but becomes worthless or even aversive when the organism is satiated.
Maaike M. H. van Swieten integrated these internal physiological states into computational models of the basal ganglia, demonstrating that motivation dynamically redefines prediction errors based on internal deficits. Formalizing this concept, Naoto Yoshida et al. introduced Homeostatically Regulated Reinforcement Learning (HRRL). By defining "reward" not as an absolute external token, but as the reduction of an internal physiological drive, they showed that complex behaviors like risk aversion naturally emerge as an agent seeks to maintain internal stability.
The Efficiency-Irrationality Trade-off
Why did evolution favor a valuation system that distorts objective reality? The primary benefit is information-theoretic efficiency: scaling value signals allows an agent to achieve high-precision learning across highly diverse environments using minimal neural resources.
However, this efficiency comes at a steep cost. As Sophie Bavard et al. demonstrated, when options are learned within isolated contexts and subsequently evaluated outside those contexts, it leads to systematically "irrational" preferences. Because the absolute values were never recorded—only their relative ranks or scaled positions within their original contexts—the agent struggles to make consistent choices when contexts are mixed.
To mitigate these contextual illusions, the human brain may supplement external rewards with internal heuristics. For example, Anne Collins et al. showed that incorporating internally generated "goal achievement" signals alongside external rewards helps stabilize context-sensitive valuation. Similarly, integrating subjective utility functions with internal reference points allows model-free RL agents to better replicate human-like risk profiles under uncertainty, as explored by Hyungil Ahn et al.
Finally, understanding these contextual boundaries provides critical insight into clinical variations. Henri Vandendriessche et al. discovered that individuals with depression do not suffer from a global learning deficit; instead, their value representation is altered specifically within low-reward environments due to a context-dependent negativity bias.
Key Foundational Work
Context dependence of the event‐related brain potential associated with reward and punishment (2004) — contributed 6 concept elements to this direction.
IMAGING VALUATION MODELS IN HUMAN CHOICE (2006) — contributed 1 concept elements to this direction.