Introduction
Classical reinforcement learning (RL) is built upon the foundation of Expected Utility Theory (EUT), which assumes that agents make rational decisions to maximize the expected sum of future rewards. However, decades of research in psychology and behavioral economics—most notably the work of Kahneman and Tversky—demonstrate that human decision-making consistently deviates from this "risk-neutral" model. Humans perceive risks, gains, and losses through the lens of Cumulative Prospect Theory (CPT), which accounts for psychological biases such as the overweighting of rare events and the asymmetric treatment of gains versus losses.
While integrating CPT into RL offers a path toward more human-aligned AI, it introduces significant mathematical hurdles. This research addresses these challenges by deriving a novel Policy Gradient Theorem for CPT objectives and introducing CPT-PG, a scalable, first-order algorithm for behavioral reinforcement learning.
The Challenge of CPT in Reinforcement Learning
Incorporating CPT into an RL framework is notoriously difficult because CPT objectives are non-linear and non-convex. Most critically, they break the Bellman equation—the cornerstone of standard RL. Without the Bellman equation, we cannot rely on traditional dynamic programming or value-based methods (like Q-learning) to decompose a long-term problem into smaller, recursive steps.
Previous attempts to solve CPT-RL problems relied on "zeroth-order" optimization methods, such as Simultaneous Perturbation Stochastic Approximation (SPSA). These methods effectively "guess and check" the policy parameters. While functional for small-scale problems, they suffer from the curse of dimensionality, with sample complexity scaling linearly with the number of parameters. This makes them impractical for complex, high-dimensional tasks.
Theoretical Contributions: The CPT Policy Gradient Theorem
The core contribution of this work is the derivation of a first-order CPT Policy Gradient Theorem. By providing a closed-form expression for the gradient of a CPT objective, the authors enable the use of calculus-based optimization in behavioral RL.
Key theoretical insights include:
- Mandatory Stochasticity: Unlike standard RL, where an optimal policy is often deterministic, CPT-RL necessitates stochastic policies. Because the objective distorts probabilities, an agent may need to maintain a specific distribution of actions to maximize its perceived value.
- Logarithmic Scalability: The authors prove that their gradient estimator is consistent and that the resulting algorithm's sample complexity scales logarithmically with the number of policy parameters. This represents a massive efficiency gain over previous zeroth-order approaches.
The CPT-PG Algorithm
Building on the new theorem, the authors designed the CPT-PG algorithm. To handle the "probability distortion" component of CPT, the algorithm utilizes a Monte Carlo estimator based on order statistics. The process involves:
- Sampling multiple trajectories (rollouts).
- Sorting the resulting rewards to estimate quantiles.
- Weighting the gradient updates according to the CPT distortion function.
This approach allows the agent to learn policies that reflect specific human-like attitudes toward risk without requiring a predefined model of the environment.
Experimental Evaluation
The algorithm was tested across a diverse range of environments to evaluate its scalability and behavioral accuracy:
- Behavioral Benchmarks: In "Gain" and "Loss" bandit tasks, the CPT-PG agent successfully captured the "reflection effect." For instance, in the loss-based scenario, the agent correctly identified a risky action as preferable—mimicking the human tendency to gamble to avoid a certain loss—whereas standard risk-sensitive agents remained overly cautious.
- Scalability in Grid Worlds: In $n \times n$ grid environments, CPT-PG remained robust as the state space grew. In contrast, previous state-of-the-art methods (like CPT-SPSA-G) saw performance degrade rapidly as the problem complexity increased.
- Real-World Applications: The algorithm was applied to financial trading (BTC-USD data) and electricity market management. In the electricity task, the agent demonstrated nuanced control: a risk-averse CPT profile minimized the danger of battery depletion, while a risk-seeking profile maximized profit by aggressively timing energy sales during price peaks.
- Continuous Control: The algorithm’s success on the MuJoCo InvertedPendulum benchmark demonstrated its applicability to high-dimensional, continuous action spaces.
Conclusion and Future Directions
This research bridges the gap between behavioral economics and modern machine learning. By providing a scalable, first-order method for CPT-RL, it opens the door for agents that can better understand or mimic human decision-making.
Future work remains in several areas. Currently, the framework assumes that CPT parameters (the specific utility and distortion functions) are known. Integrating this with Reinforcement Learning from Human Feedback (RLHF) could allow agents to learn these psychological profiles directly from human interaction. Additionally, extending the theory from finite-horizon tasks to infinite-horizon discounted rewards remains a significant theoretical frontier. Ultimately, this work advances the development of AI systems that are not just mathematically optimal, but psychologically resonant.
Summary updated