Abstract
We introduce a novel framework to account for sensitivity to rewards uncertainty in sequential decision-making problems. While risk-sensitive formulations for Markov decision processes studied so far focus on the distribution of the cumulative reward as a whole, we aim at learning policies sensitive to the uncertain/stochastic nature of the rewards, which has the advantage of being conceptually more meaningful in some cases. To this end, we present a new decomposition of the randomness contained in the cumulative reward based on the Doob decomposition of a stochastic process, and introduce a new conceptual tool - the chaotic variation - which can rigorously be interpreted as the risk measure of the martingale component associated to the cumulative reward process. We innovate on the reinforcement learning side by incorporating this new risk-sensitive approach into model-free algorithms, both policy gradient and value function based, and illustrate its relevance on grid world and portfolio optimization problems.
Introduction: The Flaw in Standard Risk-Sensitivity
In classical reinforcement learning (RL), agents typically aim to maximize the expected sum of future rewards. Risk-sensitive RL extends this by accounting for reward variability, often utilizing measures like variance or Conditional Value at Risk (CVaR). However, current methods often fail to distinguish between two fundamentally different sources of randomness: predictable variability and chaotic uncertainty.
Predictable variability occurs when an agent moves between states with different reward levels (e.g., moving from a low-reward state to a high-reward state). Chaotic uncertainty, conversely, is the inherent "noise" or stochasticity of a reward within a specific state—the "surprise" factor that cannot be predicted even with full state knowledge. Standard risk-averse agents often avoid high-paying states simply because the transition itself creates variance in the reward stream, even if the outcome is guaranteed.
Methodology: The Martingale Approach
To address this, we leverage the Doob decomposition from probability theory. This allows us to split any stochastic reward process into two distinct parts:
- The Predictable Component: The expected reward based on current information.
- The Martingale (Chaotic) Component: The "surprise" difference between the actual reward received and the expectation.
We introduce Chaotic Variation, a metric that specifically measures the risk associated with the martingale component. By penalizing only this chaotic element, we develop Chaotic Mean-Variance (CMV) algorithms. These include CMV-Q-Learning, which modifies the reward function to subtract a penalty proportional to the squared "surprise," and policy gradient methods like CMV-REINFORCE and CMV-Actor-Critic.
Experimental Evaluation
The framework was evaluated in two simulated environments designed to test the distinction between predictable and chaotic risk:
- Grid World: A robot must navigate to a goal while avoiding "trap" regions. The environment includes a 50% error probability in movement, creating high reward uncertainty near traps. As the risk-aversion parameter ($\beta$) increases, CMV agents learn to take longer, safer paths to avoid these unpredictable regions.
- Portfolio Optimization: A financial simulation involving three volatility states and two assets (one risk-free, one risky). In this setting, standard risk-averse agents often stop investing entirely to zero out variance. In contrast, CMV agents maintain investments in risk-free assets while precisely scaling back exposure to the "chaotic" risky asset as risk-aversion increases, aligning better with rational financial behavior.
Key Insights and Results
The primary contribution of this work is the realization that state transitions should not be penalized as "risk" if they are predictable. By using the Doob decomposition, the agent becomes "brave" regarding predictable changes but "cautious" regarding unpredictable noise.
Key results include:
- Financial Rationality: CMV agents demonstrate more human-like investment strategies compared to classical mean-variance baselines.
- Convergence: We provide formal proof that CMV-Q-Learning converges to the optimal chaotic policy with probability 1.
- Robustness: The algorithms successfully isolate and mitigate the impact of "surprise" spikes in reward streams without sacrificing performance derived from predictable state improvements.
Conclusion and Future Work
This martingale approach provides a mathematically rigorous way to handle risk over time by bridging the gap between overall variance and per-period variance. While current experiments focus on finite state spaces, future work will extend these principles to continuous spaces using deep neural networks (e.g., Chaotic PPO) and explore scenarios with delayed rewards where uncertainty resolution is deferred across multiple timesteps.
Summary updated