Risk-sensitive reinforcement learning: a martingale approach to reward uncertainty

Nelson Vadori, Sumitra Ganesh, Prashant P. Reddy, Manuela Veloso

TL;DR

Standard risk-sensitive reinforcement learning (RL) often fails because it treats all reward variability as "risk," even the predictable changes caused by moving between states. This paper introduces a novel framework that uses the Doob decomposition to isolate "chaotic" reward uncertainty (the "surprise" factor) from predictable transitions. By only penalizing this chaotic component, the authors develop new "Chaotic Mean-Variance" (CMV) algorithms that produce more intuitive, stable, and financially rational policies.

What Problem Does This Paper Address?

In classical RL, agents try to maximize the expected sum of future rewards. Risk-sensitive RL goes a step further by accounting for the variability of those rewards, typically using measures like variance or Conditional Value at Risk (CVaR).

However, there is a subtle but critical flaw in how current methods handle risk: they apply risk measures to the cumulative reward as a whole. This fails to distinguish between two very different sources of randomness:

  1. Predictable Variability: The change in reward that happens because an agent moves from a low-reward state to a high-reward state.
  2. Chaotic Uncertainty: The inherent "noise" or stochasticity of a reward in a specific state (e.g., a stock price change that is unknown until it happens).

Standard risk-averse agents might avoid a high-paying state simply because the transition itself creates "variance" in the reward stream, even if the outcome is guaranteed. This paper aims to fix this by isolating and penalizing only the "surprise" or non-deterministic part of the reward.

Methods

The authors leverage a fundamental concept from probability theory called the Doob decomposition. This allows them to split any stochastic reward process into two parts:

  • The Predictable Component: What we expect the reward to be based on the information we have at the current time step.
  • The Martingale (Chaotic) Component: The "surprise" difference between the actual reward received and our best guess.

They introduce a new conceptual tool called Chaotic Variation, which measures the risk associated specifically with that martingale component.

To put this into practice, they modify canonical RL algorithms to create Chaotic Mean-Variance (CMV) versions:

  • CMV-Q-Learning: A value-based approach where the reward is modified to $R^\beta$, subtracting a penalty proportional to the squared "surprise."
  • CMV-REINFORCE & CMV-Actor-Critic: Policy gradient methods that optimize a performance metric sensitive to chaotic variation.

A key technical challenge they solve is that the "expected reward" $R(s, a)$ isn't usually known upfront. Their algorithms estimate this value on-the-fly to calculate the chaotic penalty.

Datasets

The paper evaluates the framework using two simulated environments designed to highlight the difference between predictable and chaotic risk:

  1. Grid World: A 16-square grid where a robot must reach a goal while avoiding "trap" regions with negative rewards. The environment includes a 50% "error" probability where the robot moves in a random direction, creating high reward uncertainty near traps.
  2. Portfolio Optimization: A financial simulation with three volatility states (Low, Medium, High) and two assets (one risk-free, one risky). The agent must decide how to allocate capital across 21 possible investment fractions.

Key Insights and Ideas

  • Risk vs. Transition: The "Aha!" moment is realizing that moving to a better state shouldn't be considered "risky" just because it changes the reward level. By using the Doob decomposition, the agent learns to be "brave" regarding predictable changes but "cautious" regarding unpredictable noise.
  • The Martingale Bridge: The authors show that penalizing the quadratic variation of the martingale component bridges the gap between "overall variance" and "per-period variance," providing a mathematically rigorous way to handle risk over time.
  • Financial Rationality: In the portfolio task, standard risk-averse RL agents often stop investing in everything (including the risk-free asset) to minimize total variance. The CMV agent, however, only pulls back from the risky asset, which aligns much better with how human investors actually behave.

Key Results

  • Grid World: As the risk-aversion parameter ($\beta$) increases, CMV agents successfully learn to take longer, safer paths that stay far away from highly uncertain "trap" states, even when those states are adjacent to the goal.
  • Portfolio Optimization: CMV-REINFORCE outperformed the classical mean-variance baseline. While the baseline agent eventually stopped trading entirely to zero out variance, the CMV agent maintained its investment in the risk-free asset while precisely scaling back its exposure to the "chaotic" risky asset as risk-aversion increased.
  • Convergence: The authors provide a formal proof that CMV-Q-Learning converges to the optimal chaotic policy with probability 1, ensuring the algorithm is robust.

Limitations and Future Work

  • State Space: The current proofs and tabular experiments focus on finite state and action spaces. Extending this to continuous spaces using deep neural networks (e.g., Chaotic PPO or SAC) is a logical next step.
  • Estimation Error: The method relies on an accurate estimate of the expected reward $R(s, a)$. If this estimate is poor, the "chaotic" penalty will be inaccurate, potentially leading to sub-optimal policies.
  • Delayed Rewards: The current framework assumes rewards are received immediately after an action. The authors suggest that future work could extend this to "delayed" reward scenarios where the uncertainty isn't resolved for several time steps.

Summary updated

Abstract

We introduce a novel framework to account for sensitivity to rewards uncertainty in sequential decision-making problems. While risk-sensitive formulations for Markov decision processes studied so far focus on the distribution of the cumulative reward as a whole, we aim at learning policies sensitive to the uncertain/stochastic nature of the rewards, which has the advantage of being conceptually more meaningful in some cases. To this end, we present a new decomposition of the randomness contained in the cumulative reward based on the Doob decomposition of a stochastic process, and introduce a new conceptual tool - the \textit{chaotic variation} - which can rigorously be interpreted as the risk measure of the martingale component associated to the cumulative reward process. We innovate on the reinforcement learning side by incorporating this new risk-sensitive approach into model-free algorithms, both policy gradient and value function based, and illustrate its relevance on grid world and portfolio optimization problems.
BibTeX
@inproceedings{vadori2020risk,
  author = {Nelson Vadori and Sumitra Ganesh and Prashant P. Reddy and Manuela Veloso},
  title = {{Risk-sensitive reinforcement learning: a martingale approach to reward uncertainty}},
  booktitle = {ICAIF},
  year = {2020},
  doi = {10.1145/3383455.3422519},
}

Loading related content...

Figure