Iterated risk measures for risk-sensitive Markov decision processes with discounted cost

Takayuki Osogami

Abstract

Standard approaches for representing risk preferences in discounted cost scenarios, such as Discounted Expected Utility (DEU), face significant limitations. We demonstrate that DEU cannot represent certain rational preferences, particularly regarding cumulative risk. Furthermore, modifications like Expected Utility of Discounted cumulative reward (EUD) often lead to time-inconsistent decision-making. This paper proposes Iterated Risk Measures (IRM) as a solution. We show that IRMs can represent preferences beyond the reach of DEU while ensuring time-consistent decisions and compatibility with efficient dynamic programming.

Introduction

In Markov Decision Processes (MDPs), the traditional goal is to minimize expected costs. However, in domains like finance or robotics, risk sensitivity is crucial to avoid catastrophic outcomes. Standard methods for incorporating risk sensitivity often fail to balance mathematical consistency with intuitive human preferences. This paper, authored by Takayuki Osogami, tackles this challenge by introducing a framework that is sensitive to total risk while remaining compatible with efficient optimization.

Limitations of Standard Risk Measures

The paper identifies two primary flaws in existing frameworks for risk-sensitive decision-making:

  1. Discounted Expected Utility (DEU): DEU applies a utility function to rewards at each time step. This approach has a "blind spot": it fails to account for cumulative risk. It cannot represent a preference for a single certain cost over a series of risky costs that could aggregate into a disaster.
  2. Expected Utility of Discounted cumulative reward (EUD): EUD applies a utility function to the total discounted sum. While it captures cumulative risk, it is time-inconsistent. A decision-maker might prefer Plan A today but switch to Plan B tomorrow without any new information, simply due to the passage of time.

The Payment Example

To illustrate the failure of DEU, consider the "Payment Example" shown in Table 1.

Table 1: The Payment Example comparing a certain payment (A) with a risky sequence of payments (B).

In this scenario, a decision-maker might prefer a guaranteed $1,000 payment today (Option A) over a small probability of paying $1,000 every day for 20 days (Option B). The paper proves that no utility function exists such that DEU can represent this preference, effectively showing that DEU "forgets" the risk of previous steps.

Iterated Risk Measures (IRM)

The author proposes Iterated Risk Measures (IRM) to address these issues. Instead of evaluating the total sum at once, an IRM applies risk measures recursively at each time step. If $\rho$ is a risk measure (such as Expected Value or Value at Risk), the risk at time $n$ is defined as: $$\rho_n[\cdot] = \dot{\rho}_n[\rho_{n+1}[\cdot]]$$

This recursive structure allows for the derivation of a Bellman equation, enabling the use of Dynamic Programming to find optimal policies efficiently without checking every possible future path.

The Traveler Example

The "Traveler Example" (Table 2) demonstrates how different risk measures influence decision-making in the face of uncertainty.

Table 2: The Traveler Example defining two paths (P and Q) under different traffic conditions.

A traveler chooses between two paths with different traffic distributions. Depending on the risk measure used—such as Entropic Risk or Iterated Conditional Tail Expectation (ICTE)—the optimal path may change. This illustrates how IRMs can capture human-like risk aversion better than standard utility functions.

Key Theoretical Results

The paper identifies a "Golden Trio" of properties—strong monotonicity, translation-invariance, and positive homogeneity—that guarantee an IRM is both time-consistent and solvable via dynamic programming.

  • Theorem 1: If an IRM satisfies these three properties, the optimal policy found today will remain optimal in the future. This bridges the gap between complex risk theory and practical reinforcement learning.
  • Recursive Efficiency: Unlike some risk-sensitive objectives that lead to computational explosions, IRMs maintain the recursive property needed for efficient computation in large-scale environments.

Conclusion and Future Work

Iterated Risk Measures provide a robust framework for risk-sensitive decision-making in discounted MDPs. By bridging the gap between complex risk theory and practical optimization, they offer a path toward more reliable and intuitive AI agents in high-stakes environments. While the mathematical superiority of IRMs is established, future work will involve behavioral studies to determine how well specific measures like ICTE map to actual human psychological risk profiles. Additionally, implementing these measures in high-dimensional state spaces remains a significant engineering challenge for modern Deep RL.

Summary updated

Abstract

We demonstrate a limitation of discounted expected utility, a standard approach for representing the preference to risk when future cost is discounted. Specifically, we provide an example of the preference of a decision maker that appears to be rational but cannot be represented with any discounted expected utility. A straightforward modification to discounted expected utility leads to inconsistent decision making over time. We will show that an iterated risk measure can represent the preference that cannot be represented by any discounted expected utility and that the decisions based on the iterated risk measure are consistent over time.
BibTeX
@inproceedings{osogami2011iterated,
  author = {Takayuki Osogami},
  title = {{Iterated risk measures for risk-sensitive Markov decision processes with discounted cost}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2011},
  url = {https://openreview.net/forum?id=0xMIOv1GvL},
}
Figure