Abstract
Standard approaches for representing risk preferences in discounted cost scenarios, such as Discounted Expected Utility (DEU), face significant limitations. We demonstrate that DEU cannot represent certain rational preferences, particularly regarding cumulative risk. Furthermore, modifications like Expected Utility of Discounted cumulative reward (EUD) often lead to time-inconsistent decision-making. This paper proposes Iterated Risk Measures (IRM) as a solution. We show that IRMs can represent preferences beyond the reach of DEU while ensuring time-consistent decisions and compatibility with efficient dynamic programming.
Introduction
In Markov Decision Processes (MDPs), the traditional goal is to minimize expected costs. However, in domains like finance or robotics, risk sensitivity is crucial to avoid catastrophic outcomes. Standard methods for incorporating risk sensitivity often fail to balance mathematical consistency with intuitive human preferences. This paper, authored by Takayuki Osogami, tackles this challenge by introducing a framework that is sensitive to total risk while remaining compatible with efficient optimization.
Limitations of Standard Risk Measures
The paper identifies two primary flaws in existing frameworks for risk-sensitive decision-making:
- Discounted Expected Utility (DEU): DEU applies a utility function to rewards at each time step. This approach has a "blind spot": it fails to account for cumulative risk. It cannot represent a preference for a single certain cost over a series of risky costs that could aggregate into a disaster.
- Expected Utility of Discounted cumulative reward (EUD): EUD applies a utility function to the total discounted sum. While it captures cumulative risk, it is time-inconsistent. A decision-maker might prefer Plan A today but switch to Plan B tomorrow without any new information, simply due to the passage of time.
The Payment Example
To illustrate the failure of DEU, consider the "Payment Example" shown in Table 1.
In this scenario, a decision-maker might prefer a guaranteed $1,000 payment today (Option A) over a small probability of paying $1,000 every day for 20 days (Option B). The paper proves that no utility function exists such that DEU can represent this preference, effectively showing that DEU "forgets" the risk of previous steps.
Iterated Risk Measures (IRM)
The author proposes Iterated Risk Measures (IRM) to address these issues. Instead of evaluating the total sum at once, an IRM applies risk measures recursively at each time step. If $\rho$ is a risk measure (such as Expected Value or Value at Risk), the risk at time $n$ is defined as: $$\rho_n[\cdot] = \dot{\rho}_n[\rho_{n+1}[\cdot]]$$
This recursive structure allows for the derivation of a Bellman equation, enabling the use of Dynamic Programming to find optimal policies efficiently without checking every possible future path.
The Traveler Example
The "Traveler Example" (Table 2) demonstrates how different risk measures influence decision-making in the face of uncertainty.
A traveler chooses between two paths with different traffic distributions. Depending on the risk measure used—such as Entropic Risk or Iterated Conditional Tail Expectation (ICTE)—the optimal path may change. This illustrates how IRMs can capture human-like risk aversion better than standard utility functions.
Key Theoretical Results
The paper identifies a "Golden Trio" of properties—strong monotonicity, translation-invariance, and positive homogeneity—that guarantee an IRM is both time-consistent and solvable via dynamic programming.
- Theorem 1: If an IRM satisfies these three properties, the optimal policy found today will remain optimal in the future. This bridges the gap between complex risk theory and practical reinforcement learning.
- Recursive Efficiency: Unlike some risk-sensitive objectives that lead to computational explosions, IRMs maintain the recursive property needed for efficient computation in large-scale environments.
Conclusion and Future Work
Iterated Risk Measures provide a robust framework for risk-sensitive decision-making in discounted MDPs. By bridging the gap between complex risk theory and practical optimization, they offer a path toward more reliable and intuitive AI agents in high-stakes environments. While the mathematical superiority of IRMs is established, future work will involve behavioral studies to determine how well specific measures like ICTE map to actual human psychological risk profiles. Additionally, implementing these measures in high-dimensional state spaces remains a significant engineering challenge for modern Deep RL.
Summary updated