Abstract
We introduce and treat a class of Multi Objective Risk-Sensitive Markov Decision Processes (MORSMDPs), where the optimality criteria are generated by a multivariate function applied on a finite set of different running costs. To illustrate our approach, we study the example of a two-armed bandit problem. In the sequel, we show that it is possible to reformulate standard Risk-Sensitive Partially Observable Markov Decision Processes (RSPOMDPs), where risk is modeled by a utility function that is a sum of exponentials, as MORSMDPs that can be solved with the methods described in the first part. This way, we extend the treatment of RSPOMDPs with exponential utility to RSPOMDPs corresponding to a qualitatively bigger family of utility functions.
Introduction
In the field of automated decision-making, Markov Decision Processes (MDPs) serve as the foundational framework for modeling agents in stochastic environments. However, real-world applications frequently encounter two significant hurdles: partial observability and risk sensitivity. While Partially Observable MDPs (POMDPs) address environments where the state is not directly known, standard models typically focus on maximizing expected rewards. In high-stakes domains such as finance, medicine, or autonomous navigation, an agent must account for the distribution of outcomes, utilizing utility functions to penalize variance or avoid catastrophic failures.
Combining partial observability with general risk-sensitive utility functions is mathematically daunting. While solving POMDPs with simple exponential utility is well-established—leveraging the property that $e^{A+B} = e^A \cdot e^B$—general utility functions often lead to infinite-dimensional state spaces. This paper bridges this gap by introducing a method to solve Risk-Sensitive POMDPs (RSPOMDPs) by approximating complex utility functions as a sum of exponentials, transforming them into manageable, fully observable multivariate optimization tasks.
Methodology: From RSPOMDPs to MORSMDPs
The core innovation of this research lies in the "Sum of Exponentials" approximation and the subsequent reformulation into Multi Objective Risk-Sensitive Markov Decision Processes (MORSMDPs).
Utility Approximation
The authors observe that almost any increasing continuous function can be approximated with high precision by a sum of exponentials: $$U(t) = \sum_{i=1}^{i_{max}} w_i e^{\lambda_i t}$$ This approximation is key because it allows the decomposition of a complex, non-linear utility objective into a set of components that behave more predictably under expectation operators.
Information Vectors and State Augmentation
In a standard POMDP, an agent maintains a "belief state"—a probability distribution over possible hidden states. To handle the sum of exponentials, the authors introduce an "information vector." This vector tracks the expected accumulated costs associated with each specific exponential term in the utility function. By augmenting the state space with these vectors, the authors restore a form of linearity to the problem.
Change of Measure
Using a change of probability measure (specifically, a Radon-Nikodym derivative), the research proves that the original RSPOMDP is equivalent to a Completely Observable MDP on this augmented state space. This transformation is critical; it moves the problem from a partially observable domain to a fully observable one, where Bellman-style optimality equations can be derived for both finite and infinite horizons.
Experimental Framework: The Extended Tiger Problem
To evaluate the algorithm, the authors designed the "Extended Tiger Problem," a more complex version of a classic POMDP benchmark. In the traditional version, an agent must choose between two doors (one containing a treasure, the other a tiger) based on noisy "listening" observations.
The extended version introduces three specific complexities:
- Stochastic States: The tiger has a 10% chance to switch doors each turn, requiring constant monitoring.
- Hidden Rewards: The agent does not receive immediate feedback on rewards or penalties until the end of the trial, complicating the credit assignment.
- Variable Stakes: The agent can choose between "low stake" and "high stake" actions, providing a testing ground for different risk appetites.
Key Insights and Results
Restoring Linearity
In risk-neutral POMDPs, the belief state is a sufficient statistic because the expectation operator is linear. Risk sensitivity usually destroys this property. By utilizing the sum of exponentials, the authors effectively restore a structure that allows for a finite-dimensional augmented state space. The complexity of the solution scales linearly with $i_{max}$ (the number of exponential terms), making the problem computationally tractable if the utility function can be represented by a small number of terms.
Behavioral Modeling and Prospect Theory
A significant advantage of this approach is its ability to model complex human-like behaviors. By approximating S-shaped curves (such as the Sigmoid function), the model can replicate phenomena described in Prospect Theory—where agents are risk-averse regarding gains but risk-seeking regarding losses.
Simulation Outcomes
The simulations on the Extended Tiger Problem confirmed that the augmented MDP converges to the optimal policy across four distinct utility profiles:
- Risk-Neutral: Prioritizes information gathering (listening) before committing to high-stake actions.
- Risk-Seeking: Favors high-stake actions immediately, avoiding the "cost" of listening.
- Risk-Averse: Consistently prefers low-stake actions, even when confidence in the treasure's location is high.
- Sigmoid (S-shaped): Demonstrates adaptive behavior that shifts based on whether the agent's current accumulated "wealth" represents a gain or a loss.
Conclusion and Future Work
This paper provides a robust framework for solving RSPOMDPs by leveraging multivariate utility optimization. While the method is efficient for utility functions that can be approximated by a few exponential terms, the authors acknowledge that high-precision approximations requiring a large $i_{max}$ may increase the computational burden.
Future research directions include applying this framework to true Multi-Objective Optimization problems, such as government resource allocation. In such scenarios, different sectors (e.g., healthcare vs. education) may have distinct running costs and diminishing marginal utility functions, making the MORSMDP approach uniquely suited for finding optimal, risk-aware policies.
Summary updated