Standard Reinforcement Learning (RL) often simplifies complex reward functions into a basic "state-action" average. While this works for maximizing expected returns, it fails in risk-sensitive scenarios where the full distribution of rewards matters. This paper introduces State-Augmentation Transformations (SATs), a method to map complex, stochastic reward structures into simplified ones without losing the vital distributional information needed for risk measures like Value-at-Risk (VaR).
The Challenge of Stochastic Rewards
In the standard Markov Decision Process (MDP) framework, rewards are typically treated as a function of the current state and action: $r(s, a)$. However, real-world rewards are often transition-based (depending on the successor state $s'$) or stochastic.
When applying RL, researchers frequently "simplify" rewards by taking the expectation over transitions. For risk-neutral goals (maximizing average total reward), this is mathematically sound. However, for risk-sensitive applications—such as avoiding crashes in autonomous vehicles or managing financial portfolios—the full reward distribution is essential. Simplifying rewards to a mean effectively "smooths out" variance and tail risks, leading to dangerously incorrect risk assessments.
Methodology: State-Augmentation Transformations (SATs)
The authors propose three levels of SATs to bridge the gap between complex reward reality and simplified algorithmic requirements, covering Markov reward processes, MDPs with fixed policies, and general MDPs with randomized policy spaces.
The core technique involves "augmenting" the state space. Instead of averaging rewards, a new, larger state space is defined where each state is a tuple containing the previous state, the action taken, the successor state, and the specific reward outcome. By moving the "uncertainty" of the reward into the definition of the state itself, the authors create a new MDP where the reward is a deterministic function of the augmented state.
This transformation is mathematically proven to preserve the exact return distribution of the original process, allowing standard RL tools to be used for risk-sensitive objectives without information loss.
Experimental Evaluation: Inventory Control
The proposed SATs were evaluated using a simulated Inventory Control Problem. In this setup, a manager decides order quantities for a single product based on uncertain demand. The reward structure includes ordering costs, maintenance fees, and sales revenue, creating a deterministic, transition-based reward structure.
The study compared return distributions generated by the SAT method against a "naively simplified" model using expected rewards.
Key Results and the "Mean Trap"
The results highlight the "Mean Trap": reward simplification acts as an analysis of variance problem. By averaging rewards over transitions, "within-group" variance is eliminated, making the resulting return distribution appear much narrower and safer than it actually is.
Using the Kolmogorov-Smirnov ($D_{KS}$) statistic to measure the distance between the true empirical distribution and estimations:
- SAT-based estimation: Negligible error ($D_{KS} \approx 0.012$).
- Simplified model: Significantly higher error ($D_{KS} \approx 0.145$).
The simplified model significantly underestimated return variance. In a risk-sensitive context, this would mislead policy-makers into believing a strategy is safe, potentially leading to catastrophic failures during tail-risk events.
Conclusion and Future Work
While SATs provide a robust framework for risk-sensitive RL, they introduce computational complexity due to state space expansion ($S^\dagger = S^2 \times A \times J$). Future research should explore scaling these transformations to continuous or very large state spaces. Nonetheless, SATs serve as a vital "wrapper" for making real-world stochastic problems compatible with risk-sensitive theoretical tools, ensuring that the sequence of rewards—and thus the law-invariant risk measures—is preserved.
Summary updated