Abstract
Standard deep reinforcement learning (DRL) aims to maximize expected reward, considering collected experiences equally in formulating a policy. This differs from human decision-making, where gains and losses are valued differently and outlying outcomes are given increased consideration. It also fails to capitalize on opportunities to improve safety and/or performance through the incorporation of distributional context. Several approaches to distributional DRL have been investigated, with one popular strategy being to evaluate the projected distribution of returns for possible actions. We propose a more direct approach whereby risk-sensitive objectives, specified in terms of the cumulative distribution function (CDF) of the distribution of full-episode rewards, are optimized. This approach allows for outcomes to be weighed based on relative quality, can be used for both continuous and discrete action spaces, and may naturally be applied in both constrained and unconstrained settings. We show how to compute an asymptotically consistent estimate of the policy gradient for a broad class of risk-sensitive objectives via sampling, subsequently incorporating variance reduction and regularization measures to facilitate effective on-policy learning. We then demonstrate that the use of moderately "pessimistic" risk profiles, which emphasize scenarios where the agent performs poorly, leads to enhanced exploration and a continual focus on addressing deficiencies. We test the approach using different risk profiles in six OpenAI Safety Gym environments, comparing to state of the art on-policy methods. Without cost constraints, we find that pessimistic risk profiles can be used to reduce cost while improving total reward accumulation. With cost constraints, they are seen to provide higher positive rewards than risk-neutral approaches at the prescribed allowable cost.
Introduction
In Deep Reinforcement Learning (DRL), the standard objective is to maximize the expected reward, effectively seeking a policy that performs well on average. However, in high-stakes or safety-critical applications, the "average" performance can be a misleading metric. A policy that succeeds frequently but suffers rare, catastrophic failures may have a high expected reward but carries unacceptable risk.
This research addresses the limitations of risk-neutral policy optimization by introducing a framework for training agents sensitive to the entire distribution of possible outcomes. By moving beyond the mean, the proposed approach allows for the incorporation of risk-sensitive objectives that prioritize reliability and safety.
Methodology: Risk-Sensitive Policy Optimization
The core of this approach lies in optimizing objectives defined by the Cumulative Distribution Function (CDF) of full-episode rewards. Drawing inspiration from Cumulative Prospect Theory (CPT), the framework utilizes a weighting function to distort probabilities, effectively re-weighting experiences based on their relative quality.
As shown in the figure above, different risk profiles can be defined by varying the weighting function. A "pessimistic" profile (e.g., Wang with $\eta = 0.5$) over-weights low-reward outcomes, forcing the agent to focus on improving its performance in scenarios where it currently struggles. Conversely, an "optimistic" profile would emphasize the best-case scenarios.
The authors introduce CRiSP (Constrained, Risk-Sensitive Proximal) policy optimization. This algorithm computes an asymptotically consistent estimate of the policy gradient by ordering trajectories in a batch from worst to best and weighting their contributions based on their rank. To ensure stable learning, CRiSP incorporates variance reduction techniques, PPO-style clipping, and Lagrangian constraints for safety-critical environments.
Experimental Evaluation
The approach was evaluated using the OpenAI Safety Gym, featuring "Level 2" versions of Point and Car robot tasks (Goal, Button, and Push). These environments are highly stochastic with randomized layouts, providing a rigorous testbed for risk-sensitive agents.
Pessimism as a Driver for Exploration
A key insight from the experiments is that moderate pessimism can actually enhance exploration. By focusing on fixing worst-case outcomes, pessimistic agents maintain higher policy entropy for longer periods, preventing premature convergence to local optima.
Performance in Constrained and Unconstrained Settings
In unconstrained environments, a pessimistic objective ($\eta = 0.5$) consistently outperformed standard risk-neutral methods (PPO and TRPO) in terms of both higher total rewards and lower costs.
The figure above illustrates this effect in the PointPush2 environment, where the pessimistic Wang ($\eta = 0.5$) and CPT profiles achieve significantly lower mean episode costs compared to the uniform risk-neutral baseline. In constrained settings, the CRiSP algorithm achieved higher positive rewards than other state-of-the-art methods while strictly adhering to safety limits.
Conclusion
This work provides a robust mathematical framework for incorporating risk-sensitivity directly into policy optimization. By leveraging the CDF of rewards, agents can be trained to be "pessimistic," leading to improved exploration, better safety profiles, and higher overall performance. Future work will explore extending this framework to off-policy learning to improve sample efficiency and investigating its application in discrete action spaces.
Summary updated