Abstract
We present a mean-variance policy iteration (MVPI) framework for risk-averse control in a discounted infinite horizon MDP optimizing the variance of a per-step reward random variable. MVPI enjoys great flexibility in that any policy evaluation method and risk-neutral control method can be dropped in for risk-averse control off the shelf, in both on- and off-policy settings. This flexibility reduces the gap between risk-neutral control and risk-averse control and is achieved by working on a novel augmented MDP directly. We propose risk-averse TD3 as an example instantiating MVPI, which outperforms vanilla TD3 and many previous risk-averse control methods in challenging Mujoco robot simulation tasks under a risk-aware performance metric. This risk-averse TD3 is the first to introduce deterministic policies and off-policy learning into risk-averse reinforcement learning, both of which are key to the performance boost we show in Mujoco domains.
Introduction
Standard Reinforcement Learning (RL) typically focuses on maximizing the expected total reward, a "risk-neutral" objective. However, real-world deployments in fields such as healthcare, autonomous driving, and finance necessitate "risk-averse" strategies where the variance of outcomes is minimized to avoid catastrophic failures.
Historically, risk-averse RL has faced two major hurdles. First, the mean-variance objective introduces a policy-dependent reward, making the environment appear non-stationary and breaking standard Bellman equations. Second, existing risk-averse methods often rely on on-policy learning and linear models, failing to leverage the data efficiency and representational power of modern off-policy deep RL.
The MVPI Framework
To address these challenges, we introduce Mean-Variance Policy Iteration (MVPI), a flexible meta-algorithm that transforms risk-neutral RL algorithms into risk-sensitive ones.
Per-Step Reward Variance
Rather than optimizing the complex variance of the total return, MVPI optimizes the variance of the per-step reward. This serves as a computationally efficient proxy and an upper bound for the total variance, providing a more stable objective for deep RL agents.
Fenchel Duality and Augmented MDPs
MVPI overcomes the policy-dependent reward issue by utilizing Fenchel duality to reformulate the objective. This allows the problem to be solved via Block Cyclic Coordinate Ascent (BCCA), alternating between two primary steps:
- Policy Evaluation: Estimating the current policy's performance scalar.
- Policy Improvement: Solving a standard MDP with an augmented reward function that penalizes high-variance actions.
Crucially, because the augmented reward is fixed during the improvement step, any standard risk-neutral algorithm—such as TD3 or PPO—can be employed. We instantiate this framework as MVPI-TD3, the first risk-averse algorithm to utilize deterministic policies and off-policy learning.
Experimental Setup
We evaluated MVPI-TD3 on eight challenging MuJoCo robot benchmarks (e.g., Ant-v2, HalfCheetah-v2) within OpenAI Gym. To necessitate risk-aversion, Gaussian noise was added to the actions to create stochastic environments. Additionally, a Tabular MDP was used to assess the algorithm's performance in off-line learning scenarios.
Results and Discussion
MVPI-TD3 demonstrates significant improvements over existing risk-averse baselines. As shown in the performance benchmarks, MVPI-TD3 consistently achieves higher episode returns than previous risk-averse methods like TRVO and MVP across the MuJoCo suite. While risk-neutral TD3 may achieve higher raw returns in some tasks, MVPI-TD3 provides a superior balance of reward and stability.
A key advantage of MVPI is its ability to handle the risk-performance trade-off explicitly. In off-line settings, adjusting the risk-aversion parameter $\lambda$ allows the agent to transition smoothly between high-reward/high-risk policies and safer, lower-reward alternatives.
Furthermore, MVPI-TD3 improved the Sharpe Ratio (the ratio of mean return to standard deviation) in 6 out of 8 tasks compared to vanilla TD3, confirming its effectiveness in optimizing risk-aware performance metrics.
Conclusion
MVPI bridges the gap between risk-neutral and risk-averse RL by providing a modular framework that can incorporate state-of-the-art off-policy algorithms. While the per-step variance is a proxy for total variance and density ratio estimation remains challenging in high dimensions, MVPI-TD3 sets a new benchmark for risk-averse control in complex robotic simulations. Future work will explore automated tuning of the risk parameter $\lambda$ and extending the framework to other risk measures like Conditional Value at Risk (CVaR).
Summary updated