Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning

Shangtong Zhang, Bo Liu, Shimon Whiteson

Abstract

We present a mean-variance policy iteration (MVPI) framework for risk-averse control in a discounted infinite horizon MDP optimizing the variance of a per-step reward random variable. MVPI enjoys great flexibility in that any policy evaluation method and risk-neutral control method can be dropped in for risk-averse control off the shelf, in both on- and off-policy settings. This flexibility reduces the gap between risk-neutral control and risk-averse control and is achieved by working on a novel augmented MDP directly. We propose risk-averse TD3 as an example instantiating MVPI, which outperforms vanilla TD3 and many previous risk-averse control methods in challenging Mujoco robot simulation tasks under a risk-aware performance metric. This risk-averse TD3 is the first to introduce deterministic policies and off-policy learning into risk-averse reinforcement learning, both of which are key to the performance boost we show in Mujoco domains.

Introduction

Standard Reinforcement Learning (RL) typically focuses on maximizing the expected total reward, a "risk-neutral" objective. However, real-world deployments in fields such as healthcare, autonomous driving, and finance necessitate "risk-averse" strategies where the variance of outcomes is minimized to avoid catastrophic failures.

Historically, risk-averse RL has faced two major hurdles. First, the mean-variance objective introduces a policy-dependent reward, making the environment appear non-stationary and breaking standard Bellman equations. Second, existing risk-averse methods often rely on on-policy learning and linear models, failing to leverage the data efficiency and representational power of modern off-policy deep RL.

The MVPI Framework

To address these challenges, we introduce Mean-Variance Policy Iteration (MVPI), a flexible meta-algorithm that transforms risk-neutral RL algorithms into risk-sensitive ones.

Per-Step Reward Variance

Rather than optimizing the complex variance of the total return, MVPI optimizes the variance of the per-step reward. This serves as a computationally efficient proxy and an upper bound for the total variance, providing a more stable objective for deep RL agents.

Fenchel Duality and Augmented MDPs

MVPI overcomes the policy-dependent reward issue by utilizing Fenchel duality to reformulate the objective. This allows the problem to be solved via Block Cyclic Coordinate Ascent (BCCA), alternating between two primary steps:

  1. Policy Evaluation: Estimating the current policy's performance scalar.
  2. Policy Improvement: Solving a standard MDP with an augmented reward function that penalizes high-variance actions.

Crucially, because the augmented reward is fixed during the improvement step, any standard risk-neutral algorithm—such as TD3 or PPO—can be employed. We instantiate this framework as MVPI-TD3, the first risk-averse algorithm to utilize deterministic policies and off-policy learning.

Experimental Setup

We evaluated MVPI-TD3 on eight challenging MuJoCo robot benchmarks (e.g., Ant-v2, HalfCheetah-v2) within OpenAI Gym. To necessitate risk-aversion, Gaussian noise was added to the actions to create stochastic environments. Additionally, a Tabular MDP was used to assess the algorithm's performance in off-line learning scenarios.

Results and Discussion

MVPI-TD3 demonstrates significant improvements over existing risk-averse baselines. As shown in the performance benchmarks, MVPI-TD3 consistently achieves higher episode returns than previous risk-averse methods like TRVO and MVP across the MuJoCo suite. While risk-neutral TD3 may achieve higher raw returns in some tasks, MVPI-TD3 provides a superior balance of reward and stability.

Performance comparison of MVPI-TD3 against risk-neutral TD3 and several risk-averse baselines (TRVO, MVP, Prashanth, Tamar) across eight MuJoCo environments.

A key advantage of MVPI is its ability to handle the risk-performance trade-off explicitly. In off-line settings, adjusting the risk-aversion parameter $\lambda$ allows the agent to transition smoothly between high-reward/high-risk policies and safer, lower-reward alternatives.

Off-line MVPI results showing how the policy probability evolves over iterations for different values of the risk-aversion parameter $\lambda$. Higher $\lambda$ values drive the agent toward safer actions.

Furthermore, MVPI-TD3 improved the Sharpe Ratio (the ratio of mean return to standard deviation) in 6 out of 8 tasks compared to vanilla TD3, confirming its effectiveness in optimizing risk-aware performance metrics.

Conclusion

MVPI bridges the gap between risk-neutral and risk-averse RL by providing a modular framework that can incorporate state-of-the-art off-policy algorithms. While the per-step variance is a proxy for total variance and density ratio estimation remains challenging in high dimensions, MVPI-TD3 sets a new benchmark for risk-averse control in complex robotic simulations. Future work will explore automated tuning of the risk parameter $\lambda$ and extending the framework to other risk measures like Conditional Value at Risk (CVaR).

Summary updated

Abstract

We present a mean-variance policy iteration (MVPI) framework for risk-averse control in a discounted infinite horizon MDP optimizing the variance of a per-step reward random variable. MVPI enjoys great flexibility in that any policy evaluation method and risk-neutral control method can be dropped in for risk-averse control off the shelf, in both on- and off-policy settings. This flexibility reduces the gap between risk-neutral control and risk-averse control and is achieved by working on a novel augmented MDP directly. We propose risk-averse TD3 as an example instantiating MVPI, which outperforms vanilla TD3 and many previous risk-averse control methods in challenging Mujoco robot simulation tasks under a risk-aware performance metric. This risk-averse TD3 is the first to introduce deterministic policies and off-policy learning into risk-averse reinforcement learning, both of which are key to the performance boost we show in Mujoco domains.
BibTeX
@inproceedings{zhang2021mean,
  author = {Shangtong Zhang and Bo Liu and Shimon Whiteson},
  title = {{Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=rMG-mKSSSe9},
}
Figure