Abstract
We present a mean-variance policy iteration (MVPI) framework for risk-averse control in a discounted infinite horizon MDP optimizing the variance of a per-step reward random variable. MVPI enjoys great flexibility in that any policy evaluation method and risk-neutral control method can be dropped in for risk-averse control off the shelf, in both on- and off-policy settings. This flexibility reduces the gap between risk-neutral control and risk-averse control and is achieved by working on a novel augmented MDP directly. We propose risk-averse TD3 as an example instantiating MVPI, which outperforms vanilla TD3 and many previous risk-averse control methods in challenging Mujoco robot simulation tasks under a risk-aware performance metric. This risk-averse TD3 is the first to introduce deterministic policies and off-policy learning into risk-averse reinforcement learning, both of which are key to the performance boost we show in Mujoco domains.
Introduction
Standard reinforcement learning (RL) typically focuses on maximizing the expected cumulative reward, a "risk-neutral" objective. However, real-world deployments in fields like autonomous driving or healthcare necessitate "risk-averse" strategies that prioritize consistency and low volatility. Traditional risk-averse RL faces significant hurdles: the policy-dependent nature of variance objectives breaks the Bellman equation's stationarity, and existing methods often struggle to scale to high-dimensional continuous control tasks or require sample-inefficient on-policy data.
This paper introduces Mean-Variance Policy Iteration (MVPI), a flexible framework designed to bridge the gap between risk-neutral and risk-averse RL. By optimizing the variance of per-step rewards, MVPI provides a scalable approach to achieving stable, high-performance control in complex environments.
Methods
The MVPI framework redefines the risk-averse objective by focusing on the variance of the per-step reward rather than the total reward. This per-step perspective acts as a tractable proxy that bounds total variance while maintaining smoother trajectories.
To optimize this objective, the authors employ Fenchel duality to transform the variance term (which involves a squared expectation) into a more manageable form. This enables the use of Block Cyclic Coordinate Ascent (BCCA), an alternating optimization strategy:
- Policy Evaluation: Estimate the performance of the current policy.
- Policy Improvement: Update the policy using an "augmented" reward function that incorporates both mean and variance.
A key advantage of MVPI is its "plug-and-play" nature. The policy improvement step essentially reduces to a standard RL problem, allowing researchers to utilize state-of-the-art algorithms like TD3 or PPO. The authors instantiate this as MVPI-TD3, the first risk-averse algorithm to leverage deterministic policies and off-policy learning in complex domains.
Experimental Setup
The framework was evaluated on eight high-dimensional continuous control benchmarks from the MuJoCo suite (e.g., HalfCheetah, Walker2d, Ant). Additionally, a tabular MDP was used to validate the off-line MVPI setting, where agents learn from fixed datasets without environment interaction.
Results and Discussion
MVPI-TD3 demonstrates superior performance compared to vanilla TD3 and established risk-averse baselines like TRVO. While standard RL agents might achieve higher raw returns, they often exhibit significant volatility. MVPI-TD3 successfully balances return and risk, as evidenced by its performance under risk-aware metrics.
As shown in the learning curves, MVPI-TD3 (green) consistently scales to complex tasks where previous risk-averse methods often fail to progress. Quantitatively, MVPI-TD3 achieves significant improvements in risk-adjusted metrics, such as the Sharpe Ratio, across most environments.
The results highlight that deterministic policies, previously under-explored in risk-sensitive research, are highly effective at variance reduction. Furthermore, the ability to use off-policy data significantly improves sample efficiency compared to prior on-policy risk-averse methods.
Conclusion
MVPI offers a scalable and flexible framework for risk-averse RL by leveraging the per-step reward perspective and an augmented MDP formulation. While the per-step variance is a proxy for total variance, it proves to be a highly effective signal for deep RL. Future work may extend this framework to other risk measures such as Conditional Value at Risk (CVaR) and further improve the stability of density ratio learning for off-line settings.
Summary updated