Abstract
Keeping risk under control is often more crucial than maximizing expected rewards in real-world decision-making situations, such as finance, robotics, and autonomous driving. The most common choice of risk measure is variance, which penalizes upside volatility as much as downside risk. This paper focuses on the Mean-Semivariance (MSV) criterion, which captures only the negative deviation of a random variable below its mean. Because semivariance is time-inconsistent and does not satisfy the standard Bellman equation, traditional dynamic programming methods are inapplicable. To tackle this, we utilize Perturbation Analysis (PA) theory to establish a performance difference formula for MSV. We demonstrate that the MSV problem can be solved by iteratively solving a sequence of RL problems with a policy-dependent surrogate reward function. We propose two on-policy algorithms—MSVAC and MSVPO—and demonstrate their effectiveness across environments ranging from bandit problems to continuous control tasks in MuJoCo.
1. Introduction
In many practical Reinforcement Learning (RL) applications, agents must balance the pursuit of high rewards with the necessity of avoiding catastrophic outcomes. While Mean-Variance (MV) optimization is a standard approach, it is often criticized for being "risk-blind" to the direction of volatility. Variance penalizes any deviation from the mean, effectively treating unexpectedly high rewards as negatively as unexpectedly low ones.
This paper advocates for the use of Mean-Semivariance (MSV). By only penalizing the "downside" risk, MSV allows agents to remain aggressive in pursuing high-payoff opportunities while remaining conservative regarding losses.
2. Theoretical Framework: Overcoming Time-Inconsistency
The primary challenge in optimizing for semivariance is its time-inconsistency. Unlike standard expected returns, semivariance does not decompose via the Bellman equation, meaning a policy that is optimal at the start of a trajectory may not be optimal at a later state.
To bridge this gap, the authors employ Perturbation Analysis (PA). They derive a performance difference formula that characterizes how changes in a policy affect the MSV objective. A key insight from this derivation is that the MSV objective can be optimized by treating it as a standard RL problem using a specific surrogate reward function ($g$):
$$g = (1 + 2\beta\eta_-)r - \beta(r - \eta)^2_-$$
Where:
- $\eta$ is the mean reward.
- $\eta_-$ is the "semimean" (the expected value of rewards falling below the mean).
- $\beta$ is the risk-aversion parameter.
- $(r - \eta)^2_-$ represents the squared downside deviation.
3. Algorithms: MSVAC and MSVPO
Building on the surrogate reward framework, the paper introduces two algorithms designed for stable, risk-averse learning:
- Mean-Semivariance Actor-Critic (MSVAC): An extension of standard policy gradient methods that incorporates the MSV performance difference formula to update the policy.
- Mean-Semivariance Policy Optimization (MSVPO): A trust-region method inspired by PPO. It provides a "monotonic improvement guarantee," ensuring that each update step consistently improves the MSV objective without causing performance collapses, which is critical in high-stakes, risk-sensitive environments.
4. Experimental Evaluation
The authors tested their framework across three distinct scenarios to evaluate its performance against risk-neutral and Mean-Variance (MV) baselines.
4.1 Tabular Portfolio Management
In a financial simulation involving two assets and cash, the MSV agent significantly outperformed the MV agent. The MSV-optimized policy achieved a mean return ($\eta$) of 0.168, compared to the MV policy's 0.073. While the MSV agent accepted higher total variance, its downside semivariance remained low. This confirms that the MSV agent was able to capitalize on "good" volatility that the MV agent avoided.
4.2 Continuous Control (MuJoCo Walker2d)
To test the framework in high-dimensional spaces, the authors modified the MuJoCo Walker2d task by adding Gaussian noise to actions and a heavy penalty for falling.
- Robustness: The risk-averse agent ($\beta=0.1$) actually achieved a higher average reward than the risk-neutral agent. By learning to avoid "risky" poses that might lead to a fall (the ultimate downside risk), the agent became more robust to environmental noise.
- Distributional Shift: Increasing the risk-aversion parameter $eta$ successfully "thinned" the left tail of the reward distribution, demonstrating that the agent was actively minimizing the probability of catastrophic failure.
5. Conclusion and Future Work
This paper provides a theoretically grounded method for incorporating downside risk into deep RL. By using Perturbation Analysis to derive surrogate rewards, the authors enable the use of powerful on-policy algorithms like MSVPO for risk-averse tasks.
Future research directions include extending this framework to Mean-Semideviation (MSD), which is a mathematically "coherent" risk measure, and exploring the use of upside semivariance maximization as a tool for exploration in sparse-reward environments. Currently, the framework assumes the environment reaches a steady state (ergodicity), and future work may address non-stationary or finite-horizon settings.
Summary updated