An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient

Yudong Luo, Guiliang Liu, Pascal Poupart, Yangchen Pan

Abstract

Restricting the variance of a policy's return is a popular choice in risk-averse Reinforcement Learning (RL) due to its clear mathematical definition and easy interpretability. Traditional methods directly restrict the total return variance. Recent methods restrict the per-step reward variance as a proxy. We thoroughly examine the limitations of these variance-based methods, such as sensitivity to numerical scale and hindering of policy learning, and propose to use an alternative risk measure, Gini deviation, as a substitute. We study various properties of this new risk measure and derive a policy gradient algorithm to minimize it. Empirical evaluation in domains where risk-aversion can be clearly defined, shows that our algorithm can mitigate the limitations of variance-based risk measures and achieves high return with low risk in terms of variance and Gini deviation when others fail to learn a reasonable policy.

Introduction

In standard Reinforcement Learning (RL), the primary objective is to maximize the expected cumulative return. However, in high-stakes domains such as autonomous driving, healthcare, or financial trading, the expected return is an insufficient metric. In these contexts, agents must also account for risk—the potential for catastrophic outcomes or high volatility.

Traditionally, the RL community has relied on "Mean-Variance" frameworks to balance performance and safety. While variance is mathematically intuitive, it introduces significant practical challenges during optimization. This paper identifies two critical flaws in variance-based risk measures: scale sensitivity and learning hindrance. To address these, we introduce Gini Deviation (GD) as a robust alternative and derive a novel policy gradient algorithm for its optimization.

The Limitations of Variance-Based Risk Measures

1. Scale Sensitivity

Variance is defined by squared differences from the mean ($R^2$). Consequently, the variance penalty is highly sensitive to the numerical scale of the rewards. If the reward signal is doubled, the variance penalty quadruples. This quadratic growth makes hyperparameter tuning (specifically the risk-aversion coefficient $\lambda$) extremely difficult across different environments or even different reward scales within the same environment.

2. Learning Hindrance (The "Good-to-Bad" Trap)

Many modern risk-averse methods attempt to penalize per-step reward volatility as a proxy for total return variance. However, this can inadvertently turn a positive reward into a negative one. For example, if an agent receives a large positive reward for reaching a goal, but that reward is associated with high variance, the risk penalty might outweigh the reward itself. This "reward destruction" prevents the agent from exploring the goal state, effectively halting the learning process before a viable policy can be formed.

Gini Deviation: A Coherent Alternative

Gini Deviation (GD), or the Gini mean difference, measures the average absolute difference between any two independent realizations of a random variable. Unlike variance, GD is a "coherent measure of variability," satisfying a set of mathematical axioms (such as sub-additivity) that ensure stable behavior.

Key Advantages:

  • Positive Homogeneity: GD scales linearly with rewards. If rewards are doubled, the GD penalty only doubles. This property makes the risk-return trade-off much more stable and easier to tune.
  • Rank-Based Optimization: The gradient of GD is inherently linked to the rank of trajectories. It places significant weight on the differences between trajectories in the middle of the distribution, providing a more balanced learning signal than variance, which is often dominated by extreme outliers.

Methodology: Mean-GD Policy Gradient

Optimizing GD directly is non-trivial because it involves absolute differences between realizations. To solve this, we utilize the signed Choquet integral to represent GD in terms of quantiles. This representation allows us to derive a clean policy gradient formula:

  1. Quantile Representation: We express the GD objective using the cumulative distribution function (CDF) of the returns.
  2. Empirical Estimator: We develop a sampling-based estimator that uses a batch of $n$ trajectories. By sorting these trajectories by their total returns, we can approximate the area under the CDF required for the GD gradient.
  3. Integration: The resulting "Mean-GD" objective is compatible with standard architectures like PPO or REINFORCE. We employ Importance Sampling (IS) to improve sample efficiency by reusing trajectory data.

Experimental Results

We evaluated the Mean-GD algorithm against standard variance-based baselines (such as MVO and Tamar) across several risk-sensitive benchmarks.

Guarded Maze

In a tabular maze environment, agents must choose between a "safe" path (low, certain reward) and a "risky" path (high, volatile reward). When the goal reward was increased from 20 to 40, all variance-based baselines failed to maintain a risk-averse policy due to scale sensitivity. In contrast, Mean-GD remained stable, consistently choosing the safe path regardless of the reward scale.

LunarLander

We modified the LunarLander environment so that landing on one side of the pad was highly noisy. Mean-GD was the only method that successfully learned to land on the stable side of the pad consistently. Variance-based methods either failed to land at all or exhibited unstable convergence patterns.

Continuous Control (MuJoCo)

In high-dimensional tasks like HalfCheetah and Swimmer, we introduced "risky regions" where agents received high-variance rewards. Mean-GD achieved a superior balance, maintaining high total returns while significantly reducing the time spent in noisy zones compared to variance-based competitors.

Conclusion and Future Work

Gini Deviation offers a mathematically sound and practically robust alternative to variance for risk-averse RL. By providing linear scaling and avoiding the pitfalls of reward destruction, Mean-GD allows agents to learn safe policies in complex environments where traditional methods fail.

Future research will focus on improving the sample efficiency of the gradient estimator and extending the framework to fully off-policy, incremental updates using distributional RL techniques. Additionally, exploring hybrid measures that combine GD with tail-risk measures like CVaR could provide even more comprehensive risk management.

Summary updated

Abstract

Restricting the variance of a policy's return is a popular choice in risk-averse Reinforcement Learning (RL) due to its clear mathematical definition and easy interpretability. Traditional methods directly restrict the total return variance. Recent methods restrict the per-step reward variance as a proxy. We thoroughly examine the limitations of these variance-based methods, such as sensitivity to numerical scale and hindering of policy learning, and propose to use an alternative risk measure, Gini deviation, as a substitute. We study various properties of this new risk measure and derive a policy gradient algorithm to minimize it. Empirical evaluation in domains where risk-aversion can be clearly defined, shows that our algorithm can mitigate the limitations of variance-based risk measures and achieves high return with low risk in terms of variance and Gini deviation when others fail to learn a reasonable policy.
BibTeX
@inproceedings{luo2023alternative,
  author = {Yudong Luo and Guiliang Liu and Pascal Poupart and Yangchen Pan},
  title = {{An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=B7QRV4XXiK},
}
Figure