Introduction
Standard Reinforcement Learning (RL) typically focuses on maximizing the expected return. While effective in controlled environments, this risk-neutral approach is often unsuitable for high-stakes applications like autonomous driving or financial management, where catastrophic outcomes must be avoided. Risk-averse Reinforcement Learning (RARL) addresses this by incorporating risk measures into the optimization objective.
While most RARL research focuses on tail-risk measures like Conditional Value at Risk (CVaR), this paper explores a distinct class: measures of variability. These metrics quantify the "spread" or dispersion of returns. Although Variance is the industry standard for measuring variability, it introduces significant instability in RL training due to its quadratic nature. This research systematically derives policy gradient formulas for nine different variability metrics, demonstrating that alternative measures like Gini Deviation and CVaR Deviation offer superior stability and performance.
The Problem with Variance
In policy gradient methods, the gradient of the Variance involves a quadratic term ($R^2$). This makes the training process extremely sensitive to reward scaling. A single outlier return can trigger a massive spike in the gradient, leading to high-variance policy updates that can "break" the agent's learning trajectory. This "quadratic trap" often results in unstable training or convergence to sub-optimal policies.
Methodological Framework
To address these limitations, the authors derived policy gradient expressions for a suite of variability metrics, including Gini Deviation, Mean Deviation, Mean-Median Deviation, and Semi-Standard Deviation.
The Signed Choquet Integral
The researchers utilized the Signed Choquet Integral framework to make these derivations tractable. This mathematical tool allows complex variability metrics to be represented through their quantiles and specific distortion functions. By leveraging this framework, the authors could provide a unified approach to calculating gradients for metrics that were previously unstudied in the context of RARL.
Gradient Estimation and Algorithm Integration
For metrics requiring quantile estimation (such as the Inter-Quantile Range or CVaR Deviation), the authors employed Kernel Density Estimation (KDE). This ensures that gradient updates point in the correct direction by accurately estimating the underlying probability density. These metrics were integrated into standard RL architectures, specifically REINFORCE (with a baseline) and Proximal Policy Optimization (PPO). Additionally, the authors addressed the "double sampling" problem—the requirement for two independent sets of trajectories for unbiased gradient estimation—by implementing practical sampling strategies for metrics like Gini Deviation.
Experimental Evaluation
The proposed methods were evaluated across three diverse environments designed to test risk sensitivity:
- Tabular Maze: A custom environment where the shortest path includes a "risky" state with high-variance noise (Gaussian, Pareto, and Uniform).
- Discrete Control (LunarLander): A modified version where landing on specific areas provides noisy, high-variance rewards, forcing the agent to prioritize safety.
- Continuous Control (Mujoco): InvertedPendulum and HalfCheetah tasks where noise was injected into specific state-space regions to simulate risky zones.
Key Insights
L1 vs. L2 Metrics
The research highlights that metrics based on absolute differences (L1), such as Gini Deviation and Mean Deviation, are significantly more stable than L2-based metrics like Variance. These L1 metrics act as "linear" versions of variance, reducing gradient volatility and leading to more robust policy updates.
The Performance of CVaR Deviation
CVaR Deviation, which measures the distance between the mean and the tail risk, emerged as a consistently strong performer. It successfully balanced the pursuit of high returns with effective risk reduction across all tested domains.
Variability as Regularization
In the HalfCheetah environment, an unexpected result occurred: penalizing variability actually led to higher mean returns compared to standard risk-neutral PPO. This suggests that penalizing dispersion can function as a form of regularization, preventing the policy from collapsing into high-variance, sub-optimal behaviors that hinder exploration and stability.
Conclusion and Future Work
This work expands the RARL toolkit by providing the theoretical and practical foundations for using diverse variability metrics. The findings suggest a shift away from standard Variance in favor of L1-based metrics like Gini Deviation for more stable and reliable risk-averse agents.
Future research directions include developing fully unbiased policy gradient estimators for quantile-based metrics and exploring methods for the automatic tuning of the risk-aversion trade-off parameter ($\lambda$) to further enhance the autonomy of risk-averse agents.
Summary updated