Risk-averse reinforcement learning often optimizes Conditional Value-at-Risk (CVaR), the average return of the worst $\alpha$ fraction of episodes. The objective has a kink: an episode counts fully if its return falls below a moving threshold and not at all otherwise. That hard cutoff causes the two best-documented sample-efficiency failures in the area. The standard policy gradient discards the $1-\alpha$ of episodes above the cutoff, which is 95% of the data at $\alpha=0.05$. When the worst episodes all end the same way, for instance by crashing into the same hazard, their returns are identical and the expected gradient is exactly zero, so learning stops. We propose to replace the hard cutoff with a smooth one whose width shrinks on a schedule during training. The idea is borrowed from variable-smoothing methods in non-smooth optimization. Every episode then gets a weight that falls off smoothly with its return. Early in training, near-tail episodes supply gradient signal. Late in training, the objective converges to the true CVaR.
In two sentences: CVaR policy gradients waste data and stall because of a single non-differentiable hinge, and the non-smooth optimization literature has a principled answer for hinges, which is smoothing with a decreasing width. We test whether that answer beats the three RL-specific fixes already published, which are variance reduction, mixture policies, and quantile curricula. The test covers sample efficiency, the flat-tail stall, and final tail performance.
Core Claim
Write CVaR in the Rockafellar–Uryasev form, $\text{CVaR}_\alpha(\pi)=\max_\eta \{\eta-\tfrac{1}{\alpha}\mathbb{E}_\pi[(\eta-R)_+]\}$. Here $R$ is episode return, $\eta$ is the threshold that tracks the $\alpha$-quantile, and $(\cdot)_+$ is the hinge. We replace the hinge with its softplus smoothing of width $\mu$, $\mu\log(1+e^{(\eta-R)/\mu})$. The smoothed objective overestimates the true CVaR by at most $\mu\log 2/\alpha$, because softplus exceeds the hinge by at most $\mu\log 2$ pointwise. Its score-function gradient weights each episode by
$$w_\mu(R)=\sigma\!\left(\frac{\eta-R}{\mu}\right),$$
where $\sigma$ is the logistic function. This weight replaces the indicator $\mathbb{I}\{R\le\eta\}$. It is a one-line change to CVaR policy gradient.
The weights have two consequences that no existing fix provides together. First, the fraction of episodes with non-negligible weight grows from $\alpha$ to roughly $\alpha$ plus $\mu$ times the return density near $\eta$. Second, when the tail is flat, the weights still differ across the episodes just above the tail. The expected gradient therefore points toward whatever made those episodes slightly better, and the stall is broken. Because the bias is $O(\mu)$, a decreasing schedule $\mu_k$ gives up only transient accuracy. We use epoch-wise decreasing widths as in Variable Smoothing for Weakly Convex Composite Functions, which trades bias against the $1/\mu$ growth in curvature to obtain $O(\epsilon^{-3})$ rather than $O(\epsilon^{-4})$ complexity.
Relation to Prior Work
Three papers establish the gap. A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimization shows that CVaR-PG gradients vanish on flat tails in a synthetic maze and fail on hazardous HalfCheetah and Ant. Its fix mixes in a risk-neutral policy trained with Implicit Q-Learning, which changes the policy class and adds a gating network. We keep the policy class and change only the objective's kink. Risk-Sensitive Policy Optimization via Predictive CVaR Policy Gradient replaces the indicator with a learned predicted tail probability, a Rao-Blackwellization that reduces gradient variance on Continuous Blackjack. It remains an unbiased estimator of the exact CVaR gradient. On a truly flat tail that gradient is zero, so this method lowers variance around a zero mean and cannot break the stall. Our smoothing deliberately biases the mean, and the two are composable. Predictive CVaR Q-learning makes CVaR learning dense in time through Bellman-style tail functions and randomized risk budgets, but it is tabular and value-based. We target deep policy-gradient continuous control.
Two papers supply the method. The variable-smoothing schedule comes from Variable Smoothing for Weakly Convex Composite Functions. Risk-Sensitive Submodular Optimization already used smoothed gradient and threshold estimators to handle CVaR kinks, but for static submodular maximization, with a fixed smoothing and no sample-complexity question. Adaptive-Gradient Policy Optimization also treats non-smoothness as the cause of RL inefficiency, but its kinks are in contact physics rather than in the risk objective.
One constraint shapes the design. Near-Optimal Sample Complexity for Iterated CVaR Reinforcement Learning with a Generative Model proves a minimax lower bound scaling as $1/\tau^2$ in the risk tolerance. Smoothing cannot beat that rate for exact CVaR. Our claim is therefore about constants and about escaping stalls in the transient regime, not about asymptotic rates.
Research Program
We will build Annealed-Smooth CVaR-PG (AS-CVaR). It is PPO-style CVaR policy gradient with softplus-smoothed weights, $\eta$ updated as the smoothed quantile, and $\mu_k$ decreasing across epochs.
We will compare three schedules. The first sets width in return units. The second ties $\mu_k$ to a target effective sample fraction, shrinking $\mu$ as the quantile estimate stabilizes. The third decays the width polynomially, following the variable-smoothing analysis.
The analysis has two parts. In tabular bandits and small MDPs, we will bound the bias and variance of the smoothed estimator and characterize when the flat-tail stall is escaped. The condition should be that the return density just above $\eta$ is non-zero and correlated with the policy score. We will also track effective sample size, gradient variance, and gradient norm on flat-tail instances throughout training.
Baselines
The naive reference is standard CVaR-PG with the hard indicator, run in the same PPO backbone. It is the object of repair, and any method must first beat it on steps-to-threshold. We also report risk-neutral PPO as the "ignore risk" point.
The strongest variance-reduction competitor is PCVaR (Risk-Sensitive Policy Optimization via Predictive CVaR Policy Gradient). It reports gradient-estimator variance of 6.71 against 24.01 for naive CVaR on Continuous Blackjack. We will reproduce it on Blackjack first, and beating it there means lower variance at a matched bias level.
The strongest competitor on the flat-tail failure is MIX (A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimization). No reported result was found on Lacuna for its maze or MuJoCo returns, so we will rerun it from the paper's description. Beating MIX means reaching its final CVaR in fewer environment steps with no extra networks, or reaching higher final CVaR.
All methods share the environment-step budget, network sizes, seed count, and an equal hyperparameter-search budget. The search covers $\mu$ schedules for our method, the predictor for PCVaR, and the gate and IQL settings for MIX.
The ablation that isolates the key ingredient is fixed-$\mu$ smoothing at several widths. If annealing does not beat the best fixed width, the schedule is not the contribution. We also run AS-CVaR combined with PCVaR's predictor to test whether the two ideas are complementary.
Why not just…?
Why not just use PCVaR's predicted tail probability? It is the most principled fix for data waste. However, it estimates the exact CVaR gradient, which on flat tails has zero mean. Variance reduction around zero leaves the policy stalled. The maze experiment settles this directly: if PCVaR escapes the stall as fast as AS-CVaR, our flat-tail argument is wrong.
Why not just anneal $\alpha$ from 1 down to the target? This quantile curriculum is a real threat, because it also uses more data early. The difference is the shape of the weights. Annealing $\alpha$ weights everything below a looser quantile uniformly and still has a hard edge, so flat tails at the new quantile can still stall. Smoothing keeps the target quantile and grades the weights. We include $\alpha$-annealing with an equal tuning budget. If it matches AS-CVaR everywhere, we will report the smoothing view as an explanation of why curricula work rather than as a better method.
Why not just use MIX? MIX works on the reported MuJoCo domains. It adds an IQL-trained policy and a state-dependent gate, and it assumes risk matters only locally. We test whether an objective-level change recovers its gains without those assumptions. Hazards that cover much of the state space, as in Safety-Gymnasium velocity tasks, are where MIX's locality assumption should hurt.
Proposed Experiments
Continuous Blackjack from the PCVaR paper tests variance and bias at matched budget. The flat-tail synthetic maze and the hazardous HalfCheetah and Ant from the MIX paper test the stall and sample efficiency. For broader coverage we add the Safety-Gymnasium navigation and velocity tasks (Safety Gymnasium), where PPO-Lag is reported to cut safety violations by 98% at a 45% reward cost. We also use the Sequential Decision Tree from Predictive CVaR Q-learning as a sanity check on converged tail value. There PCVaR-Q reports CVaR 2.45 against an optimum of 2.50, and a converged method should come within that gap.
The metrics are true (unsmoothed) CVaR at the target $\alpha$ as a function of environment steps, steps to a fixed CVaR threshold, gradient-estimator variance, and the fraction of transitions with weight above a small cutoff.
Three results would be decisive. On the maze, AS-CVaR should show non-zero gradient norm and rising CVaR where hard CVaR-PG and PCVaR flatline. On MuJoCo, it should reach MIX's final CVaR in fewer steps. Final true CVaR under annealing should match or exceed the best fixed width, showing that shrinking $\mu$ removes the smoothing bias without giving back the early speedup.
References
- Variable Smoothing for Weakly Convex Composite Functions
- A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimization
- Risk-Sensitive Policy Optimization via Predictive CVaR Policy Gradient
- Predictive CVaR Q-learning
- Risk-Sensitive Submodular Optimization
- Adaptive-Gradient Policy Optimization: Enhancing Policy Learning in Non-Smooth Differentiable Simulations
- Near-Optimal Sample Complexity for Iterated CVaR Reinforcement Learning with a Generative Model
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark