Many deep reinforcement learning agents now learn the whole distribution of future returns instead of only its average. The most stable way to do this is the categorical critic: it places the return distribution on a fixed grid of bins and trains with cross-entropy. Cross-entropy gradients are differences of probabilities, so they stay bounded however surprising a reward is. This bounded-gradient property is the mechanism that recent theory credits for the stability and faster convergence of distributional RL. It is also the reason a risk-sensitive agent would want this critic, since conditional value-at-risk (CVaR, the average return over the worst α fraction of outcomes) can only be read off a distribution.
The grid has edges, though. Any return worse than the lowest bin is projected onto that bin, so a categorical critic cannot represent a catastrophe worse than its designer anticipated. For risk-neutral control this clipping rarely matters. For CVaR at small α under heavy-tailed hazards it removes the very quantity being optimized, and the agent believes its worst cases are milder than they are. We propose to keep the grid for the body of the distribution and add one extra "overflow" atom. The cross-entropy loss learns the atom's probability. Its location is the mean of returns beyond the grid, and we estimate it with adaptive Huber regression, a robust mean estimator whose clipping threshold grows with the number of tail samples. Gradients therefore stay bounded at every step, while the bias in the tail estimate vanishes as tail data accumulates.
Core Claim
The stabilizers that make distributional critics work all damp extreme targets so that no single sample can dominate an update. These stabilizers are fixed-support projection, the bounded cross-entropy gradient, and Huber-smoothed losses. In risk-sensitive RL those extreme targets are the signal, so the stabilizers bias CVaR estimates optimistically, and the bias grows with how heavy the tail is. We claim that stability needs bounded gradients only per step, while tail accuracy needs consistency only in the limit. A loss whose clipping threshold scales with the effective number of tail samples can satisfy both requirements, which no fixed-threshold or fixed-support design does.
The closest prior work is the generalized quantile Huber loss of A Robust Quantile Huber Loss with Interpretable Parameter Adjustment in Distributional Reinforcement Learning. It adapts the Huber threshold of quantile critics from a Gaussian model of quantile noise and evaluates only risk-neutral objectives. We differ in four ways. We target the categorical support truncation rather than the quantile kink. We set the threshold from tail sample counts and heavy-tail moment theory rather than a Gaussian noise model. We optimize CVaR explicitly. We measure tail bias against Monte Carlo ground truth rather than only measuring return.
Why This Gap Is Real
Three prior results motivate the work, and they do so in different ways. Ke Sun and colleagues prove that categorical critics trained with KL divergence are Lipschitz and smooth, and that this smoothness accelerates convergence from $O(1/\tau^4)$ to $O(1/\tau^2)$ (Interpreting Distributional Reinforcement Learning: A Regularization Perspective; Interpreting Categorical Distributional Reinforcement Learning: An Implicit Risk-Sensitive Regularization Effect). Their theorem carries an environment approximation term that grows when the fixed support cannot represent the environment's stochasticity, and they leave estimating that term open. Heavy-tailed hazards are exactly the case where it grows, so our overflow atom is a direct attempt to shrink it.
XQC (XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning) shows that cross-entropy plus normalization gives a well-conditioned critic. Compact XQC critics match larger architectures with "~4.5x fewer parameters" than SIMBA-V2, but only for risk-neutral returns. We reuse XQC as the critic backbone, which makes it a source of method rather than a competitor.
DSAC (DSAC: Distributional Soft Actor-Critic for Risk-Sensitive Reinforcement Learning) reports that hard risk measures such as CVaR at 0.1 make agents over-conservative and stop exploring, and the authors fall back to soft distortions. This finding constrains our design. Any method that makes tail estimates more pessimistic risks worsening collapse, so exploration behavior must be measured, not assumed.
The robust-statistics side supplies the estimator. Adaptive Huber regression (Adaptive Huber Regression) shows a sharp result about the clipping threshold. If the threshold grows with sample size at the right rate, the estimator has sub-Gaussian deviations under only finite variance. Under infinite variance with a finite $(1+\delta)$-th moment, it attains minimax rates. Under Student-t noise it reaches mean error 0.806 against 1.355 for least squares. On Empirical Risk Minimization with Dependent and Heavy-Tailed Data extends Huber's advantage to dependent, mixing data, which is closer to replay buffers. Its proofs assume convex losses and fixed targets. We do not inherit those guarantees for bootstrapped neural critics, so we test them empirically. The threshold schedule we borrow is a principled choice, not a proven one in our setting.
Research Program
The critic is an XQC-style categorical network over $K$ bins on $[V_{\min}, V_{\max}]$. It has one extra output: an overflow probability $p_0(s,a)$ attached to a learned location $m(s,a) \le V_{\min}$.
Bootstrapped targets are formed as $r + \gamma Z'$. Target mass landing inside the grid is projected as in C51. Mass landing below $V_{\min}$ goes to the overflow bin for the cross-entropy loss, and its realized values become regression targets $y$ for $m$. Those values include $r + \gamma\, m(s',a')$ when the next state's own overflow atom is involved.
The overflow atom location is trained with a Huber loss whose threshold is
$$\tau_t = c\,\hat\sigma_t \sqrt{n_t / \log(1/\eta)},$$
where $n_t$ is a running count of overflow targets seen so far, $\hat\sigma_t$ is a running robust scale estimate of those targets, $\eta$ is a confidence level, and $c$ is a constant. The square-root scaling is the finite-variance rate from adaptive Huber regression. Because each gradient on $m$ is at most $\tau_t$ in magnitude, the per-step Lipschitz bound of the categorical analysis survives, with a constant that grows slowly. The clipping bias meanwhile shrinks as $n_t$ grows.
CVaR at level $\alpha$ is computed from the combined distribution. If $\alpha \le p_0$, then $\text{CVaR}_\alpha = m$. Otherwise,
$$\text{CVaR}_\alpha = \frac{1}{\alpha}\Big(p_0\, m + \sum_{k} w_k\, z_k\Big),$$
where $z_k$ are bin locations in increasing order and $w_k$ are the bin probabilities truncated so that the total mass sums to $\alpha$.
The actor is SAC-style and maximizes this CVaR. For static CVaR of total return we use the threshold-augmented state of Distributional Reinforcement Learning for Risk-Sensitive Policies. For safety-constrained settings we use a CVaR cost constraint with a Lagrange multiplier, as in WCSAC: Worst-Case Soft Actor Critic for Safety-Constrained Reinforcement Learning.
The analysis has three parts. First, we measure tail bias directly: the critic's CVaR estimate at evaluation states minus a Monte Carlo CVaR from many rollouts of the current policy. We measure it as a function of tail heaviness. Second, we track gradient norms and the critic Hessian condition number with the XQC protocol, to check that adding an unbounded-location atom does not undo the conditioning. This also gives a first look at the open question of whether risk-distorted targets re-introduce ill-conditioning. Third, we sweep the tail index into the infinite-variance regime. There adaptive Huber theory prescribes a different, slower threshold rate, and we test whether that rate matters in practice.
Baselines
The naive reference point is a standard categorical critic with fixed support and the same CVaR actor. This is XQC's critic with CVaR action selection and the projection clipping left as is. It isolates what truncation costs. Every method shares the network size, replay ratio, environment steps, and seeds, and every method gets the same hyperparameter search budget. For the naive baseline that budget is spent on $V_{\min}$ and the bin count.
The strongest existing competitor is RDSAC from DSAC: Distributional Soft Actor-Critic for Risk-Sensitive Reinforcement Learning. It uses a quantile critic with unbounded support, a quantile Huber loss with δ = 1, and an explicit CVaR objective. It never truncates, so it is the method most likely to already get heavy tails right. Its weak points are that CVaR at small α rests on one or two extreme quantiles, and that the magnitude of the tail beyond the last quantile fraction is not represented. The only reported numbers for it on Lacuna are risk-neutral: DSAC reaches an average return of 8673 on Humanoid-v2 against 6547 for SAC, and 318 on BipedalWalkerHardcore-v3. No reported result was found on Lacuna for RDSAC's CVaR on any benchmark, so we rerun it ourselves with the same quantile count sweep and tuning budget.
We also run three further comparators. WCSAC is the Gaussian-cost representative; no reported result was found on Lacuna for its Safety Gym cost rates, which the paper describes only qualitatively. The generalized quantile Huber critic from A Robust Quantile Huber Loss with Interpretable Parameter Adjustment in Distributional Reinforcement Learning is the adaptive-robust-loss representative. CVaR-VaR from Boosting CVaR Policy Optimization with Quantile Gradients is run on its own tail-modified tasks, where no reported number was found on Lacuna either.
Two ablations isolate the key ingredient. The first trains the overflow atom with plain squared error, which is unbiased but has unbounded gradients. The second uses a fixed-threshold Huber, which is bounded but stays biased.
Beating the strongest baseline means two things on heavy-tailed variants. The CVaR estimation error must be lower than RDSAC's with non-overlapping confidence intervals across seeds. The realized policy CVaR must be equal or better, without losing expected return on the light-tailed originals.
Why not just…?
Why not just widen the support or add more bins? This is the most serious threat for moderate tails. A wide enough grid removes truncation entirely. The cost is resolution: with fixed $K$, widening the support makes bins coarser in the body where most learning happens, and heavy tails have no natural edge to aim for. We run a sweep over $V_{\min}$ and $K$, including a larger-$K$ variant with matched compute. If some fixed grid matches the overflow atom across all tail indices, the contribution reduces to a tuning recommendation, and we will report it that way.
Why not just use a quantile critic, which has no support edge? RDSAC is that option, and it is the main baseline. Quantile critics avoid clipping but estimate CVaR from very few quantile locations at small α. They also give up the bounded cross-entropy gradient that the stability theory relies on, since the convergence results are tied to categorical parameterizations. The tail-index sweep shows whether their CVaR error grows with tail heaviness the way we predict.
Why not just tune a fixed Huber threshold, or skip robustness on the atom altogether? A fixed threshold either clips real catastrophes, which causes bias, or barely clips, in which case it behaves like squared error and produces large gradients when an extreme target arrives. The two ablations above test exactly this. If squared error on the overflow atom shows no gradient spikes or instability under Pareto hazards, the robust-statistics part of the proposal is unnecessary, and that result is worth knowing.
Why not just assume a Gaussian tail, as WCSAC does? A Gaussian has closed-form CVaR and is cheap. It cannot represent skewed or multimodal costs, and the WCSAC authors acknowledge this limitation. Including WCSAC on the same skewed hazard variants measures how much this costs in practice rather than assuming it.
Proposed Experiments
The main testbeds are environments where tail heaviness can be dialed. We use the Safety Gym StaticEnv and DynamicEnv navigation tasks from WCSAC, with hazard costs redrawn from Pareto distributions of tail index 3, 2, and 1.5. We use the tail-modified Maze, LunarLander, and InvertedPendulum from Luo and Delage, with their noise replaced by the same Pareto family. We use DSAC's Risky Mass Point and Risky Ant. For a non-synthetic heavy-tailed domain we use the SABR option-hedging task with transaction costs from Malekzadeh et al. As a sanity check we use the sequential decision tree of Predictive CVaR Q-learning, where the optimal CVaR is known to be 2.50 and PCVaR-Q reports 2.45 against 1.96 for a risk-neutral agent. Our method should reach at least 2.45 there.
Light-tailed MuJoCo locomotion tasks (Humanoid, Walker2d, BipedalWalkerHardcore) check that the overflow atom costs nothing when tails are mild.
Metrics are reported over at least ten seeds. They are the critic's CVaR estimation error against Monte Carlo ground truth, the realized policy CVaR at α of 0.05 and 0.1, expected return, constraint violation rate where applicable, critic gradient norms, and critic Hessian condition number.
The decisive check is a single plot: CVaR estimation error against tail index. We predict that the fixed-support categorical critic's error rises steeply as tails get heavier. RDSAC's error should rise more slowly but stay above ours at α = 0.05. The overflow atom with adaptive Huber should stay roughly flat while keeping gradient norms and condition number close to the plain categorical critic. A second decisive check is behavioral: on the over-conservatism setting DSAC reported, the more accurate tail must not cause exploration collapse, which we track through state coverage and entropy.
If the widest tuned fixed grid matches us across the tail-index plot, or if squared-error tail atoms are just as stable, we will report that the support edge or the robust estimator, respectively, was not the bottleneck.
References
- A Robust Quantile Huber Loss with Interpretable Parameter Adjustment in Distributional Reinforcement Learning
- Interpreting Distributional Reinforcement Learning: A Regularization Perspective
- Interpreting Categorical Distributional Reinforcement Learning: An Implicit Risk-Sensitive Regularization Effect
- XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning
- DSAC: Distributional Soft Actor-Critic for Risk-Sensitive Reinforcement Learning
- Adaptive Huber Regression
- On Empirical Risk Minimization with Dependent and Heavy-Tailed Data
- Distributional Reinforcement Learning for Risk-Sensitive Policies
- WCSAC: Worst-Case Soft Actor Critic for Safety-Constrained Reinforcement Learning
- Boosting CVaR Policy Optimization with Quantile Gradients
- Predictive CVaR Q-learning