Learning the Supports for Categorical Critic in Reinforcement Learning

Jen-Yen Chang, Takayuki Osa, Tatsuya HaradaarXiv 2026 · arXiv

TL;DR

Histogram-based value estimation methods like HL-Gauss improve reinforcement learning stability by reframing regression as a classification task, but they traditionally require a hand-picked, fixed support interval. This paper introduces DySEL (Dynamic Support Endpoint Learning), an approach that dynamically optimizes support boundaries during training via a constrained min-max formulation. DySEL eliminates the need for manual interval tuning while matching or exceeding the performance of fixed-support baselines on complex continuous control tasks.

What Problem Does This Paper Address?

In actor-critic deep reinforcement learning (RL), critics estimate value functions to guide the agent's policy. While traditionally trained using mean-squared error (MSE) regression, recent work shows that framing value estimation as a classification problem over a histogram (such as the Gaussian Histogram Loss, or HL-Gauss) yields smoother gradients and better scalability.

However, histogram-based methods suffer from a major catch: they require a pre-defined, fixed support interval ($[\nu_{min}, \nu_{max}]$). This introduces a difficult trade-off:

  • Too narrow: The target distribution gets truncated, discarding critical tail information (truncation bias).
  • Too wide: Because the number of histogram bins is fixed, a wide interval dilutes the resolution of each bin, weakening the learning signal (quantization bias).

Because RL value targets are highly non-stationary—shifting and expanding as the agent's policy improves—no single fixed interval is optimal throughout the entire training process.

Methods

The authors propose DySEL (Dynamic Support Endpoint Learning) to automatically learn the optimal lower and upper bounds of the support interval during training.

  1. Theoretical Upper Bound: The authors mathematically demonstrate that the mean-squared Bellman error is upper-bounded by a combination of the support width, a distribution-matching term (cross-entropy), and truncation bias.
  2. Constrained Optimization: To resolve the adversarial tension between making the support narrow (for high resolution) and wide (to avoid truncation), they formulate support learning as a constrained optimization problem. The goal is to minimize the support width while ensuring it encapsulates the target distribution's probability mass.
  3. Adversarial Min-Max Game: They introduce a Lagrangian multiplier ($\lambda$) to enforce a probability mass coverage constraint based on the "leaked mass" ($1 - Z$). This sets up a min-max game where the network automatically expands the supports to capture leaked target mass, while a width penalty prevents the interval from expanding to infinity.
Figure 1: An example of a truncated Gaussian distribution q(y) and its truncation bias. The X- axis is the support interval, limited in (0,1). The Y-axis is the probability density of the truncated Gaussian. Blue denotes bins with probability as- signed, whilst grey bins have zero probability. In this work we aim to utilise (1 −Z) as one of the substitutes to learn dynamic support interval.

Datasets

The method was evaluated on 11 continuous control tasks from the DeepMind Control Suite (including cheetah-run, finger-turn_hard, fish-swim, hopper-hop, humanoid variants, and quadruped variants).

Algorithms were evaluated across 10 random seeds over 3 million training timesteps, utilizing the Inter-Quantile Mean (IQM) metric to ensure robustness against outlier seeds.

Key Insights and Ideas

  • The Danger of Degenerate Widening: Without a width penalty, a classification loss can "cheat" by expanding the support interval to infinity. Doing so eventually places all targets into a single bin, trivially yielding a zero cross-entropy loss. The absolute width penalty is essential to prevent this degenerate behavior.
  • Dynamic Scheduling Beats Fixed Intervals: The optimal support interval changes drastically depending on the policy. For some tasks, the support needs to expand gradually over time, meaning that even a perfectly chosen static interval will perform suboptimally at some point during training.
  • Tighter Bounds: Dynamically shrinking the support interval directly tightens the theoretical upper bound on the mean-squared Bellman error, leading to more accurate value estimations.

Key Results

The authors integrated DySEL into the Twin Delayed DDPG (TD3) algorithm (TD3+DySEL) and compared it against standard TD3 and TD3 with a fixed-support HL-Gauss baseline (TD3+HLG).

  • Competitive and Superior Performance: TD3+DySEL matched the performance of the hand-tuned TD3+HLG baseline on most tasks and achieved substantial performance gains on highly complex tasks like humanoid-run and humanoid-stand.
  • Ablation Verification: Ablation studies proved that removing either the mass constraint or the width penalty causes the support intervals to diverge or collapse, confirming that both components of the min-max game are critical for stability.
IQM returns for DM Control tasks comparing DySEL against baselines.

Limitations and Future Work

  • Optimization Sensitivity: The stability of the min-max saddle-point optimization is sensitive to the dual learning rate of the Lagrangian multiplier ($\lambda$). Aggressive updates can cause early training oscillations. The authors suggest exploring PID-controlled Lagrangian updates to damp these oscillations.
  • Symmetric Support Constraint: The current implementation results in support intervals that are nearly symmetric around zero. Because real-world return distributions are often highly asymmetric, decoupling the upper and lower bounds to allow asymmetric shifting is an open area for future work.
  • Fixed Smoothing: The Gaussian smoothing parameter ($\sigma$) is currently tied to a fixed ratio of the bin width. Dynamically adapting $\sigma$ alongside the support bounds could allow even finer control over distribution representation.

Summary updated

Abstract

Value functions are an essential component in actor-critic based deep reinforcement learning (RL). Conventionally, these functions are trained as a regression task by minimising the mean squared error (MSE) relative to bootstrapped target values. Meanwhile, in distributional RL, a distribution of returns is modelled based on the distributional Bellman operator. This work investigates the Gaussian Histogram Loss (HL-Gauss), a recent approach that reframes value estimation as classification by encoding each scalar Bellman target as a Gaussian-smoothed categorical target. Despite its potential, applying histogram-based losses to RL presents inherent challenges, most notably the requirement to pre-define a fixed support interval, which is often complicated by the non-stationary and stochastic nature of target values typically found in RL tasks. In this work, we propose an approach that dynamically learns the lower and upper bounds of the support instead of assigning them beforehand. We derive an objective that jointly learns these bounds whilst learning the categorical representation of the scalar values, and we show that this objective forms an upper bound on the mean-squared Bellman error. Our theoretical analysis further shows that this bound is tighter than that of non-learned supports of HL-Gauss. Empirically, the proposed objective enables stable adaptation of the support interval and matches HL-Gauss-based actor-critic algorithms on most continuous-control tasks whilst improving on a subset, without requiring a pre-specified support interval.
BibTeX
@misc{chang2026learning,
  author = {Jen-Yen Chang and Takayuki Osa and Tatsuya Harada},
  title = {{Learning the Supports for Categorical Critic in Reinforcement Learning}},
  year = {2026},
  eprint = {2607.01880},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.01880},
}
Figure