TL;DR
Histogram-based value estimation methods like HL-Gauss improve reinforcement learning stability by reframing regression as a classification task, but they traditionally require a hand-picked, fixed support interval. This paper introduces DySEL (Dynamic Support Endpoint Learning), an approach that dynamically optimizes support boundaries during training via a constrained min-max formulation. DySEL eliminates the need for manual interval tuning while matching or exceeding the performance of fixed-support baselines on complex continuous control tasks.
What Problem Does This Paper Address?
In actor-critic deep reinforcement learning (RL), critics estimate value functions to guide the agent's policy. While traditionally trained using mean-squared error (MSE) regression, recent work shows that framing value estimation as a classification problem over a histogram (such as the Gaussian Histogram Loss, or HL-Gauss) yields smoother gradients and better scalability.
However, histogram-based methods suffer from a major catch: they require a pre-defined, fixed support interval ($[\nu_{min}, \nu_{max}]$). This introduces a difficult trade-off:
- Too narrow: The target distribution gets truncated, discarding critical tail information (truncation bias).
- Too wide: Because the number of histogram bins is fixed, a wide interval dilutes the resolution of each bin, weakening the learning signal (quantization bias).
Because RL value targets are highly non-stationary—shifting and expanding as the agent's policy improves—no single fixed interval is optimal throughout the entire training process.
Methods
The authors propose DySEL (Dynamic Support Endpoint Learning) to automatically learn the optimal lower and upper bounds of the support interval during training.
- Theoretical Upper Bound: The authors mathematically demonstrate that the mean-squared Bellman error is upper-bounded by a combination of the support width, a distribution-matching term (cross-entropy), and truncation bias.
- Constrained Optimization: To resolve the adversarial tension between making the support narrow (for high resolution) and wide (to avoid truncation), they formulate support learning as a constrained optimization problem. The goal is to minimize the support width while ensuring it encapsulates the target distribution's probability mass.
- Adversarial Min-Max Game: They introduce a Lagrangian multiplier ($\lambda$) to enforce a probability mass coverage constraint based on the "leaked mass" ($1 - Z$). This sets up a min-max game where the network automatically expands the supports to capture leaked target mass, while a width penalty prevents the interval from expanding to infinity.
Datasets
The method was evaluated on 11 continuous control tasks from the DeepMind Control Suite (including cheetah-run, finger-turn_hard, fish-swim, hopper-hop, humanoid variants, and quadruped variants).
Algorithms were evaluated across 10 random seeds over 3 million training timesteps, utilizing the Inter-Quantile Mean (IQM) metric to ensure robustness against outlier seeds.
Key Insights and Ideas
- The Danger of Degenerate Widening: Without a width penalty, a classification loss can "cheat" by expanding the support interval to infinity. Doing so eventually places all targets into a single bin, trivially yielding a zero cross-entropy loss. The absolute width penalty is essential to prevent this degenerate behavior.
- Dynamic Scheduling Beats Fixed Intervals: The optimal support interval changes drastically depending on the policy. For some tasks, the support needs to expand gradually over time, meaning that even a perfectly chosen static interval will perform suboptimally at some point during training.
- Tighter Bounds: Dynamically shrinking the support interval directly tightens the theoretical upper bound on the mean-squared Bellman error, leading to more accurate value estimations.
Key Results
The authors integrated DySEL into the Twin Delayed DDPG (TD3) algorithm (TD3+DySEL) and compared it against standard TD3 and TD3 with a fixed-support HL-Gauss baseline (TD3+HLG).
- Competitive and Superior Performance:
TD3+DySELmatched the performance of the hand-tunedTD3+HLGbaseline on most tasks and achieved substantial performance gains on highly complex tasks likehumanoid-runandhumanoid-stand. - Ablation Verification: Ablation studies proved that removing either the mass constraint or the width penalty causes the support intervals to diverge or collapse, confirming that both components of the min-max game are critical for stability.
Limitations and Future Work
- Optimization Sensitivity: The stability of the min-max saddle-point optimization is sensitive to the dual learning rate of the Lagrangian multiplier ($\lambda$). Aggressive updates can cause early training oscillations. The authors suggest exploring PID-controlled Lagrangian updates to damp these oscillations.
- Symmetric Support Constraint: The current implementation results in support intervals that are nearly symmetric around zero. Because real-world return distributions are often highly asymmetric, decoupling the upper and lower bounds to allow asymmetric shifting is an open area for future work.
- Fixed Smoothing: The Gaussian smoothing parameter ($\sigma$) is currently tied to a fixed ratio of the bin width. Dynamically adapting $\sigma$ alongside the support bounds could allow even finer control over distribution representation.
Summary updated