Truncated Gaussian Policy for Debiased Continuous Control

Ganghun Lee, Minji Kim, Minsu Lee, Byoung-Tak ZhangAAAI 2025 · DOI

TL;DR

This paper tackles the "boundary action bias" in reinforcement learning (RL) for continuous control tasks, where standard Gaussian policies tend to favor extreme actions due to clipping. They propose a "scale-adjusted truncated Gaussian policy" that dynamically shrinks the action distribution's scale near boundaries, effectively balancing the bias and leading to improved performance across various control tasks.

What Problem Does This Paper Address?

Reinforcement learning (RL) has shown great promise in solving complex problems, particularly in domains with continuous action spaces (e.g., robotics, locomotion). A common approach for continuous control is to use Gaussian distributions to model the policy, offering desirable properties like differentiability and intuitive parameters. However, in many real-world scenarios, actions are constrained within a specific range. When a Gaussian policy samples actions outside these bounds, they are typically "clipped" to the nearest boundary. This clipping mechanism, while necessary, introduces a "boundary action bias," where the policy unnaturally favors sampling actions at the extremes of the allowed range. This bias can negatively impact learning, especially in algorithms like Proximal Policy Optimization (PPO), by distorting the perceived action probabilities and hindering the discovery of optimal, nuanced actions. Existing methods to address this often involve entirely different distributions (like Beta or logit-normal) or gradient corrections, which can add complexity or alter the policy's fundamental properties. This paper aims to mitigate this bias with minimal modifications to the standard Gaussian policy.

Methods

The authors introduce a novel policy called the scale-adjusted truncated Gaussian policy.

  1. Truncated Gaussian Policy: As a first step, they propose using a truncated Gaussian distribution. Unlike a standard Gaussian, this distribution inherently has bounded support, meaning it naturally samples actions within a predefined range. This eliminates the need for explicit clipping, thereby removing the direct cause of the boundary action bias. However, they observe that a plain truncated Gaussian policy can sometimes lead to a counter-bias, overly favoring interior actions.
  2. Scale-Adjusted Truncated Gaussian Policy: To address the counter-bias and achieve a better balance, they introduce a scale-adjusted truncated Gaussian policy. This approach dynamically adjusts the scale (standard deviation) of the truncated Gaussian distribution based on the action's proximity to the boundaries. Specifically, the scale is discounted (shrunk) as the action mean approaches the boundaries. This makes boundary actions more deterministic than in a plain truncated Gaussian but less so than in a standard clipped Gaussian. The scale adjustment is governed by a function d(μ; l, u) that depends on the action mean μ and the lower (l) and upper (u) bounds. This function is designed to be 1 at the center of the action range and 0 at the boundaries, with a tunable parameter k controlling the kurtosis of the discount and dmin setting a minimum scale to avoid zero.

Datasets

The paper evaluates their methods on several standard continuous control benchmarks:

  • MuJoCo: This includes six locomotion tasks (HalfCheetah, Walker2d, Ant, Hopper, Humanoid, Swimmer) and two manipulation tasks (Pusher, Reacher), all using the v4 versions.
  • HumanoidBench: Two locomotion tasks (hlhand-walk, hlhand-reach) with a higher-dimensional action space (61 dimensions).
  • DeepMind Control Suite (DMC): Two locomotion tasks (dog-walk, dog-run) with a higher-dimensional action space (38 dimensions).

The action ranges for all tasks were rescaled to [-1, 1]. No new benchmarks were introduced.

Key Insights and Ideas

  • Boundary Action Bias is Real and Detrimental: The paper provides clear explanations and formalization of the boundary action bias in standard PPO, demonstrating how action clipping can lead to a feedback loop that reinforces extreme action sampling, potentially hindering learning.
  • Truncated Gaussian Offers a Simple Fix, But Has Its Own Issues: While a truncated Gaussian policy directly addresses the clipping problem, it can introduce a "counter-bias," making the policy too hesitant to explore boundary actions, which can be crucial in some tasks.
  • Dynamic Scale Adjustment is Key to Balance: The core insight is that simply bounding actions isn't enough; the confidence in those actions needs to be managed. Shrinking the action distribution's scale near boundaries allows the policy to be more decisive about boundary actions when necessary, without overshooting into the problematic bias.
  • Task-Dependent Nature of Boundary Actions: The ablation studies reveal that the optimal balance between boundary and interior actions can vary significantly across tasks. Some tasks (like Swimmer) benefit from boundary actions, while others (like Ant) are harmed by them. This highlights the need for adaptive methods.

Key Results

  • Reduced Boundary Action Rate (aBAR): The proposed truncated Gaussian and scale-adjusted truncated Gaussian policies significantly reduced the average boundary action rate (aBAR) compared to standard Gaussian and CAPG methods, often bringing it close to 0%.
  • Improved Performance: The scale-adjusted truncated Gaussian policy consistently achieved the highest overall normalization scores across the tested MuJoCo tasks, outperforming Gaussian, CAPG, Beta, and Logit policies.
  • Competitive Performance in High Dimensions: The SA-TGaussian policy also demonstrated strong performance in high-dimensional action spaces (HumanoidBench and DMC), outperforming other methods and showing robustness.
  • Balancing Act: While plain truncated Gaussian reduced aBAR, its performance was often suboptimal. The scale-adjusted version successfully balanced the bias and counter-bias, leading to superior results. For instance, in HalfCheetah and Walker2d, where Gaussian had high aBAR and performance degradation, SA-TGaussian excelled.

Limitations and Future Work

  • Task-Specific Hyperparameters: The optimal values for the scale-adjustment hyperparameters (dmin and k) can vary across tasks, suggesting that automatic tuning or more sophisticated adaptive mechanisms might be beneficial.
  • Internal Mechanisms: The paper acknowledges that a deeper understanding of why suppressing boundary actions can lead to better performance in some cases, and how this interacts with task characteristics, is an area for future research.
  • Generalizability: While the proposed method shows promise, further exploration of its effectiveness across a wider range of RL algorithms and task types is warranted.
  • Adaptation vs. Distributional Choice: The authors ponder whether the observed performance similarities between some methods with different aBARs suggest an inherent adaptability of RL agents to various distributions, rather than being solely constrained by them. This opens avenues for investigating how RL can leverage diverse solution pathways.

Summary updated

BibTeX
@inproceedings{lee2025truncated,
  author = {Ganghun Lee and Minji Kim and Minsu Lee and Byoung-Tak Zhang},
  title = {{Truncated Gaussian Policy for Debiased Continuous Control}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i17.33988},
}
Figure