Dimension-Wise Importance Sampling Weight Clipping for Sample-Efficient Reinforcement Learning

Seungyul Han, Youngchul Sung

This paper introduces DISC (Dimension-wise Importance Sampling weight Clipping), a refinement of the popular Proximal Policy Optimization (PPO) algorithm designed to handle the "curse of dimensionality" in continuous control tasks. By proposing a simple but powerful change to how importance sampling weights are clipped, the authors prevent gradients from vanishing in high-dimensional tasks, enabling significantly better sample efficiency and the safe reuse of off-policy data.

The Problem: The Curse of Dimensionality in PPO

In Reinforcement Learning (RL), we often use Importance Sampling (IS) to estimate how a new policy would perform based on data collected by an older policy. To keep training stable, algorithms like PPO "clip" these IS weights—essentially telling the model, "Don't change too much at once."

However, in high-dimensional action spaces (e.g., a robot with 17 joints), the IS weight is the product of the weights for every single dimension. If even one joint's action is slightly "too different" from the old policy, the entire product can trigger the clipping mechanism.

When this happens, the gradient becomes zero, and the model learns nothing from that sample. Standard PPO is too blunt for high dimensions; by treating the action vector as a single unit, it throws away perfectly good data because of minor deviations in a single dimension. In complex tasks like Humanoid, PPO's gradients vanish for over 60% of samples, leading to massive sample inefficiency.

Amount of gradient vanishing across different environments.

The Solution: DISC

The authors introduce four key technical innovations to create the DISC algorithm:

  1. Dimension-Wise Clipping: Instead of clipping the total likelihood ratio, DISC factorizes the policy (assuming independent Gaussian distributions for each action dimension) and clips the IS weight for each dimension separately. This ensures that even if one dimension is clipped, the model can still learn from the others, keeping the gradient "alive."
  2. IS Weight Control Loss ($J_{IS}$): To prevent the policy from drifting too far from the data-generating policy, they add a logarithmic loss term that adaptively penalizes large deviations in IS weights.
  3. GAE-V (GAE with V-trace): To better reuse old data (off-policy learning), they combine the Generalized Advantage Estimator (GAE) with V-trace. V-trace helps correct the "lag" in value estimation when using trajectories generated by older versions of the policy.
  4. Modified Batch Inclusion: They propose a new criterion for deciding when an old batch of data is "safe" to reuse, based on the average dimension-wise IS weight rather than the global ratio.

Key Results

The algorithm was evaluated on standard continuous control benchmarks, including MuJoCo (Ant, HalfCheetah, Hopper, Humanoid, Walker2d) and Box2D (BipedalWalker). These environments range from simple 3-dimensional tasks to highly complex 17-dimensional tasks.

The results show that DISC is particularly dominant in high-dimensional environments:

  • Humanoid-v1: DISC achieved a maximum average return of ~6705, dwarfing PPO’s ~821. It also significantly outperformed state-of-the-art off-policy algorithms like SAC (~6883, but with much higher variance and a slower start) and TD3 (~63).
  • HumanoidStandup-v1: In this 17-dimensional task, DISC reached a score of ~246,435, while PPO plateaued at ~154,048.
  • Sample Efficiency: On-policy algorithms (like PPO) are usually safer but less efficient than off-policy ones. DISC bridges this gap by using dimension-wise metrics to safely incorporate older data that PPO would normally reject as "too stale." Because DISC can reuse old batches more effectively, it reaches higher performance levels with fewer environment interactions compared to PPO-AMBER and standard PPO.
Number of sample batches reused over time.

Limitations and Future Work

While DISC offers substantial improvements, it comes with a few limitations:

  • Factorization Assumption: The method assumes the policy can be factorized into independent dimensions (e.g., a diagonal Gaussian). While standard in many RL implementations, it might not apply to architectures with complex dependencies between action dimensions.
  • Hyperparameter Sensitivity: While the authors suggest a "best" setup that works across most tasks, the clipping factor $\epsilon$ and the IS target $J_{targ}$ still require some tuning for optimal performance.

For future work, the authors suggest a deeper theoretical analysis of the bias/variance trade-offs introduced by dimension-wise clipping and a formal proof of convergence for the DISC objective function.

Summary updated

Abstract

In importance sampling (IS)-based reinforcement learning algorithms such as Proximal Policy Optimization (PPO), IS weights are typically clipped to avoid large variance in learning. However, policy...
BibTeX
@inproceedings{han2019dimension,
  author = {Seungyul Han and Youngchul Sung},
  title = {{Dimension-Wise Importance Sampling Weight Clipping for Sample-Efficient Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=SJ-Dl3Zu-H},
}
Figure