Abstract
Reparameterization policy gradient (RPG) is promising for improving sample efficiency by leveraging differentiable dynamics. However, a critical barrier is its training instability, where high-variance gradients can destabilize the learning process. To address this, we draw inspiration from Proximal Policy Optimization (PPO), which uses a surrogate objective to enable stable sample reuse in the model-free setting. We first establish a connection between this surrogate objective and RPG, which has been largely unexplored and is non-trivial. Then, we bridge this gap by demonstrating that the reparameterization gradient of a PPO-like surrogate objective can be computed efficiently using backpropagation through time. Based on this key insight, we propose Reparameterization Proximal Policy Optimization (RPO), a stable and sample-efficient RPG-based method. RPO enables stable sample reuse over multiple epochs by employing a policy gradient clipping mechanism tailored for RPG. It is further stabilized by Kullback-Leibler (KL) divergence regularization and remains fully compatible with existing variance reduction methods. We evaluate RPO on a suite of challenging locomotion and manipulation tasks, where experiments demonstrate that our method achieves superior sample efficiency and strong performance.
Introduction
In Reinforcement Learning (RL), Reparameterization Policy Gradient (RPG) is a powerful tool for sample efficiency. Unlike standard model-free methods that treat the environment as a "black box," RPG leverages differentiable simulators to calculate exactly how a change in action affects future rewards by backpropagating through the physics engine itself.
However, RPG faces two major challenges:
- Computational Cost: Calculating Jacobians (physics gradients) is expensive, and most current methods discard this data after a single update.
- Instability: RPG is prone to "exploding" or "vanishing" gradients, particularly in complex environments with long horizons or contact-rich dynamics.
The RPO Framework
To address these issues, we introduce Reparameterization Proximal Policy Optimization (RPO). RPO provides a principled way to reuse expensive gradients without compromising training stability.
The method is built upon three key components:
- Unified Surrogate Objective: We prove that reusing action-gradients is mathematically equivalent to optimizing a PPO-style surrogate objective via Backpropagation Through Time (BPTT). This enables a formal framework for both on-policy and off-policy updates.
- Action Regeneration: To reuse old data, the model "re-traces" the path from current policy parameters to old actions by recovering the original noise used to generate them. This creates a new differentiable path for gradient flow.
- Stability Mechanisms: RPO employs Asymmetric Clipping of importance weights and KL Regularization to ensure the new policy does not drift too far from the behavior policy that collected the data, preventing divergence during volatile training phases.
Experimental Evaluation
We evaluated RPO on five challenging continuous control tasks across two differentiable simulators: DFlex (locomotion tasks like Ant, Anymal, and Humanoid) and Rewarped (the Hand Reorient task).
Key Results
RPO consistently outperformed state-of-the-art methods like SHAC and SAPO in both performance and efficiency:
- Performance: In the Hand Reorient task, RPO achieved a score of ~237, significantly higher than SAPO (~213) and SHAC (~174).
- Wall-Clock Speedup: On the Anymal task, RPO reached peak performance in just 81 minutes, whereas SHAC required 313 minutes to reach a similar level.
- Sample Efficiency: RPO reached performance benchmarks millions of steps earlier than competitors. For example, it hit the "8000" return mark on the Humanoid task roughly 3 million steps faster than SAPO.
Conclusion
RPO bridges the gap between the high efficiency of differentiable simulators and the stability of proximal policy methods. By enabling stable sample reuse and providing robust regularization, RPO significantly reduces the wall-clock time required for training complex robotic policies. Future work will investigate the framework's performance in stochastic environments and its potential for direct sim-to-real transfer on physical hardware.
Summary updated