TL;DR
Reparameterization Policy Gradient (RPG) methods exploit differentiable physics simulators to yield sample-efficient updates, but they often suffer from training instability and cannot safely reuse samples across multiple epochs. Reparameterization Proximal Policy Optimization (RPO) bridges the gap between RPG and PPO-style surrogate objectives, combining a single backpropagation through time (BPTT) pass with action-regeneration, importance-weight clipping, and KL regularization to achieve stable, multi-epoch sample reuse.
What Problem Does This Paper Address?
Reinforcement learning (RL) in robotics frequently faces a dilemma: model-free policy gradients (like REINFORCE or standard PPO) have high variance and require massive sample counts, whereas model-based Reparameterization Policy Gradients (RPG) leverage differentiable simulators to backpropagate through trajectory dynamics directly.
While RPG provides lower-variance gradient estimates, it is prone to exploding or vanishing gradients—particularly in contact-rich environments like locomotion and manipulation. Existing variance-reduction methods (such as SHAC and SAPO) truncate rollout horizons and add entropy regularization, but they still lack an explicit mechanism to bound policy update sizes. As a result, policies experience sudden performance collapses triggered by massive spikes in policy divergence. Furthermore, because previous RPG methods cannot safely perform off-policy updates on collected rollouts, they update the policy only once per trajectory batch, underutilizing costly BPTT computations.
Methods
RPO introduces a mathematically grounded way to optimize a PPO-like surrogate objective using reparameterization gradients while enabling multi-epoch sample reuse:
- Surrogate Objective via BPTT: RPO runs a single BPTT pass on a rollout batch to compute and cache action-gradients ($\nabla_a R(\tau)$). For the initial update (on-policy), these gradients are backpropagated directly through the policy network.
- Action Regeneration for Off-Policy Updates: For subsequent epochs ($m > 1$), the behavior policy no longer matches the current policy. RPO inverts the policy transformation to find the noise vector $\epsilon_{\text{reg}} = f_\theta^{-1}(a; s)$ that would reproduce the buffered action under the current policy, re-establishing a differentiable path from parameters to actions without re-running simulation rollouts.
- RPG-Tailored Gradient Clipping: Unlike standard PPO, which clips based on advantage signs, RPO clips based purely on the importance sampling ratio $\rho(\theta) = \frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}$. Updates are zeroed out if $\rho(\theta)$ falls outside $[1 - c_{\text{low}}, 1 + c_{\text{high}}]$, filtering out numerical instability caused by extreme action-ratio mismatches.
- KL Divergence Regularization: Because clipping does not constrain the initial on-policy step and overly strict clipping reduces sample reuse, RPO adds an explicit Kullback–Leibler (KL) penalty ($\mathcal{L}_{\text{KL}}$) to penalize policy drift.
[ Single BPTT Pass over Trajectory ]
│
(Cache Action Gradients)
│
┌───────────────────┴───────────────────┐
▼ ▼
[ Epoch 1: On-Policy ] [ Epochs 2-M: Off-Policy ]
Backpropagate cached gradients Regenerate noise: ε_reg = f_θ⁻¹(a;s)
directly into policy network Weight by ratio: ρ(θ) = π_θ / π_old
Clip gradients outside [1-c_low, 1+c_high]
Penalize drift via KL regularization
Datasets
The authors evaluate RPO across five continuous-control benchmarks from two differentiable physics simulators:
- DFlex Tasks (Locomotion):
- Hopper ($S \in \mathbb{R}^{11}, A \in \mathbb{R}^3$)
- Ant ($S \in \mathbb{R}^{37}, A \in \mathbb{R}^8$)
- Anymal ($S \in \mathbb{R}^{49}, A \in \mathbb{R}^{12}$)
- Humanoid ($S \in \mathbb{R}^{76}, A \in \mathbb{R}^{21}$)
- Rewarped Task (Manipulation):
- Hand Reorient ($S \in \mathbb{R}^{72}, A \in \mathbb{R}^{16}$), where an Allegro Hand reorients a cube to match target orientations.
No new datasets were introduced; tasks follow official implementations from DiffRL and Rewarped.
Key Insights and Ideas
- BPTT action gradients are surrogate-objective gradients: The authors prove that the cached gradient of cumulative return with respect to action $\nabla_a R(\tau)$ is an unbiased Monte Carlo estimate of the advantage gradient $\nabla_a A^{\pi_{\theta_{\text{old}}}}(s, a)$.
- One BPTT pass is enough for multi-epoch reuse: Prior RPG methods threw away trajectories after a single update. By combining action regeneration with importance sampling, RPO extracts up to 5 updates from the same trajectory without running the simulator again.
- Clipping alone is insufficient for RPG: In model-free PPO, clipping surrogate objectives restricts update steps naturally. In RPG, clipping alone either allows destructive early policy updates (since the ratio starts at 1) or, if made too strict, throws away over 70% of off-policy samples. Pairing asymmetric clipping with explicit KL regularization resolves both issues.
Key Results
RPO consistently outperforms SOTA baselines (SAPO, SHAC, GI-PPO, PPO, and SAC) in both sample efficiency and asymptotic return:
- Final Deterministic Performance (Mean Return):
- Humanoid: RPO achieves $8958.41 \pm 373.64$ (vs. SAPO's $8676.25$, SHAC's $7722.20$, GI-PPO's $7576.97$, and PPO's $2140.61$).
- Anymal: RPO reaches $16006.42 \pm 342.49$ (vs. SAPO's $14783.27$ and SHAC's $14560.59$).
- Hand Reorient: RPO reaches $230.15 \pm 33.12$ (vs. SAPO's $225.17$, SHAC's $175.13$, and PPO's $20.88$).
- Hopper & Ant: RPO achieves top scores ($5534.32$ and $9249.50$).
- Sample Efficiency Gains: On Humanoid, RPO reaches an 8000-return threshold roughly 3 million steps faster than SAPO. In Hopper, it reaches a score of 5000 approximately 5 million steps sooner than all baselines.
- Stability: Unlike SAPO and SHAC, which exhibit intermittent catastrophic drops in returns caused by KL spikes, RPO maintains smooth, monotonic learning curves.
Limitations and Future Work
- Deterministic Dynamics Assumption: The mathematical derivation assumes deterministic system dynamics ($s_{t+1} = g(s_t, a_t)$), leaving extensions to highly stochastic transitions unaddressed.
- Sim-to-Real Transfer: All evaluations were conducted purely within differentiable simulation engines (DFlex and Rewarped); the paper explicitly leaves physical hardware deployment and sim-to-real transfer to future work.
Summary updated