TL;DR
DexPolicy demonstrates that explicitly scheduling the reduction of action noise during reinforcement learning (RL) training is a simple yet powerful way to boost the performance of dexterous robotic hands. By annealing exploration noise from broad to narrow, the researchers achieved dramatic success rate improvements—up to 85% on real-world hardware—across multiple policy optimization algorithms.
What Problem Does This Paper Address?
Dexterous manipulation—using multi-fingered hands to move objects—presents a fundamental "exploration-exploitation" dilemma. To learn how to grasp an object, an RL agent needs high action noise to "stumble" upon the right finger-object contacts. However, once a grasp is established, that same noise becomes a liability, causing the hand to jitter and drop the object during relocation.
The authors observed that standard algorithms like Proximal Policy Optimization (PPO) often fail to reduce this noise naturally. Even after 5 million training steps, agents often remain stuck with high exploration noise, leading to mediocre final performance. DexPolicy addresses this by taking manual control of the noise scale, ensuring the agent transitions from a "discovery" phase to a "precision" phase.
Methods
The core contribution is DexPolicy, a method that makes the exploration scale (the standard deviation of Gaussian action noise) an explicit function of training steps.
The researchers integrated this schedule into three different policy-optimization settings:
- PPO: The industry-standard on-policy RL algorithm.
- GRPO (Group Relative Policy Optimization): A critic-free algorithm that uses group-relative returns instead of a learned value baseline.
- FPO (Flow-Parameterized PPO): A variant that uses conditional flow matching to generate the action mean, providing more expressive policy representations.
The DexPolicy schedule typically starts at a high noise level (e.g., $\sigma = 0.20$) and anneals down to a low plateau (e.g., $\sigma = 0.04$) over several million steps. Crucially, they held all other hyperparameters—like learning rates and reward functions—fixed to isolate the effect of noise scheduling.
Datasets
The study utilized the YCB Object and Model Set, a standard benchmark for robotic manipulation.
- Simulation: Policies were trained on eight objects, including a mustard bottle, mug, banana, sugar box, tomato can, pitcher base, wood block, and an extra-large clamp.
- Hardware: Evaluation was conducted on a RealMan RM75 arm equipped with an Inspire/RH56 dexterous hand. The real-world trials focused on three specific geometries: the tomato can, sugar box, and mustard bottle.
Key Insights and Ideas
- Dissociation of Metrics: One of the most important findings is that training return, deterministic success (executing without noise), and noise tolerance do not always correlate. A policy might show high rewards during training but fail completely when the noise is removed for deployment. This suggests that "Target Success" under intended execution conditions is the only metric that truly matters.
- Noise Control > Optimizer Tweaks: Through "component screening," the authors found that the performance gains came primarily from controlling the noise scale itself, rather than from shrinking the optimizer's learning rate or clipping range (optimizer contraction).
- Schedule Transferability: A noise schedule tuned on a single object (the mustard bottle) transferred effectively to other objects without further tuning. This indicates that the rule for exploration is more generalizable than the resulting policies.
- Learning Dynamics: As shown in Figure 4, scheduled noise (DexPolicy) consistently leads to higher stochastic evaluation returns compared to baselines across different architectures.
Key Results
The impact of DexPolicy was significant across both simulation and physical hardware:
- Simulation (5-object mean):
- FPO: Success rose from 49.4% to 68.1%.
- GRPO: Success rose from 14.1% to 45.4%.
- PPO: Success rose from 32.0% to 35.7%.
- Real Robot Hardware:
- FPO: Mean success jumped from 25.0% to 85.0%.
- GRPO: Mean success jumped from 10.0% to 63.3%.
- PPO: Mean success jumped from 8.3% to 43.3%.
Limitations and Future Work
- Object Specificity: The trained policies are object-specific. While the scheduling rule transfers, the resulting policy does not yet offer zero-shot generalization to new objects.
- Unsolved Objects: Some complex objects (like the pitcher base and wood block) remained effectively unsolved by all methods, suggesting that noise scheduling is a piece of the puzzle, but not a complete solution for all geometries.
- Sim-to-Real Gap: While hardware results were strong, the authors noted that their hardware simulator differed from the benchmark simulator, meaning the results don't provide a perfectly controlled measure of the sim-to-real gap.
- Future Directions: The authors suggest that future work should investigate how to automatically adapt these schedules per task and explore the interaction between noise scheduling and visual-policy distillation.
Summary updated