DexPolicy: Scheduled Exploration for Trajectory-Guided Dexterous Manipulation

Haoyu Wang, Siyuan Qian, Yanjun Li, Zeyu Zhang, Yandong Guo, Boxin Shi, Hao TangarXiv 2026 · arXiv

TL;DR

DexPolicy demonstrates that explicitly scheduling the reduction of action noise during reinforcement learning (RL) training is a simple yet powerful way to boost the performance of dexterous robotic hands. By annealing exploration noise from broad to narrow, the researchers achieved dramatic success rate improvements—up to 85% on real-world hardware—across multiple policy optimization algorithms.

What Problem Does This Paper Address?

Dexterous manipulation—using multi-fingered hands to move objects—presents a fundamental "exploration-exploitation" dilemma. To learn how to grasp an object, an RL agent needs high action noise to "stumble" upon the right finger-object contacts. However, once a grasp is established, that same noise becomes a liability, causing the hand to jitter and drop the object during relocation.

The authors observed that standard algorithms like Proximal Policy Optimization (PPO) often fail to reduce this noise naturally. Even after 5 million training steps, agents often remain stuck with high exploration noise, leading to mediocre final performance. DexPolicy addresses this by taking manual control of the noise scale, ensuring the agent transitions from a "discovery" phase to a "precision" phase.

Methods

The core contribution is DexPolicy, a method that makes the exploration scale (the standard deviation of Gaussian action noise) an explicit function of training steps.

Overview of DexPolicy, showing the noise scheduling process and its impact on different RL algorithms.

The researchers integrated this schedule into three different policy-optimization settings:

  1. PPO: The industry-standard on-policy RL algorithm.
  2. GRPO (Group Relative Policy Optimization): A critic-free algorithm that uses group-relative returns instead of a learned value baseline.
  3. FPO (Flow-Parameterized PPO): A variant that uses conditional flow matching to generate the action mean, providing more expressive policy representations.

The DexPolicy schedule typically starts at a high noise level (e.g., $\sigma = 0.20$) and anneals down to a low plateau (e.g., $\sigma = 0.04$) over several million steps. Crucially, they held all other hyperparameters—like learning rates and reward functions—fixed to isolate the effect of noise scheduling.

Datasets

The study utilized the YCB Object and Model Set, a standard benchmark for robotic manipulation.

  • Simulation: Policies were trained on eight objects, including a mustard bottle, mug, banana, sugar box, tomato can, pitcher base, wood block, and an extra-large clamp.
  • Hardware: Evaluation was conducted on a RealMan RM75 arm equipped with an Inspire/RH56 dexterous hand. The real-world trials focused on three specific geometries: the tomato can, sugar box, and mustard bottle.
Simulation environments showing the benchmark hand and the hardware-matched embodiment.

Key Insights and Ideas

  • Dissociation of Metrics: One of the most important findings is that training return, deterministic success (executing without noise), and noise tolerance do not always correlate. A policy might show high rewards during training but fail completely when the noise is removed for deployment. This suggests that "Target Success" under intended execution conditions is the only metric that truly matters.
  • Noise Control > Optimizer Tweaks: Through "component screening," the authors found that the performance gains came primarily from controlling the noise scale itself, rather than from shrinking the optimizer's learning rate or clipping range (optimizer contraction).
  • Schedule Transferability: A noise schedule tuned on a single object (the mustard bottle) transferred effectively to other objects without further tuning. This indicates that the rule for exploration is more generalizable than the resulting policies.
  • Learning Dynamics: As shown in Figure 4, scheduled noise (DexPolicy) consistently leads to higher stochastic evaluation returns compared to baselines across different architectures.
Learning histories comparing baseline PPO/FPO/GRPO against DexPolicy across different objects.

Key Results

The impact of DexPolicy was significant across both simulation and physical hardware:

  • Simulation (5-object mean):
    • FPO: Success rose from 49.4% to 68.1%.
  • GRPO: Success rose from 14.1% to 45.4%.
  • PPO: Success rose from 32.0% to 35.7%.
  • Real Robot Hardware:
    • FPO: Mean success jumped from 25.0% to 85.0%.
  • GRPO: Mean success jumped from 10.0% to 63.3%.
  • PPO: Mean success jumped from 8.3% to 43.3%.
Real-world sequences of the robot successfully grasping and lifting various YCB objects.

Limitations and Future Work

  • Object Specificity: The trained policies are object-specific. While the scheduling rule transfers, the resulting policy does not yet offer zero-shot generalization to new objects.
  • Unsolved Objects: Some complex objects (like the pitcher base and wood block) remained effectively unsolved by all methods, suggesting that noise scheduling is a piece of the puzzle, but not a complete solution for all geometries.
  • Sim-to-Real Gap: While hardware results were strong, the authors noted that their hardware simulator differed from the benchmark simulator, meaning the results don't provide a perfectly controlled measure of the sim-to-real gap.
  • Future Directions: The authors suggest that future work should investigate how to automatically adapt these schedules per task and explore the interaction between noise scheduling and visual-policy distillation.

Summary updated

Abstract

Reinforcement learning (RL) for dexterous manipulation must discover finger-object contacts and then control the object precisely; the action noise that serves the first goal can interfere with the second. In trajectory-guided settings such as ViViDex, where RL refine hand-object trajectories from human video, our baseline PPO runs end near their initial action noise after 5M steps, motivating explicit control of exploration scale. DexPolicy makes that scale an explicit function of training steps, annealing from broad to narrow exploration while holding loss, architecture, reward, and optimizer settings fixed. We study three policy-optimization settings: PPO, critic-free GRPO continuation, and a flow-parameterized PPO variant (FPO). Across five YCB objects and three training seeds, mean deterministic Target success rises from 49.4% to 68.1% (FPO), 14.1% to 45.4% (GRPO), and 32.0% to 35.7% (PPO). On a RealMan RM75 arm with an Inspire/RH56 hand, 360 trials over three objects raise mean Target success from 25.0% to 85.0% (FPO), 10.0% to 63.3% (GRPO), and 8.3% to 43.3% (PPO), with one trained model per object-method condition. PPO component screening favors noise control over the tested optimizer contraction; the selected PPO schedule yields higher mean Target success than linear decay with the same endpoints on three tested objects. Training return, deterministic Target success, and tolerance to execution noise dissociate; schedules should therefore be judged by terminal task success under the intended execution conditions, per task and policy-optimization setting. Code: https://github.com/AIGeeksGroup/DexPolicy. Website: https://aigeeksgroup.github.io/DexPolicy.
BibTeX
@misc{wang2026dexpolicy,
  author = {Haoyu Wang and Siyuan Qian and Yanjun Li and Zeyu Zhang and Yandong Guo and Boxin Shi and Hao Tang},
  title = {{DexPolicy: Scheduled Exploration for Trajectory-Guided Dexterous Manipulation}},
  year = {2026},
  eprint = {2610.00360},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.00360},
}
Figure