Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation

Yuanpei Chen, Chen Wang, Li Fei-Fei, C. Karen Liu

Many real-world manipulation tasks consist of a series of subtasks that are significantly different from one another. Such long-horizon, complex tasks highlight the potential of dexterous hands, which possess adaptability and versatility, capable of seamlessly transitioning between different modes of functionality without the need for re-grasping or external tools. However, the challenges arise due to the high-dimensional action space of dexterous hands and the complex compositional dynamics of long-horizon tasks. We present Sequential Dexterity, a general system based on reinforcement learning (RL) that chains multiple dexterous policies for achieving long-horizon task goals.

The Challenge of Long-Horizon Dexterity

Multi-fingered dexterous hands are incredibly versatile, but controlling them for long tasks—such as building a structure from a pile of blocks—is notoriously difficult. Most current research focuses on "atomic" skills: just grasping or just in-hand re-orientation.

The "chaining" problem arises because a robot might successfully complete one sub-task (e.g., grasping a block) but leave the object in a state that makes the next sub-task (e.g., inserting it into a slot) impossible. Naively running policies one after another often fails due to these "out-of-distribution" transitions. To solve this, the robot needs a global understanding of how early actions influence late-stage goals.

Sequential stages of manipulation (Search, Orient, Grasp, Insert) in both simulation and real-world environments.

Methodology: Bi-directional Optimization

The core of the Sequential Dexterity system is a bi-directional optimization framework that uses Reinforcement Learning (PPO) to train sub-policies ($\pi^1, \pi^2, ... \pi^K$).

  1. Forward Initialization: The robot trains sub-policies in chronological order. The successful end states of policy $i$ are used as the starting distribution for policy $i+1$. This ensures that the later policies are trained on realistic data they will actually encounter.
  2. Backward Fine-tuning: This is the key innovation. The authors introduce a Transition Feasibility Function (TFF). This function learns to predict the expected future reward of a policy based on its starting state. The TFF of a subsequent task is used as an auxiliary reward to fine-tune the preceding task. For example, the "Insertion" TFF tells the "Grasping" policy to pick up the block in a specific orientation that facilitates easier insertion later.
  3. Autonomous Policy Switching: During deployment, the TFFs act as "stage estimators." If the robot fails a step, the TFFs can trigger a fallback to a previous policy. Conversely, if a task is already partially done, the robot can bypass redundant stages.
The bi-directional optimization framework illustrating the flow between forward initialization and backward fine-tuning using the Transition Feasibility Function.

Experimental Setup and Results

Training was conducted in Isaac Gym using "Mega Bloks" models. The system was evaluated on its ability to generalize to unseen block shapes and zero-shot transfer to a real-world Franka Emika Panda arm equipped with a 16-degree-of-freedom Allegro Hand.

Key Results

  • Superior Success Rates: In the "Building Blocks" task, Sequential Dexterity achieved a 46% overall success rate, significantly outperforming prior state-of-the-art chaining methods like T-STAR (19%) and V-Chain (8%).
  • Real-World Robustness: In real-world tests, the system successfully built a "Double Block" stack (an 8-step long-horizon task) with a 33% success rate, while all baseline methods failed completely (0%).
  • Efficiency through Switching: The ability to switch policies autonomously improved success rates by over 30% compared to a fixed sequential execution, as it allowed the robot to recover from slips or unstable grasps.

Conclusion and Future Work

Sequential Dexterity demonstrates that complex, long-horizon manipulation can be achieved by intelligently chaining specialized policies. By backpropagating the "feasibility" of future tasks into the rewards of current tasks, the system achieves a form of global optimization without the computational burden of training one massive, end-to-end policy.

Future work will look into better integration of high-resolution tactile sensors and extending this bi-directional chaining to bimanual (two-handed) coordination, further expanding the capabilities of dexterous robotic systems.

Summary updated

Abstract

Many real-world manipulation tasks consist of a series of subtasks that are significantly different from one another. Such long-horizon, complex tasks highlight the potential of dexterous hands, which possess adaptability and versatility, capable of seamlessly transitioning between different modes of functionality without the need for re-grasping or external tools. However, the challenges arise due to the high-dimensional action space of dexterous hand and complex compositional dynamics of the long-horizon tasks. We present Sequential Dexterity, a general system based on reinforcement learning (RL) that chains multiple dexterous policies for achieving long-horizon task goals. The core of the system is a transition feasibility function that progressively finetunes the sub-policies for enhancing chaining success rate, while also enables autonomous policy-switching for recovery from failures and bypassing redundant stages. Despite being trained only in simulation with a few task objects, our system demonstrates generalization capability to novel object shapes and is able to zero-shot transfer to a real-world robot equipped with a dexterous hand. Code and videos are available at https://sequential-dexterity.github.io
BibTeX
@inproceedings{chen2023sequential,
  author = {Yuanpei Chen and Chen Wang and Li Fei-Fei and C. Karen Liu},
  title = {{Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation}},
  booktitle = {Conference on Robot Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=2Qrd-Yw4YmF},
}
Figure