Many real-world manipulation tasks consist of a series of subtasks that are significantly different from one another. Such long-horizon, complex tasks highlight the potential of dexterous hands, which possess adaptability and versatility, capable of seamlessly transitioning between different modes of functionality without the need for re-grasping or external tools. However, the challenges arise due to the high-dimensional action space of dexterous hands and the complex compositional dynamics of long-horizon tasks. We present Sequential Dexterity, a general system based on reinforcement learning (RL) that chains multiple dexterous policies for achieving long-horizon task goals.
The Challenge of Long-Horizon Dexterity
Multi-fingered dexterous hands are incredibly versatile, but controlling them for long tasks—such as building a structure from a pile of blocks—is notoriously difficult. Most current research focuses on "atomic" skills: just grasping or just in-hand re-orientation.
The "chaining" problem arises because a robot might successfully complete one sub-task (e.g., grasping a block) but leave the object in a state that makes the next sub-task (e.g., inserting it into a slot) impossible. Naively running policies one after another often fails due to these "out-of-distribution" transitions. To solve this, the robot needs a global understanding of how early actions influence late-stage goals.
Methodology: Bi-directional Optimization
The core of the Sequential Dexterity system is a bi-directional optimization framework that uses Reinforcement Learning (PPO) to train sub-policies ($\pi^1, \pi^2, ... \pi^K$).
- Forward Initialization: The robot trains sub-policies in chronological order. The successful end states of policy $i$ are used as the starting distribution for policy $i+1$. This ensures that the later policies are trained on realistic data they will actually encounter.
- Backward Fine-tuning: This is the key innovation. The authors introduce a Transition Feasibility Function (TFF). This function learns to predict the expected future reward of a policy based on its starting state. The TFF of a subsequent task is used as an auxiliary reward to fine-tune the preceding task. For example, the "Insertion" TFF tells the "Grasping" policy to pick up the block in a specific orientation that facilitates easier insertion later.
- Autonomous Policy Switching: During deployment, the TFFs act as "stage estimators." If the robot fails a step, the TFFs can trigger a fallback to a previous policy. Conversely, if a task is already partially done, the robot can bypass redundant stages.
Experimental Setup and Results
Training was conducted in Isaac Gym using "Mega Bloks" models. The system was evaluated on its ability to generalize to unseen block shapes and zero-shot transfer to a real-world Franka Emika Panda arm equipped with a 16-degree-of-freedom Allegro Hand.
Key Results
- Superior Success Rates: In the "Building Blocks" task, Sequential Dexterity achieved a 46% overall success rate, significantly outperforming prior state-of-the-art chaining methods like T-STAR (19%) and V-Chain (8%).
- Real-World Robustness: In real-world tests, the system successfully built a "Double Block" stack (an 8-step long-horizon task) with a 33% success rate, while all baseline methods failed completely (0%).
- Efficiency through Switching: The ability to switch policies autonomously improved success rates by over 30% compared to a fixed sequential execution, as it allowed the robot to recover from slips or unstable grasps.
Conclusion and Future Work
Sequential Dexterity demonstrates that complex, long-horizon manipulation can be achieved by intelligently chaining specialized policies. By backpropagating the "feasibility" of future tasks into the rewards of current tasks, the system achieves a form of global optimization without the computational burden of training one massive, end-to-end policy.
Future work will look into better integration of high-resolution tactile sensors and extending this bi-directional chaining to bimanual (two-handed) coordination, further expanding the capabilities of dexterous robotic systems.
Summary updated