Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidance

Kelvin Xu, Zheyuan Hu, Ria Doshi, Aaron Rovinsky, Vikash Kumar, Abhishek Gupta, Sergey Levine

Introduction

Dexterous manipulation with multi-fingered hands represents a significant challenge in robotics. While Reinforcement Learning (RL) offers a path toward mastering these skills, its real-world application is often hindered by the need for extensive human intervention, including complex reward engineering, precise state estimation via motion capture, and manual resets after failures. This work introduces AVAIL (Autonomy ViA mILestones), a system designed to enable autonomous, programming-free learning of multi-stage manipulation tasks directly from raw visual input.

Methods: Autonomy via Milestones (AVAIL)

The AVAIL framework simplifies complex tasks into a sequence of manageable sub-skills guided by a "milestones graph." Instead of requiring full demonstrations, which are difficult to provide for high-degree-of-freedom (DoF) hands, the system relies on simple image snapshots provided by the user to define key task states.

The AVAIL system architecture, illustrating the workflow from user-provided visual milestones to autonomous practice loops.

The system incorporates several key components:

  • Learned Rewards (VICE): Success classifiers are trained on milestone images using the VICE algorithm. The robot receives rewards based on the probability that its current visual observation matches the target milestone.
  • Autonomous Task Selection: A learned task dynamics model ($p_{task}$) evaluates the current state and determines which sub-skill to practice. This allows the robot to autonomously recover from failures; for instance, if an object is dropped, the system reverts to a "grasp" policy.
  • Sample-Efficient RL (DroQ): To facilitate learning from pixels in the real world, the authors utilize DroQ, which combines Soft Actor-Critic (SAC) with data augmentation and dropout for robust performance.

Experimental Evaluation

The system was evaluated on three intricate real-world tasks requiring high dexterity:

  1. Kitchen Brush: Grasping, reorienting bristles, and scrubbing a plate.
  2. Hose Connector: Reaching, grasping, and inserting a connector onto a peg.
  3. Rope Hooking: Latching a carabiner attached to a rope onto a handle.
Sequence of the kitchen brush task, demonstrating the robot's ability to grasp, reorient, and perform functional scrubbing.

Key Results and Insights

AVAIL demonstrated significant success in enabling unattended learning:

  • High Success Rates: After 18–48 hours of autonomous practice, the robot achieved success rates of 80% or higher across all real-world sub-tasks.
  • Efficiency: The learned task graph converged 25% faster than hand-coded heuristic graphs in the hose insertion task, as the learned model better accounted for specific robot failure modes.
  • Robustness: In simulated benchmarks, AVAIL outperformed prior autonomous RL methods like R3L, which struggled with high-DoF tasks without granular milestone guidance.
  • Snapshots vs. Demonstrations: Providing "outcome snapshots" is significantly lower effort for users than kinesthetic demonstrations, which are often awkward for high-DoF hands and can occlude camera views.

Conclusion and Future Work

AVAIL proves that complex dexterous skills can be learned with minimal instrumentation and human effort. By treating failures as transitions within a milestone graph, the system maintains a continuous loop of practice. Future work will focus on removing current constraints, such as the use of tethers to keep objects within reach, and integrating safety critics to handle potentially irreversible states during autonomous exploration.

Summary updated

Abstract

Complex and contact-rich robotic manipulation tasks, particularly those that involve multi-fingered hands and underactuated object manipulation, present a significant challenge to any control method. Methods based on reinforcement learning offer an appealing choice for such settings, as they can enable robots to learn to delicately balance contact forces and dexterously reposition objects without strong modeling assumptions. However, running reinforcement learning on real-world dexterous manipulation systems often requires significant manual engineering. This negates the benefits of autonomous data collection and ease of use that reinforcement learning should in principle provide. In this paper, we describe a system for vision-based dexterous manipulation that provides a "programming-free" approach for users to define new tasks and enable robots with complex multi-fingered hands to learn to perform them through interaction. The core principle underlying our system is that, in a vision-based setting, users should be able to provide high-level intermediate supervision that circumvents challenges in teleoperation or kinesthetic teaching which allow a robot to not only learn a task efficiently but also to autonomously practice. Our system includes a framework for users to define a final task and intermediate sub-tasks with image examples, a reinforcement learning procedure that learns the task autonomously without interventions, and experimental results with a four-finger robotic hand learning multi-stage object manipulation tasks directly in the real world, without simulation, manual modeling, or reward engineering.
BibTeX
@inproceedings{xu2023dexterous,
  author = {Kelvin Xu and Zheyuan Hu and Ria Doshi and Aaron Rovinsky and Vikash Kumar and Abhishek Gupta and Sergey Levine},
  title = {{Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidance}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10161493},
}
Figure