Introduction
Dexterous manipulation with multi-fingered hands represents a significant challenge in robotics. While Reinforcement Learning (RL) offers a path toward mastering these skills, its real-world application is often hindered by the need for extensive human intervention, including complex reward engineering, precise state estimation via motion capture, and manual resets after failures. This work introduces AVAIL (Autonomy ViA mILestones), a system designed to enable autonomous, programming-free learning of multi-stage manipulation tasks directly from raw visual input.
Methods: Autonomy via Milestones (AVAIL)
The AVAIL framework simplifies complex tasks into a sequence of manageable sub-skills guided by a "milestones graph." Instead of requiring full demonstrations, which are difficult to provide for high-degree-of-freedom (DoF) hands, the system relies on simple image snapshots provided by the user to define key task states.
The system incorporates several key components:
- Learned Rewards (VICE): Success classifiers are trained on milestone images using the VICE algorithm. The robot receives rewards based on the probability that its current visual observation matches the target milestone.
- Autonomous Task Selection: A learned task dynamics model ($p_{task}$) evaluates the current state and determines which sub-skill to practice. This allows the robot to autonomously recover from failures; for instance, if an object is dropped, the system reverts to a "grasp" policy.
- Sample-Efficient RL (DroQ): To facilitate learning from pixels in the real world, the authors utilize DroQ, which combines Soft Actor-Critic (SAC) with data augmentation and dropout for robust performance.
Experimental Evaluation
The system was evaluated on three intricate real-world tasks requiring high dexterity:
- Kitchen Brush: Grasping, reorienting bristles, and scrubbing a plate.
- Hose Connector: Reaching, grasping, and inserting a connector onto a peg.
- Rope Hooking: Latching a carabiner attached to a rope onto a handle.
Key Results and Insights
AVAIL demonstrated significant success in enabling unattended learning:
- High Success Rates: After 18–48 hours of autonomous practice, the robot achieved success rates of 80% or higher across all real-world sub-tasks.
- Efficiency: The learned task graph converged 25% faster than hand-coded heuristic graphs in the hose insertion task, as the learned model better accounted for specific robot failure modes.
- Robustness: In simulated benchmarks, AVAIL outperformed prior autonomous RL methods like R3L, which struggled with high-DoF tasks without granular milestone guidance.
- Snapshots vs. Demonstrations: Providing "outcome snapshots" is significantly lower effort for users than kinesthetic demonstrations, which are often awkward for high-DoF hands and can occlude camera views.
Conclusion and Future Work
AVAIL proves that complex dexterous skills can be learned with minimal instrumentation and human effort. By treating failures as transitions within a milestone graph, the system maintains a continuous loop of practice. Future work will focus on removing current constraints, such as the use of tethers to keep objects within reach, and integrating safety critics to handle potentially irreversible states during autonomous exploration.
Summary updated