Introduction
Learning complex, multi-stage tasks from human demonstrations remains a significant challenge in robotics. Traditional "flat" imitation learning often requires an impractical number of demonstrations to generalize effectively. This paper introduces Discovery of Deep Continuous Options (DDCO), a framework that automatically discovers hierarchical sub-skills—or "options"—from continuous control demonstrations. By decomposing complex tasks into reusable sub-tasks, DDCO enables robots to learn more efficiently, requiring significantly fewer demonstrations than standard methods.
Problem Statement
Most Learning from Demonstrations (LfD) approaches attempt to learn a single policy mapping observations directly to actions. While effective for simple tasks, this "flat" approach struggles with multi-stage procedures, such as surgical needle insertion, which naturally consist of distinct phases like grasping, orienting, and inserting. While Hierarchical Reinforcement Learning (HRL) has utilized "options" to address this, previous work was largely restricted to discrete action spaces or low-dimensional inputs. DDCO addresses the need for automatic option discovery in continuous control spaces using high-dimensional inputs (e.g., images) and deep neural networks.
Methodology
DDCO extends the Deep Discovery of Options (DDO) algorithm through two primary innovations:
- Hybrid Categorical–Continuous Model: DDCO employs a hybrid high-level policy that can either invoke a discrete learned option or execute a direct continuous control action. This flexibility allows the system to utilize specialized sub-skills when applicable while maintaining general control capabilities for novel situations.
- Automated Parameter Tuning via Cross-Validation: Determining the optimal number of options is a common bottleneck in hierarchical learning. DDCO introduces an offline cross-validation method that maximizes the log-likelihood of a holdout demonstration set. This allows researchers to determine the ideal model complexity without requiring risky physical trials.
The learning process utilizes an Expectation-Gradient (EG) approach, alternating between identifying which option was active during a demonstration segment (E-step) and updating policy parameters (G-step).
Experimental Evaluation
The researchers evaluated DDCO across three environments:
- Box2D Simulation: A 3-link robot arm tasked with pushing a block to a goal position.
- Physical Needle Insertion: A da Vinci Research Kit (DVRK) surgical robot grasping and inserting a needle into a silicone phantom.
- Physical Surgical Bin Picking: The DVRK robot clearing a pile of needles and pins and sorting them into bins.
Results and Analysis
The experimental results highlight the advantages of hierarchical decomposition:
- Sample Efficiency: In the 3-link arm simulation, DDCO achieved the same reward as flat baselines using 3x fewer demonstrations. This suggests that hierarchy acts as a powerful regularizer during the learning process.
- Offline Validation: The peak in offline cross-validated log-likelihood correlated strongly with online success rates, validating the automated tuning approach.
- Robustness and Recovery: In the bin-picking task, the hierarchical structure allowed for natural error recovery. If a grasp failed, the high-level policy could simply re-invoke the "grasp" option. The robot successfully recovered from failed grasps in 97% of cases.
- Task Performance: In the needle insertion task, DDCO achieved an 80% success rate, outperforming the best imitation learning baseline (60%).
Conclusion
DDCO demonstrates that discovering deep continuous options can significantly improve the sample efficiency and robustness of robot imitation learning. By automatically segmenting demonstrations into intuitive sub-skills, the framework bridges the gap between high-level reasoning and low-level continuous control. Future work will explore heterogeneous options and the reuse of discovered option libraries across diverse multi-task settings.
Summary updated