DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations

Sanjay Krishnan, Roy Fox, Ion Stoica, Ken Goldberg

Introduction

Learning complex, multi-stage tasks from human demonstrations remains a significant challenge in robotics. Traditional "flat" imitation learning often requires an impractical number of demonstrations to generalize effectively. This paper introduces Discovery of Deep Continuous Options (DDCO), a framework that automatically discovers hierarchical sub-skills—or "options"—from continuous control demonstrations. By decomposing complex tasks into reusable sub-tasks, DDCO enables robots to learn more efficiently, requiring significantly fewer demonstrations than standard methods.

Problem Statement

Most Learning from Demonstrations (LfD) approaches attempt to learn a single policy mapping observations directly to actions. While effective for simple tasks, this "flat" approach struggles with multi-stage procedures, such as surgical needle insertion, which naturally consist of distinct phases like grasping, orienting, and inserting. While Hierarchical Reinforcement Learning (HRL) has utilized "options" to address this, previous work was largely restricted to discrete action spaces or low-dimensional inputs. DDCO addresses the need for automatic option discovery in continuous control spaces using high-dimensional inputs (e.g., images) and deep neural networks.

Methodology

DDCO extends the Deep Discovery of Options (DDO) algorithm through two primary innovations:

  1. Hybrid Categorical–Continuous Model: DDCO employs a hybrid high-level policy that can either invoke a discrete learned option or execute a direct continuous control action. This flexibility allows the system to utilize specialized sub-skills when applicable while maintaining general control capabilities for novel situations.
  2. Automated Parameter Tuning via Cross-Validation: Determining the optimal number of options is a common bottleneck in hierarchical learning. DDCO introduces an offline cross-validation method that maximizes the log-likelihood of a holdout demonstration set. This allows researchers to determine the ideal model complexity without requiring risky physical trials.

The learning process utilizes an Expectation-Gradient (EG) approach, alternating between identifying which option was active during a demonstration segment (E-step) and updating policy parameters (G-step).

Experimental Evaluation

The researchers evaluated DDCO across three environments:

  • Box2D Simulation: A 3-link robot arm tasked with pushing a block to a goal position.
  • Physical Needle Insertion: A da Vinci Research Kit (DVRK) surgical robot grasping and inserting a needle into a silicone phantom.
  • Physical Surgical Bin Picking: The DVRK robot clearing a pile of needles and pins and sorting them into bins.
Sequence of a 3-link robot arm pushing a block to a goal in the Box2D simulation.

Results and Analysis

The experimental results highlight the advantages of hierarchical decomposition:

  • Sample Efficiency: In the 3-link arm simulation, DDCO achieved the same reward as flat baselines using 3x fewer demonstrations. This suggests that hierarchy acts as a powerful regularizer during the learning process.
  • Offline Validation: The peak in offline cross-validated log-likelihood correlated strongly with online success rates, validating the automated tuning approach.
  • Robustness and Recovery: In the bin-picking task, the hierarchical structure allowed for natural error recovery. If a grasp failed, the high-level policy could simply re-invoke the "grasp" option. The robot successfully recovered from failed grasps in 97% of cases.
  • Task Performance: In the needle insertion task, DDCO achieved an 80% success rate, outperforming the best imitation learning baseline (60%).
Performance metrics showing sample efficiency and the effectiveness of cross-validation for option selection.

Conclusion

DDCO demonstrates that discovering deep continuous options can significantly improve the sample efficiency and robustness of robot imitation learning. By automatically segmenting demonstrations into intuitive sub-skills, the framework bridges the gap between high-level reasoning and low-level continuous control. Future work will explore heterogeneous options and the reuse of discovered option libraries across diverse multi-task settings.

Summary updated

Abstract

An option is a short-term skill consisting of a control policy for a specified region of the state space, and a termination condition recognizing leaving that region. In prior work, we proposed an algorithm called Deep Discovery of Options (DDO) to discover options to accelerate reinforcement learning in Atari games. This paper studies an extension to robot imitation learning, called Discovery of Deep Continuous Options (DDCO), where low-level continuous control skills parametrized by deep neural networks are learned from demonstrations. We extend DDO with: (1) a hybrid categorical-continuous distribution model to parametrize high-level policies that can invoke discrete options as well continuous control actions, and (2) a cross-validation method that relaxes DDO’s requirement that users specify the number of options to be discovered. We evaluate DDCO in simulation of a 3-link robot in the vertical plane pushing a block with friction and gravity, and in two physical experiments on the da Vinci surgical robot, needle insertion where a needle is grasped and inserted into a silicone tissue phantom, and needle bin picking where needles and pins are grasped from a pile and categorized into bins. In the 3-link arm simulation, results suggest that DDCO can take 3x fewer demonstrations to achieve the same reward compared to a baseline imitation learning approach. In the needle insertion task, DDCO was successful 8/10 times compared to the next most accurate imitation learning baseline 6/10. In the surgical bin picking task, the learned policy successfully grasps a single object in 66 out of 99 attempted grasps, and in all but one case successfully recovered from failed grasps by retrying a second time.
BibTeX
@inproceedings{krishnan2017ddco,
  author = {Sanjay Krishnan and Roy Fox and Ion Stoica and Ken Goldberg},
  title = {{DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations}},
  booktitle = {Conference on Robot Learning},
  year = {2017},
  url = {https://openreview.net/forum?id=hjeEJKCBtW},
}
Figure