TL;DR
This paper introduces a framework for sequencing diverse robot control policies—ranging from classical motion planners to deep neural networks—to solve complex, multi-stage tasks. By learning a "Goal Score Estimator" from expert demonstrations, the system can use Model Predictive Control (MPC) to select the best controller for the current situation, achieving a 100% success rate on a challenging physical gear assembly task.
What Problem Does This Paper Address?
Robots often struggle with "temporally extended tasks"—long sequences of actions like assembling a piece of machinery. These tasks usually require different types of skills: reaching for a part needs precise geometric planning, while inserting a gear into a tight slot might require a reactive, vision-based neural network policy.
The core challenge is compositionality: how do you combine these diverse controllers into a single, cohesive system? Most existing methods assume all controllers share the same internal representation (e.g., all are joint-space controllers). In reality, a "reaching" controller might care about joint angles, while an "insertion" controller cares about camera pixels. Manually designing state machines to switch between these is brittle and doesn't scale. The authors aim to automate this sequencing without needing to manually define complex pre- and post-conditions for every sub-task.
Methods
The authors propose a hybrid hierarchical control strategy consisting of three main components:
- Controller Library: A collection of pre-existing policies, such as Motion Planning (MP) primitives, Dynamic Motion Primitives (DMPs), and Convolutional Neural Networks (CNNs).
- Goal Score Estimator: This is the "progress bar" of the system. It uses a CNN with a Mixture Density Network (MDN) tail—a type of network that predicts a probability distribution rather than a single value—to map image observations to a score between 0 (start) and 1 (goal). It is trained on expert demonstrations to provide a universal metric of task completion.
- Forward Dynamics Models: Each controller has its own learned dynamics model that predicts the next latent state (a compressed representation of the world) if that specific controller is executed.
To choose a controller, the system uses Model Predictive Control (MPC). It looks $n$ steps into the future for every available controller using their respective dynamics models, evaluates the predicted future state with the Goal Score Estimator, and selects the controller that promises the most progress toward the goal.
Datasets
- Simulated MDP: A 19-state (and later 100-state) random walk benchmark used to test the framework's robustness to noise in the dynamics models and goal scoring.
- Gear Assembly Task: A physical experiment on a PR2 robot. The dataset consists of 50 teleoperated demonstrations performed by a human expert using an HTC Vive controller to teach the robot how to pick, move, and insert a gear.
Key Insights and Ideas
- Universal Currency for Progress: By using images as a shared observation space for the Goal Score Estimator, the framework can compare the "goodness" of vastly different controllers (e.g., one working on joint angles and another on pixels) using a single metric.
- Planning over Policies: Instead of learning a high-level policy from scratch (which is sample-inefficient), the framework treats the selection of sub-policies as a planning problem. This allows it to leverage the strengths of classical robotics (precision) and modern AI (reactivity).
- Robustness through Foresight: The MPC look-ahead allows the robot to avoid controllers that might look good in the short term but lead to "dead ends" or regressive states where progress is lost.
Key Results
The framework was evaluated on a complex gear assembly task where the robot had to pick a gear from a table, move it, and insert it onto a peg.
- 100% Success Rate: The hybrid "MP & CNN" approach achieved 10/10 successful trials.
- Failure of Single-Method Approaches: Pure Motion Planning (MP) or DMP approaches failed the insertion phase (1/10 success) because they couldn't handle the small inaccuracies and physical tolerances of the fit. Conversely, pure CNN policies were often too "jerky" for the initial pick-up and violated safety constraints.
- Noise Tolerance: In simulations, the system remained effective even with significant noise (up to 20% error) in the dynamics and goal scoring models, eventually converging to the optimal sequence of actions.
Limitations and Future Work
While powerful, the method relies on having a library of pre-existing controllers. If a necessary skill is missing from the library, the task cannot be completed. Additionally, the Goal Score Estimator requires expert demonstrations, which may be difficult to obtain for extremely long or dangerous tasks.
Future work could involve automatically discovering new controllers when the current library is insufficient or extending the framework to handle even more diverse sensor modalities beyond vision and joint encoders.
Summary updated