Active learning of manipulation sequences

David Martínez Martínez, Guillem Alenyà, Pablo Jiménez, Carme Torras, Jürgen Rossmann, Nils Wantia, Eren Erdal Aksoy, Simon Haller, Justus H. PiaterICRA 2014 · OpenReview

TL;DR

This paper presents an active learning system for robots to learn manipulation sequences, combining exploration with teacher guidance. The system learns symbolic rules for actions on the fly, allowing for dynamic replanning and adaptation to new situations, significantly outperforming traditional methods in learning efficiency.

What Problem Does This Paper Address?

Industrial robots typically execute pre-programmed, fixed sequences of actions, requiring extensive and costly manual programming. Modern applications demand more flexibility, enabling robots to learn and adapt to new tasks and environments through simpler means like human demonstration. The challenge lies in developing a system that can learn these manipulation sequences efficiently, handle uncertainty, and leverage human expertise strategically without overwhelming the teacher. Existing methods often require pre-defined action sets or large amounts of data, and lack the ability to dynamically refine their knowledge base during operation.

Methods

The core of the system is a Decision Maker (DM), a symbolic reasoning engine that learns and plans manipulation sequences. The DM operates on Noisy Indeterministic Deictic (NID) rules, which define actions with preconditions and probabilistic effects. These rules are learned incrementally through a combination of:

  • Observation: The system observes a teacher performing manipulations in a simulated Virtual Reality (VR) environment.
  • Exploration: The robot actively tests actions to maximize learning progress.
  • Teacher Interaction: The system can actively request demonstrations from a teacher when it encounters uncertainty or needs to learn new actions. This is a key aspect of their REX-D algorithm, an extension of relational reinforcement learning.

The process involves:

  1. Semantic Event Chain (SEC) Generation: Sensory data from the VR environment is converted into a sequence of graphs representing object relationships.
  2. Manipulation Recognition: The system parses these SECs to identify known or novel manipulation actions.
  3. Rule Learning: The DM uses a model-based reinforcement learning approach (specifically, Pasula et al.'s [11] learner) to acquire and refine NID rules based on observed experiences and teacher demonstrations.
  4. Planning: A probabilistic planner (PRADA [8]) generates action sequences to achieve goals using the learned rule base.

The system operates in two modes: Learning and Execution (where the robot learns and performs tasks) and Monitoring (where the robot observes a human and provides feedback).

Datasets

The paper describes experiments conducted in a simulated Virtual Reality (VR) environment. Two specific scenarios are detailed:

  • LABEX: A simulated laboratory environment in the International Space Station (ISS) involving tasks like opening/closing cupboards and manipulating experimental containers. This scenario was primarily used for testing the monitoring capabilities.
  • AUTAS: Based on the Cranfield assembly benchmark, involving the assembly of mechanical parts. This scenario was used for evaluating the learning of manipulation sequences, including deterministic and noisy versions.

No specific publicly available datasets were mentioned; the experiments were conducted within their custom VR simulation.

Key Insights and Ideas

  • Active Learning for Robots: The system's ability to actively request teacher demonstrations when needed, rather than passively receiving them, is crucial for efficient learning. This minimizes teacher burden and focuses human intervention on critical learning points.
  • On-the-Fly Rule Refinement: The system doesn't require a complete set of rules upfront. It continuously learns and refines action rules, including preconditions and effects, as it encounters new situations or failures.
  • Handling Uncertainty: The use of NID rules and a probabilistic planner allows the system to operate effectively in stochastic environments and to recover from unexpected action outcomes.
  • Seamless Learning and Execution: There are no distinct learning and application phases. Learned rules are immediately available for planning and execution, enabling a fluid transition from learning to performing tasks.
  • Teacher as a Strategic Resource: The teacher's role is not just to demonstrate, but to guide the learning process by providing information when the system is most likely to benefit from it, such as when a new action needs to be learned or an unexpected outcome occurs.

Key Results

  • Fast Convergence: Quantitative results in the AUTAS scenario show that REX-D (their proposed algorithm) achieves significantly faster convergence in terms of the number of actions required to complete tasks compared to the original REX algorithm, especially in the initial learning stages.
  • Reduced Teacher Intervention: REX-D requires fewer teacher demonstrations overall, particularly after the initial learning episode, indicating efficient knowledge acquisition.
  • Effective Handling of Novelty: The system successfully learned new actions and adapted to variations in task execution (e.g., a peg in a horizontal position requiring a new "put_vertical" action) through teacher demonstrations.
  • Outperformance in Stochastic Environments: In a noisy AUTAS scenario with a 60% success ratio for actions, REX-D still demonstrated significant advantages over REX, highlighting its robustness.
  • Monitoring Capabilities: The system demonstrated the ability to monitor human performance, provide feedback (green/yellow/red traffic light signals), and recommend actions.

Limitations and Future Work

  • VR Simulation: The current work is performed in a simulated VR environment. The authors acknowledge that extending this to real robotic platforms is a crucial next step and is currently in progress.
  • Teacher Dependence: While the system aims to minimize teacher intervention, it still relies on a human teacher for learning new actions and resolving complex ambiguities.
  • Complexity of Actions: The paper focuses on object-level manipulations. More complex, multi-step actions or tasks requiring fine motor control might pose further challenges.
  • Generalization to Unseen Environments: While the system learns relational rules, the extent to which these generalize to entirely novel environments or object types is not fully explored.

Future work will focus on deploying the system on real robotic platforms and exploring more complex manipulation tasks.

Summary updated

Abstract

We describe a system allowing a robot to learn goal-directed manipulation sequences such as steps of an assembly task. Learning is based on a free mix of exploration and instruction by an external teacher, and may be active in the sense that the system tests actions to maximize learning progress and asks the teacher if needed. The main component is a symbolic planning engine that operates on learned rules, defined by actions and their pre- and postconditions. Learned by model-based reinforcement learning, rules are immediately available for planning. Thus, there are no distinct learning and application phases. We show how dynamic plans, replanned after every action if necessary, can be used for automatic execution of manipulation sequences, for monitoring of observed manipulation sequences, or a mix of the two, all while extending and refining the rule base on the fly. Quantitative results indicate fast convergence using few training examples, and highly effective teacher intervention at early stages of learning.
BibTeX
@inproceedings{martinez2014active,
  author = {David Martínez Martínez and Guillem Alenyà and Pablo Jiménez and Carme Torras and Jürgen Rossmann and Nils Wantia and Eren Erdal Aksoy and Simon Haller and Justus H. Piater},
  title = {{Active learning of manipulation sequences}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2014},
  url = {https://openreview.net/forum?id=l6WXMzI7NPm},
}
Figure