Teaching Robots through Situated Interactive Dialogue and Visual Demonstrations

Jose L. Part, Oliver LemonIJCAI 2017 · DOI

TL;DR

This paper tackles the challenge of teaching robots new tasks and objects incrementally, mimicking how humans learn. They propose a framework combining natural language instructions, visual demonstrations, and interactive dialogue, enabling robots to learn "on-the-fly" with minimal examples, crucial for unstructured environments.

What Problem Does This Paper Address?

Robots operating in real-world, unstructured environments (like homes) need to be adaptable and learn new things quickly, especially from non-expert users. Current methods often require extensive pre-programmed knowledge or large datasets for offline training, which isn't practical for dynamic, human-centric scenarios. This paper addresses the gap in enabling robots to learn about new objects and how to perform tasks in an online, interactive manner, directly from human guidance.

Methods

The core of the proposed framework involves two key modules:

  • Incremental Online Object Learning: This module allows the robot to learn new objects on-the-fly. It uses pre-trained Convolutional Neural Networks (CNNs) to extract features from RGB images and colourized surface normals (derived from depth maps). These features are then fed into an adapted Load-Balancing Self-Organizing Incremental Neural Network (LB-SOINN) classifier. The system learns by associating visual input with labels provided through dialogue, enabling it to update its knowledge base incrementally.
  • Situated Interactive Task Learning: This module focuses on learning hierarchical task structures. It leverages natural language instructions and visual demonstrations. The goal is to learn from a single user interaction, rather than requiring multiple examples. The robot engages in clarifying dialogue to resolve references, correct mistakes, and refine its understanding of the task.

These modules are designed to be integrated, allowing task learning to be grounded in the robot's incrementally learned object knowledge.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Learning on-the-fly is Crucial: For robots to be useful in dynamic, unstructured environments, they must be able to acquire new knowledge and skills without extensive pre-training or offline data collection.
  • Dialogue as a Learning Tool: Interactive dialogue is not just for clarification but is a fundamental mechanism for grounding and refining the robot's understanding of objects and tasks.
  • Combining Modalities: Integrating visual information (RGB, depth) with natural language instructions and interactive feedback provides a richer learning signal than any single modality alone.
  • Incremental Learning for Adaptability: The proposed incremental object learning approach allows the robot to adapt to new objects encountered during task execution, a key requirement for real-world deployment.

Key Results

The paper reports on the development and integration of these modules. While specific quantitative results are not detailed in this excerpt, the authors state their aim is to evaluate the full integrated system on a real robotic platform (a PR2 robot) for a specific task. The key qualitative outcome is the demonstration of a robot's ability to learn objects and tasks through situated interaction, moving towards more adaptable and user-friendly robotic systems.

Limitations and Future Work

  • Current Object Learning Limitations: The current object learning module treats each class as a single entity in the feature space. This can lead to issues with overlapping classes and difficulties in recognizing sparsely distributed clusters within a class.
  • Future Object Learning Improvements: The authors plan to address these limitations by exploring independent networks for each class and using similarity heuristics to identify unknown objects, enabling a self-driven mechanism for addressing knowledge gaps.
  • Task Learning Integration: A major future direction is the full integration of the object learning and task learning modules to enable grounding and inference over objects involved in learned tasks.
  • Real-World Evaluation: The ultimate goal is to evaluate the complete integrated system on a PR2 robot performing a real task.

Summary updated

BibTeX
@inproceedings{part2017teaching,
  author = {Jose L. Part and Oliver Lemon},
  title = {{Teaching Robots through Situated Interactive Dialogue and Visual Demonstrations}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/760},
}
Figure