Learning from Implicit Information in Natural Language Instructions for Robotic Manipulations

Ozan Arkan Can, Pedro Zuidberg Dos Martires, Andreas Persson, Julian Gaal, Amy Loutfi, Luc De Raedt, Deniz Yuret, Alessandro Saffiotti

TL;DR

This paper tackles the challenge of robots understanding natural language instructions by proposing a novel approach that separately learns the robot's internal world representation and how to ground language within it. They then introduce a Bayesian learning method to resolve inconsistencies between these two learned components, leveraging implicit spatial information within instructions.

What Problem Does This Paper Address?

Robots performing tasks based on human instructions require a shared understanding of the world. This means the robot needs to build its own internal model of the environment (world representation) and then connect natural language commands to that model (language grounding). Ideally, these two processes would be learned jointly. However, in robotics, acquiring enough data for such joint learning is extremely difficult and costly. This paper addresses this data scarcity problem by proposing to learn these components separately. The challenge then becomes how to reconcile potential inconsistencies that arise when the robot's world model and its language understanding don't perfectly align, especially when instructions rely on spatial relationships.

Methods

The authors propose a two-pronged approach:

  • Separate Learning: They first learn the robot's world representation using "perceptual anchoring," where sensory data is used to create symbolic representations (anchors) of objects. Simultaneously, they train a language grounder on synthetic data that simulates these anchors.
  • Bayesian Learning for Inconsistency Resolution: To bridge the gap between the separately learned components, they introduce a Bayesian learning framework. This framework probabilistically reasons about the relationship between the instruction and the robot's world representation. Crucially, it leverages implicit spatio-relational information within the natural language instructions (e.g., "next to," "to the right of") to update and align the robot's world model with the human's intended meaning.

The language grounding system itself is built using a neural module network approach, consisting of:

  • Preprocessor: Converts anchor information into a grid representation.
  • Instruction Parser: Uses a dependency parser to create a computation graph from the instruction.
  • Program Executor: Runs the computation graph to produce robot commands.

Datasets

The paper mentions training the language grounding component on synthetic data that simulates the world represented as anchors. For the visual object classification, they used a CNN trained on 101 object categories expected in a kitchen domain. The experiments also involve generating artificial learning tasks with random grid worlds to test different scenarios of increasing difficulty. No specific named datasets are detailed for the overall system training or evaluation.

Key Insights and Ideas

  • Separation is Key for Data Scarcity: Learning world representation and language grounding separately is a pragmatic solution to the data bottleneck in robotics.
  • Implicit Spatial Information is Powerful: Natural language instructions contain rich, often implicit, spatial cues that can be leveraged to resolve ambiguities and align world models.
  • Bayesian Reasoning as a Unifier: Probabilistic reasoning provides a robust framework for reconciling discrepancies between perception and language.
  • Modular Neural Networks for Compositionality: The use of neural module networks allows for effective composition of learned components, enabling the system to handle complex instructions.

Key Results

The paper demonstrates the feasibility of their approach on a showcase scenario involving a robotic arm in a physical kitchen environment. They show that their system can successfully interpret and execute instructions that involve spatial relationships, even when there are initial ambiguities or misclassifications in the robot's perception. A key example highlighted is correcting a misclassification of a "pot" as a "mug" based on an instruction, showcasing the system's ability to resolve perceptual inconsistencies.

Limitations and Future Work

  • Curse of Dimensionality: The Bayesian learning step suffers from a "curse of dimensionality" due to the large number of possible label configurations for anchors. The authors mitigate this by limiting their experiments to the two most probable labels per anchor.
  • Limited Vocabulary: The current system relies on a predefined vocabulary.
  • Future Work:
    • Perform more quantitative analysis to address the dimensionality issue.
  • Extend the approach to natural language beyond instructions (e.g., descriptions).
  • Enable the robot to extend its vocabulary through dialogue.
  • Learn a richer probabilistic model to resolve inconsistencies, potentially incorporating unknown vocabulary.

Summary updated

Abstract

Human-robot interaction often occurs in the form of instructions given from a human to a robot. For a robot to successfully follow instructions, a common representation of the world and objects in it should be shared between humans and the robot so that the instructions can be grounded. Achieving this representation can be done via learning, where both the world representation and the language grounding are learned simultaneously. However, in robotics this can be a difficult task due to the cost and scarcity of data. In this paper, we tackle the problem by separately learning the world representation of the robot and the language grounding. While this approach can address the challenges in getting sufficient data, it may give rise to inconsistencies between both learned components. Therefore, we further propose Bayesian learning to resolve such inconsistencies between the natural language grounding and a robot's world representation by exploiting spatio-relational information that is implicitly present in instructions given by a human. Moreover, we demonstrate the feasibility of our approach on a scenario involving a robotic arm in the physical world.
BibTeX
@inproceedings{can2019learning,
  author = {Ozan Arkan Can and Pedro Zuidberg Dos Martires and Andreas Persson and Julian Gaal and Amy Loutfi and Luc De Raedt and Deniz Yuret and Alessandro Saffiotti},
  title = {{Learning from Implicit Information in Natural Language Instructions for Robotic Manipulations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=0gFI-szlKp8},
}
Figure