Learning from Implicit Information in Natural Language Instructions for Robotic Manipulations

Ozan Arkan Can, Pedro Zuidberg Dos Martires, Andreas Persson, Julian Gaal, Amy Loutfi, Luc De Raedt, Deniz Yuret, Alessandro Saffiotti

Introduction

Robots often struggle to follow natural language instructions because their sensor-based perception of the world frequently conflicts with human descriptions. For instance, a robot might misidentify a "green apple" as a "pear" due to lighting or sensor noise. Traditionally, resolving such ambiguities requires the robot to stop and ask for clarification. This research proposes a Bayesian learning framework that leverages the implicit spatial information within a command—such as "the apple next to the mug"—to automatically correct perceptual errors without human intervention.

Methodology: Bridging Perception and Language

The proposed architecture is modular, designed to reconcile raw visual data with linguistic instructions through three primary components:

Visual Object Anchoring

Using a Kinect RGB-D sensor, the system segments the scene into object anchors. A fine-tuned GoogLeNet CNN categorizes these objects into 101 kitchen-related classes, extracting attributes like color and position.

Natural Language Grounding via Neural Module Networks (NMNs)

Instead of a monolithic model, the system parses instructions into a computation graph of functional units. As shown in Figure 1, the instruction "pick up the apple to the right of the black mug" is decomposed into modules like Detect (for nouns/adjectives), Shift (for spatial relations), and Locate (to find the target).

Figure 1: The modular architecture decomposes instructions into functional units (NMNs) to ground language in a grid-world representation of the physical scene.

Bayesian Belief Revision

The core innovation is treating the human instruction as "evidence" to update the robot's internal beliefs. If the spatial relations in a command only hold true if a mislabeled "pot" is actually a "mug," the Bayesian update re-labels the object anchor. To maintain computational tractability, the system only considers the top two most likely labels for each object.

Experimental Results

The framework was evaluated using both synthetic grid worlds and a real-world setup featuring a Kinova Jaco robotic arm.

Learning Efficiency and Complexity

Synthetic tests revealed a significant disparity in learning difficulty between different linguistic components. While the model mastered nouns and adjectives within approximately 300,000 instances, prepositional phrases (e.g., "next to") required over 1.3 million instances to reach comparable accuracy. This is attributed to the complexity of the Shift modules, which must learn many-to-many spatial relations across the scene.

Figure 2: Learning curves for different linguistic complexities, showing that prepositional relations require significantly more data than simple nouns or adjectives.

Real-World Validation

In a physical demonstration, the robot initially misclassified a black mug as a "pot." Upon receiving the instruction "drop it [the ball] in front of the mug," the system utilized the spatial context to correctly identify the "pot" as the intended "mug," successfully completing the manipulation task.

Conclusion and Future Work

This research demonstrates that implicit information in natural language can serve as a powerful corrective signal for robotic perception. Future work aims to expand the system's vocabulary and explore active dialogue, allowing robots to learn new concepts through a combination of implicit reasoning and explicit human feedback.

Summary updated

BibTeX
@inproceedings{can2019learning,
  author = {Ozan Arkan Can and Pedro Zuidberg Dos Martires and Andreas Persson and Julian Gaal and Amy Loutfi and Luc De Raedt and Deniz Yuret and Alessandro Saffiotti},
  title = {{Learning from Implicit Information in Natural Language Instructions for Robotic Manipulations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=0gFI-szlKp8},
}
Figure