Learning to Recognize Novel Objects in One Shot through Human-Robot Interactions in Natural Language Dialogues

Evan A. Krause, Michael Zillich, Thomas Emrys Williams, Matthias ScheutzAAAI 2014 · DOI

Abstract

Traditional computer vision models require vast amounts of data to recognize new objects, a limitation that hinders robots in dynamic, open-world environments. This paper introduces a framework for one-shot learning that leverages natural language descriptions to bridge the gap between vision and language. By decomposing verbal instructions into known visual primitives and spatial relationships, the system enables a robot to identify and learn novel objects instantly through dialogue.

Introduction

Modern computer vision typically relies on data-driven techniques, requiring thousands of labeled images to achieve high accuracy. However, in scenarios like search-and-rescue, a robot may encounter unique objects—such as a specific medical kit—that it must recognize immediately based on a human's description. This challenge is characterized by extreme data scarcity and a contextual gap: while traditional systems treat labels as atomic names, humans describe objects compositionally based on parts, colors, and spatial arrangements.

Methodology: Integrating NLU and Vision

The proposed solution integrates a Natural Language Understanding (NLU) system with a Modular Vision System within the DIARC (Distributed Integrated Affect Reflection Cognition) architecture.

The NLU Pipeline

When a human provides a description, such as "It is a white box with a red cross on it," the NLU system parses the input into a quantifier-free logical form. For example: color(X, white) ∧ type(X, box) ∧ color(Y, red) ∧ type(Y, cross) ∧ on(Y, X).

The Vision Pipeline and Search Manager

The vision system is organized hierarchically into Saliency Operators, Detectors, and Validators. The Search Manager acts as the critical link; instead of searching for an unknown label like "medkit," it uses the logical form to instantiate a search for the constituent parts (e.g., a "white box" and a "red cross") and verifies their spatial relationship ("on"). Once these conditions are met, the robot binds the visual structure to the new label.

The integrated architecture of the Vision Component and Search Manager.

Experimental Evaluation

The system was evaluated using a Willow Garage PR2 robot in a real-world laboratory setting. The robot was tasked with learning three novel objects: a "medkit" (white box with a red cross), a "radioactive box" (yellow box with a radioactive symbol), and a "biohazard cylinder" (orange cylinder with a biohazard symbol).

Experimental setup (top) and the robot's visual perception of the novel objects (bottom).

Key Results

  • One-Shot Success: The robot successfully identified the "medkit" among distractor objects after only a single verbal description.
  • Pose Invariance: By representing objects as collections of parts and relations rather than static templates, the robot maintained recognition despite changes in the object's location or orientation.
  • Generalization: The system effectively differentiated between multiple complex objects using the same underlying logic, demonstrating the flexibility of structured visual representations.

Discussion and Future Work

A primary insight of this work is that language serves as a structural blueprint for visual recognition. However, the current system depends on pre-existing knowledge of "primitives" (e.g., "box," "red"). Future research will focus on allowing robots to learn these basic terms through interactive grounding. Additionally, while the current implementation handles simple noun phrases, expanding linguistic complexity to include embedded clauses and resolving visual-linguistic ambiguities through further dialogue remain key areas for development.

Summary updated

BibTeX
@inproceedings{krause2014learning,
  author = {Evan A. Krause and Michael Zillich and Thomas Emrys Williams and Matthias Scheutz},
  title = {{Learning to Recognize Novel Objects in One Shot through Human-Robot Interactions in Natural Language Dialogues}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2014},
  doi = {10.1609/aaai.v28i1.9143},
}
Figure