Abstract
Traditional computer vision models require vast amounts of data to recognize new objects, a limitation that hinders robots in dynamic, open-world environments. This paper introduces a framework for one-shot learning that leverages natural language descriptions to bridge the gap between vision and language. By decomposing verbal instructions into known visual primitives and spatial relationships, the system enables a robot to identify and learn novel objects instantly through dialogue.
Introduction
Modern computer vision typically relies on data-driven techniques, requiring thousands of labeled images to achieve high accuracy. However, in scenarios like search-and-rescue, a robot may encounter unique objects—such as a specific medical kit—that it must recognize immediately based on a human's description. This challenge is characterized by extreme data scarcity and a contextual gap: while traditional systems treat labels as atomic names, humans describe objects compositionally based on parts, colors, and spatial arrangements.
Methodology: Integrating NLU and Vision
The proposed solution integrates a Natural Language Understanding (NLU) system with a Modular Vision System within the DIARC (Distributed Integrated Affect Reflection Cognition) architecture.
The NLU Pipeline
When a human provides a description, such as "It is a white box with a red cross on it," the NLU system parses the input into a quantifier-free logical form. For example: color(X, white) ∧ type(X, box) ∧ color(Y, red) ∧ type(Y, cross) ∧ on(Y, X).
The Vision Pipeline and Search Manager
The vision system is organized hierarchically into Saliency Operators, Detectors, and Validators. The Search Manager acts as the critical link; instead of searching for an unknown label like "medkit," it uses the logical form to instantiate a search for the constituent parts (e.g., a "white box" and a "red cross") and verifies their spatial relationship ("on"). Once these conditions are met, the robot binds the visual structure to the new label.
Experimental Evaluation
The system was evaluated using a Willow Garage PR2 robot in a real-world laboratory setting. The robot was tasked with learning three novel objects: a "medkit" (white box with a red cross), a "radioactive box" (yellow box with a radioactive symbol), and a "biohazard cylinder" (orange cylinder with a biohazard symbol).
Key Results
- One-Shot Success: The robot successfully identified the "medkit" among distractor objects after only a single verbal description.
- Pose Invariance: By representing objects as collections of parts and relations rather than static templates, the robot maintained recognition despite changes in the object's location or orientation.
- Generalization: The system effectively differentiated between multiple complex objects using the same underlying logic, demonstrating the flexibility of structured visual representations.
Discussion and Future Work
A primary insight of this work is that language serves as a structural blueprint for visual recognition. However, the current system depends on pre-existing knowledge of "primitives" (e.g., "box," "red"). Future research will focus on allowing robots to learn these basic terms through interactive grounding. Additionally, while the current implementation handles simple noun phrases, expanding linguistic complexity to include embedded clauses and resolving visual-linguistic ambiguities through further dialogue remain key areas for development.
Summary updated