Interactive Visual Task Learning for Robots

Weiwei Gu, Anant Sah, Nakul GopalanAAAI 2024 · DOI

Abstract

Current robotic systems often struggle with generalizing learned tasks to novel objects or variations without extensive retraining. We present Hi-Viscont (HIerarchical VISual CONcept learner for Task), a framework that enables robots to learn complex visual concepts and tasks from a single human demonstration. By employing hierarchical knowledge propagation via an Ancestor Relational GNN (ARGNN) and representing tasks as scene graphs, Hi-Viscont allows robots to update their general understanding of object properties and execute modified tasks zero-shot. Our evaluation shows a 30% improvement in identifying functional properties and a significantly higher success rate in real-world human-robot interaction scenarios compared to state-of-the-art baselines.

Introduction

The deployment of robots in dynamic household environments requires the ability to learn new concepts on the fly. However, existing few-shot learning models typically suffer from two primary limitations: isolated learning and rigid task execution. In isolated learning, a robot might learn that a specific "bowl" is a "container" but fails to update its broader internal representation of what constitutes a "container." Consequently, its knowledge remains "bottom-heavy," focused on specific leaf-level instances rather than generalizable categories. Furthermore, task execution is often tied to specific sequences of moves or exact object instances, preventing the robot from adapting to minor variations—such as swapping a component's color—without a complete new demonstration.

Methodology: The Hi-Viscont Framework

Hi-Viscont addresses these challenges by integrating hierarchical reasoning into the visual learning process. The architecture builds upon the FALCON model but introduces several key innovations.

Hierarchical Knowledge Propagation

The core of the system is the Ancestor Relational GNN (ARGNN). When a human teacher introduces a new concept, such as a "green curve block," the ARGNN propagates this information upward through a knowledge hierarchy. This ensures that the robot's understanding of parent nodes—such as "roof" or "sheltering property"—evolves dynamically. This "two-way street" approach means that every new specific example refines the robot's general definitions of object categories and affordances.

Hierarchical knowledge propagation and scene graph representation in Hi-Viscont.

Scene Graph Task Representation

Rather than treating a task as a fixed sequence of coordinates, Hi-Viscont converts human demonstrations into a scene graph. In this graph, nodes represent objects (linked to linguistic descriptions via BERT-based encoders), and edges represent spatial relationships (e.g., "on-top-of"). This structural representation allows for zero-shot generalization. If a user requests a "house with a green roof" after being shown a demo with a yellow roof, the robot performs node-wise classification to identify the required change, locates a suitable "green" object with "sheltering" properties in its environment, and automatically updates the plan.

Interactive learning process: (a) initial demonstration, (b) teaching a new concept, and (c) zero-shot task execution.

Experimental Evaluation

The model's visual reasoning capabilities were tested across three domains: CUB-200-2011 (bird species), House-Construction (building blocks), and a custom Zoo dataset. To evaluate real-world performance, a human-subjects experiment was conducted using a Franka Emika Research 3 robotic arm. Eighteen participants interacted with the robot in real-time to teach and modify construction tasks, providing a robust test of the system's interactive capabilities.

Results and Discussion

Hi-Viscont demonstrated superior performance over the FALCON baseline across multiple metrics. In the House-Construction domain, the model improved accuracy for "Affordance" concepts by over 30%, indicating a much stronger grasp of functional properties like "sheltering."

In robotic trials, Hi-Viscont achieved a 50% success rate in completing novel task requests, a significant jump from the baseline's 16.67%. Object identification accuracy reached 81.25%, representing a 19% improvement. Beyond technical metrics, human participants reported higher levels of trust and perceived intelligence when interacting with the Hi-Viscont system, noting its ability to understand abstract properties through language rather than just simple color-matching.

Conclusion and Future Work

Hi-Viscont represents a step toward more intuitive and flexible human-robot collaboration. By decoupling task structure from specific instances and allowing knowledge to flow hierarchically, robots can learn more like humans do—from single examples that inform a broader worldview. Future research will focus on scaling the system to handle the complexity of real-world household environments and moving toward more dynamic, open-ended interactions with a more diverse range of users.

Summary updated

BibTeX
@inproceedings{gu2024interactive,
  author = {Weiwei Gu and Anant Sah and Nakul Gopalan},
  title = {{Interactive Visual Task Learning for Robots}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i9.28896},
}
Figure