Tools Identification By On-Board Adaptation of Vision-and-Language Models

Jun Hu, Phil Miller, Michael Lomnitz, Saurabh Farkya, Emre Yilmaz, Aswin Raghavan, David C. Zhang, Michael R. PiacentinoAAAI 2024 · DOI

Abstract

This paper introduces a system designed for robotic workshop assistants to identify tools in cluttered, real-world environments. By combining the semantic power of Vision-and-Language models (VLMs) with the efficiency of Hyperdimensional Computing (HD), the system enables robots to understand tool names and functional affordances while performing on-board adaptation on low-power edge hardware.

Introduction

While Large Vision-and-Language models like CLIP are impressive, they often struggle when transitioned from "clean" internet datasets to the "messy" reality of a physical workshop. These models face three primary hurdles: domain shift (where a tool on a pegboard looks different from a stock photo), visual clutter, and a functional gap. A workshop assistant needs to understand affordances—the purpose of a tool—rather than just its label. For instance, a robot should be able to find a tool when asked to "find something to drive these nails," even if it hasn't explicitly mapped the word "hammer" to that specific object in its current environment.

Proposed Method

The researchers developed a two-stage pipeline that bridges high-level semantic understanding with fast, local execution on edge devices like the NVIDIA Jetson Orin.

The system pipeline integrates speech-to-text and zero-shot visual grounding to identify tools based on user instructions.
  1. Region Proposal & Embedding: A Region Proposal Network (RPN) identifies potential object boxes in an image. Each box is cropped and processed by CLIP to generate a high-dimensional embedding.
  2. Hyperdimensional Computing (HD) Adaptation: This is the core innovation. Instead of using heavy neural network layers for classification, the system projects CLIP embeddings into a high-dimensional binary vector space. Using bitwise operations like XOR (binding) and averaging (bundling), the system creates "exemplars" for tool categories.
  3. Purpose-Driven Queries: The system is trained with captions that link tool names to their primary functions (e.g., "Hammer: drive nails, remove nails").

Because HD operations are computationally light, the system can adapt to a new environment or "enroll" a new tool almost instantly without requiring a constant internet connection or expensive backpropagation.

Experimental Results

The system was evaluated using the ALET (Automated Labeling of Equipment and Tools) and SKIML datasets. The results highlighted the efficiency of the HD approach and the power of functional reasoning.

Detection results on a cluttered pegboard, demonstrating the system's ability to identify multiple tools in a real-world setting.
  • Purpose as a Generalizer: Querying by purpose (what the tool does) was significantly more effective than querying by name. Using purpose-derived exemplars boosted Mean Average Precision (mAP) from 27.61 to 43.57.
  • Zero-Shot Performance: The system achieved a zero-shot mAP roughly 50% as high as a model fully fine-tuned on target data, representing a major step forward for "out-of-the-box" robotic utility.
  • Edge Efficiency: The HD classification step takes only 1.95 seconds on an NVIDIA Jetson Orin, though the initial RPN stage remains a bottleneck at approximately 18 seconds.

Conclusion and Future Work

This research demonstrates that combining VLMs with Hyperdimensional Computing allows for efficient, on-board adaptation for robotic assistants. By focusing on tool affordances, the system becomes more robust to environmental changes. Future work will focus on optimizing the initial region proposal stage and integrating this vision system with robotic grasping and motion planning to enable physical tool retrieval.

Summary updated

BibTeX
@inproceedings{hu2024tools,
  author = {Jun Hu and Phil Miller and Michael Lomnitz and Saurabh Farkya and Emre Yilmaz and Aswin Raghavan and David C. Zhang and Michael R. Piacentino},
  title = {{Tools Identification By On-Board Adaptation of Vision-and-Language Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30569},
}
Figure