Abstract
This paper introduces a system designed for robotic workshop assistants to identify tools in cluttered, real-world environments. By combining the semantic power of Vision-and-Language models (VLMs) with the efficiency of Hyperdimensional Computing (HD), the system enables robots to understand tool names and functional affordances while performing on-board adaptation on low-power edge hardware.
Introduction
While Large Vision-and-Language models like CLIP are impressive, they often struggle when transitioned from "clean" internet datasets to the "messy" reality of a physical workshop. These models face three primary hurdles: domain shift (where a tool on a pegboard looks different from a stock photo), visual clutter, and a functional gap. A workshop assistant needs to understand affordances—the purpose of a tool—rather than just its label. For instance, a robot should be able to find a tool when asked to "find something to drive these nails," even if it hasn't explicitly mapped the word "hammer" to that specific object in its current environment.
Proposed Method
The researchers developed a two-stage pipeline that bridges high-level semantic understanding with fast, local execution on edge devices like the NVIDIA Jetson Orin.
- Region Proposal & Embedding: A Region Proposal Network (RPN) identifies potential object boxes in an image. Each box is cropped and processed by CLIP to generate a high-dimensional embedding.
- Hyperdimensional Computing (HD) Adaptation: This is the core innovation. Instead of using heavy neural network layers for classification, the system projects CLIP embeddings into a high-dimensional binary vector space. Using bitwise operations like XOR (binding) and averaging (bundling), the system creates "exemplars" for tool categories.
- Purpose-Driven Queries: The system is trained with captions that link tool names to their primary functions (e.g., "Hammer: drive nails, remove nails").
Because HD operations are computationally light, the system can adapt to a new environment or "enroll" a new tool almost instantly without requiring a constant internet connection or expensive backpropagation.
Experimental Results
The system was evaluated using the ALET (Automated Labeling of Equipment and Tools) and SKIML datasets. The results highlighted the efficiency of the HD approach and the power of functional reasoning.
- Purpose as a Generalizer: Querying by purpose (what the tool does) was significantly more effective than querying by name. Using purpose-derived exemplars boosted Mean Average Precision (mAP) from 27.61 to 43.57.
- Zero-Shot Performance: The system achieved a zero-shot mAP roughly 50% as high as a model fully fine-tuned on target data, representing a major step forward for "out-of-the-box" robotic utility.
- Edge Efficiency: The HD classification step takes only 1.95 seconds on an NVIDIA Jetson Orin, though the initial RPN stage remains a bottleneck at approximately 18 seconds.
Conclusion and Future Work
This research demonstrates that combining VLMs with Hyperdimensional Computing allows for efficient, on-board adaptation for robotic assistants. By focusing on tool affordances, the system becomes more robust to environmental changes. Future work will focus on optimizing the initial region proposal stage and integrating this vision system with robotic grasping and motion planning to enable physical tool retrieval.
Summary updated