Abstract
Human robot interaction is an exciting task, which aimed to guide robots following instructions from human. Since huge gap lies between human natural language and machine codes, end to end human robot interaction models is fair challenging. Further, visual information receiving from sensors of robot is also a hard language for robot to perceive. In this work, HuBo-VLM is proposed to tackle perception tasks associated with human robot interaction including object detection and visual grounding by a unified transformer based vision language model. Extensive experiments on the Talk2Car benchmark demonstrate the effectiveness of our approach. Code would be publicly available in https://github.com/dzcgaara/HuBo-VLM.
Introduction
Human-Robot Interaction (HRI) faces a significant "translation gap" between nuanced human natural language and the rigid machine code or raw sensor data processed by robots. Existing Vision-Language Models (VLMs) often struggle with three primary issues: dependency on pre-extracted Regions of Interest (ROI), task rigidity that prevents switching between different perception modes, and limited reasoning capabilities for complex, multi-step instructions.
HuBo-VLM addresses these challenges by introducing a flexible, instruction-aware architecture capable of handling diverse perception tasks within a single unified framework.
Architecture and Methodology
The HuBo-VLM architecture is designed as a unified task solver that treats perception as a sequence-generation problem. It consists of three core modules:
- Image Encoder: Utilizes a ResNet152 backbone to transform raw visual input into feature embeddings.
- Instruction Encoder: Employs BERT to encode natural language commands into task-specific embeddings, guiding the model's focus.
- Unified Task Solver: A Transformer-based encoder-decoder that integrates visual and linguistic embeddings to generate output sequences.
A key innovation in HuBo-VLM is the elimination of traditional object detection "heads." Instead of complex regression losses, the model outputs coordinates directly as a string of tokens (e.g., x0, y0, x1, y1), which are then post-processed into bounding boxes. This approach simplifies the pipeline and enables end-to-end training.
Training and Evaluation
The model's development involved a two-stage process:
- Unified Pre-training: Leveraging large-scale multi-modal datasets such as Conceptual Captions (CC12M/CC3M), MSCOCO, Visual Genome, and ImageNet-21K to build a robust foundation.
- Fine-tuning: Specialized training on the Talk2Car benchmark, which contains nearly 12,000 natural language commands mapped to the nuScenes autonomous driving dataset.
Results and Discussion
HuBo-VLM achieved state-of-the-art performance on the Talk2Car benchmark with an AP50 of 76.74, significantly outperforming previous models like Deformable-MDETR (74.4) and Stacked VLBert (71).
Beyond quantitative metrics, the model demonstrates superior qualitative reasoning. It successfully interprets complex spatial logic and relative positioning, which are often stumbling blocks for traditional vision systems.
Key Insights
- Instruction-Driven Scalability: The dedicated instruction encoder allows the model to adapt to new tasks via text prompts without architectural changes.
- Language-Centric Reasoning: By utilizing language-style architectures, the model better handles the logical flow required for complex HRI tasks.
- Operational Simplicity: Removing ROI dependency makes the model more suitable for real-time robotics applications.
Conclusion and Future Work
While HuBo-VLM sets a new benchmark for HRI perception, future work will focus on replacing the BERT-based encoder with a Large Language Model (LLM) like LLaMA to further enhance reasoning. Additionally, the authors aim to expand the model's generalist capabilities across broader robotic tasks while maintaining efficiency for embedded hardware deployment.
Summary updated