HuBo-VLM: Unified Vision-Language Model designed for HUman roBOt interaction tasks

Zichao Dong, Weikun Zhang, Xufeng Huang, Hang Ji, Xin Zhan, Junbo ChenarXiv 2023 · OpenReview

Abstract

Human robot interaction is an exciting task, which aimed to guide robots following instructions from human. Since huge gap lies between human natural language and machine codes, end to end human robot interaction models is fair challenging. Further, visual information receiving from sensors of robot is also a hard language for robot to perceive. In this work, HuBo-VLM is proposed to tackle perception tasks associated with human robot interaction including object detection and visual grounding by a unified transformer based vision language model. Extensive experiments on the Talk2Car benchmark demonstrate the effectiveness of our approach. Code would be publicly available in https://github.com/dzcgaara/HuBo-VLM.

Introduction

Human-Robot Interaction (HRI) faces a significant "translation gap" between nuanced human natural language and the rigid machine code or raw sensor data processed by robots. Existing Vision-Language Models (VLMs) often struggle with three primary issues: dependency on pre-extracted Regions of Interest (ROI), task rigidity that prevents switching between different perception modes, and limited reasoning capabilities for complex, multi-step instructions.

HuBo-VLM addresses these challenges by introducing a flexible, instruction-aware architecture capable of handling diverse perception tasks within a single unified framework.

Architecture and Methodology

The HuBo-VLM architecture is designed as a unified task solver that treats perception as a sequence-generation problem. It consists of three core modules:

  1. Image Encoder: Utilizes a ResNet152 backbone to transform raw visual input into feature embeddings.
  2. Instruction Encoder: Employs BERT to encode natural language commands into task-specific embeddings, guiding the model's focus.
  3. Unified Task Solver: A Transformer-based encoder-decoder that integrates visual and linguistic embeddings to generate output sequences.
The HuBo-VLM architecture integrates image and instruction encoders into a unified Transformer-based task solver.

A key innovation in HuBo-VLM is the elimination of traditional object detection "heads." Instead of complex regression losses, the model outputs coordinates directly as a string of tokens (e.g., x0, y0, x1, y1), which are then post-processed into bounding boxes. This approach simplifies the pipeline and enables end-to-end training.

Training and Evaluation

The model's development involved a two-stage process:

  • Unified Pre-training: Leveraging large-scale multi-modal datasets such as Conceptual Captions (CC12M/CC3M), MSCOCO, Visual Genome, and ImageNet-21K to build a robust foundation.
  • Fine-tuning: Specialized training on the Talk2Car benchmark, which contains nearly 12,000 natural language commands mapped to the nuScenes autonomous driving dataset.

Results and Discussion

HuBo-VLM achieved state-of-the-art performance on the Talk2Car benchmark with an AP50 of 76.74, significantly outperforming previous models like Deformable-MDETR (74.4) and Stacked VLBert (71).

Beyond quantitative metrics, the model demonstrates superior qualitative reasoning. It successfully interprets complex spatial logic and relative positioning, which are often stumbling blocks for traditional vision systems.

HuBo-VLM successfully identifies a target car by reasoning through the multi-step instruction: "park behind the car in front of the mail truck."

Key Insights

  • Instruction-Driven Scalability: The dedicated instruction encoder allows the model to adapt to new tasks via text prompts without architectural changes.
  • Language-Centric Reasoning: By utilizing language-style architectures, the model better handles the logical flow required for complex HRI tasks.
  • Operational Simplicity: Removing ROI dependency makes the model more suitable for real-time robotics applications.

Conclusion and Future Work

While HuBo-VLM sets a new benchmark for HRI perception, future work will focus on replacing the BERT-based encoder with a Large Language Model (LLM) like LLaMA to further enhance reasoning. Additionally, the authors aim to expand the model's generalist capabilities across broader robotic tasks while maintaining efficiency for embedded hardware deployment.

Summary updated

Abstract

Human robot interaction is an exciting task, which aimed to guide robots following instructions from human. Since huge gap lies between human natural language and machine codes, end to end human robot interaction models is fair challenging. Further, visual information receiving from sensors of robot is also a hard language for robot to perceive. In this work, HuBo-VLM is proposed to tackle perception tasks associated with human robot interaction including object detection and visual grounding by a unified transformer based vision language model. Extensive experiments on the Talk2Car benchmark demonstrate the effectiveness of our approach. Code would be publicly available in https://github.com/dzcgaara/HuBo-VLM.
BibTeX
@misc{dong2023hubo,
  author = {Zichao Dong and Weikun Zhang and Xufeng Huang and Hang Ji and Xin Zhan and Junbo Chen},
  title = {{HuBo-VLM: Unified Vision-Language Model designed for HUman roBOt interaction tasks}},
  year = {2023},
  eprint = {2308.12537},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.12537},
}
Figure