OVGNet: A Unified Visual-Linguistic Framework for Open-Vocabulary Robotic Grasping

Meng Li, Qi Zhao, Shuchang Lyu, Chunlei Wang, Yujing Ma, Guangliang Cheng, Chenguang YangIROS 2024 · DOI · OpenReview

TL;DR

This paper introduces OVGNet, a novel framework that enables robots to grasp objects they've never seen before, using natural language descriptions. They achieve this by developing a new large-scale dataset for open-vocabulary grasping and a unified visual-linguistic model that leverages foundation models and fine-tuning on base categories.

What Problem Does This Paper Address?

Robots are increasingly expected to perform tasks in diverse, real-world environments, such as warehouses or homes. A major hurdle is their inability to handle "novel" or "unseen" objects – items not present in their training data. Traditional robotic grasping systems are limited to recognizing a predefined set of categories, making them impractical for dynamic environments where new objects are constantly introduced. While open-vocabulary learning (OVL) has shown promise in object detection and segmentation, its application to robotic grasping, particularly for novel objects, remains largely unexplored. This paper aims to bridge this gap by enabling robots to understand and grasp objects based on natural language descriptions, even if those objects are entirely new to them.

Methods

OVGNet is built upon two core components: a visual-linguistic perception system and a grasping system.

  • Visual-Linguistic Perception System: This system is inspired by models like GroundingDINO and is designed to locate target objects based on image and text inputs. It extracts features from both modalities and uses a cross-modality encoder. Crucially, it introduces two novel alignment modules:
  • Image Guided Language Attention (IGLA): This module enhances text features by aligning them with image features using multi-head attention.
  • Language Guided Image Attention (LGIA): This module refines image features by paying closer attention to regions described in the language, improving comprehension. These modules work together to improve the consistency between visual and linguistic features, leading to better generalization for novel objects. The system outputs a bounding box for the target object.

  • Grasping System: Once the target object is located, this system generates grasping poses. It takes the bounding box from the perception system to segment the relevant point cloud of the target object. This segmented point cloud is then fed into a pre-trained GraspNet model to generate 6-DOF (six degrees of freedom) grasping poses. The system further filters these poses using grasp scores and angles, and finally selects the best grasp pose based on the Euclidean distance to the bounding box center.

Datasets

The authors introduce a new benchmark dataset called OVGrasping.

  • OVGrasping Dataset: This dataset comprises 63,385 instances across 117 categories, strategically divided into "base" (seen) and "novel" (unseen) categories. It's constructed by reorganizing data from RoboRefIt and GraspNet, and incorporating newly collected simulated data. A key feature is the inclusion of subject-object relationships in the language descriptions, allowing for more precise localization, especially when dealing with identical objects.

Key Insights and Ideas

  • Bridging the Gap: The core insight is that by combining open-vocabulary learning with robotic grasping, robots can move beyond their pre-trained object sets and adapt to new objects in real-time.
  • Dataset is Crucial: The creation of the OVGrasping dataset, with its explicit division of base and novel categories and detailed language annotations, is a significant contribution for advancing research in this area.
  • Attention is Key for Generalization: The IGLA and LGIA modules highlight the importance of fine-grained attention mechanisms for aligning visual and linguistic information, which is critical for the model to generalize to unseen objects.
  • Leveraging Foundation Models: The framework effectively leverages pre-trained foundation models (like GroundingDINO) and fine-tunes them on base categories to achieve strong performance on novel categories, demonstrating a powerful transfer learning strategy.

Key Results

  • Perceptual Performance: OVGNet achieves an average accuracy of 71.2% for base objects and 64.4% for novel objects in grasping tasks.
  • Improved Novel Object Grasping: Compared to VL-Grasp, OVGNet shows a significant improvement of 47.38% in novel category perception.
  • Grasping Success Rate: In a virtual environment, OVGNet achieves a remarkable grasping success rate, reaching 71.2% for base objects and 64.4% for novel objects within three attempts.
  • Ablation Studies: The proposed IGLA and LGIA modules individually contribute to performance improvements, with their combined use leading to the best results.

Limitations and Future Work

  • Limitations:
    • The paper notes that OVGNet's performance in the base category is marginally lower than VL-Grasp, potentially due to fewer fine-tuning epochs.
  • Grasping failures can occur if the visual-linguistic perception system makes an incorrect detection.
  • Occlusion and irregular object shapes can interfere with accurate grasping pose generation.
  • Future Work: The authors aim to continue exploring the open-vocabulary grasping task, with a long-term goal of designing an end-to-end open-vocabulary grasping system.

Summary updated

Abstract

Recognizing and grasping novel-category objects remains a crucial yet challenging problem in real-world robotic applications. Despite its significance, limited research has been conducted in this specific domain. To address this, we seamlessly propose a novel framework that integrates open-vocabulary learning into the domain of robotic grasping, empowering robots with the capability to adeptly handle novel objects. Our contributions are threefold. Firstly, we present a large-scale benchmark dataset specifically tailored for evaluating the performance of open-vocabulary grasping tasks. Secondly, we propose a unified visual-linguistic framework that serves as a guide for robots in successfully grasping both base and novel objects. Thirdly, we introduce two alignment modules designed to enhance visual-linguistic perception in the robotic grasping process. Extensive experiments validate the efficacy and utility of our approach. Notably, our framework achieves an average accuracy of 71.2\% and 64.4\% on base and novel categories in our new dataset, respectively.
BibTeX
@inproceedings{li2024ovgnet,
  author = {Meng Li and Qi Zhao and Shuchang Lyu and Chunlei Wang and Yujing Ma and Guangliang Cheng and Chenguang Yang},
  title = {{OVGNet: A Unified Visual-Linguistic Framework for Open-Vocabulary Robotic Grasping}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10802654},
}
Figure