NBMOD: Find It and Grasp It in Noisy Background

Boyuan Cao, Xinyu Zhou, Congmin Guo, Baohua Zhang, Yuchen Liu, Qianqiu Tan

Introduction

In robotic manipulation, grasp detection—the process of identifying the optimal position and orientation for a gripper—is a fundamental challenge. While existing models have achieved high performance on standard benchmarks like the Cornell or Jacquard datasets, these successes often fail to translate to real-world settings. Traditional datasets typically feature single objects against pristine, high-contrast backgrounds. In contrast, real-world industrial and domestic environments are "noisy," filled with shadows, overlapping items, and distracting clutter. This paper introduces NBMOD (Noisy Background Multi-Object Dataset) and a novel Rotation Anchor Mechanism (RAM) to bridge this "sim-to-real" gap.

The Rotation Anchor Mechanism (RAM)

A significant hurdle in grasp detection is predicting the exact rotation angle (θ) of a grasp. Standard neural networks often struggle with the vast hypothesis space of possible orientations. To address this, the researchers developed the Rotation Anchor Mechanism (RAM).

RAM functions by partitioning the input image into a grid. Each cell in this grid is assigned a set of "anchors" representing predefined angles. Instead of predicting an absolute angle, the model learns to predict the offset from the nearest anchor. This inductive bias significantly narrows the search space, leading to more stable training and higher precision in oriented grasping.

RA-GraspNet Architectures

To leverage RAM, the authors proposed three lightweight RA-GraspNet architectures, each balancing global context with local feature extraction:

  • RARA (Region Attention): Utilizes a convolutional decoder with a specialized attention mechanism to focus on relevant grid areas.
  • RAST (Semi-Transformer): A hybrid model that integrates Transformer blocks within a CNN framework to capture long-range dependencies.
  • RAGT (Global Transformer): A model designed for maximum scene context, processing downsampled feature maps to maintain high speed while understanding the entire environment.

The NBMOD Dataset

The core contribution of this work is the NBMOD dataset, comprising 31,500 RGB-D images. It is specifically designed to test robustness against background noise and object density.

Samples from the NBMOD dataset showcasing the progression from simple single-object scenes to complex, noisy multi-object environments.

The dataset is divided into three distinct categories:

  1. SSS (Simple Single-object): 13,500 images with clean backgrounds, serving as a baseline.
  2. NSS (Noisy Single-object): 13,000 images where office supplies and other clutter surround the target object.
  3. MOS (Multi-Object): 5,000 complex scenes containing multiple target objects amidst significant noise.

Key Results and Performance

The proposed models demonstrate exceptional performance and efficiency. The RAGT-3/3 variant achieved a peak accuracy of 99% on the NBMOD dataset. Furthermore, the models are optimized for real-time industrial use; the RAGT-2/3 version can process up to 844 images per second.

Grasp detection results on the NBMOD dataset, illustrating the model's ability to identify optimal grasp points despite significant background clutter.

Beyond raw accuracy, the models proved robust under strict evaluation metrics, maintaining high correctness even when requiring minimal angular deviation and high Intersection over Union (IoU) scores. This stability is largely attributed to the anchor-based approach, which provides the necessary geometric constraints for complex grasping tasks.

Conclusion and Future Directions

NBMOD and the RA-GraspNet family represent a significant step toward reliable robotic grasping in unstructured environments. While currently limited to a closed set of 20 fruit types, the success of the Rotation Anchor Mechanism provides a blueprint for future "open-vocabulary" grasping. Future work aims to integrate multimodal foundation models like CLIP or SAM, enabling robots to grasp novel objects based on visual cues or natural language instructions.

Summary updated

Abstract

Grasping objects is a fundamental yet important capability of robots, and many tasks such as sorting and picking rely on this skill. The prerequisite for stable grasping is the ability to correctly identify suitable grasping positions. However, finding appropriate grasping points is challenging due to the diverse shapes, varying density distributions, and significant differences between the barycenter of various objects. In the past few years, researchers have proposed many methods to address the above-mentioned issues and achieved very good results on publicly available datasets such as the Cornell dataset and the Jacquard dataset. The problem is that the backgrounds of Cornell and Jacquard datasets are relatively simple - typically just a whiteboard, while in real-world operational environments, the background could be complex and noisy. Moreover, in real-world scenarios, robots usually only need to grasp fixed types of objects. To address the aforementioned issues, we proposed a large-scale grasp detection dataset called NBMOD: Noisy Background Multi-Object Dataset for grasp detection, which consists of 31,500 RGB-D images of 20 different types of fruits. Accurate prediction of angles has always been a challenging problem in the detection task of oriented bounding boxes. This paper presents a Rotation Anchor Mechanism (RAM) to address this issue. Considering the high real-time requirement of robotic systems, we propose a series of lightweight architectures called RA-GraspNet (GraspNet with Rotation Anchor): RARA (network with Rotation Anchor and Region Attention), RAST (network with Rotation Anchor and Semi Transformer), and RAGT (network with Rotation Anchor and Global Transformer) to tackle this problem. Among them, the RAGT-3/3 model achieves an accuracy of 99% on the NBMOD dataset. The NBMOD and our code are available at https://github.com/kmittle/Grasp-Detection-NBMOD.
BibTeX
@misc{cao2023nbmod,
  author = {Boyuan Cao and Xinyu Zhou and Congmin Guo and Baohua Zhang and Yuchen Liu and Qianqiu Tan},
  title = {{NBMOD: Find It and Grasp It in Noisy Background}},
  year = {2023},
  eprint = {2306.10265},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.10265},
}
Figure