A Single Multi-Task Deep Neural Network with Post-Processing for Object Detection with Reasoning and Robotic Grasp Detection

Dongwon Park, Yonghyeok Seo, Dongju Shin, Jaesik Choi, Se Young Chun

Abstract

Recently, robotic grasp detection (GD) and object detection (OD) with reasoning have been investigated using deep neural networks (DNNs). There have been works to combine these multi-tasks using separate networks so that robots can deal with situations of grasping specific target objects in the cluttered, stacked, complex piles of novel objects from a single RGB-D camera. We propose a single multi-task DNN that yields the information on GD, OD and relationship reasoning among objects with a simple post-processing. Our proposed methods yielded state-of-the-art performance with the accuracy of 98.6% and 74.2% and the computation speed of 33 and 62 frame per second on VMRD and Cornell datasets, respectively. Our methods also yielded 95.3% grasp success rate for single novel object grasping with a 4-axis robot arm and 86.7% grasp success rate in cluttered novel objects with a Baxter robot.

Introduction

Robotic grasping in cluttered environments—such as a bin of tools or a pile of household items—presents a significant challenge. Beyond simply identifying a grasp point, a robot must understand the scene's hierarchy. If Object A is stacked atop Object B, the robot must prioritize grasping A first to prevent collisions or damage to the underlying items.

Historically, this has been addressed through modular pipelines involving separate networks for detection, grasp planning, and relationship reasoning. However, these multi-stage systems are often computationally expensive and suffer from "cascading errors," where a failure in the initial detection stage ruins the entire process. This paper introduces a unified, single deep neural network that simultaneously performs object detection, robotic grasp detection, and relationship reasoning, achieving real-time performance without sacrificing accuracy.

Methodology

The proposed architecture is built upon the Darknet-53 backbone, extending the YOLOv3 framework to handle multi-task outputs through a single forward pass.

Multi-Scale Architecture

The network generates predictions at three distinct resolutions ($1\times, 2\times, \text{ and } 4\times$ up-sampling). This multi-scale approach ensures the model can detect both large objects and fine-grained features, such as the narrow handles of scissors or small tools, which are critical for precise grasping.

The proposed multi-task DNN architecture based on Darknet-53, featuring multi-scale predictions and post-processing for unified output.

Unified Representation and Loss

The standard bounding box is expanded into a 15-dimensional vector. This vector encapsulates:

  • Object bounding box coordinates and class.
  • Grasp parameters (coordinates, gripper width, and angle).
  • Stacking relationships (identifying "Father" and "Child" classes for grid cells).

To address the inherent class imbalance in robotic datasets (e.g., many more notebooks than chargers), the authors employed a custom loss function combining Mean Squared Error for spatial coordinates and Binary Focal Loss for classification, forcing the model to focus on harder-to-learn examples.

Experimental Results

The model was evaluated on the Cornell Grasping Dataset and the Visual Manipulation Relationship Dataset (VMRD), which contains complex scenes of stacked objects.

Quantitative Performance

The system achieved state-of-the-art results, reaching 98.6% accuracy on the Cornell dataset and 74.2% mAP on the VMRD dataset. Crucially, the unified architecture allows for real-time inference, running at 33 FPS for high-resolution $608 \times 608$ images and up to 62 FPS at lower resolutions.

Visual results of object detection, grasp detection, and relationship reasoning, alongside a performance comparison graph showing the speed-accuracy trade-off.

Real-World Validation

Physical testing was conducted using a 4-axis Dobot Magician and a 7-axis Baxter robot. The system demonstrated a 95.3% grasp success rate for single novel objects and an 86.7% success rate in cluttered environments, proving its robustness in practical scenarios.

Discussion and Conclusion

A key insight from the study is that the network is more proficient at identifying the "Child" (top) object than the "Father" (bottom) object in a stack. By prioritizing Child Class predictions during post-processing, the system effectively builds a relationship graph for sequential grasping. This enables "search-and-clear" strategies where a robot can systematically remove occluding items to reach a hidden target.

While highly effective, performance can degrade in extremely dense piles where bottom objects are almost entirely occluded. Future research will explore integrating temporal data or active vision to resolve these complex 3D ambiguities.

Summary updated

Abstract

Recently, robotic grasp detection (GD) and object detection (OD) with reasoning have been investigated using deep neural networks (DNNs). There have been works to combine these multi-tasks using separate networks so that robots can deal with situations of grasping specific target objects in the cluttered, stacked, complex piles of novel objects from a single RGB-D camera. We propose a single multi-task DNN that yields the information on GD, OD and relationship reasoning among objects with a simple post-processing. Our proposed methods yielded state-of-the-art performance with the accuracy of 98.6% and 74.2% and the computation speed of 33 and 62 frame per second on VMRD and Cornell datasets, respectively. Our methods also yielded 95.3% grasp success rate for single novel object grasping with a 4-axis robot arm and 86.7% grasp success rate in cluttered novel objects with a Baxter robot.
BibTeX
@inproceedings{park2020single,
  author = {Dongwon Park and Yonghyeok Seo and Dongju Shin and Jaesik Choi and Se Young Chun},
  title = {{A Single Multi-Task Deep Neural Network with Post-Processing for Object Detection with Reasoning and Robotic Grasp Detection}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  url = {https://openreview.net/forum?id=8ly6ds8OAmX},
}
Figure