A Single Multi-Task Deep Neural Network with Post-Processing for Object Detection with Reasoning and Robotic Grasp Detection

Dongwon Park, Yonghyeok Seo, Dongju Shin, Jaesik Choi, Se Young Chun

TL;DR

This paper introduces a unified, single deep neural network that simultaneously performs object detection, robotic grasp detection, and relationship reasoning (determining which objects are stacked on others). By extending the YOLOv3 architecture, the authors achieved state-of-the-art accuracy and real-time speeds (up to 62 FPS), enabling robots to efficiently clear clutter and retrieve specific targets.

What Problem Does This Paper Address?

Robotic grasping in "cluttered" environments—like a pile of tools or a messy bin—is notoriously difficult. A robot can't just find a handle; it needs to understand the scene's hierarchy. If Object A is resting on Object B, the robot must grasp A first to avoid a collision or damaging the items.

Traditionally, researchers solved this using modular pipelines: one network for detection, another for grasp planning, and a third for "reasoning" about stacking order. These modular systems are often slow, memory-intensive, and prone to "cascading errors" where a mistake in the first step ruins the entire process. The authors sought to create a single, streamlined network that handles all these tasks at once.

Methods

The authors built their system on Darknet-53 (the backbone of YOLOv3) and introduced several key innovations:

  • 15D Representation: They expanded the standard bounding box to a 15-dimensional vector that includes grasp coordinates, gripper width/angle, object class, and stacking order.
  • Multi-Scale Predictions: The network makes predictions at three different resolutions ($1\times, 2\times, \text{ and } 4\times$ up-sampling). This allows the robot to detect both large objects and fine details, like the thin handles of scissors.
  • Relationship Reasoning: Instead of just detecting boxes, the network predicts "Father" and "Child" classes for each grid cell. This identifies which object is "over" or "under" another, allowing the robot to build a "relation graph" for sequential grasping.
  • Custom Loss Function: They utilized a complex loss function combining Mean Squared Error (for coordinates) and Binary Focal Loss (to handle the class imbalance common in robotic datasets).

Datasets

The model was trained and evaluated on two primary benchmarks:

  1. Cornell Grasping Dataset: A standard dataset featuring single objects with multiple labeled "good" grasps.
  2. VMRD (Visual Manipulation Relationship Dataset): A more complex dataset containing 4,233 images of stacked objects (2–5 per image) with labels for object classes and their spatial relationships.

The authors also conducted real-world testing using a 4-axis Dobot Magician and a 7-axis Baxter robot.

Key Insights and Ideas

  • Unified > Modular: Moving from a modular "RPN + three separate DNNs" approach to a single multi-task network improved speed from ~6 FPS to over 33 FPS without sacrificing accuracy.
  • Focal Loss is Critical: Robotic datasets are often unbalanced (e.g., the VMRD has 2,061 notebooks but only 93 chargers). Using Focal Loss helped the model focus on "hard" examples rather than getting lazy on the common ones.
  • "Child" is easier than "Father": The authors discovered that the network was much more accurate at identifying which object was on top (Child Class) than which was underneath (Father Class). Consequently, they optimized their reasoning post-processing to rely primarily on the Child Class predictions.
  • The "Invisible" Scene Strategy: By combining reasoning with detection, the robot can perform "search-and-clear" maneuvers. If a target is hidden, the robot uses the relationship graph to remove occluding objects one by one until the target is revealed.

Key Results

  • Accuracy: Achieved 98.6% accuracy on the Cornell dataset and 74.6% mAPg (mean Average Precision for grasping) on the VMRD dataset, setting new state-of-the-art benchmarks at the time.
  • Speed: The system runs at 33 FPS for high-resolution images ($608 \times 608$) and up to 62 FPS for lower resolutions ($320 \times 320$), making it highly suitable for real-time robotic control.
  • Real-World Success: In physical tests, the robot achieved a 95.3% grasp success rate for single novel objects and an 86.7% success rate in cluttered environments.

Limitations and Future Work

While the system is fast and accurate, the authors noted that as the number of objects in a pile increases, the grasp success rate tends to drop. This is largely due to severe occlusions where the "Father Class" (the bottom object) becomes nearly impossible to see or predict accurately.

Future work could involve integrating temporal data (video) or active vision (moving the camera) to better understand complex 3D stacks that are partially hidden from a single top-down view.

Summary updated

Abstract

Applications of deep neural network (DNN) based object and grasp detections could be expanded significantly when the network output is processed by a high-level reasoning over relationship of objects. Recently, robotic grasp detection and object detection with reasoning have been investigated using DNNs. There have been efforts to combine these multitasks using separate networks so that robots can deal with situations of grasping specific target objects in the cluttered, stacked, complex piles of novel objects from a single RGB-D camera. We propose a single multi-task DNN that yields accurate detections of objects, grasp position and relationship reasoning among objects. Our proposed methods yield state-of-the-art performance with the accuracy of 98.6% and 74.2% with the computation speed of 33 and 62 frame per second on VMRD and Cornell datasets, respectively. Our methods also yielded 95.3% grasp success rate for novel object grasping tasks with a 4-axis robot arm and 86.7% grasp success rate in cluttered novel objects with a humanoid robot.
BibTeX
@inproceedings{park2020single,
  author = {Dongwon Park and Yonghyeok Seo and Dongju Shin and Jaesik Choi and Se Young Chun},
  title = {{A Single Multi-Task Deep Neural Network with Post-Processing for Object Detection with Reasoning and Robotic Grasp Detection}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  url = {https://openreview.net/forum?id=8ly6ds8OAmX},
}

Loading related content...

Figure