A Multi-task Convolutional Neural Network for Autonomous Robotic Grasping in Object Stacking Scenes

Hanbo Zhang, Xuguang Lan, Site Bai, Lipeng Wan, Chenjie Yang, Nanning Zheng

Abstract

Autonomous robotic grasping in stacked scenes faces two major hurdles: identifying what and how to grasp, and managing hidden or covered targets. This paper introduces a multi-task convolutional neural network (CNN) that integrates vision-based grasp detection and visual manipulation relationship reasoning. By planning a sequence of moves based on a predicted relationship tree, the system enables a Baxter robot to achieve high success rates in cluttered (90.6%) and complex stacking (59.4%) environments.

What Problem Does This Paper Address?

While robots are becoming adept at picking up isolated objects, "object stacking scenes"—like a messy desk or a bin of parts—remain a major hurdle. There are two primary challenges:

  1. The "What and How": The robot must simultaneously identify the target and determine the physical orientation of a successful grasp.
  2. The "Hidden Target": In a stack, the target is often partially or completely obscured. Grasping it directly might damage other objects or fail because the target is pinned down.
Figure 1: An example of an object stacking scene where the target (tape) is buried. The system generates a Manipulation Relationship Tree to determine the order of removal.

Previous solutions often treated these as separate problems, leading to slow, redundant computations. This paper aims to create a unified, efficient system that can perceive the scene and reason about the "manipulation relationships" (which object is on top of which) to plan a sequence of moves.

Methods

The authors propose a vision-based multi-task CNN architecture based on ResNet-101. The network is divided into three functional modules:

  • Perception: This part uses a Region Proposal Network (RPN) to identify objects. Crucially, it performs Grasp Detection on the features of individual objects rather than the whole scene. It outputs "oriented anchor boxes" that suggest how a gripper should align with an object.
  • Reasoning: This module predicts the visual manipulation relationships between pairs of objects. It builds a Manipulation Relationship Tree, where "leaf nodes" represent objects that are not covered by anything else and are safe to move first.
  • Grasping: This is the execution layer. It selects the best grasp candidate, transforms image coordinates into 3D robot coordinates using depth data (from a Kinect sensor), and executes the physical move.
Figure 5: The multi-task CNN architecture, showing the integration of perception, reasoning, and grasping modules within a single deep network.

The entire network is trained end-to-end using a multi-task loss function that balances object detection, grasp accuracy, and relationship prediction.

Datasets

The model was trained and validated using the VMRD (Visual Manipulation Relationship Dataset).

  • Scale: 4,683 images containing over 100k labeled grasps.
  • Complexity: Images feature 2-5 objects stacked with varying degrees of overlap and occlusion.
  • Labels: Each image includes object bounding boxes, grasp rectangles, and a manipulation relationship tree.

Key Results

The system was tested on a physical Baxter robot across three scenarios:

  • Object Cluttered Scenes: 90.6% success rate.
  • Familiar Stacking Scenes (2-5 objects): 71.9% success rate.
  • Complex Stacking Scenes (6-9 objects): 59.4% success rate.

In all cases, the proposed multi-task network significantly outperformed the baseline (VMRN), particularly in complex stacks where the baseline's success rate dropped to a mere 6.3%. The model runs at roughly 6.5 to 7.6 FPS, providing a good balance between accuracy and speed.

Limitations and Future Work

Despite the improvements, the authors noted several failure modes:

  • Extreme Occlusion: If an object is almost entirely covered, the detector may fail to find it or assign it a low confidence score.
  • Grasp Confusion: In very dense stacks, the model sometimes struggles to distinguish which grasp belongs to which object when they overlap significantly.
  • Physical Constraints: Occasionally, the robot might plan a correct grasp that is physically impossible for its gripper to execute due to the size of the object.

Future work will focus on improving detection in high-overlap scenarios and refining the coordination between the vision system and the robot's physical constraints.

Summary updated

Abstract

Autonomous robotic grasping plays an important role in intelligent robotics. However, how to help the robot grasp specific objects in object stacking scenes is still an open problem, because there are two main challenges for autonomous robots: (1)it is a comprehensive task to know what and how to grasp; (2)it is hard to deal with the situations in which the target is hidden or covered by other objects. In this paper, we propose a multi-task convolutional neural network for autonomous robotic grasping, which can help the robot find the target, make the plan for grasping and finally grasp the target step by step in object stacking scenes. We integrate vision-based robotic grasping detection and visual manipulation relationship reasoning in one single deep network and build the autonomous robotic grasping system. Experimental results demonstrate that with our model, Baxter robot can autonomously grasp the target with a success rate of 90.6%, 71.9% and 59.4% in object cluttered scenes, familiar stacking scenes and complex stacking scenes respectively.
BibTeX
@inproceedings{zhang2019multi,
  author = {Hanbo Zhang and Xuguang Lan and Site Bai and Lipeng Wan and Chenjie Yang and Nanning Zheng},
  title = {{A Multi-task Convolutional Neural Network for Autonomous Robotic Grasping in Object Stacking Scenes}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  doi = {10.1109/iros40897.2019.8967977},
}
Figure