A Multi-task Convolutional Neural Network for Autonomous Robotic Grasping in Object Stacking Scenes

Hanbo Zhang, Xuguang Lan, Site Bai, Lipeng Wan, Chenjie Yang, Nanning Zheng

TL;DR

Researchers have developed a multi-task convolutional neural network (CNN) that allows robots to grasp specific objects buried in cluttered stacks. By integrating object detection, grasp planning, and relationship reasoning into a single model, the system can "think" through the order of operations needed to uncover and retrieve a target object.

What Problem Does This Paper Address?

While robots are becoming adept at picking up isolated objects, "object stacking scenes"—like a messy desk or a bin of parts—remain a major hurdle. There are two primary challenges:

  1. The "What and How": The robot must simultaneously identify the target and determine the physical orientation of a successful grasp.
  2. The "Hidden Target": In a stack, the target is often partially or completely obscured. Grasping it directly might damage other objects or fail because the target is pinned down.

Previous solutions often treated these as separate problems, leading to slow, redundant computations. This paper aims to create a unified, efficient system that can perceive the scene and reason about the "manipulation relationships" (which object is on top of which) to plan a sequence of moves.

Methods

The authors propose a vision-based multi-task CNN architecture based on ResNet-101. The network is divided into three functional modules:

  • Perception: This part uses a Region Proposal Network (RPN) to identify objects. Crucially, it performs Grasp Detection on the features of individual objects rather than the whole scene. It outputs "oriented anchor boxes" that suggest how a gripper should align with an object.
  • Reasoning: This module predicts the visual manipulation relationships between pairs of objects. It builds a Manipulation Relationship Tree, where "leaf nodes" represent objects that are not covered by anything else and are safe to move first.
  • Grasping: This is the execution layer. It selects the best grasp candidate, transforms image coordinates into 3D robot coordinates using depth data (from a Kinect sensor), and executes the physical move.

The entire network is trained end-to-end using a multi-task loss function that balances object detection, grasp accuracy, and relationship prediction.

Datasets

The model was trained and validated using the VMRD (Visual Manipulation Relationship Dataset).

  • Scale: 4,683 images containing over 100k labeled grasps.
  • Complexity: Images feature 2-5 objects stacked with varying degrees of overlap and occlusion.
  • Labels: Each image includes object bounding boxes, grasp rectangles, and a manipulation relationship tree.

Key Insights and Ideas

  • Object-Centric Grasping: By detecting grasps on specific object features rather than the entire image, the model avoids getting "confused" by dense clutter.
  • The Relationship Tree: The core "aha moment" is treating unstacking as a tree-search problem. By identifying which objects are "parents" (covering others) and which are "leaves" (free to move), the robot can systematically clear a path to a hidden target.
  • Shared Features: Using a single backbone (ResNet-101) for both detection and reasoning significantly reduces redundant calculations, making the system faster and more suitable for real-time robotics.

Key Results

The system was tested on a physical Baxter robot across three scenarios:

  • Object Cluttered Scenes: 90.6% success rate.
  • Familiar Stacking Scenes (2-5 objects): 71.9% success rate.
  • Complex Stacking Scenes (6-9 objects): 59.4% success rate.

In all cases, the proposed multi-task network significantly outperformed the baseline (VMRN), particularly in complex stacks where the baseline's success rate dropped to a mere 6.3%. The model runs at roughly 6.5 to 7.6 FPS, providing a good balance between accuracy and speed.

Limitations and Future Work

Despite the improvements, the authors noted several failure modes:

  • Extreme Occlusion: If an object is almost entirely covered, the detector may fail to find it or assign it a low confidence score.
  • Grasp Confusion: In very dense stacks, the model sometimes struggles to distinguish which grasp belongs to which object when they overlap significantly.
  • Physical Constraints: Occasionally, the robot might plan a correct grasp that is physically impossible for its gripper to execute due to the size of the object.

Future work will focus on improving detection in high-overlap scenarios and refining the coordination between the vision system and the robot's physical constraints.

Summary updated

Abstract

Autonomous robotic grasping plays an important role in intelligent robotics. However, how to help the robot grasp specific objects in object stacking scenes is still an open problem, because there are two main challenges for autonomous robots: (1) it is a comprehensive task to know what and how to grasp; (2) it is hard to deal with the situations in which the target is hidden or covered by other objects. In this paper, we propose a multi-task convolutional neural network for autonomous robotic grasping, which can help the robot find the target, make the plan for grasping and finally grasp the target step by step in object stacking scenes. We integrate vision-based robotic grasping detection and visual manipulation relationship reasoning in one single deep network and build the autonomous robotic grasping system. Experimental results demonstrate that with our model, Baxter robot can autonomously grasp the target with a success rate of 90.6%, 71.9% and 59.4% in object cluttered scenes, familiar stacking scenes and complex stacking scenes respectively.
BibTeX
@inproceedings{zhang2019multi,
  author = {Hanbo Zhang and Xuguang Lan and Site Bai and Lipeng Wan and Chenjie Yang and Nanning Zheng},
  title = {{A Multi-task Convolutional Neural Network for Autonomous Robotic Grasping in Object Stacking Scenes}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  doi = {10.1109/iros40897.2019.8967977},
}

Loading related content...

Figure