Abstract
Autonomous robotic grasping plays an important role in intelligent robotics. However, how to help the robot grasp specific objects in object stacking scenes is still an open problem, because there are two main challenges for autonomous robots: (1)it is a comprehensive task to know what and how to grasp; (2)it is hard to deal with the situations in which the target is hidden or covered by other objects. In this paper, we propose a multi-task convolutional neural network for autonomous robotic grasping, which can help the robot find the target, make the plan for grasping and finally grasp the target step by step in object stacking scenes. We integrate vision-based robotic grasping detection and visual manipulation relationship reasoning in one single deep network and build the autonomous robotic grasping system. Experimental results demonstrate that with our model, Baxter robot can autonomously grasp the target with a success rate of 90.6%, 71.9% and 59.4% in object cluttered scenes, familiar stacking scenes and complex stacking scenes respectively.
Introduction
Robotic grasping in cluttered or stacked environments remains a significant challenge. While many systems can pick up isolated objects, they often fail when a specific target is buried beneath others. To address this, a robot requires more than simple detection; it needs the ability to reason about the physical hierarchy of a scene—understanding, for instance, that a stapler must be moved before a roll of tape can be accessed. Traditional approaches often rely on separate, slow pipelines for detection and reasoning. This work proposes a unified, real-time solution that integrates perception and logic into a single end-to-end framework.
Methodology
The proposed architecture, RPRG, is built upon a ResNet-101 backbone and performs three primary tasks simultaneously:
- Perception: A Region Proposal Network (RPN) identifies object bounding boxes and classes. A specialized Grasp Detector predicts oriented grasp rectangles (position, width, height, and angle) for each detected object.
- Reasoning: An Object Pairing Pooling Layer (OP2L) extracts features from pairs of objects to predict their spatial relationships (e.g., "A is on top of B"). These relationships form a Manipulation Relationship Tree.
- Grasping: A non-trainable module selects the optimal grasp for the target and converts image coordinates into real-world motor commands.
The network is trained using the Visual Manipulation Relationship Dataset (VMRD), which contains 4,683 images with over 100,000 labeled grasps and explicit "parent-child" relationship labels.
Visual Relationship Reasoning
The core innovation lies in the Manipulation Relationship Tree. By identifying "leaf nodes"—objects with no other items resting on them—the system determines a safe sequence of moves to uncover a buried target. This allows the robot to handle occlusions by moving obstacles in the correct order.
Experimental Results
The system was evaluated on a Baxter robot equipped with a Kinect v2 sensor. RPRG demonstrated significant improvements over the VMRN baseline across various scene complexities:
- Cluttered Scenes: 90.6% success rate.
- Familiar Stacking: 71.9% success rate.
- Complex Stacking: 59.4% success rate.
The model successfully executed "human-like" strategies, such as clearing a screwdriver and pliers to reach a buried roll of tape.
Conclusion
RPRG provides a robust framework for real-time robotic manipulation in complex environments. While performance decreases in extremely dense scenes with heavy occlusion, the integration of grasp-object affiliation and relationship reasoning marks a significant step toward autonomous robotic intelligence. Future work will explore deeper integration of depth information to improve accuracy in highly packed environments.
Summary updated