Abstract
Autonomous robotic grasping in stacked scenes faces two major hurdles: identifying what and how to grasp, and managing hidden or covered targets. This paper introduces a multi-task convolutional neural network (CNN) that integrates vision-based grasp detection and visual manipulation relationship reasoning. By planning a sequence of moves based on a predicted relationship tree, the system enables a Baxter robot to achieve high success rates in cluttered (90.6%) and complex stacking (59.4%) environments.
What Problem Does This Paper Address?
While robots are becoming adept at picking up isolated objects, "object stacking scenes"—like a messy desk or a bin of parts—remain a major hurdle. There are two primary challenges:
- The "What and How": The robot must simultaneously identify the target and determine the physical orientation of a successful grasp.
- The "Hidden Target": In a stack, the target is often partially or completely obscured. Grasping it directly might damage other objects or fail because the target is pinned down.
Previous solutions often treated these as separate problems, leading to slow, redundant computations. This paper aims to create a unified, efficient system that can perceive the scene and reason about the "manipulation relationships" (which object is on top of which) to plan a sequence of moves.
Methods
The authors propose a vision-based multi-task CNN architecture based on ResNet-101. The network is divided into three functional modules:
- Perception: This part uses a Region Proposal Network (RPN) to identify objects. Crucially, it performs Grasp Detection on the features of individual objects rather than the whole scene. It outputs "oriented anchor boxes" that suggest how a gripper should align with an object.
- Reasoning: This module predicts the visual manipulation relationships between pairs of objects. It builds a Manipulation Relationship Tree, where "leaf nodes" represent objects that are not covered by anything else and are safe to move first.
- Grasping: This is the execution layer. It selects the best grasp candidate, transforms image coordinates into 3D robot coordinates using depth data (from a Kinect sensor), and executes the physical move.
The entire network is trained end-to-end using a multi-task loss function that balances object detection, grasp accuracy, and relationship prediction.
Datasets
The model was trained and validated using the VMRD (Visual Manipulation Relationship Dataset).
- Scale: 4,683 images containing over 100k labeled grasps.
- Complexity: Images feature 2-5 objects stacked with varying degrees of overlap and occlusion.
- Labels: Each image includes object bounding boxes, grasp rectangles, and a manipulation relationship tree.
Key Results
The system was tested on a physical Baxter robot across three scenarios:
- Object Cluttered Scenes: 90.6% success rate.
- Familiar Stacking Scenes (2-5 objects): 71.9% success rate.
- Complex Stacking Scenes (6-9 objects): 59.4% success rate.
In all cases, the proposed multi-task network significantly outperformed the baseline (VMRN), particularly in complex stacks where the baseline's success rate dropped to a mere 6.3%. The model runs at roughly 6.5 to 7.6 FPS, providing a good balance between accuracy and speed.
Limitations and Future Work
Despite the improvements, the authors noted several failure modes:
- Extreme Occlusion: If an object is almost entirely covered, the detector may fail to find it or assign it a low confidence score.
- Grasp Confusion: In very dense stacks, the model sometimes struggles to distinguish which grasp belongs to which object when they overlap significantly.
- Physical Constraints: Occasionally, the robot might plan a correct grasp that is physically impossible for its gripper to execute due to the size of the object.
Future work will focus on improving detection in high-overlap scenarios and refining the coordination between the vision system and the robot's physical constraints.
Summary updated