Abstract
This research introduces an end-to-end framework for semantic grasping, enabling robotic systems to identify and retrieve specific object classes from cluttered environments using only monocular RGB images. By mimicking the human brain's "two-stream" visual system—separating object identification ("what") from geometric localization ("where")—the proposed architecture overcomes the fragility of traditional modular pipelines. Leveraging a massive dataset of 11 million images and a novel self-supervised label propagation technique, the system achieves a significant performance leap over standard bounding-box baselines.
Introduction
Most robotic grasping research focuses on the geometric challenge: determining the finger movements required to lift an arbitrary shape. However, real-world utility often demands semantic reasoning, where a robot must pick up a specific item, such as a "spatula" or a "soda can," from a pile of diverse objects.
Traditional approaches typically rely on a modular "pipeline": an object detector identifies a bounding box, a classifier labels the contents, and a grasp planner executes the movement. This modularity is inherently fragile; failures in detection due to clutter, occlusion, or poor lighting propagate through the system. Furthermore, standard computer vision models trained on datasets like ImageNet often struggle with the unique "robot's-eye view"—top-down, often blurry, and highly occluded images of objects in a bin.
Methods
The authors propose an end-to-end deep learning approach inspired by the two-stream hypothesis of visual processing in the brain:
- The Ventral Stream ("What"): A network branch dedicated to recognizing the object class.
- The Dorsal Stream ("Where/How"): A branch that interprets geometric relationships to predict the probability of grasp success.
Key Innovations
- Separated Architecture: The researchers found that a dedicated dorsal stream, pre-trained on millions of generic grasps, paired with a ventral stream for classification, yielded the best performance.
- Label Propagation: To avoid the prohibitive cost of hand-labeling millions of cluttered images, the team utilized a "present" pose. When a robot successfully grasps an object, it holds it up to the camera. This isolated, centered image is easily labeled (manually or via automated classifiers), and that label is then "propagated" back to the original cluttered scene image used for the grasp.
- Joint Training: The model is trained to simultaneously optimize for grasp success probability and the probability that the grasped object matches the target class.
Datasets and Experimental Setup
The scale of the data is a cornerstone of this work:
- Primary Dataset: 500 unique object instances across 16 categories (e.g., erasers, spoons, staplers, soda cans).
- Scale: 11 million total grasping images collected across 8 KUKA robot arms.
- Semantic Labels: 330,000 images were semantically labeled via the label propagation method.
To further improve the ventral stream's feature representation, the researchers incorporated auxiliary data from ImageNet and JFT, though they noted that the benefits were dependent on the degree of domain shift.
Key Insights
- End-to-End Superiority: Modular baselines struggle with the domain shift between clean training data and messy bin environments. End-to-end models learn features specifically optimized for grasping in clutter.
- Self-Supervision via the "Present" Pose: By transforming a difficult perception problem (classification in clutter) into an easy one (classification of a centered object), the researchers enabled massive-scale training without manual intervention for every frame.
- Attention Mechanisms: The inclusion of a "soft keypoint attention" mechanism allowed a 9-layer CNN to match the performance of much deeper 16-layer architectures, significantly improving computational efficiency.
Results
The "Separated + Attention" model emerged as the most effective configuration:
- Class Success Rate: On unseen test objects, the model achieved a 39.0% success rate in picking up the correct object, nearly doubling the 19.3% achieved by the bounding-box baseline.
- Grasp Success: The dorsal stream remained highly effective even when the semantic target was missed, maintaining a general grasp success rate of over 60% on unseen objects.
Discussion and Future Work
Despite the progress, a 39% success rate underscores the difficulty of semantic grasping in high clutter. Robots frequently grasp objects adjacent to the target rather than the target itself. The authors identify the gap between "present" pose images and "cluttered" bin images as a primary bottleneck. Future work will focus on collecting more diverse "present" poses—varying angles and rotations—to improve label propagation accuracy and help the ventral stream generalize better to the complexities of real-world robotic environments.
Summary updated