TL;DR
This paper introduces an award-winning "grasp-first-then-recognize" robotic pick-and-place system capable of handling novel, unseen objects in heavily cluttered environments without any task-specific retraining. By decoupling object-agnostic affordance learning (where to pick) from cross-domain metric learning (matching the isolated object to standard web product images), the system achieves robust out-of-the-box generalization, securing 1st place in the stowing task at the 2017 Amazon Robotics Challenge.
What Problem Does This Paper Address?
Traditional robotic pick-and-place systems rely on a "recognize-then-grasp" pipeline: they segment objects, estimate their 3D poses, and then plan a grasp based on known 3D models. However, this approach breaks down in heavy clutter, under severe occlusions, or when encountering novel objects for which no 3D models or extensive training datasets exist.
Collecting physical training data or retraining models for every new product is highly impractical for dynamic real-world environments like warehouse automation or service robotics. The authors tackle the challenge of picking and identifying completely novel objects in messy bins using only a few standard product images (e.g., scraped from the web) as a reference.
Methods
The system flips the traditional script by adopting a grasp-first-then-recognize paradigm using two decoupled components:
- Multi-Affordance Object-Agnostic Grasping: The robot uses Fully Convolutional Networks (FCNs) based on ResNet-101 to map visual observations (RGB-D images and heightmaps) directly to pixel-wise affordance maps. These maps represent the probability of success for four distinct motion primitives: suction down, suction side, grasp down, and flush grasp.
To handle various gripper orientations, the system orthographically projects the scene into a heightmap, rotates it across 16 different angles, and feeds each rotation through the FCN. This allows a single network to share visual features across all orientations, maximizing sample efficiency.
- Cross-Domain Image Matching: Once an object is isolated from the clutter and held up to the cameras, the system recognizes it by matching the observed image to a gallery of web product images. This is achieved via a two-stream ConvNet architecture trained with a triplet loss (distance ratio loss) to map both domains into a shared feature embedding space. To prevent "metric collapse" (overfitting to known categories), the product stream uses a fixed ImageNet-pretrained model, while only the observation stream is trained.
Datasets
- Grasping Dataset: A custom dataset containing 1,837 manually annotated RGB-D images/heightmaps of cluttered bins. Users painted green (positive) and red (negative) brushstrokes to label successful and failing suction/grasp locations based on real-world experience.
- ARC 2017 Benchmark: Used for evaluation, containing a mix of 40 different household items (20 known, 20 novel) featuring complex properties like deformability, transparency, and varying weights.
Key Insights and Ideas
- Decoupling Perception and Manipulation: Isolating an object before trying to recognize it drastically simplifies the perception problem. It transforms a complex multi-object segmentation and pose estimation problem in heavy clutter into a much cleaner single-object verification task.
- Leveraging the Web as a Prior: By framing recognition as a cross-domain metric learning problem, the system can utilize readily available web product images as its only knowledge base for novel items, bypassing the need for physical object scans or task-specific data collection.
- Input Rotation over Network Branches: Instead of using separate network branches for different grasp angles (which suffers from uneven data distribution and poor convergence), rotating the input heightmap allows a single network to share visual features across all orientations.
Key Results
- Grasping Accuracy: The proposed ConvNet achieved a 92.4% precision for top-1 suction proposals and 96.7% precision for top-1 grasping proposals, significantly outperforming simulation-trained alternatives like Dex-Net (69.3% and 80.4% respectively) due to the domain gap in handling heavy, non-porous, or noisy real-world data.
- Recognition Accuracy: The two-stage recognition framework achieved 88.6% mixed accuracy (93.6% for known, 77.5% for novel objects) on a 1-vs-20 classification benchmark.
- Real-World Validation: In the 2017 Amazon Robotics Challenge final stowing task, the system achieved a 58.3% suction success rate, 75% grasp success rate, and 100% recognition accuracy, successfully stowing all 20 objects within the allotted time frame.
Limitations and Future Work
- Open-Loop Execution: The system operates blindly during execution (open-loop). Small errors in affordance prediction can cause grasp failures if the object shifts. Incorporating closed-loop tactile or visual feedback during the approach-grasp phase is a key future direction.
- State Tracking Vulnerability: The state tracking system is highly sensitive to drastic changes (e.g., multiple objects shifting locations simultaneously), which can cause it to lose track without recovery.
- Complex Manipulation Policies: The current system is limited to direct, accessible grasps. It cannot plan non-prehensile actions like pushing or scooping to deliberately uncover hidden objects, which requires a longer planning horizon.
Summary updated