This paper presents the approach taken by team NimbRo Picking (University of Bonn) for the 2017 Amazon Robotics Challenge (ARC). The system was designed to handle the "stow" and "pick" tasks in a warehouse setting, specifically focusing on the challenge's new requirement: learning to handle previously unseen objects in just 45 minutes.
TL;DR
Team NimbRo Picking developed a dual-arm robotic system capable of rapidly learning new objects through an automated turntable capture system and synthetic data generation. By combining a deep semantic segmentation pipeline with a "clutter graph" for occlusion handling and online dual-arm coordination, they secured second place in multiple categories at the ARC 2017.
What Problem Does This Paper Address?
Automating warehouse tasks like "stowing" (putting items away) and "picking" (retrieving specific items) is difficult because of clutter and novelty. Robots must identify objects piled haphazardly in bins and determine how to grab them without damaging them or colliding with the environment.
The 2017 ARC added a significant hurdle: teams were given "surprise" items shortly before the competition. This meant systems could no longer rely on pre-trained models for every object; they needed a way to capture, learn, and integrate new items into their perception and manipulation pipelines almost in real-time.
Methods
The authors describe a full-stack mechatronic and software solution:
- Hardware: Two Universal Robots UR5 arms equipped with hybrid grippers (suction + a 2-DoF "pinch" finger). A gantry-mounted Nikon DSLR and a Photoneo 3D scanner provide high-resolution workspace perception.
- Fast Object Learning: To handle new items, they built an automated turntable. It captures 60 views of an object in about a minute. They use background subtraction to automatically mask the item, creating a library of segmented object views.
- Perception Pipeline: They utilize RefineNet (with a ResNeXt-101 backbone) for semantic segmentation. To train this quickly, they generate synthetic cluttered scenes on-the-fly by "pasting" the turntable images onto real backgrounds of bins and totes.
- The Clutter Graph: To decide the picking order, the system builds a directed graph where an edge from object A to B means A occludes B. They solve this using a "minimum feedback arc set" approach to determine which items must be moved first.
- Dual-arm Coordination: Instead of pre-planning, the system uses a high-level planner that assigns tasks to arms based on reachability and current occupancy. They use "cost function planes" in the Inverse Kinematics (IK) solver to keep the arms from colliding while working in the same shared workspace.
Datasets
- Turntable Dataset: A custom-built dataset of 60 views per item for the 40 known and several "surprise" items.
- Synthetic Scenes: Thousands of images generated on-the-fly during training by compositing turntable items onto manually annotated background frames of the storage system.
- ARC 2017 Competition Data: The system was evaluated on the actual item sets and bin configurations provided by Amazon during the Nagoya competition.
Key Insights and Ideas
- Heuristics over End-to-End Learning: While some teams tried to learn grasp poses directly, NimbRo used a robust heuristic (the pole of inaccessibility—the point furthest from any edge) for suction. This was more reliable for new items where grasp annotations weren't available.
- Synthetic Data is Sufficient: The paper demonstrates that you don't need real-world images of every possible cluttered configuration. High-quality segments pasted onto real backgrounds with random occlusions provide enough signal for a deep network to generalize.
- The "Aha" Moment on Coordination: The authors realized that dual-arm speedup is limited by "resource conflicts" (both arms wanting the same bin). They found that while two arms increase throughput, the speedup is roughly 1.2x–1.3x, not 2x, due to these spatial constraints.
Key Results
- Competition Success: The team placed 2nd in the Picking Task (245 points) and 2nd in the Final Task (235 points).
- Perception Accuracy: Their segmentation model achieved a Mean IoU of 0.720 with the full synthetic pipeline, significantly outperforming models trained on simpler data.
- Speed: The system could learn a new item and be ready for the task in under 45 minutes, meeting the strict competition requirements.
- Pose Estimation: Their 6D pose estimation network could predict item translation within a few pixels and orientation within a few degrees.
Limitations and Future Work
- Thin/Transparent Objects: The system struggled with items like a wire-mesh cup, which are notoriously difficult for both RGB cameras and 3D scanners to "see" and segment correctly.
- Hardware Fragility: The authors noted that a power supply failure and precision issues with their scales cost them points during the official runs, highlighting the "reality gap" in robotics competitions.
- Storage Layout: The physical layout of the bins was a bottleneck for dual-arm efficiency. Future work suggests that a more "global" planning approach or a redesigned storage geometry could reduce arm-to-arm interference.
Summary updated