Fast Object Learning and Dual-arm Coordination for Cluttered Stowing, Picking, and Packing

Max Schwarz, Christian Lenz, Germán Martín García, Seongyong Koo, Arul Selvam Periyasamy, Michael Schreiber, Sven Behnke

Abstract

Robotic picking from cluttered bins is a demanding task, for which Amazon Robotics holds challenges. The 2017 Amazon Robotics Challenge (ARC) required stowing items into a storage system, picking specific items, and packing them into boxes. In this paper, we describe the entry of team NimbRo Picking. Our deep object perception pipeline can be quickly and efficiently adapted to new items using a custom turntable capture system and transfer learning. It produces high-quality item segments, on which grasp poses are found. A planning component coordinates manipulation actions between two robot arms, minimizing execution time. The system has been demonstrated successfully at ARC, where our team reached second places in both the picking task and the final stow-and-pick task. We also evaluate individual components.

Introduction

The 2017 Amazon Robotics Challenge (ARC) introduced a significant hurdle for automated warehouse systems: the requirement to handle "surprise" items with only 45 minutes of preparation time. This shift moved the challenge beyond pre-trained models toward systems capable of rapid, on-site learning and integration. Team NimbRo Picking from the University of Bonn addressed this by developing a dual-arm robotic system optimized for fast object learning, robust perception in clutter, and efficient multi-arm coordination.

System Architecture and Hardware

The NimbRo Picking system utilizes a mechatronic design centered around two Universal Robots UR5 arms. The workspace is monitored by a gantry-mounted Nikon DSLR and a Photoneo 3D scanner, providing high-resolution RGB and depth data.

CAD model of the NimbRo Picking system setup, illustrating the sensor arrangement, vacuum systems, and storage layout.

A critical component of the hardware is the custom hybrid gripper. Each arm is equipped with a tool that combines a suction cup with a 2-DOF "pinch" finger. This dual-modality allows the robot to handle a wide variety of object geometries and surface properties, switching between suction for flat surfaces and pinching for more complex shapes.

The hybrid gripper design featuring a suction cup and a 2-DOF pinch finger for versatile object manipulation.

Fast Object Learning and Perception

To meet the 45-minute learning requirement, the team built an automated turntable system. This setup captures 60 views of a new item in approximately one minute. By using background subtraction, the system automatically generates a library of segmented object views.

The perception pipeline employs RefineNet (with a ResNeXt-101 backbone) for semantic segmentation. Training is accelerated by generating synthetic cluttered scenes on-the-fly. This process involves "pasting" the segmented turntable images onto real background frames of bins and totes, incorporating random occlusions. This approach demonstrated that high-quality synthetic data is sufficient for deep networks to generalize to real-world cluttered environments without requiring thousands of manually annotated real images.

Manipulation and Coordination

Picking order is determined via a clutter graph, a directed graph where edges represent occlusions between objects. The system identifies which items must be moved first by solving for the "minimum feedback arc set." Instead of end-to-end learning for grasping, the team utilized a robust heuristic: the pole of inaccessibility (the point furthest from any edge) to determine suction targets. This proved more reliable for novel items where specific grasp annotations were unavailable.

For dual-arm coordination, the system avoids complex pre-planning in favor of a high-level planner that assigns tasks based on reachability. To prevent collisions in the shared workspace, the Inverse Kinematics (IK) solver utilizes "cost function planes," ensuring the arms operate safely and efficiently. While dual-arm operation increased throughput, the team noted a speedup of roughly 1.2x–1.3x rather than 2x, primarily due to spatial resource conflicts at the bins.

Results and Conclusion

The NimbRo Picking system proved highly effective during the ARC 2017 competition, securing 2nd place in the Picking Task (245 points) and 2nd place in the Final Task (235 points). The segmentation model achieved a Mean IoU of 0.720, and the system successfully integrated new items within the strict time limits.

Despite these successes, challenges remain. The system struggled with thin or transparent objects (e.g., wire-mesh cups) that are difficult for 3D scanners to resolve. Future work will focus on improving hardware robustness and exploring more global planning strategies to further optimize dual-arm efficiency.

Summary updated

Abstract

Robotic picking from cluttered bins is a demanding task, for which Amazon Robotics holds challenges. The 2017 Amazon Robotics Challenge (ARC) required stowing items into a storage system, picking specific items, and packing them into boxes. In this paper, we describe the entry of team NimbRo Picking. Our deep object perception pipeline can be quickly and efficiently adapted to new items using a custom turntable capture system and transfer learning. It produces high-quality item segments, on which grasp poses are found. A planning component coordinates manipulation actions between two robot arms, minimizing execution time. The system has been demonstrated successfully at ARC, where our team reached second places in both the picking task and the final stow-and-pick task. We also evaluate individual components.
BibTeX
@inproceedings{schwarz2018fast,
  author = {Max Schwarz and Christian Lenz and Germán Martín García and Seongyong Koo and Arul Selvam Periyasamy and Michael Schreiber and Sven Behnke},
  title = {{Fast Object Learning and Dual-arm Coordination for Cluttered Stowing, Picking, and Packing}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2018},
  url = {https://openreview.net/forum?id=9SPtAasYJ5z},
}
Figure