TL;DR
This paper introduces Form2Fit, a novel robotic assembly system that learns to generalize to new objects and kits by treating assembly as a shape-matching problem. Instead of relying on precise 3D models or extensive task-specific data, Form2Fit learns geometric correspondences between object shapes and their target locations, enabling robust and adaptable assembly. A key innovation is its self-supervised data collection method, which uses time-reversed disassembly to generate training data.
What Problem Does This Paper Address?
Traditional robotic assembly methods often struggle with generalization. They typically rely on accurate 6D object pose estimation, which requires either detailed 3D CAD models of every object or substantial task-specific training data. This makes them brittle and slow to adapt to new objects or variations in kit configurations, a common challenge in real-world scenarios like retail logistics where product packaging can change weekly. The core problem is how to enable robots to assemble novel objects into new kits without explicit prior knowledge, moving beyond rigid, pre-programmed approaches.
Methods
Form2Fit tackles this by formulating assembly as a shape-matching problem. Its approach has three main components:
- Shape-Driven Assembly for Generalization: The system learns a "shape descriptor" that maps visual input to dense, pixel-wise feature representations. This descriptor is trained to establish geometric correspondences between object surfaces and their target placement locations (e.g., cavities in packaging). By training on diverse objects and kits, the network learns a general understanding of how shapes fit together.
- Learning Assembly from Disassembly: To overcome the difficulty and cost of collecting assembly training data, Form2Fit uses a self-supervised pipeline. It disassembles complete kits through trial-and-error pick-and-place actions. By reversing these disassembly sequences, it generates ground-truth labels for object-to-placement correspondences and motion trajectories, effectively teaching the robot how to assemble by observing how to take things apart.
- Network Modules:
- Suction Module: Predicts favorable picking locations on objects using a deep network that outputs suction success probabilities.
- Place Module: Predicts dense, pixel-wise confidence maps for placing objects into the kit, implicitly learning sequential ordering constraints.
- Matching Module: A Siamese network that learns rotation-sensitive feature descriptors to match object parts to their corresponding locations in the kit and infer the necessary object orientation.
- Planner: Integrates outputs from the suction, place, and matching modules to determine the final pick location, place location, and object orientation.
Datasets
The paper does not explicitly mention pre-existing public datasets for training. Instead, it introduces a novel self-supervised data collection pipeline that generates its own training data by disassembling complete kits. For each kit in the training set, approximately 500 disassembly sequences are collected, taking 8-10 hours per kit. This data is then used to train the suction, placing, and matching networks. The evaluation uses various kits, including tape-runner, black-floss, fruits, zoo-animals, and deodorants, with specific training and testing distributions detailed.
Key Insights and Ideas
- Assembly as Shape Matching: Reframing assembly as a shape-matching problem, rather than solely a pose-estimation problem, is crucial for generalization. This allows the robot to learn fundamental geometric relationships that transcend specific object identities.
- Time-Reversed Disassembly as Supervision: The ingenious use of time-reversed disassembly to generate assembly training data is a significant contribution. It bypasses the need for manual demonstrations or complex motion planning, making data collection far more efficient and scalable.
- Learning Rotation-Sensitive Descriptors: The matching module's ability to learn descriptors that are sensitive to rotation is key to inferring the correct object orientation for placement, a critical aspect of successful assembly.
- Implicit Sequencing: The placing module, trained on reversed disassembly, implicitly learns the correct order in which objects should be placed into a kit, even for complex multi-layer arrangements.
Key Results
Form2Fit demonstrates impressive performance across various generalization scenarios:
- Generalization to Initial Conditions: Achieves 90% average assembly success rate with varying object and kit poses.
- Generalization to Novel Configurations: Reaches 94.27% success rate on new combinations of kits (multiple instances of the same kit or mixtures of different kits) that were not seen during training.
- Generalization to Novel Objects and Kits: Achieves over 86% generalization accuracy on completely new objects and kits that the system has never encountered before.
- Comparison to Baseline: Outperforms the ORB-PE pose estimation baseline in terms of average distance metric for pose accuracy, particularly in its ability to generalize.
Limitations and Future Work
The authors acknowledge several limitations:
- 2D Rotations Only: The current system primarily handles 2D rotations (planar object rotations) and assumes objects are face-down.
- Transparent Objects: The system struggles with fully transparent objects, requiring workarounds like spray-painting.
Future work directions include:
- Higher DoF Assembly: Extending the action representation to handle more complex 3D assembly tasks with higher degrees of freedom.
- Handling Transparent Objects: Investigating the use of external vision algorithms to estimate the geometry of transparent kits for improved handling.
Summary updated