Abstract
Deep neural network (DNN) architectures have been shown to outperform traditional pipelines for object segmentation and pose estimation using RGBD data, but the performance of these DNN pipelines is directly tied to how representative the training data is of the true data. Hence a key requirement for employing these methods in practice is to have a large set of labeled data for your specific robotic manipulation task, a requirement that is not generally satisfied by existing datasets. In this paper we develop a pipeline to rapidly generate high quality RGBD data with pixelwise labels and object poses. We use an RGBD camera to collect video of a scene from multiple viewpoints and leverage existing reconstruction techniques to produce a 3D dense reconstruction. We label the 3D reconstruction using a human assisted ICP-fitting of object meshes. By reprojecting the results of labeling the 3D scene we can produce labels for each RGBD image of the scene. This pipeline enabled us to collect over 1,000,000 labeled object instances in just a few days. We use this dataset to answer questions related to how much training data is required, and of what quality the data must be, to achieve high performance from a DNN architecture.
Introduction
Deep Neural Networks (DNNs) have revolutionized robotic perception, particularly for tasks like object segmentation and 6DOF (six degrees of freedom) pose estimation. However, these models are data-hungry. To perform reliably in cluttered, real-world scenes with varying lighting and occlusions, they require massive datasets. Existing solutions often rely on manual 2D labeling, which is prohibitively slow, or synthetic data, which often fails to capture the nuances of real-world sensors (the "sim-to-real gap"). LabelFusion aims to bridge this gap by providing "real data at a synthetic scale" using consumer-grade RGBD cameras.
The LabelFusion Pipeline
The LabelFusion pipeline simplifies the difficult 2D labeling problem by transforming it into a 3D alignment task. The process begins with data collection, where a user moves an RGBD camera around a cluttered scene. These video frames are then fused into a single, high-fidelity 3D point cloud using ElasticFusion, a dense SLAM (Simultaneous Localization and Mapping) algorithm.
Because the SLAM algorithm tracks the camera's exact trajectory, any label applied to the 3D reconstruction can be automatically projected back into every 2D frame of the original video. This allows a single 30-second human intervention to generate pixel-perfect segmentation masks and 6DOF poses for thousands of frames.
Human-Assisted 3D Alignment
To make the annotation process efficient, LabelFusion employs a "3-click" initialization. Instead of manually nudging a 3D mesh into place, the user simply clicks three corresponding points on the object in the 3D scan and on a pre-existing CAD model.
Following this rough alignment, an Iterative Closest Point (ICP) algorithm "snaps" the mesh perfectly onto the point cloud data. This hybrid approach combines human intuition for global alignment with algorithmic precision for local fine-tuning.
Dataset and Key Insights
Using this pipeline, the authors produced the LabelFusion Dataset, featuring 352,000 labeled images and over 1,000,000 object instances across 138 scenes. The research yielded several critical insights for training robotic perception models:
- Multi-Object Training is Superior: Training on multi-object cluttered scenes is significantly more valuable than training on isolated objects. Even with fewer total labeled pixels due to occlusions, the network's ability to generalize improved by 190%.
- Logarithmic Returns on Environments: Generalization to new backgrounds improves logarithmically. High-performance generalization typically requires around 50 different training environments.
- Efficiency and Accuracy: The method achieved an Intersection over Union (IoU) above 80% compared to manual labeling, while requiring 10,000x less human effort per image.
Limitations and Future Work
While highly effective, LabelFusion is currently limited to rigid objects, as it relies on matching fixed CAD meshes to scans. Additionally, the underlying SLAM algorithm can struggle in featureless environments, such as those with blank white walls. Future work may involve integrating active learning to identify and prioritize the labeling of scenes that are most challenging for the model.
Summary updated