A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes

Pat Marion, Peter R. Florence, Lucas Manuelli, Russ Tedrake

Abstract

Deep neural network (DNN) architectures have been shown to outperform traditional pipelines for object segmentation and pose estimation using RGBD data, but the performance of these DNN pipelines is directly tied to how representative the training data is of the true data. Hence a key requirement for employing these methods in practice is to have a large set of labeled data for your specific robotic manipulation task, a requirement that is not generally satisfied by existing datasets. In this paper we develop a pipeline to rapidly generate high quality RGBD data with pixelwise labels and object poses. We use an RGBD camera to collect video of a scene from multiple viewpoints and leverage existing reconstruction techniques to produce a 3D dense reconstruction. We label the 3D reconstruction using a human assisted ICP-fitting of object meshes. By reprojecting the results of labeling the 3D scene we can produce labels for each RGBD image of the scene. This pipeline enabled us to collect over 1,000,000 labeled object instances in just a few days. We use this dataset to answer questions related to how much training data is required, and of what quality the data must be, to achieve high performance from a DNN architecture.

Introduction

Deep Neural Networks (DNNs) have revolutionized robotic perception, particularly for tasks like object segmentation and 6DOF (six degrees of freedom) pose estimation. However, these models are data-hungry. To perform reliably in cluttered, real-world scenes with varying lighting and occlusions, they require massive datasets. Existing solutions often rely on manual 2D labeling, which is prohibitively slow, or synthetic data, which often fails to capture the nuances of real-world sensors (the "sim-to-real gap"). LabelFusion aims to bridge this gap by providing "real data at a synthetic scale" using consumer-grade RGBD cameras.

The LabelFusion Pipeline

The LabelFusion pipeline simplifies the difficult 2D labeling problem by transforming it into a 3D alignment task. The process begins with data collection, where a user moves an RGBD camera around a cluttered scene. These video frames are then fused into a single, high-fidelity 3D point cloud using ElasticFusion, a dense SLAM (Simultaneous Localization and Mapping) algorithm.

The complete LabelFusion pipeline: (a) raw scene, (b) 3D reconstruction, (c) human-assisted alignment, (d) ICP refinement, (e) segmented 3D model, and (f) final labeled 2D frames.

Because the SLAM algorithm tracks the camera's exact trajectory, any label applied to the 3D reconstruction can be automatically projected back into every 2D frame of the original video. This allows a single 30-second human intervention to generate pixel-perfect segmentation masks and 6DOF poses for thousands of frames.

Human-Assisted 3D Alignment

To make the annotation process efficient, LabelFusion employs a "3-click" initialization. Instead of manually nudging a 3D mesh into place, the user simply clicks three corresponding points on the object in the 3D scan and on a pre-existing CAD model.

The human-assisted "3-click" initialization process for aligning a CAD model to the 3D point cloud scan.

Following this rough alignment, an Iterative Closest Point (ICP) algorithm "snaps" the mesh perfectly onto the point cloud data. This hybrid approach combines human intuition for global alignment with algorithmic precision for local fine-tuning.

Dataset and Key Insights

Using this pipeline, the authors produced the LabelFusion Dataset, featuring 352,000 labeled images and over 1,000,000 object instances across 138 scenes. The research yielded several critical insights for training robotic perception models:

  • Multi-Object Training is Superior: Training on multi-object cluttered scenes is significantly more valuable than training on isolated objects. Even with fewer total labeled pixels due to occlusions, the network's ability to generalize improved by 190%.
  • Logarithmic Returns on Environments: Generalization to new backgrounds improves logarithmically. High-performance generalization typically requires around 50 different training environments.
  • Efficiency and Accuracy: The method achieved an Intersection over Union (IoU) above 80% compared to manual labeling, while requiring 10,000x less human effort per image.

Limitations and Future Work

While highly effective, LabelFusion is currently limited to rigid objects, as it relies on matching fixed CAD meshes to scans. Additionally, the underlying SLAM algorithm can struggle in featureless environments, such as those with blank white walls. Future work may involve integrating active learning to identify and prioritize the labeling of scenes that are most challenging for the model.

Summary updated

Abstract

Deep neural network (DNN) architectures have been shown to outperform traditional pipelines for object segmentation and pose estimation using RGBD data, but the performance of these DNN pipelines is directly tied to how representative the training data is of the true data. Hence a key requirement for employing these methods in practice is to have a large set of labeled data for your specific robotic manipulation task, a requirement that is not generally satisfied by existing datasets. In this paper we develop a pipeline to rapidly generate high quality RGBD data with pixelwise labels and object poses. We use an RGBD camera to collect video of a scene from multiple viewpoints and leverage existing reconstruction techniques to produce a 3D dense reconstruction. We label the 3D reconstruction using a human assisted ICP-fitting of object meshes. By reprojecting the results of labeling the 3D scene we can produce labels for each RGBD image of the scene. This pipeline enabled us to collect over 1,000,000 labeled object instances in just a few days. We use this dataset to answer questions related to how much training data is required, and of what quality the data must be, to achieve high performance from a DNN architecture.
BibTeX
@inproceedings{marion2018pipeline,
  author = {Pat Marion and Peter R. Florence and Lucas Manuelli and Russ Tedrake},
  title = {{A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2018},
  url = {https://openreview.net/forum?id=cWOrSG8ci8a},
}
Figure