Label Fusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes

Pat Marion, Peter R. Florence, Lucas Manuelli, Russ Tedrake

TL;DR

This paper introduces LabelFusion, a novel pipeline for efficiently generating pixel-wise labeled RGBD data for robotic manipulation tasks. By leveraging 3D reconstruction and human-assisted mesh fitting, they can label thousands of images from a single scene in just a few days, significantly reducing annotation time and enabling large-scale dataset creation.

What Problem Does This Paper Address?

Deep neural networks (DNNs) have revolutionized object segmentation and pose estimation in robotics, often outperforming traditional methods. However, their performance is heavily reliant on the quality and representativeness of the training data. For specific robotic manipulation tasks, this means needing large datasets of labeled scenes featuring the target objects in realistic clutter and varying conditions. Acquiring such datasets through manual, per-image labeling is prohibitively time-consuming and expensive. Existing datasets often lack per-pixel labels or 6DOF object poses, or require highly specialized and restrictive data collection setups. This paper aims to bridge this gap by providing a scalable, efficient pipeline for generating high-quality, labeled RGBD data for real-world cluttered scenes.

Methods

The LabelFusion pipeline consists of several key stages:

  1. RGBD Data Collection: Raw RGBD data is captured using an RGBD sensor. This can be done either by hand-carrying the sensor for flexibility or by mounting it on a robot arm for automated scanning patterns.
  2. Dense 3D Reconstruction: The collected RGBD frames are fused into a dense 3D point cloud reconstruction of the scene. The authors use ElasticFusion, an open-source implementation known for its real-time performance and camera pose tracking capabilities.
  3. Object Mesh Generation: Pre-existing 3D mesh models of the target objects are required. The paper mentions using various methods for mesh generation, including 3D scanners, existing datasets (like YCB), and even a volumetric meshing method that operates directly on the reconstructed point cloud.
  4. Human Assisted Annotation: This is the core innovation for reducing annotation time. Instead of labeling each 2D image, users interact with the 3D reconstruction. They provide a rough initial alignment of an object's mesh to the point cloud using a simple 3-click initialization. This rough pose is then used as a seed for an ICP (Iterative Closest Point) algorithm to achieve a fine, accurate alignment of the mesh to the 3D scene. This process takes approximately 30 seconds per object per scene.
  5. Rendering of Labeled Images and Object Poses: Once the object meshes are aligned in the 3D reconstruction, their poses are known for each camera view. By projecting the aligned 3D object meshes back into the original RGB images, pixel-wise segmentation masks and 6DOF object poses are automatically generated for thousands of frames within a single scene.

Datasets

The paper introduces the LabelFusion dataset, which is described as the largest available RGBD dataset with object-pose labels.

  • Statistics: It contains 352,000 labeled RGBD images and over 1,000,000 labeled object instances.
  • Collection: The data was collected from 112 scenes using a handheld sensor and 26 scenes using a robot arm.
  • Content: The dataset features multi-object scenes with clutter, occlusions, and varied lighting conditions, aiming to be representative of real-world robotic manipulation scenarios.
  • Availability: The dataset and the annotation pipeline are made publicly available.

Key Insights and Ideas

  • 3D Reconstruction as an Annotation Bottleneck Solver: Shifting annotation from 2D images to 3D reconstructions dramatically reduces human effort. A single annotation effort in 3D can label thousands of 2D frames.
  • Human-Assisted ICP for Efficient Alignment: Combining a quick human-provided initial pose with automated ICP alignment offers a sweet spot between manual precision and full automation, achieving high-quality results with minimal human input.
  • Value of Cluttered, Multi-Object Scenes: Training data that includes occlusions and multiple objects in a scene significantly improves generalization performance, even for single-object test scenes, suggesting that learning to handle complexity is crucial.
  • Data Diversity is Key: The experiments highlight the importance of diverse backgrounds and object arrangements in the training data for robust generalization.

Key Results

  • Speed and Scale: The pipeline enabled the collection of over 1,000,000 labeled object instances in just a few days with minimal human annotation time (approx. 30 seconds per object per scene).
  • Dataset Size: The LabelFusion dataset comprises 352,000 labeled RGBD images.
  • Performance Gains from Multi-Object Data: Training on multi-object scenes with occlusions led to a significant increase in mean IoU (Intersection over Union) performance (up to 190% increase) compared to training on single-object scenes, even when tested on novel scenes.
  • Impact of Background Diversity: Generalization performance improved logarithmically with the number of background environments in the training data.
  • Data Rate vs. Views: While higher sensor rates (closer to 30 Hz) generally improve performance, the gains diminish significantly after 0.3 Hz for slower robot arm movements, suggesting that the number of views per scene is often more critical than raw frame rate for certain scenarios.

Limitations and Future Work

  • Rigid Objects: The current pipeline requires rigid objects for which mesh models are available. Non-rigid objects or objects without existing models would require different approaches.
  • Mesh Quality: The quality of the generated labels is dependent on the quality of the input 3D object meshes.
  • Future Work: The authors suggest that their pipeline can be used to answer further questions about data requirements for DNNs in robotics. They also hope their work lowers the barrier for researchers to create custom datasets for their specific robotic manipulation tasks. The analysis of data quantity and quality provides guidance for future data collection efforts.

Summary updated

Abstract

Deep neural network (DNN) architectures have been shown to outperform traditional pipelines for object segmentation and pose estimation using RGBD data, but the performance of these DNN pipelines is directly tied to how representative the training data is of the true data. Hence a key requirement for employing these methods in practice is to have a large set of labeled data for your specific robotic manipulation task, a requirement that is not generally satisfied by existing datasets. In this paper we develop a pipeline to rapidly generate high quality RGBD data with pixelwise labels and object poses. We use an RGBD camera to collect video of a scene from multiple viewpoints and leverage existing reconstruction techniques to produce a 3D dense reconstruction. We label the 3D reconstruction using a human assisted ICP-fitting of object meshes. By reprojecting the results of labeling the 3D scene we can produce labels for each RGBD image of the scene. This pipeline enabled us to collect over 1,000,000 labeled object instances in just a few days. We use this dataset to answer questions related to how much training data is required, and of what quality the data must be, to achieve high performance from a DNN architecture. Our dataset and annotation pipeline are available at labelfusion.csail.mit.edu.
BibTeX
@inproceedings{marion2018label,
  author = {Pat Marion and Peter R. Florence and Lucas Manuelli and Russ Tedrake},
  title = {{Label Fusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2018},
  url = {https://openreview.net/forum?id=cWOrSG8ci8a},
}
Figure