Effects of Markers in Training Datasets on the Accuracy of 6D Pose Estimation

Janis Rosskamp, Rene Weller, Gabriel ZachmannWACV 2024

TL;DR

This paper investigates how using markers in training data affects the accuracy of 6D object pose estimation. They found that while marker-based tracking simplifies data collection, removing markers via inpainting or careful augmentation can maintain or even improve accuracy compared to using raw marker-laden images, especially when ground truth data is imprecise.

What Problem Does This Paper Address?

Estimating the 6D pose (position and orientation) of objects in 3D space from 2D images is crucial for applications like robotics, augmented reality, and medical guidance. Deep learning methods have shown great promise, but they rely heavily on large, high-quality training datasets with accurate ground truth poses. Collecting this data is often time-consuming and labor-intensive, requiring manual labeling. Marker-based tracking systems offer a way to automate this data collection by providing precise ground truth poses. However, a key question arises: does training a model on images that contain these markers, which are absent during real-world prediction, negatively impact the model's accuracy? This paper aims to quantify this effect and explore methods to mitigate potential issues.

Methods

The authors propose and evaluate four strategies for handling marker-based training data:

  1. Baseline (No Markers): Training on images without any markers.
  2. With Markers: Training directly on images that include markers used for ground truth generation.
  3. Marker Removal (Inpainting): Using a pre-trained inpainting model to remove markers from training images. They employ a double-masking technique to reduce artifacts.
  4. Marker Augmentation: Randomly generating 1-5 markers on the object's surface in synthetic training images. This aims to teach the network to be invariant to marker presence.

They also investigate the impact of annotation errors by artificially introducing noise into the ground truth poses of synthetic data.

Datasets

  • Teabox Dataset: A custom synthetic dataset created using BlenderProc2, featuring a teabox object. This dataset was used for detailed analysis of marker effects and annotation errors. It includes both synthetic and real images.
  • YCBV Dataset: A larger, publicly available dataset used for evaluating the best-performing methods on a more diverse set of objects.
  • BOP Challenge Test Set: Real images from the BOP challenge were used for evaluation.

Key Insights and Ideas

  • Marker-based data collection is a viable shortcut: The paper confirms that marker-based tracking significantly simplifies the process of acquiring precise ground truth data for pose estimation.
  • Markers can introduce bias: Training directly on images with markers can lead to a performance drop because the network might learn marker-specific features that are not present during inference.
  • Inpainting is a powerful solution: Removing markers via inpainting is highly effective. The inpainted images, even with minor residual artifacts, allow the model to achieve accuracy comparable to training on purely markerless images.
  • Annotation errors are costly: Imprecise ground truth labels, even small ones, can significantly degrade pose estimation accuracy, often more so than using unedited marker-based data.
  • Synthetic data is a good proxy: The experiments on synthetic data largely align with findings on real data, validating the approach and the effectiveness of the proposed methods.

Key Results

  • Training on images with markers (Method II) resulted in a performance reduction of up to 5.6 percent points in AUC of ADD-S compared to the markerless baseline (Method I).
  • Marker augmentation (Method IV) improved accuracy by about 3 percent points over Method II but was still slightly behind the baseline.
  • Marker removal via inpainting (Method III) achieved accuracy comparable to the markerless baseline (Method I), demonstrating its effectiveness.
  • For the teabox dataset, the average AUC of ADD-S for Method I was 97.1%, and for Method III (inpainting) it was 97.0%.
  • When evaluating the impact of annotation errors, even small errors in translation (e.g., 2-4 mm) or rotation (e.g., 2.5°) could reduce accuracy to levels comparable to or worse than training on unedited marker-based data (Method II).

Limitations and Future Work

  • Inpainting Artifacts: While generally effective, inpainting can leave minor artifacts, which are noted but not manually post-processed for automatic pipeline demonstration.
  • Object-Specific Performance: The paper notes that for certain objects (025_mug and 036_wood_block) in the YCBV dataset, performance discrepancies between synthetic and real data were observed, leading to their exclusion from further detailed analysis.
  • Future Work: The authors suggest investigating whether 6D pose estimation is even more precise using real marker-based training data, and exploring the impact of imprecise marker-object registration. They also propose further research into how domain randomization and other techniques can be combined with marker-based approaches.

Summary updated

BibTeX
@inproceedings{rosskamp2024effects,
  author = {Janis Rosskamp and Rene Weller and Gabriel Zachmann},
  title = {{Effects of Markers in Training Datasets on the Accuracy of 6D Pose Estimation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}

Loading related content...

Figure