Classification dense d'images de profondeur entraînée sans données réelles

Howard Mahe, Denis Marraud, Andrew I. ComportHAL 2017

TL;DR

This paper tackles the challenge of training deep learning models for semantic segmentation of depth images without relying on large, manually labeled real-world datasets. They achieve this by developing a sophisticated sensor noise model to generate realistic synthetic training data from 3D models, demonstrating that this approach can yield competitive results on real-world data.

What Problem Does This Paper Address?

Robots and autonomous systems often rely on depth cameras to perceive their environment. For tasks like Simultaneous Localization and Mapping (SLAM), understanding the semantic meaning of objects in the scene (e.g., "chair," "table") is crucial for more robust and intelligent navigation and interaction. However, training deep learning models for semantic segmentation typically requires vast amounts of accurately labeled real-world data, which is expensive and time-consuming to acquire. This paper addresses the problem of how to train effective semantic segmentation models for depth images when such large labeled real-world datasets are unavailable, by leveraging synthetic data.

Methods

The core of the paper's contribution lies in its approach to generating synthetic training data that closely mimics real-world depth images.

  • Synthetic Data Generation: They start with 3D CAD models of the target environment (an aircraft assembly workshop) and use a rendering engine to generate synthetic depth images and their corresponding ground truth semantic labels.
  • Sophisticated Sensor Noise Model: To bridge the gap between synthetic and real data, they introduce a novel probabilistic sensor noise model. This model goes beyond simple noise addition and incorporates:
  • Quantization and Gaussian Noise: Simulates the inherent noise in depth measurements and the rounding effects of quantization.
  • Correspondence Noise: Models the challenges of depth measurement due to misalignments in stereo matching, especially at surface boundaries and with varying surface orientations. This includes simulating lateral shifts and missing depth values.
  • HHA Encoding: They utilize the HHA encoding, which represents depth images using three channels: horizontal disparity, height relative to the ground, and the angle between the surface normal and gravity. This encoding is designed to be robust to viewpoint changes and height variations.
  • Handling Missing Pixels: Two strategies are explored for dealing with "holes" (pixels with no depth information) in the synthetic data:
  • Ignoring Missing Pixels: The model is trained to ignore these pixels during the cost function calculation.
  • Inpainting: Techniques are used to fill in the missing depth values in the synthetic data.

Datasets

  • Training and Validation Data: 29,000 synthetic depth images and their corresponding semantic labels were generated from a 3D CAD model of an aircraft assembly workshop.
  • Test Data: 25 real-world depth images were acquired using an ASUS Xtion PRO LIVE camera in the same workshop environment. Ground truth for this test set was obtained by aligning the RGB-D images with the CAD model.

Key Insights and Ideas

  • Realistic Synthetic Data is Key: The paper strongly suggests that the quality of synthetic data, particularly its ability to accurately model real-world sensor noise, is paramount for successful training without real-world labels.
  • Sophisticated Noise Modeling Outperforms Simple Noise: The proposed comprehensive sensor noise model significantly improves performance compared to simpler noise models or no noise modeling at all. This highlights the importance of understanding and simulating the specific failure modes of depth sensors.
  • HHA Encoding Benefits Depth-Only Segmentation: The HHA encoding proves beneficial for depth-only semantic segmentation, improving performance by providing richer geometric cues.
  • "Ignoring Pixels" vs. "Inpainting": For sparse semantic segmentation (where only pixels with depth are considered), ignoring missing pixels during training is a viable and effective strategy. For dense segmentation, inpainting (especially colorization-based) shows promise but is less effective with the HHA encoding.

Key Results

  • Improved Performance with Noise Modeling: The best results were achieved using the complete noise model. For HHA encoded images, this led to a global precision of 68.9% for sparse semantic segmentation.
  • Inpainting Effectiveness: For depth-only (D) images, inpainting by colorization (using RGB information) achieved a global precision of 63.1%, outperforming the best noise model without inpainting.
  • HHA Encoding Gains: The HHA encoding improved global precision by an average of 13.4% compared to using raw depth images, particularly benefiting classes like "floor" and "upper deck."

Limitations and Future Work

  • Model Limitations: The authors acknowledge that their noise model is an approximation of real-world sensor noise.
  • Future Work: They suggest improving the sensor noise modeling further, potentially using more advanced machine learning techniques, and exploring more sophisticated inpainting methods. They also aim to improve the robustness of the HHA encoding to noise.

Summary updated

Abstract

International audience
BibTeX
@inproceedings{mahe2017classification,
  author = {Howard Mahe and Denis Marraud and Andrew I. Comport},
  title = {{Classification dense d'images de profondeur entraînée sans données réelles}},
  booktitle = {HAL},
  year = {2017},
}
Figure