Single-Shot Dense Depth Sensing with Color Sequence Coded Fringe Pattern

Fu Li, Baoyu Zhang, Guangming Shi, Yi Niu, Ruodai Li, Lili Yang, Xuemei Xie2017 · DOI

TL;DR

This paper introduces a novel single-shot structured light method for dense depth sensing in dynamic scenes. By encoding a De Bruijn sequence into color channels of a fringe pattern, the method effectively tackles the phase unwrapping problem, leading to more precise and higher-resolution depth maps compared to existing technologies like Kinect and ToF cameras.

What Problem Does This Paper Address?

Accurate and dense depth sensing is crucial for various applications, from robotics and augmented reality to industrial inspection. Structured Light Illumination (SLI) is a popular technique for this, but traditional methods often struggle with dynamic scenes due to synchronization requirements or fail to produce dense depth maps. Phase-based SLI methods, while capable of high accuracy, are particularly susceptible to the "phase unwrapping" problem, where the periodic nature of phase measurements leads to ambiguity. Existing solutions often require multiple projected patterns, making them unsuitable for fast-moving objects, or rely on markers that can obscure parts of the scene. This paper aims to overcome these limitations by proposing a single-shot method that is robust to dynamic scenes and achieves dense, accurate depth without relying on external markers.

Methods

The core innovation lies in the color sequence coded fringe pattern. Here's a breakdown:

  • Phase Encoding: The intensity of the fringe pattern itself is modulated sinusoidally along one direction, providing the fundamental information for phase calculation. This is a standard approach in SLI.
  • Color-Coded De Bruijn Sequence: This is the novel part. A De Bruijn sequence, known for its unique properties in error detection and decoding, is used to encode the period order of the phase. This sequence is embedded into the color channels (specifically, red and green) of the projected pattern. The blue channel carries the phase information.
  • Phase Unwrapping via Color Decoding: By projecting this color-coded fringe pattern and capturing it with a camera, the system can extract both intensity (for phase) and color (for sequence decoding). The De Bruijn sequence in the color channels allows for unambiguous determination of the phase period, thus solving the phase unwrapping problem.
  • Gabor Filter for Phase Estimation: A Gabor filter is employed to extract the wrapped phase from the intensity information, which is considered more robust than traditional Fourier Transform methods, especially at edges.
  • Stereo Matching: After phase unwrapping, standard stereo matching techniques between the projector and camera are used to triangulate and compute the final depth map.

The key advantage of this approach is that the color information directly aids in phase unwrapping, eliminating the need for multiple shots or complex calibration for ambiguity resolution. The use of multiple channels for phase information (blue channel for phase, red/green channels for sequence) also contributes to improved phase estimation quality.

Datasets

The paper does not explicitly mention training datasets. Instead, it focuses on experimental validation using:

  • Simulated Data: Experiments were conducted using 3ds Max software to simulate the SLI system and evaluate the method's procedure.
  • Real-World Scenes: The authors conducted experiments on various real-world objects and scenes, including:
  • Planes at different depths (0.9-1.4m) for quantitative analysis.
  • A discontinuous surface (cuboid next to a cube) to test edge recovery.
  • Plaster geometries and human body sculptures to assess edge and detail recovery.
  • Color discontinuous scenes with objects of varying materials (specular and diffuse) and colors to test robustness.

The paper also uses Kinect and ToF cameras as benchmarks for comparison, providing their depth maps alongside the proposed method's results.

Key Insights and Ideas

  • Color as a Direct Phase Unwrapping Aid: The most significant insight is using color channels to directly encode the phase period order via a De Bruijn sequence. This bypasses the need for multiple temporal patterns or complex spatial encoding schemes that can be sensitive to scene content.
  • De Bruijn Sequence for Robustness: The choice of De Bruijn sequence is crucial. Its mathematical properties allow for efficient and robust decoding of the period order, even in the presence of minor errors, by leveraging neighborhood information.
  • Multi-Channel Phase Information: Distributing phase information across multiple channels (blue for phase, red/green contributing to phase via color overlay) enhances the signal-to-noise ratio and improves the accuracy of phase estimation.
  • Single-Shot Advantage for Dynamic Scenes: The single-shot nature of the method, enabled by the color-coded De Bruijn sequence, makes it inherently suitable for capturing dynamic scenes where objects are in motion.

Key Results

  • Quantitative Performance:
    • The proposed method demonstrated lower Root Mean Square Error (RMSE) compared to Kinect and ToF cameras across various depths (0.9-1.4m). For example, at 1.4m, the proposed method had an RMSE of approximately 5mm, while Kinect was around 6mm and ToF around 7mm.
  • On a discontinuous surface, the proposed method achieved significantly lower mean absolute errors (e.g., 0.8366mm for a cube) compared to Kinect (1.4665mm) and ToF (1.8699mm).
  • Qualitative Performance:
    • The method successfully recovers dense depth maps with sharp edges and fine details, even on curved surfaces like a vase.
  • It outperforms Kinect and ToF in recovering details, with less blurring and a coarser surface appearance in the competitor's results.
  • The method shows robustness to different colors and materials, producing clear and dense depth maps in complex scenes where competitors struggle with color interference or granular effects.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that direct color decoding can be sensitive to the color of the target surface. While a voting mechanism is used to mitigate this, it's still a potential area for improvement.
  • The performance is affected by the optical absorption of materials, which can lead to errors in color decoding.
  • The paper mentions that the De Bruijn sequence is embedded into the fringe pattern to eliminate phase ambiguity, but the "trip point of the wrapped phase assists in overcoming the measurement sensitivity caused by the De Bruijn coding." This suggests a potential sensitivity that is partially addressed.
  • Future Work:
    • The authors suggest that a longer De Bruijn sequence could be used for greater robustness, though this would increase decoding complexity.
  • While not explicitly stated as future work, the robustness to color and material variations implies potential for further exploration in challenging real-world environments. The paper's success in complex scenes suggests it's a strong foundation for more advanced applications.

Summary updated

BibTeX
@misc{li2017single,
  author = {Fu Li and Baoyu Zhang and Guangming Shi and Yi Niu and Ruodai Li and Lili Yang and Xuemei Xie},
  title = {{Single-Shot Dense Depth Sensing with Color Sequence Coded Fringe Pattern}},
  year = {2017},
  doi = {10.3390/s17112558},
}
Figure