Micro-Baseline Structured Light

Vishwanath Saragadam, Raja Venkata, Jian Wang, Shree K. Nayar, Mohit GuptaICCV 2019 · OpenReview

Abstract

We propose Micro-baseline Structured Light (MSL), a novel 3D imaging approach designed for small form-factor devices such as cell-phones and miniature robots. MSL operates with small projector-camera baseline and low-cost projection hardware, and can recover scene depths with computationally lightweight algorithms. The main observation is that a small baseline leads to small disparities, enabling a first-order approximation of the non-linear SL image formation model. This leads to the key theoretical result of the paper: the MSL equation, a linearized version of SL image formation. MSL equation is under-constrained due to two unknowns (depth and albedo) at each pixel, but can be efficiently solved using a local least squares approach. We analyze the performance of MSL in terms of various system parameters such as projected pattern and baseline, and provide guidelines for optimizing performance. Armed with these insights, we build a prototype to experimentally examine the theory and its practicality.

Introduction

Traditional Structured Light (SL) systems, such as those used in facial recognition, typically require a significant physical distance (baseline) between the projector and the camera to achieve high-resolution depth sensing. However, the trend toward miniaturization in consumer electronics—from ultra-thin smartphones to micro-drones and medical endoscopes—leaves little room for such configurations.

Target applications for MSL include smartphones, micro-drones, and endoscopy.

When the baseline is reduced to a "micro" scale, the shifts in the projected pattern (disparities) become extremely small. Standard SL algorithms often fail in this regime because they are not optimized for sub-pixel precision under low-power constraints. Micro-Baseline Structured Light (MSL) addresses this by rethinking the fundamental image formation model.

The MSL Equation and Linearization

The core innovation of MSL is the linearization of the traditionally non-linear SL image formation model. In a micro-baseline setup, the pixel disparity ($u$) is sufficiently small that the projected pattern's intensity at a shifted position can be approximated using a first-order Taylor expansion.

The MSL system setup and the resulting linearized image formation model.

This leads to the MSL Equation: $$I(\mathbf{x}) = \rho(\mathbf{x})P(\mathbf{x}) + u(\mathbf{x})P'(\mathbf{x})$$ where $I$ is the measured intensity, $\rho$ is the scene albedo, $P$ is the projected pattern, and $P'$ is its spatial derivative. This linear relationship allows depth and albedo to be recovered simultaneously through a simple $2 \times 2$ matrix inversion at each pixel, making the process highly efficient and parallelizable for mobile GPUs.

Guided MSL for Complex Textures

A common failure point for simple SL systems is highly textured objects, where the object's own color interferes with the projected pattern. To mitigate this, the researchers introduced Guided MSL. By capturing two images—one with the pattern active and one with it off—the "off" image serves as a guide to isolate the scene's albedo. This significantly improves depth accuracy on complex surfaces.

Furthermore, the team determined that a periodic symmetric triangle pattern is mathematically optimal for MSL. Unlike sinusoidal patterns, triangle patterns provide a constant derivative, which minimizes bias in depth estimation.

Experimental Results

The researchers validated MSL using both simulated data from the Middlebury dataset and a custom hardware prototype.

  • Computational Performance: On a Google Pixel 2 XL, the MSL algorithm processed 3MP images in just 27ms, outperforming standard block-matching techniques by nearly 5x.
  • Accuracy: The system maintained a depth error of less than 8mm across various test scenarios, even with a baseline as small as 15mm.
  • Real-time Capability: The prototype demonstrated the ability to track hand gestures in 3D at 15 frames per second, proving its utility for interactive applications.

Conclusion and Future Work

MSL demonstrates that a small baseline, often viewed as a limitation, can be leveraged to simplify the computational complexity of 3D sensing. While the current implementation faces challenges with "edge adhesion" and global illumination (interreflections), it provides a robust framework for depth sensing on power-constrained, small-form-factor devices. Future research may integrate deep learning to better handle complex light transport and improve edge preservation.

Summary updated

Abstract

We propose Micro-baseline Structured Light (MSL), a novel 3D imaging approach designed for small form-factor devices such as cell-phones and miniature robots. MSL operates with small projector-camera baseline and low-cost projection hardware, and can recover scene depths with computationally lightweight algorithms. The main observation is that a small baseline leads to small disparities, enabling a first-order approximation of the non-linear SL image formation model. This leads to the key theoretical result of the paper: the MSL equation, a linearized version of SL image formation. MSL equation is under-constrained due to two unknowns (depth and albedo) at each pixel, but can be efficiently solved using a local least squares approach. We analyze the performance of MSL in terms of various system parameters such as projected pattern and baseline, and provide guidelines for optimizing performance. Armed with these insights, we build a prototype to experimentally examine the theory and its practicality.
BibTeX
@inproceedings{saragadam2019micro,
  author = {Vishwanath Saragadam and Raja Venkata and Jian Wang and Shree K. Nayar and Mohit Gupta},
  title = {{Micro-Baseline Structured Light}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=ava9-fSi8e2},
}
Figure