IR Shape From Shading Enhanced RGBD for 3D Scanning

Nolan Lunscher, Georges Younes, David Abou Chacra, Henry A. Leopold, Jeremy Pinto, John ZelekJournal of Computational Vision and Imaging Systems 2016 · DOI

Abstract: RGBD Cameras such as the Microsoft Kinect that can quickly provide usable depth maps have become very affordable, and thus very popular and abundant in recent years. Beyond gaming, RGBD cameras can have numerous applications, including their use in affordable 3D scanners. These cameras however are limited in their ability to capture finer details. We explore the use of additional 3D reconstruction algorithms to enhance the depth maps produced from RGBD cameras, allowing them to capture more detail.

Introduction: The Resolution Gap in Consumer RGBD

Low-cost RGBD (Red, Green, Blue + Depth) cameras have revolutionized robotics and 3D scanning, but they face a significant "resolution" problem. While effective at capturing the general layout of a scene, they struggle with fine surface details. Resulting 3D models often appear "noisy" or "blocky" due to quantization errors and sensor limitations. Existing solutions like KinectFusion address this by averaging data from multiple viewpoints, but this requires static subjects. This paper explores a single-view refinement method to improve depth quality from a single snapshot, bridging the gap between "cheap and noisy" and "high-precision" scanning.

Methodology: Leveraging Infrared Shape from Shading

The researchers build upon the RGBD-Fusion algorithm. Traditionally, Shape from Shading (SfS) estimates 3D geometry by analyzing light reflection in standard color (RGB) images. However, RGB images are often "polluted" by ambient light, shadows, and complex textures that can mislead the algorithm.

The authors propose using the Infrared (IR) image already captured by the sensor. This offers two major advantages:

  1. Controlled Lighting: Most RGBD cameras use an internal IR projector. Treating this projected light as the primary source simplifies the shading analysis.
  2. Ambient Filtering: IR cameras typically use narrow-band filters to block visible light, making the system robust in indoor environments where overhead lights might otherwise interfere.
The infrared image captured by the sensor provides the high-frequency shading information necessary for surface refinement.

By fusing the initial, coarse depth map with high-frequency detail extracted from the IR shading information, the system produces a refined 3D point cloud.

Results and Qualitative Analysis

The method was evaluated using live data from an Intel RealSense F200 RGBD camera. The results demonstrate a "noticeable improvement" over raw sensor output.

Comparison between the raw point cloud (top) and the refined output using IR-based RGBD-Fusion (bottom).

As seen in the comparison, the raw output (top) appears "stair-stepped" and lacks definition on subtle surfaces. After applying the IR-based refinement (bottom), the surfaces appear significantly smoother, and fine-grained details that were previously lost in sensor noise become visible. Interestingly, the non-uniform nature of the projected IR pattern did not significantly degrade the shading-based refinement.

Discussion and Future Work

While promising, this preliminary work has limitations. The evaluation is currently qualitative; future research should include numerical metrics (such as Root Mean Square Error) against ground-truth laser scans. Additionally, the current approach assumes a simple lighting model. Explicitly modeling the highly non-uniform pattern of IR projectors could further enhance accuracy. Finally, the impact of material properties—such as shiny or dark objects that absorb or specularly reflect IR light—remains an area for further investigation.

Summary updated

Abstract

RGBD Cameras such as the Microsoft Kinect that can quickly provideusable depth maps have become very affordable, and thusvery popular and abundant in recent years. Beyond gaming, RGBDcameras can have numerous applications, including their use in affordable3D scanners. These cameras however are limited in theirability to capture finer details. We explore the use of additional3D reconstruction algorithms to enhance the depth maps producedfrom RGBD cameras, allowing them to capture more detail.
BibTeX
@article{lunscher2016ir,
  author = {Nolan Lunscher and Georges Younes and David Abou Chacra and Henry A. Leopold and Jeremy Pinto and John Zelek},
  title = {{IR Shape From Shading Enhanced RGBD for 3D Scanning}},
  journal = {Journal of Computational Vision and Imaging Systems},
  year = {2016},
  doi = {10.15353/vsnl.v2i1.104},
}
Figure