Shape and Reflectance Reconstruction Using Concentric Multi-Spectral Light Field

Mingyuan Zhou, Yuqi Ding, Yu Ji, S. Susan Young, Jingyi Yu, Jinwei YeIEEE TPAMI 2020 · OpenReview

TL;DR

This paper introduces a novel "concentric multi-spectral light field" (CMSLF) system that captures shape and material properties of complex, non-Lambertian surfaces in a single shot. By arranging cameras and lights in concentric rings, each tuned to a specific spectrum, the system exploits unique specular highlight patterns to robustly reconstruct detailed 3D geometry and multi-spectral reflectance maps, outperforming existing methods.

What Problem Does This Paper Address?

Reconstructing the 3D shape and material properties (reflectance) of objects is crucial for many applications, from robotics to virtual reality. Traditional methods often struggle with "non-Lambertian" surfaces – those that don't reflect light uniformly in all directions, like shiny or metallic objects. These surfaces exhibit view-dependent effects, such as specular highlights (glints), which violate the assumptions of simpler models like the Lambertian model. This makes it difficult to accurately estimate shape and material properties using standard techniques like multi-view stereo or photometric stereo. Existing light-field methods, while powerful, often require additional assumptions about surface priors (like smoothness) or struggle with complex specularities.

Methods

The core innovation is the Concentric Multi-Spectral Light Field (CMSLF) acquisition system. This system features:

  • Concentric Camera Rings: Cameras are arranged in multiple concentric circles. Each ring is equipped with a specific spectral filter, meaning each ring captures the scene in a distinct color band.
  • Multi-Spectral Ring Light: A ring light surrounds the camera array, also emitting light in specific spectral bands that correspond to the camera filters. This ensures that for each spectral band, there's a corresponding camera capturing it under controlled, direction-varying illumination.
  • Spectral Multiplexing: By using narrowband spectral filters for both cameras and lights, the system can simultaneously capture multiple viewpoints and lighting variations without interference. This allows for a single-shot acquisition.

The paper then formulates a Phong Dichromatic Reflectance Model adapted for the CMSLF. This model separates surface reflectance into diffuse (body) and specular (interface) components. The key insight is that the concentric camera arrangement creates a unique, periodic pattern in specular highlights across different views within a "multi-spectral surface camera" (MSS-Cam) representation. This pattern is exploited for:

  1. Depth Initialization: A photo-consistency measure is used, and for non-Lambertian points, a periodicity consistency measure based on specular highlight patterns refines the initial depth estimate.
  2. Specular Component Removal: By analyzing the gradients of the MSS-Cams and fitting Fourier series to the periodic specular variations, the specular components can be robustly identified and removed. This yields a "specular-free" MSS-Cam.
  3. Shape and Reflectance Estimation: A multi-spectral photometric stereo approach is applied to the specular-free MSS-Cams to recover the surface normal and the multi-spectral reflectance coefficients. This is done iteratively to refine both depth and reflectance.

Datasets

The paper evaluates its method on both synthetic datasets and real-world experiments.

  • Synthetic Data: Generated using a multi-spectral renderer for spheres with varying diffuse and specular properties, as well as more complex models like a "buddha head" and "jadeware." These synthetic scenes allow for ground truth comparison.
  • Real-World Data: The authors construct their own CMSLF acquisition system and capture images of various real objects with different materials (plastic, ceramics).

No new benchmark datasets are introduced, but comparisons are made against existing methods using their respective input data formats (e.g., RGB images for 2D light fields).

Key Insights and Ideas

  • Concentricity is Key: The specific arrangement of cameras and lights in concentric rings, coupled with spectral filtering, is the crucial element. It generates a predictable, periodic specular highlight pattern that is highly informative for non-Lambertian surfaces.
  • Specular Highlights as Information, Not Noise: Instead of trying to remove specularities as outliers, the paper leverages their predictable variation across views to robustly estimate depth and material properties.
  • Single-Shot Acquisition for Complex Scenes: The ability to capture all necessary information in one go is a significant advantage, simplifying acquisition and avoiding issues with object movement or changing illumination.
  • Spectral Information Enhances Robustness: Using multi-spectral data allows for better separation of diffuse and specular components and provides richer information for reflectance reconstruction.

Key Results

  • Outperforms State-of-the-Art: The method demonstrates superior performance compared to existing light-field-based techniques for shape and reflectance reconstruction, particularly on non-Lambertian scenes.
  • Accurate Depth and Normal Estimation: Synthetic experiments show normal errors less than 2 degrees for simple spheres and less than 3 degrees for complex scenes.
  • High-Quality Reflectance Reconstruction: The method successfully recovers dense spectral reflectance maps, visualized in RGB, which closely match ground truth for synthetic data and show favorable results for real objects.
  • Robustness to Specularity: The technique effectively handles specular highlights, which are a major challenge for traditional methods.

Limitations and Future Work

  • Occlusions and Shadows: The paper acknowledges that in the presence of large occlusions and shadows, the method might fail to recover accurate depths and reflectances because these points may not receive enough spectral samples.
  • Computational Cost: While the acquisition is single-shot, the reconstruction process involves iterative refinement and optimization, which can be computationally intensive.
  • Future Work: The authors suggest further exploration of the method's capabilities, potentially extending it to more dynamic scenes or different material types. The current setup relies on specific camera and light configurations, and future work might explore more flexible or compact designs.

Summary updated

Abstract

Recovering the shape and reflectance of non-Lambertian surfaces remains a challenging problem in computer vision since the view-dependent appearance invalidates traditional photo-consistency constraint. In this paper, we introduce a novel concentric multi-spectral light field (CMSLF) design that is able to recover the shape and reflectance of surfaces of various materials in one shot. Our CMSLF system consists of an array of cameras arranged on concentric circles where each ring captures a specific spectrum. Coupled with a multi-spectral ring light, we are able to sample viewpoint and lighting variations in a single shot via spectral multiplexing. We further show that our concentric camera and light source setting results in a unique single-peak pattern in specularity variations across viewpoints. This property enables robust depth estimation for specular points. To estimate depth and multi-spectral reflectance map, we formulate a physics-based reflectance model for the CMSLF under the surface camera (S-Cam) representation. Extensive synthetic and real experiments show that our method outperforms the state-of-the-art shape reconstruction methods, especially for non-Lambertian surfaces.
BibTeX
@article{zhou2020shape,
  author = {Mingyuan Zhou and Yuqi Ding and Yu Ji and S. Susan Young and Jingyi Yu and Jinwei Ye},
  title = {{Shape and Reflectance Reconstruction Using Concentric Multi-Spectral Light Field}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2020},
  url = {https://openreview.net/forum?id=_R7_qzGj6ne},
}
Figure