Non-Lambertian Surface Shape and Reflectance Reconstruction Using Concentric Multi-Spectral Light Field

Mingyuan Zhou, Yu Ji, Yuqi Ding, Jinwei Ye, S. Susan Young, Jingyi YuarXiv 2019 · OpenReview

TL;DR

This paper introduces a novel "concentric multi-spectral light field" (CMSLF) system that captures shape and reflectance of complex, non-Lambertian surfaces in a single shot. By arranging cameras and lights in concentric rings, each capturing a specific spectrum, the system exploits unique specular highlight patterns to robustly estimate depth and material properties, outperforming existing light field methods.

What Problem Does This Paper Address?

Reconstructing the 3D shape and material properties (reflectance) of objects is a fundamental problem in computer vision with applications ranging from graphics to robotics. Traditional methods often assume "Lambertian" surfaces, meaning they reflect light equally in all directions. However, real-world objects frequently exhibit non-Lambertian behavior, characterized by view-dependent effects like specular highlights (shiny reflections). These highlights violate the Lambertian assumption, leading to inaccurate shape and reflectance estimations. Existing light field techniques, while powerful, often require additional assumptions about surface priors (like smoothness) to handle these complex materials. This paper aims to overcome these limitations by developing a system and method that can robustly reconstruct shape and reflectance for arbitrary non-Lambertian materials in a single capture.

Methods

The core innovation is the Concentric Multi-Spectral Light Field (CMSLF) acquisition system. This system features:

  • Concentric Camera Rings: Cameras are arranged in concentric circles, with each ring capturing a specific spectral band. This allows for sampling multiple viewpoints and spectral information simultaneously.
  • Multi-Spectral Ring Light: A ring light surrounds the camera array, also emitting light in specific spectral bands. This synchronized lighting and camera setup enables spectral multiplexing, where each camera ring effectively receives light from a corresponding spectral band of the light source, preventing interference.
  • Physical-Based Reflectance Model: The authors adapt a dichromatic Phong reflectance model within the Surface Camera (S-Cam) representation. This model decomposes reflectance into diffuse (body) and specular (interface) components.
  • Specular Analysis for Depth: A key insight is that the concentric camera/light configuration creates a unique, periodic pattern in specular highlights across different views. By analyzing the standard deviation of pixel intensities within MSS-Cam columns (representing different viewpoints for a single spectrum), they can distinguish diffuse from specular points. For specular points, they fit Fourier series to the intensity variations to capture their periodic nature.
  • Specular Component Removal: Using the derived specular patterns and a physical model, they iteratively estimate surface normals, specular reflectivity, and roughness. This allows them to effectively remove specular components from the captured data, yielding "specular-free" MSS-Cams.
  • Shape and Reflectance Estimation: Finally, they employ a multi-spectral photometric stereo approach on the specular-free MSS-Cams to recover the surface normal and multi-spectral reflectance coefficients.

The overall pipeline involves depth initialization (using photo-consistency and then specular consistency), specular component removal, and iterative refinement of shape and reflectance.

Datasets

The paper evaluates its method on both synthetic datasets and real-world data.

  • Synthetic Data: Generated using a multi-spectral renderer with spheres and complex models (Buddha head, jadeware) exhibiting varying reflectances (diffuse, specular, textured).
  • Real-World Data: A custom multi-spectral light field camera array was built and used to capture images of various objects with different materials (plastic, ceramics).

No new benchmark datasets are explicitly introduced, but the real-world captures serve as a new evaluation set for their proposed method.

Key Insights and Ideas

  • Specular Signatures in Concentric Light Fields: The concentric arrangement of cameras and lights creates a predictable, periodic pattern in specular highlights. This pattern is not just noise but a rich signal that can be exploited for robust depth estimation and specular removal.
  • Spectral Multiplexing for Efficiency: By matching camera spectral filters with light source spectral bands, the system efficiently captures multi-spectral information in a single shot without complex synchronization or interference.
  • Diffuse vs. Specular Separation via Consistency: The standard deviation of pixel intensities across different viewpoints (within a single spectral band) provides a simple yet effective way to distinguish diffuse from specular regions.
  • Fourier Analysis of Specularity: The periodic nature of specular highlight variations across views, when captured by the concentric setup, can be effectively modeled and analyzed using Fourier series, enabling precise specular removal.
  • Single-Shot Reconstruction for Non-Lambertian Surfaces: The primary breakthrough is achieving robust shape and reflectance reconstruction for complex, non-Lambertian materials in a single capture, a significant improvement over methods requiring multiple views or specific priors.

Key Results

  • Quantitative Performance: The method outperforms state-of-the-art light field-based techniques, particularly for non-Lambertian scenes. For synthetic experiments, normal errors are reported to be less than 2° for simple spheres and less than 3° for complex scenes.
  • Qualitative Results: The method successfully reconstructs detailed 3D shapes and multi-spectral reflectance maps for both synthetic and real-world objects with diverse materials. The reconstructed reflectance maps, visualized in RGB, show favorable results.
  • Robustness: The approach demonstrates robustness in separating and removing specular components, leading to high-frequency details in the reconstructed surfaces.

Limitations and Future Work

  • Occlusions and Shadows: The paper acknowledges that in the presence of large occlusions and shadows, the method might fail to recover accurate depths and reflectances because these points may not receive enough spectral samples.
  • Hardware Setup: The current hardware setup involves a translation stage for the camera, which might be less ideal than a dedicated multi-camera array for capturing dynamic scenes.
  • Future Work: While not explicitly detailed, the authors' success suggests future work could focus on extending the system to dynamic scenes, exploring more complex reflectance models, and potentially integrating with other sensing modalities. The current method's reliance on specific spectral bands might also be a point for future exploration to adapt to broader spectral ranges or different material types.

Summary updated

Abstract

Recovering the shape and reflectance of non-Lambertian surfaces remains a challenging problem in computer vision since the view-dependent appearance invalidates traditional photo-consistency constraint. In this paper, we introduce a novel concentric multi-spectral light field (CMSLF) design that is able to recover the shape and reflectance of surfaces with arbitrary material in one shot. Our CMSLF system consists of an array of cameras arranged on concentric circles where each ring captures a specific spectrum. Coupled with a multi-spectral ring light, we are able to sample viewpoint and lighting variations in a single shot via spectral multiplexing. We further show that such concentric camera/light setting results in a unique pattern of specular changes across views that enables robust depth estimation. We formulate a physical-based reflectance model on CMSLF to estimate depth and multi-spectral reflectance map without imposing any surface prior. Extensive synthetic and real experiments show that our method outperforms state-of-the-art light field-based techniques, especially in non-Lambertian scenes.
BibTeX
@misc{zhou2019non,
  author = {Mingyuan Zhou and Yu Ji and Yuqi Ding and Jinwei Ye and S. Susan Young and Jingyi Yu},
  title = {{Non-Lambertian Surface Shape and Reflectance Reconstruction Using Concentric Multi-Spectral Light Field}},
  year = {2019},
  eprint = {1904.04875},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.04875},
}
Figure