Surface Normal Estimation From Optimized and Distributed Light Sources Using DNN-Based Photometric Stereo

Takafumi Iwaguchi, Hiroshi KawasakiWACV 2023

TL;DR

This paper introduces a novel method for estimating surface normals using photometric stereo (PS) by optimizing the distribution of light sources with a deep neural network (DNN). By using a projector and an optical diffuser to create arbitrary, distributed light patterns, and training a DNN to interpret these patterns, the method significantly reduces the number of images required for accurate surface reconstruction, even for complex materials.

What Problem Does This Paper Address?

Photometric stereo (PS) is a powerful technique for reconstructing 3D surface information by analyzing how an object's appearance changes under different lighting conditions. Traditionally, PS relies on assumptions of Lambertian surfaces and directional light sources. However, real-world objects often exhibit complex reflectance properties (like specularity) and are illuminated by non-directional light sources (like area lights). To accurately estimate surface normals under these conditions, a large number of images are typically needed to account for noise and outliers, making the process cumbersome and time-consuming. Existing methods for handling non-Lambertian surfaces or non-directional lights often require extensive calibration or a large number of measurements, limiting their practical applicability. This paper aims to address the challenge of achieving accurate surface normal estimation with a significantly reduced number of images by intelligently designing the light sources themselves.

Methods

The core innovation of this work lies in the simultaneous optimization of light source patterns and the surface normal estimation network. The authors propose a system that uses a video projector to illuminate an optical diffuser, creating an arbitrary-shaped area light source. This setup is novel for PS as it deviates from traditional directional lighting.

The proposed method employs a deep neural network (DNN) with a two-part architecture:

  • Renderer Module: This module is a differentiable renderer that synthesizes shading images. It takes surface normals, depth maps, and light parameters (including the optimized lighting patterns) as input. The key here is that the lighting patterns themselves are learned.
  • PS Module: This module is a DNN designed for near-light photometric stereo (NLPS). It takes shading images and light source parameters as input to estimate the surface normal.

The system is trained end-to-end. The differentiable renderer allows the lighting patterns to be optimized by backpropagating the error from the PS module. This means the network learns not only how to estimate normals but also how to generate the "best" light patterns for the task, minimizing the number of images needed. The authors also introduce a "light branch" in the PS module that explicitly processes light source parameters, allowing the network to generalize to different lighting configurations beyond those seen during training.

Datasets

The paper introduces two datasets for training and evaluation:

  • Online Rendering (OR) Dataset: This dataset is generated on-the-fly during training. It uses rasterized meshes (from the blobby shape dataset) with varying surface normals and depth maps. Shading images are synthesized using linear matrix calculations for diffuse and Phong materials, with random variations in object position, rotation, and size. This dataset is crucial for learning efficient light patterns.
  • Eevee Prerender (EP) Dataset: This dataset is generated using the Eevee renderer in Blender. It includes a variety of materials (diffuse, specular, metallic) and shapes, with parameters for Disney's principled BSDF. This dataset is used for fine-tuning and evaluating the model's generalization capabilities, especially for real-world scenarios.

Key Insights and Ideas

  • Optimizing Light Patterns is Key: Instead of just adapting to existing light sources, the paper demonstrates the power of designing the light sources (patterns) to be optimal for the estimation task. This is a significant shift in perspective.
  • End-to-End Learning for Light and Normals: The integration of a differentiable renderer with the PS network enables simultaneous learning of lighting patterns and surface normal estimation, leading to more efficient and robust results.
  • Generalization through Light Parameters: Explicitly feeding light source parameters into the PS network allows it to generalize to unseen lighting configurations, a crucial step beyond traditional PS methods.
  • Diffuser + Projector for Arbitrary Area Lights: The use of a projector and diffuser to create controllable, arbitrary area lights is a practical and novel approach for generating complex illumination scenarios without physically arranging numerous lights.
  • Reduced Measurements, High Accuracy: The core promise of the method is realized: achieving high-quality surface normal estimation with a significantly smaller number of input images.

Key Results

  • Quantitative Accuracy: The proposed method achieved state-of-the-art results in terms of Mean Squared Error (MSE) and Mean Angular Error (MAE) when compared to other near-light PS methods on synthesized data. For example, the MAE was reduced to 13.68 compared to the best previous method's 21.57.
  • Robustness to Materials: The method demonstrated stable surface normal estimation across various materials, including diffuse, glossy, specular, and metallic, even in the presence of occlusion boundaries.
  • Effectiveness of Optimized Patterns: Experiments showed that optimized patterns significantly outperform random patterns, especially for specular materials, by preventing overlapping reflections and better utilizing shading information.
  • Real-World Validation: Experiments on real objects confirmed the method's effectiveness, with MAE values outperforming existing techniques, particularly for specular materials.
  • Generalization: The inclusion of a "light branch" in the network improved generalization performance for different camera-diffuser-projector configurations.

Limitations and Future Work

  • Low Exposure Conditions: While effective, the paper notes that the method is still influenced by low exposure conditions, particularly when dealing with materials that have weak sub-surface scattering.
  • Orthographic vs. Perspective Projection: In real-world 3D reconstruction, distortions were observed when using an orthographic projection model for reconstruction, despite the camera using perspective projection. This suggests a need for better integration with perspective projection models.
  • Future Work: The authors suggest exploring the construction of optimal patterns for textured objects and achieving high-frequency shape reconstruction using non-Lambertian photometric stereo (NSPS).

Summary updated

BibTeX
@inproceedings{iwaguchi2023surface,
  author = {Takafumi Iwaguchi and Hiroshi Kawasaki},
  title = {{Surface Normal Estimation From Optimized and Distributed Light Sources Using DNN-Based Photometric Stereo}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure