Differentiable Display Photometric Stereo

Seokjun Choi, Seungwoo Yoon, Giljoo Nam, Seungyong Lee, Seung-Hwan Baek

Abstract

Photometric stereo is a powerful technique for 3D reconstruction, but its performance is often limited by the choice of illumination patterns. This paper introduces Differentiable Display Photometric Stereo (DDPS), a framework that optimizes the patterns displayed on a standard computer monitor to maximize the accuracy of surface normal estimation. By making the entire pipeline—from light emission to reconstruction—differentiable, the system learns optimal patterns tailored to specific camera-monitor setups. Our approach leverages the inherent polarization of LCD monitors to handle specularities and utilizes a 3D-printed dataset for supervised training. Results demonstrate that learned patterns significantly outperform traditional heuristics, achieving higher accuracy with fewer images.

Introduction

Reconstructing the 3D geometry of objects from 2D images is a fundamental task in computer vision. Photometric stereo achieves this by capturing multiple images of a static scene under varying lighting conditions. While high-end "light stages" provide controlled illumination, many researchers opt for programmable computer monitors as accessible light sources. However, the optimal patterns for such displays remain an open question. Traditional methods rely on manually designed patterns like color gradients or random noise, which may not be optimal for the specific geometry or material of the target object.

Problem Statement

The primary challenge in display-based photometric stereo is the design of illumination patterns. The design space is vast, and finding patterns that minimize reconstruction error across diverse materials and geometries is difficult. Furthermore, real-world factors such as monitor-camera calibration, non-Lambertian reflections (specularities), and ambient light introduce noise that traditional heuristic patterns often fail to handle robustly.

Methodology

DDPS addresses these challenges through an end-to-end differentiable framework that connects image formation with reconstruction.

Overview of the DDPS pipeline: (a) learned patterns, (b) experimental setup, (c) captured images under learned patterns, and (d) estimated surface normals.

Differentiable Image Formation

To avoid the computational cost of physics-based rendering, the authors employ a "basis image" approach. By capturing the scene under a set of "superpixel" lights, any complex display pattern can be simulated through a linear combination of these basis images. This formulation is fast, realistic, and fully differentiable.

Analytic Reconstructor and Optimization

The system uses a differentiable least-squares solver to estimate surface normals. The reconstruction error is backpropagated through the solver to the display patterns, allowing for direct optimization of the input light. This ensures the system learns to compensate for calibration errors and non-ideal object placement.

Polarization and Specularity Handling

LCD monitors naturally emit polarized light. By using a polarization camera, the system can optically separate specular reflections from diffuse ones. This allows the reconstructor to operate on the more stable diffuse component, significantly improving accuracy for shiny objects without needing complex material modeling.

Datasets and Evaluation

The system was trained on a novel dataset of 3D-printed objects with known ground-truth normals. This dataset includes 40 training and 4 test scenes with diverse geometries (busts, animals, characters) and materials (various PLA and PETG filaments). "In-the-wild" testing on human faces and household items demonstrated the generalizability of the learned patterns beyond the training set.

Key Results

Learned patterns achieved a significant reduction in reconstruction error compared to standard heuristics. For instance, the reconstruction error dropped from 0.1707 (for the standard OLAT pattern) to 0.0486 with optimized patterns. Furthermore, the system demonstrated high efficiency, with just two learned patterns providing better results than four traditional heuristic patterns.

High-fidelity surface normal map reconstructed using DDPS, capturing fine details like skin pores and wrinkles.

Despite being trained on 3D-printed plastic, the patterns worked exceptionally well on human skin and other "in-the-wild" materials, capturing fine details like pores and wrinkles that are often lost in traditional photometric stereo setups.

Conclusion

Differentiable Display Photometric Stereo represents a significant step forward in accessible 3D reconstruction. By optimizing the light source itself, the method achieves professional-grade results using consumer hardware. Future work will address global illumination effects, such as inter-reflections in concave shapes, and hardware-level synchronization for high-speed capture of moving objects.

Summary updated

Abstract

Photometric stereo leverages variations in illumination conditions to reconstruct surface normals. Display photometric stereo, which employs a conventional monitor as an illumination source, has the potential to overcome limitations often encountered in bulky and difficult-to-use conventional setups. In this paper, we present differentiable display photometric stereo (DDPS), addressing an often overlooked challenge in display photometric stereo: the design of display patterns. Departing from using heuristic display patterns, DDPS learns the display patterns that yield accurate normal reconstruction for a target system in an end-to-end manner. To this end, we propose a differentiable framework that couples basis-illumination image formation with analytic photometric-stereo reconstruction. The differentiable framework facilitates the effective learning of display patterns via auto-differentiation. Also, for training supervision, we propose to use 3D printing for creating a real-world training dataset, enabling accurate reconstruction on the target real-world setup. Finally, we exploit that conventional LCD monitors emit polarized light, which allows for the optical separation of diffuse and specular reflections when combined with a polarization camera, leading to accurate normal reconstruction. Extensive evaluation of DDPS shows improved normal-reconstruction accuracy compared to heuristic patterns and demonstrates compelling properties such as robustness to pattern initialization, calibration errors, and simplifications in image formation and reconstruction.
BibTeX
@inproceedings{choi2023differentiable,
  author = {Seokjun Choi and Seungwoo Yoon and Giljoo Nam and Seungyong Lee and Seung-Hwan Baek},
  title = {{Differentiable Display Photometric Stereo}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure