Abstract
Photometric stereo is a powerful technique for 3D reconstruction, but its performance is often limited by the choice of illumination patterns. This paper introduces Differentiable Display Photometric Stereo (DDPS), a framework that optimizes the patterns displayed on a standard computer monitor to maximize the accuracy of surface normal estimation. By making the entire pipeline—from light emission to reconstruction—differentiable, the system learns optimal patterns tailored to specific camera-monitor setups. Our approach leverages the inherent polarization of LCD monitors to handle specularities and utilizes a 3D-printed dataset for supervised training. Results demonstrate that learned patterns significantly outperform traditional heuristics, achieving higher accuracy with fewer images.
Introduction
Reconstructing the 3D geometry of objects from 2D images is a fundamental task in computer vision. Photometric stereo achieves this by capturing multiple images of a static scene under varying lighting conditions. While high-end "light stages" provide controlled illumination, many researchers opt for programmable computer monitors as accessible light sources. However, the optimal patterns for such displays remain an open question. Traditional methods rely on manually designed patterns like color gradients or random noise, which may not be optimal for the specific geometry or material of the target object.
Problem Statement
The primary challenge in display-based photometric stereo is the design of illumination patterns. The design space is vast, and finding patterns that minimize reconstruction error across diverse materials and geometries is difficult. Furthermore, real-world factors such as monitor-camera calibration, non-Lambertian reflections (specularities), and ambient light introduce noise that traditional heuristic patterns often fail to handle robustly.
Methodology
DDPS addresses these challenges through an end-to-end differentiable framework that connects image formation with reconstruction.
Differentiable Image Formation
To avoid the computational cost of physics-based rendering, the authors employ a "basis image" approach. By capturing the scene under a set of "superpixel" lights, any complex display pattern can be simulated through a linear combination of these basis images. This formulation is fast, realistic, and fully differentiable.
Analytic Reconstructor and Optimization
The system uses a differentiable least-squares solver to estimate surface normals. The reconstruction error is backpropagated through the solver to the display patterns, allowing for direct optimization of the input light. This ensures the system learns to compensate for calibration errors and non-ideal object placement.
Polarization and Specularity Handling
LCD monitors naturally emit polarized light. By using a polarization camera, the system can optically separate specular reflections from diffuse ones. This allows the reconstructor to operate on the more stable diffuse component, significantly improving accuracy for shiny objects without needing complex material modeling.
Datasets and Evaluation
The system was trained on a novel dataset of 3D-printed objects with known ground-truth normals. This dataset includes 40 training and 4 test scenes with diverse geometries (busts, animals, characters) and materials (various PLA and PETG filaments). "In-the-wild" testing on human faces and household items demonstrated the generalizability of the learned patterns beyond the training set.
Key Results
Learned patterns achieved a significant reduction in reconstruction error compared to standard heuristics. For instance, the reconstruction error dropped from 0.1707 (for the standard OLAT pattern) to 0.0486 with optimized patterns. Furthermore, the system demonstrated high efficiency, with just two learned patterns providing better results than four traditional heuristic patterns.
Despite being trained on 3D-printed plastic, the patterns worked exceptionally well on human skin and other "in-the-wild" materials, capturing fine details like pores and wrinkles that are often lost in traditional photometric stereo setups.
Conclusion
Differentiable Display Photometric Stereo represents a significant step forward in accessible 3D reconstruction. By optimizing the light source itself, the method achieves professional-grade results using consumer hardware. Future work will address global illumination effects, such as inter-reflections in concave shapes, and hardware-level synchronization for high-speed capture of moving objects.
Summary updated