What Is Learned in Deep Uncalibrated Photometric Stereo?

Guanying Chen, Michael Waechter, Boxin Shi, Kwan-Yee K. Wong, Yasuyuki MatsushitaECCV 2020 · OpenReview

Abstract

This paper targets at discovering what a deep uncalibrated photometric stereo network learns to resolve the problem’s inherent ambiguity, and designing an effective network architecture based on the new insight to improve the performance. The recently proposed deep uncalibrated photometric stereo method achieved promising results in estimating directional lightings. However, what specifically inside the network contributes to its success remains a mystery. In this paper, we analyze the features learned by this method and find that they strikingly resemble attached shadows, shadings, and specular highlights, which are known to provide useful clues in resolving the generalized bas-relief (GBR) ambiguity. Based on this insight, we propose a guided calibration network, named GCNet, that explicitly leverages object shape and shading information for improved lighting estimation. Experiments on synthetic and real datasets show that GCNet achieves improved results in lighting estimation for photometric stereo, which echoes the findings of our analysis.

Introduction

Photometric stereo is a fundamental task in computer vision that aims to recover the detailed 3D surface normals of an object from multiple images taken under different lighting conditions. The "uncalibrated" version of this problem—where light source directions and intensities are unknown—is particularly challenging due to the Generalized Bas-Relief (GBR) ambiguity. This mathematical property implies that infinitely many combinations of surface shapes and light directions can produce the same image set.

While deep learning models like LCNet have recently achieved impressive results in resolving this ambiguity, they have largely functioned as "black boxes." This research seeks to demystify how these networks solve the problem and uses those findings to design a more effective architecture.

Analysis: The Network as a Physicist

To understand how deep networks resolve GBR ambiguity, the authors visualized the internal feature maps of LCNet. The analysis revealed that the network naturally learns to identify critical physical cues: attached shadows, shading gradients, and specular highlights. These are the same features that human experts and traditional physical models use to break mathematical ambiguities.

Visualization of learned features in LCNet, showing how the network identifies highlights, shadows, and shading gradients to resolve lighting ambiguity.

Through ablation studies, the researchers found that shading information is the most powerful cue for the network, proving more influential than shadows or highlights alone. This discovery suggests that the network essentially "learned physics" from data to solve the uncalibrated problem.

Proposed Architecture: GCNet

Armed with the insight that shape and shading are vital for lighting estimation, the authors developed the Guided Calibration Network (GCNet). GCNet employs a multi-stage, cyclic approach to break the "chicken-and-egg" cycle of shape and light estimation:

  1. L-Net1: Generates an initial, rough estimate of the light source parameters.
  2. N-Net: Uses the initial light estimates to predict the object's surface normals.
  3. L-Net2 (Guided Calibration): Re-estimates the lighting by using the original images "guided" by the surface normals and calculated shading from the previous steps.
The architecture of GCNet, featuring an initial lighting estimation (L-Net1), surface normal estimation (N-Net), and a guided refinement stage (L-Net2).

This iterative refinement allows the network to "check its work," using its own shape estimates to improve the accuracy of its lighting predictions.

Experimental Results

GCNet was evaluated on both synthetic datasets (rendered with Mitsuba using MERL BRDFs) and the real-world DiLiGenT benchmark.

  • Lighting Accuracy: On the DiLiGenT benchmark, GCNet achieved a Mean Angular Error (MAE) of 3.32° for light directions, a significant improvement over the previous state-of-the-art LCNet (4.92°) and traditional methods like PF14 (13.75°).
  • Surface Normal Estimation: By providing more accurate light calibration, GCNet also improved the final 3D shape recovery. When integrated with existing calibrated methods, it reduced shape errors from 16.5° to 8.6° in specific test cases.
  • Efficiency: The model is highly practical, processing a set of 96 images in approximately 0.5 seconds on a standard GPU.

Conclusion and Future Work

This research demonstrates that deep networks for uncalibrated photometric stereo succeed by learning to exploit physical cues like shading and shadows. By explicitly incorporating these cues into the GCNet architecture, the authors achieved state-of-the-art performance.

Future work remains in addressing cast shadows (shadows thrown by one part of an object onto another), which current models often treat as noise. Additionally, the method relies on surface orientation variety; perfectly flat surfaces remain a challenge as they provide fewer geometric cues for the network to interpret.

Summary updated

Abstract

This paper targets at discovering what a deep uncalibrated photometric stereo network learns to resolve the problem’s inherent ambiguity, and designing an effective network architecture based on the new insight to improve the performance. The recently proposed deep uncalibrated photometric stereo method achieved promising results in estimating directional lightings. However, what specifically inside the network contributes to its success remains a mystery. In this paper, we analyze the features learned by this method and find that they strikingly resemble attached shadows, shadings, and specular highlights, which are known to provide useful clues in resolving the generalized bas-relief (GBR) ambiguity. Based on this insight, we propose a guided calibration network, named GCNet, that explicitly leverages object shape and shading information for improved lighting estimation. Experiments on synthetic and real datasets show that GCNet achieves improved results in lighting estimation for photometric stereo, which echoes the findings of our analysis. We further demonstrate that GCNet can be directly integrated with existing calibrated methods to achieve improved results on surface normal estimation. Our code and model can be found at https://guanyingc.github.io/UPS-GCNet .
BibTeX
@inproceedings{chen2020what,
  author = {Guanying Chen and Michael Waechter and Boxin Shi and Kwan-Yee K. Wong and Yasuyuki Matsushita},
  title = {{What Is Learned in Deep Uncalibrated Photometric Stereo?}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=1IHbyMxVksQ},
}
Figure