Abstract
This paper targets at discovering what a deep uncalibrated photometric stereo network learns to resolve the problem’s inherent ambiguity, and designing an effective network architecture based on the new insight to improve the performance. The recently proposed deep uncalibrated photometric stereo method achieved promising results in estimating directional lightings. However, what specifically inside the network contributes to its success remains a mystery. In this paper, we analyze the features learned by this method and find that they strikingly resemble attached shadows, shadings, and specular highlights, which are known to provide useful clues in resolving the generalized bas-relief (GBR) ambiguity. Based on this insight, we propose a guided calibration network, named GCNet, that explicitly leverages object shape and shading information for improved lighting estimation. Experiments on synthetic and real datasets show that GCNet achieves improved results in lighting estimation for photometric stereo, which echoes the findings of our analysis.
Introduction
Photometric stereo is a fundamental task in computer vision that aims to recover the detailed 3D surface normals of an object from multiple images taken under different lighting conditions. The "uncalibrated" version of this problem—where light source directions and intensities are unknown—is particularly challenging due to the Generalized Bas-Relief (GBR) ambiguity. This mathematical property implies that infinitely many combinations of surface shapes and light directions can produce the same image set.
While deep learning models like LCNet have recently achieved impressive results in resolving this ambiguity, they have largely functioned as "black boxes." This research seeks to demystify how these networks solve the problem and uses those findings to design a more effective architecture.
Analysis: The Network as a Physicist
To understand how deep networks resolve GBR ambiguity, the authors visualized the internal feature maps of LCNet. The analysis revealed that the network naturally learns to identify critical physical cues: attached shadows, shading gradients, and specular highlights. These are the same features that human experts and traditional physical models use to break mathematical ambiguities.
Through ablation studies, the researchers found that shading information is the most powerful cue for the network, proving more influential than shadows or highlights alone. This discovery suggests that the network essentially "learned physics" from data to solve the uncalibrated problem.
Proposed Architecture: GCNet
Armed with the insight that shape and shading are vital for lighting estimation, the authors developed the Guided Calibration Network (GCNet). GCNet employs a multi-stage, cyclic approach to break the "chicken-and-egg" cycle of shape and light estimation:
- L-Net1: Generates an initial, rough estimate of the light source parameters.
- N-Net: Uses the initial light estimates to predict the object's surface normals.
- L-Net2 (Guided Calibration): Re-estimates the lighting by using the original images "guided" by the surface normals and calculated shading from the previous steps.
This iterative refinement allows the network to "check its work," using its own shape estimates to improve the accuracy of its lighting predictions.
Experimental Results
GCNet was evaluated on both synthetic datasets (rendered with Mitsuba using MERL BRDFs) and the real-world DiLiGenT benchmark.
- Lighting Accuracy: On the DiLiGenT benchmark, GCNet achieved a Mean Angular Error (MAE) of 3.32° for light directions, a significant improvement over the previous state-of-the-art LCNet (4.92°) and traditional methods like PF14 (13.75°).
- Surface Normal Estimation: By providing more accurate light calibration, GCNet also improved the final 3D shape recovery. When integrated with existing calibrated methods, it reduced shape errors from 16.5° to 8.6° in specific test cases.
- Efficiency: The model is highly practical, processing a set of 96 images in approximately 0.5 seconds on a standard GPU.
Conclusion and Future Work
This research demonstrates that deep networks for uncalibrated photometric stereo succeed by learning to exploit physical cues like shading and shadows. By explicitly incorporating these cues into the GCNet architecture, the authors achieved state-of-the-art performance.
Future work remains in addressing cast shadows (shadows thrown by one part of an object onto another), which current models often treat as noise. Additionally, the method relies on surface orientation variety; perfectly flat surfaces remain a challenge as they provide fewer geometric cues for the network to interpret.
Summary updated