Leveraging Spatial and Photometric Context for Calibrated Non-Lambertian Photometric Stereo

David Honzátko, Engin Türetken, Pascal Fua, L. Andrea Dunbar

TL;DR

This paper introduces a highly efficient fully-convolutional network for calibrated photometric stereo that simultaneously leverages both spatial and photometric context. By utilizing separable 4D convolutions and reframing surface normal estimation as a 2D Gaussian heat-map regression problem, the authors achieve state-of-the-art accuracy while reducing network parameters and computational costs by nearly an order of magnitude.

What Problem Does This Paper Address?

Photometric stereo is a classic computer vision task aimed at estimating the 3D shape (surface normals) of an object from multiple images taken from a fixed viewpoint under varying illumination directions. While straightforward for ideal, diffuse (Lambertian) surfaces, real-world objects present severe challenges like specular highlights, cast shadows, and inter-reflections.

To handle these non-Lambertian effects, recent deep learning methods have split into two camps:

  1. Spatial-first approaches: Use 2D CNNs to capture spatial context (relationships between neighboring pixels) but aggregate illumination data poorly using basic pooling layers.
  2. Photometric-first approaches: Convert pixel intensities across different light directions into "observation maps" to model complex material reflectance per pixel, completely ignoring spatial context.

The authors address this dichotomy by bridging the gap, proposing a method that extracts both spatial and photometric cues simultaneously without incurring an intractable computational penalty.

Methods

The authors construct a 5D tensor input ($Height \times Width \times w \times w \times c$) that combines local spatial image patches with their corresponding 2D observation maps (which encode photometric context).

Processing a 5D tensor directly is incredibly memory-intensive. To make this practical, the pipeline introduces two key architectural innovations:

  • Separable 4D Convolutions: Instead of full 4D convolutions, the network applies two distinct 2D convolutional layers sequentially—one over the photometric dimensions (with same padding) and one over the spatial dimensions (with valid padding). This reduces a local $5 \times 5$ spatial patch down to $1 \times 1$ efficiently.
  • Gaussian Heat-map Regression: Instead of directly regressing to 3D coordinate values $(x, y, z)$ for surface normals, the network outputs a 2D Gaussian heat-map representing the normal's projection onto a unit hemisphere. The final normal is extracted by finding the center of mass of the highest intensity region.

Additionally, the architecture is entirely bias-free, which enforces scale equivariance, making the network inherently robust to variations in light intensity and preventing overfitting.

Figure 1. Our pipeline in a nutshell: A block of observation maps created from the local image patches around a pixel and known illumination directions are given as input to our 4D fully- convolutional network, which outputs a Gaussian heat-map. The surface normal can be estimated from this heat-map by a projec- tion onto a unit hemisphere.

Datasets

  • CyclesPS (Training & Testing): A synthetic dataset consisting of 15 objects rendered with complex geometries and Disney's principled BRDFs (metallic, specular, diffuse). To simulate real-world variety, scenes are broken into superpixels with distinct material properties.
  • DiLiGenT Benchmark (Evaluation): The standard real-world benchmark for photometric stereo, containing 10 objects captured under 96 distinct, calibrated light directions.

Key Insights and Ideas

  • The Power of Heat-maps over Direct Regression: Moving from direct coordinate regression to heat-map regression was the primary driver behind the massive reduction in network size. Heat-maps simplify the optimization landscape, allowing a much smaller U-Net structure to achieve superior accuracy.
  • Scale Equivariance via Bias-Free Layers: Omitting bias terms acts as a powerful regularizer. Because the network is scale-equivariant, it focuses strictly on relative intensity differences—the core physical cue required to infer shape from shading.
  • Spatio-Photometric Synergy: Incorporating even a small local spatial context ($5 \times 5$ pixels) significantly smooths out normal estimations and resolves ambiguities in challenging regions like thin hinge lines and specular boundaries.

Key Results

The proposed method was evaluated against several leading techniques on the DiLiGenT benchmark:

  • Efficiency: The network requires only 0.5M parameters and 1M MACs (Multiply-Accumulate operations), representing nearly an order of magnitude reduction compared to the state-of-the-art PX-NET (4.5M parameters, 9M MACs).
  • Accuracy: On the DiLiGenT benchmark, the model achieved a state-of-the-art mean angular error of 6.37° (using 12 evaluation rotations), outperforming its direct predecessor CNN-PS (7.2°).
  • Qualitative Improvement: The method produces visibly smoother and more accurate normal maps on complex, shiny surfaces such as the Harvest, Reading, and Cow objects from the DiLiGenT dataset.
Figure 4. Ground truth and predicted normal maps for our and CNN-PS [6] methods and their respective angular errors in degrees. From top to bottom, the results for the Harvest, Reading, and Cow objects and their representative color pictures.

Limitations and Future Work

Despite its success, the model's spatial context is strictly local ($5 \times 5$ patches) due to the hardware memory constraints of handling high-dimensional tensors. Consequently, the network still struggles with large global illumination effects, such as massive cast shadows or broad inter-reflections that span across large areas of an object.

The authors note that future work will focus on overcoming these memory bottlenecks to allow for a more holistic, large-scale integration of spatial context.

Summary updated

Abstract

The problem of estimating a surface shape from its observed reflectance properties still remains a challenging task in computer vision. The presence of global illumination effects such as inter-reflections or cast shadows makes the task particularly difficult for non-convex real-world surfaces. State-of-the-art methods for calibrated photometric stereo address these issues using convolutional neural networks (CNNs) that primarily aim to capture either the spatial context among adjacent pixels or the photometric one formed by illuminating a sample from adjacent directions. In this paper, we bridge these two objectives and introduce an efficient fully-convolutional architecture that can leverage both spatial and photometric context simultaneously. In contrast to existing approaches that rely on standard 2D CNNs and regress directly to surface normals, we argue that using separable 4D convolutions and regressing to 2D Gaussian heat-maps severely reduces the size of the network and makes inference more efficient. Our experimental results on a real-world photometric stereo benchmark show that the proposed approach outperforms the existing methods both in efficiency and accuracy.
BibTeX
@inproceedings{honzatko2021leveraging,
  author = {David Honzátko and Engin Türetken and Pascal Fua and L. Andrea Dunbar},
  title = {{Leveraging Spatial and Photometric Context for Calibrated Non-Lambertian Photometric Stereo}},
  booktitle = {2021},
  year = {2021},
  doi = {10.1109/3dv53792.2021.00049},
}
Figure