Single Image Multi-Spectral Photometric Stereo Using a Split U-Shaped CNN

Doris Antensteiner, Svorad Stolc, Daniel SoukupWorkshops @ CVPR 2019 · DOI

Introduction

Photometric Stereo (PS) is a fundamental computer vision technique used to recover high-detail surface geometry and albedo by observing an object under varying lighting conditions. Traditionally, this requires a "multi-shot" approach, capturing dozens of images while systematically moving a light source. This requirement limits the technique's utility in dynamic environments, such as high-speed industrial inspection or handheld scanning.

While "single-shot" methods exist—typically utilizing three colored light sources (Red, Green, Blue) from different angles—they often rely on traditional mathematical models like Least Squares (LS). These methods are highly sensitive to image noise, shadows, and non-Lambertian (shiny) surfaces, frequently resulting in reconstructions with significant artifacts. This paper proposes a deep learning solution to these challenges using a specialized "Split U-Net" architecture.

The Split U-Net Architecture

The authors introduce a Convolutional Neural Network (CNN) designed specifically for the dual task of geometry and appearance estimation. The architecture, termed a Split U-Net, utilizes a shared encoder to compress input features and two distinct decoder paths.

The Split U-Net architecture featuring a shared encoder and dual decoder paths for independent albedo and normal estimation.

As shown in the architecture diagram, the network processes an $M \times N \times 3$ RGB input. The shared encoder extracts high-level features, which are then passed through "pinhole connections" to two separate decoders. One decoder is dedicated to estimating surface normals (the 3D orientation of each pixel), while the other estimates the albedo (the base reflectivity). This separation allows the network to learn specialized features for geometry and appearance independently while benefiting from a shared contextual understanding of the scene.

Methodology and Data

The hardware setup involves three light sources (RGB) positioned at 120° intervals. A single camera capture produces one RGB image where each color channel represents the object illuminated from a unique direction.

To train the network, the researchers developed a custom dataset using a specialized light dome with 32 illumination sources. Ground truth (GT) data was generated using all 32 lights and standard PS algorithms. The training input was then simulated by selecting three specific light directions to mimic the single-shot RGB setup. Although the dataset is relatively small (78 training, 20 validation, and 28 test samples), it consists of high-quality captures of coins and textured surfaces that provide a rigorous test for the model's precision.

Results and Qualitative Analysis

The proposed RGB U-Net was compared against the standard 3-Light Least Squares (3L-PS) method. The results demonstrate that the CNN acts as a powerful regularizer. In regions where traditional mathematical fits fail due to noise or low light, the CNN utilizes learned priors to produce smooth, clean reconstructions.

Qualitative comparison between the proposed RGB U-Net and traditional 3-light photometric stereo (3L-PS) against the 32-light ground truth (GT).

Quantitatively, the U-Net achieved a Mean Squared Error (MSE) for normals of 0.00353 on the test set, outperforming the 0.00371 achieved by the traditional method. The improvement in albedo estimation was even more significant, with the U-Net reaching an MSE of 0.00148 compared to 0.06832 for the 3L-PS method. Visually, the U-Net results are noticeably cleaner, particularly on background planes where traditional methods produce significant "salt and pepper" noise.

Conclusion and Future Work

The Split U-Net approach provides a robust framework for single-shot photometric stereo, offering superior noise resilience and accuracy over traditional methods. While currently tested on a domain-specific dataset of coins and flat textures, the architecture's success suggests significant potential for industrial applications, such as real-time quality control on moving conveyor belts. Future research may focus on expanding the dataset to include more complex 3D geometries and investigating the network's sensitivity to varying light configurations.

Summary updated

BibTeX
@inproceedings{antensteiner2019single,
  author = {Doris Antensteiner and Svorad Stolc and Daniel Soukup},
  title = {{Single Image Multi-Spectral Photometric Stereo Using a Split U-Shaped CNN}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  doi = {10.1109/cvprw.2019.00065},
}
Figure