GPS-Net: Graph-based Photometric Stereo Network

Zhuokun Yao, Kun Li, Ying Fu, Haofeng Hu, Boxin Shi

TL;DR

GPS-Net is a hybrid deep learning architecture that solves the "unstructured data" problem in photometric stereo by combining graph convolutions with spatial CNNs. By treating pixel observations under varying light as nodes in a graph, it adaptively weights shadows and highlights to estimate high-resolution surface normals more accurately than previous per-pixel or all-pixel methods.

The Challenge of Unstructured Data in Photometric Stereo

Photometric stereo (PS) aims to recover the 3D shape of an object—specifically its surface normals—by taking multiple images from a fixed camera position under different lighting directions. The fundamental challenge for modern deep learning approaches is handling unstructured input: a sequence of images where the number of lights and their directions can vary arbitrarily between training and testing.

Existing data-driven methods generally fall into two flawed camps:

  1. Per-pixel methods: These analyze the intensity variation of a single pixel across all images. While they handle lighting math well, they ignore spatial context, often resulting in noisy or blurry results.
  2. All-pixel methods: These process entire images through a Convolutional Neural Network (CNN). They capture spatial details well but often struggle to model the complex "inter-image" relationship between different lighting conditions, leading to over-smoothed surfaces.
Comparison of existing per-pixel and all-pixel methods

GPS-Net bridges this gap, aiming for a "best of both worlds" approach that captures both inter-image lighting variations and intra-image spatial structures.

The GPS-Net Architecture

The authors introduce novel components that form the GPS-Net architecture to handle both unstructured lighting and spatial detail:

The GPS-Net architecture featuring the UFE-Layer and NR-Net
  • Unstructured Feature Extraction (UFE) Layer: Instead of forcing pixel data into a fixed-size "observation map," the UFE-Layer connects an arbitrary number of observations for a single pixel into a graph structure.
  • Structure-aware Graph Convolution (SGC) Filters: These filters operate on the unstructured graph. Crucially, they learn adaptive weights for each observation. This allows the network to automatically "down-weight" harmful outliers like attached or cast shadows and "up-weight" useful clues like specular highlights.
  • Normal Regression Network (NR-Net): After the graph layer extracts features, the NR-Net uses a multi-scale, multi-branch CNN to regress the final normal map. It specifically uses a mix of $1 \times 1$ and $3 \times 3$ convolutions to prevent the over-smoothing common in standard all-pixel networks, preserving fine surface details.

Datasets

The model was trained and evaluated using a mix of synthetic and real-world data:

  • Training: A synthetic dataset of 85,212 samples rendered from "Blobby" and "Sculpture" shapes using the MERL BRDF dataset to simulate realistic material reflections.
  • Evaluation (Real-world): Tested on the DiLiGenT benchmark (10 objects under 96 different lightings), the Light Stage Data Gallery, and the Gourd&Apple dataset.
  • Evaluation (Synthetic): Custom-rendered "Sphere" and "Bunny" shapes with 100 different materials to test robustness across varying BRDFs.

Key Insights

  • Graphs for Flexibility: Using a graph representation naturally handles the "arbitrary number of inputs" problem. Unlike a fixed-size grid, a graph doesn't care if you have 4 neighbors (lights) or 400.
  • Learning to Ignore Shadows: One of the core breakthroughs is the SGC filter's ability to act as a learnable outlier rejection mechanism. The network learns to assign near-zero weights to shadowed regions without being explicitly programmed to do so.
  • Multi-Branch Spatial Processing: Standard $3 \times 3$ CNNs can be too aggressive at smoothing. By adding a parallel $1 \times 1$ branch (which processes pixels independently), GPS-Net maintains the high-frequency details of the surface while still benefiting from the spatial context of the $3 \times 3$ branch.

Key Results

  • State-of-the-Art Accuracy: On the DiLiGenT benchmark, GPS-Net achieved a Mean Angular Error (MAE) of 9.34°, significantly outperforming previous leaders like PS-FCN (10.33°) and CNN-PS (16.16°).
  • Robustness to Light Density: Most models perform well either with very few lights (sparse) or many lights (dense), but not both. GPS-Net remains stable across the spectrum, achieving the best results for 4, 8, 10, 16, and 32 light sources.
  • Visual Fidelity: Qualitative results show that GPS-Net recovers fine details (like intricate carvings on complex objects) that other all-pixel methods blur away.

Limitations and Future Work

While GPS-Net is highly robust, the authors note that it does not yet achieve the absolute best possible accuracy under extremely dense lighting distributions (e.g., 96+ images) compared to some specialized, less flexible models. Future work will investigate more efficient ways to aggregate information from extremely dense photometric stereo observations to further push accuracy limits.

Summary updated

Abstract

Learning-based photometric stereo methods predict the surface normal either in a per-pixel or an all-pixel manner. Per-pixel methods explore the inter-image intensity variation of each pixel but ignore features from the intra-image spatial domain. All-pixel methods explore the intra-image intensity variation of each input image but pay less attention to the inter-image lighting variation. In this paper, we present a Graph-based Photometric Stereo Network, which unifies per-pixel and all-pixel processings to explore both inter-image and intra-image information. For per-pixel operation, we propose the Unstructured Feature Extraction Layer to connect an arbitrary number of input image-light pairs into graph structures, and introduce Structure-aware Graph Convolution filters to balance the input data by appropriately weighting shadows and specular highlights. For all-pixel operation, we propose the Normal Regression Network to make efficient use of the intra-image spatial information for predicting a surface normal map with rich details. Experimental results on the real-world benchmark show that our method achieves excellent performance under both sparse and dense lighting distributions.
BibTeX
@inproceedings{yao2020gps,
  author = {Zhuokun Yao and Kun Li and Ying Fu and Haofeng Hu and Boxin Shi},
  title = {{GPS-Net: Graph-based Photometric Stereo Network}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
}
Figure