TL;DR
GPS-Net is a hybrid deep learning architecture that solves the "unstructured data" problem in photometric stereo by combining graph convolutions with spatial CNNs. By treating pixel observations under varying light as nodes in a graph, it adaptively weights shadows and highlights to estimate high-resolution surface normals more accurately than previous per-pixel or all-pixel methods.
The Challenge of Unstructured Data in Photometric Stereo
Photometric stereo (PS) aims to recover the 3D shape of an object—specifically its surface normals—by taking multiple images from a fixed camera position under different lighting directions. The fundamental challenge for modern deep learning approaches is handling unstructured input: a sequence of images where the number of lights and their directions can vary arbitrarily between training and testing.
Existing data-driven methods generally fall into two flawed camps:
- Per-pixel methods: These analyze the intensity variation of a single pixel across all images. While they handle lighting math well, they ignore spatial context, often resulting in noisy or blurry results.
- All-pixel methods: These process entire images through a Convolutional Neural Network (CNN). They capture spatial details well but often struggle to model the complex "inter-image" relationship between different lighting conditions, leading to over-smoothed surfaces.
GPS-Net bridges this gap, aiming for a "best of both worlds" approach that captures both inter-image lighting variations and intra-image spatial structures.
The GPS-Net Architecture
The authors introduce novel components that form the GPS-Net architecture to handle both unstructured lighting and spatial detail:
- Unstructured Feature Extraction (UFE) Layer: Instead of forcing pixel data into a fixed-size "observation map," the UFE-Layer connects an arbitrary number of observations for a single pixel into a graph structure.
- Structure-aware Graph Convolution (SGC) Filters: These filters operate on the unstructured graph. Crucially, they learn adaptive weights for each observation. This allows the network to automatically "down-weight" harmful outliers like attached or cast shadows and "up-weight" useful clues like specular highlights.
- Normal Regression Network (NR-Net): After the graph layer extracts features, the NR-Net uses a multi-scale, multi-branch CNN to regress the final normal map. It specifically uses a mix of $1 \times 1$ and $3 \times 3$ convolutions to prevent the over-smoothing common in standard all-pixel networks, preserving fine surface details.
Datasets
The model was trained and evaluated using a mix of synthetic and real-world data:
- Training: A synthetic dataset of 85,212 samples rendered from "Blobby" and "Sculpture" shapes using the MERL BRDF dataset to simulate realistic material reflections.
- Evaluation (Real-world): Tested on the DiLiGenT benchmark (10 objects under 96 different lightings), the Light Stage Data Gallery, and the Gourd&Apple dataset.
- Evaluation (Synthetic): Custom-rendered "Sphere" and "Bunny" shapes with 100 different materials to test robustness across varying BRDFs.
Key Insights
- Graphs for Flexibility: Using a graph representation naturally handles the "arbitrary number of inputs" problem. Unlike a fixed-size grid, a graph doesn't care if you have 4 neighbors (lights) or 400.
- Learning to Ignore Shadows: One of the core breakthroughs is the SGC filter's ability to act as a learnable outlier rejection mechanism. The network learns to assign near-zero weights to shadowed regions without being explicitly programmed to do so.
- Multi-Branch Spatial Processing: Standard $3 \times 3$ CNNs can be too aggressive at smoothing. By adding a parallel $1 \times 1$ branch (which processes pixels independently), GPS-Net maintains the high-frequency details of the surface while still benefiting from the spatial context of the $3 \times 3$ branch.
Key Results
- State-of-the-Art Accuracy: On the DiLiGenT benchmark, GPS-Net achieved a Mean Angular Error (MAE) of 9.34°, significantly outperforming previous leaders like PS-FCN (10.33°) and CNN-PS (16.16°).
- Robustness to Light Density: Most models perform well either with very few lights (sparse) or many lights (dense), but not both. GPS-Net remains stable across the spectrum, achieving the best results for 4, 8, 10, 16, and 32 light sources.
- Visual Fidelity: Qualitative results show that GPS-Net recovers fine details (like intricate carvings on complex objects) that other all-pixel methods blur away.
Limitations and Future Work
While GPS-Net is highly robust, the authors note that it does not yet achieve the absolute best possible accuracy under extremely dense lighting distributions (e.g., 96+ images) compared to some specialized, less flexible models. Future work will investigate more efficient ways to aggregate information from extremely dense photometric stereo observations to further push accuracy limits.
Summary updated