Abstract
This paper makes a first attempt to bring the Shape from Polarization (SfP) problem to the realm of deep learning. The previous state-of-the-art methods for SfP have been purely physics-based. We see value in these principled models, and blend these physical models as priors into a neural network architecture. This proposed approach achieves results that exceed the previous state-of-the-art on a challenging dataset we introduce. This dataset consists of polarization images taken over a range of object textures, paints, and lighting conditions. We report that our proposed method achieves the lowest test error on each tested condition in our dataset, showing the value of blending data-driven and physics-driven approaches.
Introduction
Shape from Polarization (SfP) has long been a pursuit in computer vision, aiming to recover 3D geometry from a single viewpoint by analyzing the polarization state of reflected light. Traditionally, this field has relied on rigid physical equations. However, physics-based models face significant hurdles: the $\pi$-ambiguity problem (where multiple surface normals are mathematically possible), dependency on unknown material refractive indices, and high sensitivity to Poisson shot noise due to light intensity reduction by filters.
Methodology: Blending Physics with Deep Learning
The authors propose a hybrid architecture that treats traditional physical models as priors for a neural network rather than replacing them entirely.
The network architecture is a fully convolutional encoder-decoder that processes two types of inputs:
- Polarization Images: Four images captured at $0^\circ, 45^\circ, 90^\circ, \text{and } 135^\circ$.
- Physical Priors: Rough surface normal maps calculated using traditional diffuse and specular reflection equations.
To preserve subtle polarization signals during downsampling, the model utilizes Spatially-Adaptive Normalization (SPADE) blocks. These blocks hierarchically re-inject the original polarization data into the upsampling layers, preventing the loss of critical geometric cues. The model is optimized using a cosine similarity loss to minimize the angular difference between estimated and ground-truth surface normals.
The Deep Shape from Polarization Dataset
A significant contribution of this work is the introduction of the first large-scale real-world SfP dataset. It includes 25 diverse objects captured in four orientations across three lighting environments: Indoors, Outdoor Overcast, and Outdoor Sunlight. Ground truth 3D shapes were obtained via a structured-light scanner, resulting in 300 unique image sets that serve as a new benchmark for the field.
Key Results
The proposed method demonstrates a substantial improvement over purely physics-based approaches.
Quantitatively, the model achieved a Mean Angular Error (MAE) of 18.52° on the test set, compared to 41.44° for the previous state-of-the-art (Smith et al.). Notably, the model maintained robustness in challenging outdoor sunlight conditions where traditional methods often fail. Qualitatively, the network successfully resolves the $\pi$-ambiguity by leveraging global context and recovers fine geometric details that were previously obscured by noise or "texture-copy" artifacts.
Discussion and Limitations
While this work represents a major leap forward, limitations remain. The current pipeline involves a pre-processing step to calculate physical priors, which introduces computational latency (approx. 1 second per frame). Furthermore, the model does not yet explicitly account for inter-reflections between object surfaces, which can distort polarization signals. Future research will likely focus on end-to-end optimization and the integration of more complex light-transport models.
Summary updated