Photometric stereo is a classic technique in computer vision used to recover the fine-grained 3D geometry of an object by observing it under varying lighting conditions. By keeping the camera fixed and moving the light source, we can observe how surface brightness changes, allowing us to mathematically "solve" for the orientation of the surface at every pixel.
However, for decades, this technique was largely confined to laboratory settings. The reason lies in a fundamental simplification: the Lambertian reflectance assumption. Most classical models assume that surfaces are perfectly matte, reflecting light uniformly in all directions. In the real world, objects are shiny, metallic, transparent, or have complex shapes that cast shadows upon themselves. Modern research in this field focuses on breaking the Lambertian barrier to reconstruct 3D information from these "difficult" materials.
The Lambertian Assumption and Its Failure
The core of photometric stereo relies on the relationship between light direction, surface normal (the direction the surface is facing), and observed intensity. In a Lambertian model, the intensity is simply the dot product of the light vector and the normal vector.
As John Isidoro and Stan Sclaroff observed in Stochastic Refinement of the Visual Hull, this assumption fails for highly reflective or transparent materials. When a surface is shiny, it produces specular highlights—bright spots where the light reflects directly into the camera. To a Lambertian solver, these highlights look like the surface is tilted sharply toward the light, leading to massive geometric distortions.
Furthermore, Satoshi Ikehata points out in CNN-PS that "non-convex" surfaces (like a bowl or a folded cloth) introduce inter-reflections. Light bounces off one part of the object and hits another, violating the assumption that each point is lit only by the primary light source.
Learning the Physics of Reflection
To handle these non-linear light interactions, researchers have moved toward data-driven models. Instead of using a rigid mathematical formula for reflection, deep learning frameworks learn to recognize and ignore "corrupted" pixels like shadows and highlights.
Guanying Chen, Kwan-Yee K. Wong, and their colleagues introduced PS-FCN, a fully convolutional network that can take an arbitrary number of images as input. By using a max-pooling fusion layer, the network learns to extract the most reliable geometric information from a sequence of images, effectively "filtering out" the specularities that would baffle a traditional algorithm.
Another approach involves dictionary learning. Brian E. Moore and Raj Rao Nadakuditi demonstrated in Robust Photometric Stereo via Dictionary Learning that surface normals can be regularized by learning the underlying structural "atoms" of an object. This allows the system to fill in gaps caused by shadows or noise by relying on the learned patterns of how surfaces typically curve.
Resolving Ambiguity with Multi-Modal Cues
Even with deep learning, some materials remain ambiguous. For example, a perfectly flat mirror and a curved matte surface might produce similar intensity patterns under certain lights. To solve this, researchers fuse photometric stereo with other physical cues.
- Polarization: Light changes its polarization state when it reflects off a surface. In Polarized 3D, researchers used polarization cues to provide a "rough" estimate of surface normals, which then acts as a guide to resolve the fine-grained ambiguities in the shading data.
- Depth Fusion: Consumer sensors like the Kinect provide "blobby" but reliable depth maps. Songyou Peng, Bjoern Haefner, and Daniel Cremers developed a variational framework for Photometric Depth Super-Resolution. By combining low-resolution depth with high-resolution shading, the two methods "fix" each other: the depth map prevents the shading solver from getting lost in specularities, while the shading adds the fine details (like wrinkles or pores) that the depth sensor misses.
Moving Toward Uncalibrated Environments
The final frontier of this research is removing the need for specialized hardware. Early work required "light stages" or precisely measured LED rigs. Recent advances, such as those by Lu Sang in High-Quality RGB-D Reconstruction, utilize Gradient-SDF representations to jointly optimize camera poses, lighting, and geometry.
By treating the lighting as an unknown variable to be solved alongside the shape, these methods allow for high-fidelity 3D capture using nothing more than a handheld camera or a smartphone with a small ring light, as explored by Jian Wang and Yasuyuki Matsushita in Photometric Stereo with Small Angular Variations. This shift from rigid lab assumptions to flexible, physics-aware models is what enables the reconstruction of the complex, non-Lambertian world we actually live in.