Photometric stereo is one of the most elegant techniques in computer vision for recovering high-resolution surface details. The core intuition is simple: the apparent brightness of a surface element changes depending on its orientation relative to a light source. By capturing multiple images of a static object from a fixed viewpoint under varying illumination, we can solve for the surface normal $\mathbf{N}$ at every single pixel.
In its classical formulation, this problem is linear and straightforward because we assume the lighting directions and intensities ($\mathbf{L}$) are perfectly known. Under a Lambertian reflectance model, the image intensity $I$ is simply the dot product of the surface normal and the light vector, scaled by the surface albedo $\rho$:
$$I = \rho (\mathbf{N} \cdot \mathbf{L})$$
If we know $\mathbf{L}$ for at least three distinct light sources, we can invert this system of equations to recover both the albedo and the fine-grained surface geometry.
However, this reliance on pre-calibrated lighting creates a severe real-world bottleneck. It demands specialized hardware—such as darkrooms, precisely calibrated light stages, or gantries—making the technology impractical for handheld devices, outdoor environments, or casual consumer applications.
To bring photometric stereo into the wild, researchers have developed uncalibrated photometric stereo. This paradigm eliminates the need for external lighting calibration by estimating the lighting parameters directly from the image data alongside the shape and material properties.
The Core Challenge: The Generalized Bas-Relief Ambiguity
When both the surface normals $\mathbf{N}$ and the light sources $\mathbf{L}$ are unknown, we face a classic chicken-and-egg problem. Is a pixel bright because the surface is tilted directly toward the light, or because the light source itself has shifted?
Mathematically, this joint estimation suffers from an inherent ambiguity known as the Generalized Bas-Relief (GBR) ambiguity. If we transform the true surface normals by an arbitrary invertible $3 \times 3$ matrix $\mathbf{A}$, we can apply the inverse transformation $\mathbf{A}^{-1}$ to the light sources, and the resulting image intensities remain completely identical:
$$I = (\rho \mathbf{N} \mathbf{A}) \cdot (\mathbf{A}^{-1} \mathbf{L}) = \rho (\mathbf{N} \cdot \mathbf{L})$$
This means an infinite family of flattened, stretched, or skewed shapes under corresponding distorted lighting setups will produce the exact same set of photographs. To resolve this ambiguity and make uncalibrated photometric stereo work, we must introduce additional physical, statistical, or geometric constraints.
Resolving Ambiguity via Geometric Proxies
One of the most intuitive ways to break the GBR ambiguity is to pair uncalibrated photometric stereo with a coarse geometric "anchor." While photometric stereo excels at capturing micro-textures, wrinkles, and high-frequency details, alternative sensors or priors can provide the global, low-frequency shape structure needed to pin down the lighting.
For example, commodity RGB-D sensors (like a Microsoft Kinect or an Intel RealSense) provide real-time depth maps, but their data is often noisy and lacks fine detail. Peng et al. leverage this in Depth Super-Resolution Meets Uncalibrated Photometric Stereo by formulating a variational framework where the low-resolution depth map acts as a regularizer. The coarse depth constrains the global low-frequency shape—effectively eliminating the GBR distortion—while the uncalibrated photometric images fill in the high-resolution surface normals.
[Uncalibrated Images] ───► [Photometric Cues] ──┐
├──► [Joint Optimization] ──► Sharp, Detailed 3D
[Commodity RGB-D] ───► [Coarse Geometry] ──┘ (No GBR Ambiguity)
This concept applies to other geometric priors as well:
- Silhouettes: Vogiatzis et al. demonstrate in Reconstruction in the Round Using Photometric Normals and Silhouettes that the visual hull of an object obtained from silhouettes can serve as a geometric bounding constraint to resolve lighting parameters for complex, textureless objects.
- Statistical Shape Models: For specific object classes like human faces, we can use domain-specific knowledge. Cao et al. show in Sparse Photometric 3D Face Reconstruction Guided by Morphable Models that a smooth 3D Morphable Model (3DMM) can serve as a proxy to calibrate sparse, near-field light sources, enabling movie-grade facial capture without a professional light stage.
Exploiting Statistical and Physical Invariants
What if we do not have a geometric proxy? We can instead exploit the statistical properties of how light interacts with materials over time.
If we observe a single pixel across many different lighting directions, the sequence of observed intensities forms an intensity profile. For objects with isotropic reflectance (meaning the material reflects light symmetrically regardless of the orientation of the light around the normal), these profiles contain structural invariants.
Lu et al. exploit this physical reality in Uncalibrated Photometric Stereo for Unknown Isotropic Reflectances. They observe that for a wide range of non-Lambertian materials, the statistical distribution (such as the skewness) of a pixel's intensity profile is directly tied to its surface normal. By analyzing these profiles across the surface, the authors bypass the need to know either the light directions or the explicit Bidirectional Reflectance Distribution Function (BRDF) of the material.
Similarly, shadows and specular highlights—traditionally treated as nuisances or corruptions in linear photometric stereo—can be repurposed as valuable signals. Chung et al. demonstrate in Efficient photometric stereo on glossy surfaces with wide specular lobes that cast shadow boundaries provide hard geometric constraints about where a light source must be positioned, turning shadows into an implicit calibration tool.
Variational Joint Optimization
Another powerful strategy is to treat uncalibrated photometric stereo as a joint energy minimization problem. Instead of attempting to solve for the lighting first and the shape second, we optimize for shape, albedo, and lighting simultaneously within a unified variational framework.
To make this optimization well-posed without explicit calibration, the energy function must penalize physically impossible solutions. Quéau introduces a robust approach to this in A Non-convex Variational Approach to Photometric Stereo under Inaccurate Lighting. By employing non-convex M-estimators and a "scaled albedo" substitution, this framework can take a set of poorly calibrated or completely uncalibrated images and iteratively refine the light source intensities and directions while simultaneously correcting for shadows and specularities.
Alternatively, researchers can split the difference by using semi-calibrated setups. As shown by Logothetis et al. in Semi-Calibrated Near Field Photometric Stereo, treating light source positions as fixed, known parameters while treating their varying intensities as unknown variables drastically simplifies the physical setup requirements. The mathematical framework can then solve for the fluctuating light intensities automatically during the shape reconstruction phase.
Neural Inverse Rendering
With the advent of deep learning, the field has shifted from hand-crafted statistical invariants to learned priors. Neural networks can learn the complex, non-linear relationships between shading, material properties, and shape from large datasets, allowing them to resolve ambiguities that are analytically intractable.
In Uncalibrated Neural Inverse Rendering for Photometric Stereo of General Surfaces, Kaya et al. demonstrate how a neural network can ingest an uncalibrated image sequence of an object with arbitrary, unknown material properties. Instead of relying on rigid Lambertian assumptions, the network parameterizes the surface geometry and the material's BRDF using neural networks. By optimizing a reconstruction loss that ensures the rendered combination of estimated shape, material, and light matches the input images, the network successfully disentangles the components without requiring ground-truth 3D shapes for training.
Ultimately, by moving away from rigid calibration requirements and embracing geometric proxies, statistical invariants, and neural priors, uncalibrated photometric stereo transforms a brittle laboratory technique into a flexible, robust tool capable of capturing the rich geometry of the real world.