Per-Pixel Solution of Multispectral Photometric Stereo

Shin Ishihara, Imari SatoWACV 2025 · DOI

Abstract

Multispectral Photometric Stereo (MPS) aims to recover 3D surface geometry from a single multispectral image. However, the technique is often hampered by the "reflectance-shape ambiguity," where it is difficult to distinguish between variations in surface orientation and variations in material color. While traditional methods rely on spatial smoothness constraints to resolve this, such assumptions fail for objects with complex textures. This paper introduces a per-pixel solution that utilizes strategic lighting configurations and a Local Linear Approximation (LLA) of spectral reflectance. By canceling out unknown reflectance terms through clever geometric design, the method achieves high-accuracy shape and reflectance recovery without spatial priors.

Introduction

Photometric Stereo (PS) is a cornerstone of computer vision, enabling high-resolution topography estimation by observing a surface under varying illumination. Multispectral Photometric Stereo (MPS) extends this by using different wavelengths of light simultaneously, potentially enabling "one-shot" acquisition. The primary challenge in MPS is the "Catch-22": a dark pixel could represent a surface tilted away from the light or a material with low reflectance at that wavelength. Most existing solutions impose spatial smoothness, assuming neighboring pixels share similar properties. This paper proposes a robust alternative that treats every pixel independently.

Methodology

The proposed framework relies on two key pillars: geometric lighting design and spectral smoothness.

  1. Strategic Lighting Configuration: Light sources are arranged in specific subsets where their direction vectors are linearly dependent. For instance, in a five-light subset, three "anchor" lights are supplemented by two intermediate lights placed at their midpoints.
  2. Local Linear Approximation (LLA): Real-world materials typically exhibit smooth spectral reflectance. The authors assume that over small wavelength intervals (10–20 nm), reflectance can be modeled as a linear function.
Overview of the proposed per-pixel MPS framework, illustrating the lighting subsets, the one-shot acquisition process, and the independent recovery of surface normals ($n$) and spectral reflectance ($r$).

By combining these elements, the authors derive a mathematical formulation where the unknown reflectance terms cancel out, allowing the surface normal to be solved directly from image intensities. To handle noise and specularities, a simultaneous optimization is performed across the spectral domain, treating specular highlights as outliers via a moving median filter.

Validating the LLA Assumption

The core of the method—the Local Linear Approximation—was validated using the Munsell Color Matt Dataset, comprising 1,269 material samples. As shown in the experimental analysis, the regression error remains remarkably low for narrow observation intervals, confirming that the linear model is a physically sound approximation for natural materials.

Empirical validation of the LLA assumption. (a) Reflectance curves for the Munsell dataset. (b) The relationship between regression error and the spectral observation interval, demonstrating the validity of the linear model at small intervals.

Results and Discussion

The method was evaluated on both synthetic and real-world datasets. In synthetic tests (e.g., "Bunny" and "Dragon" models with high-frequency textures), the per-pixel approach achieved a Mean Angular Error (MAE) for surface normals as low as 0.085, significantly outperforming methods that rely on spatial masks. Robust simultaneous optimization also improved the Root Mean Square Error (RMSE) of recovered spectral reflectance from ~0.5 to ~0.3. Real-world experiments using a custom hardware setup confirmed the method's robustness to complex color patterns and specularities, even when the spectral sampling interval was increased to 20nm.

Conclusion

This research demonstrates that the reflectance-shape ambiguity in MPS can be resolved at the pixel level by leveraging the interplay between lighting geometry and spectral properties. By moving away from spatial smoothness constraints, the method enables accurate 3D reconstruction of highly textured and multi-colored objects, paving the way for more robust one-shot photometric stereo applications.

Summary updated

BibTeX
@inproceedings{ishihara2025per,
  author = {Shin Ishihara and Imari Sato},
  title = {{Per-Pixel Solution of Multispectral Photometric Stereo}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00888},
}
Figure