Photometric Stereo from Maximum Feasible Lambertian Reflections

Chanki Yu, Yongduek Seo, Sang Wook LeeECCV 2010 · DOI

TL;DR

This paper introduces a novel algorithm for photometric stereo that robustly estimates surface shape and albedo even in the presence of specular reflections and shadows. By formulating the problem as a "Maximum Feasible Subsystem" (Max FS) optimization, the algorithm efficiently identifies and utilizes only the Lambertian components of the image data, leading to more accurate reconstructions.

What Problem Does This Paper Address?

Photometric stereo is a powerful technique for reconstructing 3D shapes and surface properties by analyzing how an object's appearance changes under different lighting conditions. The core assumption is that the object's surface is Lambertian, meaning it reflects light equally in all directions. However, real-world objects often exhibit non-Lambertian behaviors like specular reflections (shininess) and shadows, which violate this assumption and can lead to significant errors in shape and albedo estimation. Existing methods struggle to handle these non-Lambertian effects, especially when dealing with a large number of images or complex object geometries. This paper aims to address this by developing a method that can effectively filter out non-Lambertian pixels and reliably recover surface properties from a set of images, even if many of them contain specularities or shadows.

Methods

The paper proposes a novel algorithm based on the Maximum Feasible Subsystem (Max FS) framework. The core idea is to treat the problem as finding the largest possible subset of input images that consistently adhere to the Lambertian reflection model.

Here's a breakdown of the key techniques:

  • Max FS Formulation: The problem is framed as finding a subset of image constraints (representing Lambertian behavior) that are simultaneously satisfiable. This is achieved using a Big-M method, a mixed-integer linear programming (MILP) formulation.
  • Big-M Method: This technique introduces binary variables for each image constraint. If a constraint is deemed infeasible (due to specularities or shadows), its corresponding binary variable is set to 1, effectively deactivating that constraint. The objective is to minimize the number of deactivated constraints (i.e., maximize the number of feasible, or "inlier," constraints).
  • Algebraic Optimization: The algorithm employs a purely algebraic, pixel-wise optimization approach. This means it doesn't rely on probabilistic reasoning, physical models of reflectance, or initial guesses for surface normals, which can be prone to errors.
  • Handling Shadows: The method explicitly incorporates a mechanism to handle attached shadows by ensuring that the estimated irradiance is non-negative, effectively discarding shadowed regions.
  • GLPK Solver: The MILP problem is solved using the GLPK (GNU Linear Programming Kit) library, which employs a Branch-and-Cut algorithm to guarantee global optimality.

Datasets

The paper evaluates its method using two types of datasets:

  • Synthetic Sphere: Images were generated from a synthetic sphere using a variant of the Torrance-Sparrow model with a surface roughness of 0.07. This dataset provides "ground truth" for surface normals and albedo, allowing for quantitative error analysis.
  • Real Objects: Four real-world datasets were used: a baseball, a leaf, a picture frame, and a teapot. These datasets were captured using a Nikon D-200 camera and processed into linear 24-bit RGB images.

No new benchmarks were introduced; the focus was on evaluating the proposed algorithm on existing types of data.

Key Insights and Ideas

  • Robustness through Max FS: The most significant insight is that by framing the problem as finding the "maximum feasible subsystem," the algorithm can effectively isolate and utilize only the Lambertian pixels, even when they are a minority in the overall image set. This is a powerful way to handle the inherent messiness of real-world data.
  • Purely Algebraic Approach: The reliance on algebraic optimization, rather than probabilistic or physical models, simplifies the algorithm and avoids the need for potentially problematic initializations or assumptions about reflectance properties. This makes the method more generalizable.
  • Guaranteed Global Optimality: The use of MILP and a robust solver like GLPK ensures that the algorithm finds the true optimal solution, avoiding local minima that can plague other optimization techniques.
  • Single Parameter for System Noise: The algorithm requires only one parameter, which represents the system noise and distortion. This parameter can be determined once for a given system, making it relatively easy to use.

Key Results

  • Accuracy: The algorithm demonstrates very low average angular errors for recovered surface normals, often in the range of fractions of a degree, even with varying tolerance levels. For example, with 24 images and ε=5, the error was 0.153°.
  • Speed: The computation time is reported to be reasonably short, especially for medium numbers of images (10-100). For instance, with 60 images and ε=5, the running time per pixel was 5,624.73 msec (which is still quite fast for a per-pixel computation). The paper also notes that the computation time does not increase substantially with more images if averaging is performed.
  • Qualitative Results: Experiments on real objects (baseball, leaf, picture frame, teapot) show that the algorithm successfully recovers detailed 3D shapes and albedos, even for objects with complex geometries and challenging reflective properties. The rendered views clearly demonstrate the accurate reconstruction of concave regions and fine details.

Limitations and Future Work

  • Comparison with Dense Methods: The authors acknowledge that they have yet to experimentally compare their algorithm with recent "dense" photometric stereo methods that use a very large number of images (hundreds). They suggest that such methods might benefit from surface normal averaging, potentially leading to higher signal-to-noise results, and that their algorithm could be adapted for this.
  • Very Rough Surfaces: The current method might struggle with "very rough surfaces where non-Lambertian reflections from specularities are wide spread spatially." For such surfaces, fewer Lambertian reflections might be observed, making it harder to find a feasible subsystem.
  • Two-Image Photometric Stereo for Rough Surfaces: The authors express interest in developing methods that utilize spatial constraints from the neighborhood to solve the two-image photometric stereo problem for rough surfaces.
  • Automatic Noise Estimation: While the algorithm requires a noise/distortion parameter, the paper mentions that this "system noise can be possibly estimated automatically from captured images," suggesting this as a potential area for future work.

Summary updated

BibTeX
@inproceedings{yu2010photometric,
  author = {Chanki Yu and Yongduek Seo and Sang Wook Lee},
  title = {{Photometric Stereo from Maximum Feasible Lambertian Reflections}},
  booktitle = {European Conference on Computer Vision},
  year = {2010},
  doi = {10.1007/978-3-642-15561-1\_9},
}
Figure