Surface reflectance and normal estimation from photometric stereo

Qingxiong Yang, Narendra AhujaCVIU 2012 · DOI

TL;DR

This paper introduces a novel photometric stereo method that accurately estimates surface normals and separates diffuse and specular reflections from color images. By leveraging the dichromatic reflection model and a new "surface chromaticity" invariant, the method simplifies surface normal estimation into a solvable linear system, outperforming traditional approaches, especially on specular surfaces.

What Problem Does This Paper Address?

Estimating the 3D shape of objects from images is a fundamental problem in computer vision. Photometric stereo (PS) is a technique that uses multiple images of an object taken under different lighting conditions to infer its surface orientation (normals). However, traditional PS methods often struggle with non-Lambertian surfaces, particularly those with specular highlights (shiny reflections). These highlights can be misinterpreted as surface features, leading to inaccurate shape estimations. Existing methods to handle specularities fall into a few categories: ignoring highlight regions (which can lose information), assuming prior knowledge of reflectance properties (which is often unavailable), or attempting to separate diffuse and specular components. The latter often requires segmentation, which is difficult for textured surfaces, or makes assumptions about the nature of specular highlights that don't always hold. This paper aims to provide a robust method for estimating surface normals and separating reflection components that works well even in the presence of specularities, without requiring explicit segmentation or detailed prior knowledge of the material's reflectance.

Methods

The core of this paper lies in its novel application of the dichromatic reflection model and the introduction of surface chromaticity as a key invariant. The dichromatic model posits that the color of a surface point is a combination of diffuse (color of the object) and specular (color of the light source) reflections. The authors define surface chromaticity ($A_c$) as the fraction of a specific color component (e.g., red, green, blue) in the diffuse reflection. Crucially, they show this surface chromaticity is invariant to changes in light direction and intensity.

Building on this invariant, they propose a reflection components separation method. The key assumption here is specular-free observability: for any given pixel, at least one of the captured images is free of specular highlights. Under this assumption, they derive a direct correlation between the color of a diffuse pixel and the diffuse component of a specular pixel in other views. This allows them to estimate the specular component and, by subtraction, the diffuse component.

For surface normal estimation, they leverage the fact that for Lambertian surfaces, pixel intensity is linearly related to the surface normal and illumination. By extending this to non-Lambertian surfaces using their diffuse component estimation, they derive a linear relationship between the image color and the surface normal. This transforms the problem of surface normal estimation into solving a linear system, which can be done efficiently using least-squares optimization.

Datasets

The paper evaluates its method on both synthetic and real datasets.

  • Synthetic Data: Generated using the Phong reflectance model. Ground truth for diffuse/specular components and surface normals are available, allowing for quantitative evaluation.
  • Real Data:
    • Pear dataset
    • Apple dataset
    • Flash light dataset
    • Gourd dataset (using high dynamic range images to avoid saturation)

For real-world experiments, they used a Sony DFW-X700 camera and estimated light source direction and intensity using a mirror sphere and a diffuse sphere. Lighting calibration is noted as not being required for their reflection component separation method.

Key Insights and Ideas

  • Surface Chromaticity as a Universal Invariant: The paper's central idea is that surface chromaticity, derived from the dichromatic reflection model, is invariant to illumination changes. This invariant is the cornerstone for robustly separating diffuse and specular reflections.
  • Specular-Free Observability Simplifies Separation: The assumption that at least one image per pixel is specular-free significantly simplifies the derivation of the diffuse component, avoiding complex segmentation.
  • Linearization of Surface Normal Estimation: By reformulating the problem based on the estimated diffuse components and the dichromatic model, surface normal estimation becomes a linear system, making it computationally efficient and accurate.
  • Robustness to Assumption Violations: The paper demonstrates that even when the specular-free observability assumption is violated (e.g., in highly specular regions), their method still effectively reduces specular highlights and improves results, showcasing practical robustness.

Key Results

  • Reflection Component Separation: The method successfully separates diffuse and specular components, producing visually appealing diffuse images even for challenging specular objects. Quantitative evaluation on synthetic data shows low average errors, which increase slightly when specular highlights are more pronounced (lower shininess constant).
  • Surface Normal Estimation: The proposed method significantly outperforms conventional photometric stereo, especially in regions with specular highlights. On synthetic data, the root-mean-square angular difference in specular regions was reduced from 0.1720 (conventional) to 0.0070 (proposed), and 0.0075 (conventional with estimated diffuse components).
  • Real-World Performance: Visual results on real objects (pear, apple, flash light, gourd) demonstrate substantial improvements in shape reconstruction, particularly by eliminating artifacts caused by specular reflections. The method's performance is consistent with using estimated diffuse components, validating both aspects of their approach.

Limitations and Future Work

  • Specular-Free Observability Assumption: While the method shows robustness when this assumption is violated, it is still a core assumption. Highly specular objects where all captured images contain specular highlights might still pose challenges.
  • Saturation: The paper notes that saturated pixels in real images can decrease performance, a common issue in PS. Using High Dynamic Range (HDR) images mitigates this, suggesting HDR as a valuable extension.
  • Illumination Chromaticity: While not requiring lighting calibration for separation, the surface normal estimation does require knowledge of illumination chromaticity. The paper mentions this can be estimated from white reference images or existing methods, but it's an additional requirement.
  • Future Work: The paper implicitly suggests exploring methods to further improve robustness when the specular-free observability assumption is strongly violated and investigating more advanced ways to estimate illumination chromaticity. The focus on accurate normal estimation for visual evaluation of reconstructed shapes also hints at potential future work in improving the 3D reconstruction itself.

Summary updated

BibTeX
@article{yang2012surface,
  author = {Qingxiong Yang and Narendra Ahuja},
  title = {{Surface reflectance and normal estimation from photometric stereo}},
  journal = {CVIU},
  year = {2012},
  doi = {10.1016/j.cviu.2012.03.001},
}
Figure