Separating Partially-Polarized Diffuse and Specular Reflection Components Under Unpolarized Light Sources

Soma Kajiyama, Taihe Piao, Ryo Kawahara, Takahiro OkabeWACV 2023

TL;DR

This paper introduces a novel method for separating diffuse and specular reflection components from a single color polarization image. Unlike previous methods that assume diffuse reflection is unpolarized, this work accounts for the partial polarization of diffuse light, leading to more accurate separation and enabling the estimation of surface polarimetric properties.

What Problem Does This Paper Address?

In computer vision and graphics, understanding how light reflects off surfaces is crucial for tasks like 3D reconstruction, material editing, and relighting. Reflected light is typically modeled as a combination of diffuse (scattered) and specular (mirror-like) components. Separating these components is a fundamental challenge. While color information can help, it's often insufficient for a pixel-wise solution. Polarization offers an additional cue, as specular reflections are generally polarized. However, existing methods often assume that diffuse reflections are unpolarized, which is an oversimplification. Diffuse reflectance can be weakly polarized due to Fresnel effects, especially when the outgoing plane aligns with the polarization direction. This paper addresses the limitation of previous methods by proposing a more physically accurate model that accounts for the partial polarization of both diffuse and specular reflections under unpolarized light sources.

Methods

The core of the proposed method lies in a new reflection model that extends the dichromatic reflection model by incorporating polarization. The model describes the observed pixel intensity as a combination of partially polarized diffuse and specular components. The key innovation is the assumption that the diffuse reflection component is also partially polarized, with its polarization properties related to the Fresnel transmittance.

The method uses a color polarization camera, which captures radiance through color filters and linear polarizers at multiple angles (e.g., 0°, 45°, 90°, 135°). This provides 12 radiance values per pixel. The algorithm then proceeds in several steps:

  1. Light Source Color Estimation: Using the observed colors and the polarization model, the method estimates the light source color. This is achieved by treating the observed colors at each pixel as lying on a plane defined by the light source color and the diffuse reflection color. RANSAC is employed for robust estimation.
  2. Phase Angle Estimation: The phase angles of the specular and diffuse reflection components are estimated. This is done by fitting a cosine curve to the observed pixel values across different polarization angles, similar to grayscale polarization image analysis, but adapted for color.
  3. Remaining Unknowns Estimation: With the light source color and phase angles known, the method estimates the remaining unknowns: the diffuse and specular reflection colors, and their degrees of linear polarization (DoLP). This is formulated as a nonlinear minimization problem, incorporating the polarization reflection model and the dichromatic reflection model. A total variation regularization term is added to encourage smooth DoLP variations across the surface.

Datasets

The paper evaluates its method using both synthetic and real images.

  • Synthetic Images: Experiments were conducted on synthetic images of a sphere and the Stanford Bunny, illuminated by single or multiple white light sources, and a single bluish light source. Ground truth data for all components (diffuse, specular, phase angle, DoLP) was available for these synthetic scenes.
  • Real Images: Experiments were performed on real images of various objects (plastic pumpkin, cup, ceramic figurines, plastic sphere, holder) captured using a color polarization camera. For real images, ground truth for light source colors was measured directly, but ground truth for diffuse/specular separation and polarimetric properties was not available, making qualitative evaluation and comparison with other methods the primary assessment.

Key Insights and Ideas

  • Diffuse Light is Partially Polarized: The most significant insight is the explicit modeling of partially polarized diffuse reflection. This moves beyond the common assumption of unpolarized diffuse light, which is a key limitation of prior work.
  • Single Image Solution: The method achieves diffuse-specular separation and polarimetric property estimation from a single color polarization image, making it practical for real-world applications.
  • Polarimetric Properties Estimation: Beyond separation, the method can estimate the DoLP of both diffuse and specular components, providing richer information about the surface material.
  • Robustness to Noise and Complex Shapes: The method demonstrates effectiveness on both simple and complex shapes (sphere vs. Stanford Bunny) and shows robustness to added noise in synthetic experiments.

Key Results

  • Improved Separation Accuracy: The proposed "Polarized Diffuse" method consistently outperforms the "Unpolarized Diffuse" and "Only Color" methods in both synthetic and real image experiments, as evidenced by higher PSNR and SSIM values on synthetic data and qualitative visual improvements.
  • Accurate Polarimetric Estimation: The method successfully estimates the DoLP for diffuse and specular components, even in challenging areas like occluding contours. The phase angle estimation is also more stable than methods assuming unpolarized diffuse light, which can suffer from phase flips.
  • Accurate Light Source Color Estimation: The method accurately estimates light source colors, with small angular errors compared to ground truth in synthetic experiments and consistent results on real images.

Limitations and Future Work

  • Smooth Surfaces Assumption: The method assumes smooth surfaces with small surface roughness values.
  • Piece-wise Constant Albedo: For estimating remaining unknowns, the method assumes piece-wise constant albedo, requiring segmentation into regions with similar diffuse reflection colors.
  • Ambiguity in Phase Angle: The phase angle estimation still has an ambiguity of $\pi$.
  • Future Work: The authors suggest extending the method to handle fully or partially polarized light sources. They also envision applications in shape from polarization and image-based material editing.

Summary updated

BibTeX
@inproceedings{kajiyama2023separating,
  author = {Soma Kajiyama and Taihe Piao and Ryo Kawahara and Takahiro Okabe},
  title = {{Separating Partially-Polarized Diffuse and Specular Reflection Components Under Unpolarized Light Sources}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure