Polarized Light Field Imaging for Single-Shot Reflectance Separation

Jaewon Kim (김재완), Jaewon Kim, Abhijeet GhoshSensors 2018 · DOI

TL;DR

This paper introduces a novel single-shot imaging technique using polarized light fields to efficiently separate diffuse and specular reflectance from objects, including complex layered materials like skin. This advancement significantly reduces capture time and complexity compared to existing multi-shot methods, enabling more realistic digital rendering and appearance editing.

What Problem Does This Paper Address?

Accurately modeling and rendering realistic appearances of objects, especially human faces, is a long-standing challenge in computer vision and graphics. A key step in this process is separating an object's surface appearance into its constituent components, primarily diffuse (light scattered within the material) and specular (light reflected directly from the surface). Traditional methods for this separation often require multiple captures under different lighting or polarization conditions, which is time-consuming and impractical for dynamic scenes or subjects. Furthermore, separating more complex layered reflectance, like that found in skin (specular, single scattering, shallow scattering, and deep scattering), has been even more challenging and has historically demanded multi-shot acquisition. This paper aims to address the need for a faster, more practical, and more comprehensive method for reflectance separation.

Methods

The core innovation lies in the development of two polarized light-field (LF) cameras:

  • Two-Way Polarized Light-Field (TPLF) Imaging: This system modifies a standard light-field camera by incorporating a two-way linear polarizer in front of the main lens. This polarizer is split vertically and horizontally. When illuminated with vertically polarized light, specular rays are filtered by one half of the polarizer (cross-polarization), while diffuse rays, which depolarize light, pass through both halves. This spatial separation within the microlens image allows for the direct separation of diffuse and specular components in a single shot.
  • Four-Way Polarized Light-Field (FPLF) Imaging: This extends the TPLF concept by using four linear polarizers (at 0°, 45°, 90°, and 135°) in the pupil plane. This allows for the capture of four polarization states simultaneously, enabling more robust separation under uncontrolled or partially polarized illumination (like sunlight) and facilitating the separation of layered reflectance components.

The paper also introduces a novel computational method that leverages the angular sampling capabilities of light-field data to perform layered reflectance separation. By analyzing the intensity variations within the microlens images captured by the TPLF and FPLF cameras, the authors can distinguish between specular, single scattering, shallow scattering, and deep scattering components. This is achieved by exploiting the different angular lobe widths and intensity characteristics of these components.

Datasets

The paper does not explicitly mention the use of pre-existing datasets for training. Instead, it focuses on the acquisition of new data using their custom-built TPLF and FPLF cameras. The experiments involve capturing images of various subjects (mannequins, human faces, and a piece of Persian relief) under controlled and uncontrolled lighting conditions. The results are then presented as separated reflectance components (albedos and normal maps).

Key Insights and Ideas

  • Single-Shot Layered Reflectance Separation: The most significant insight is the demonstration that layered reflectance (specular, single, shallow, and deep scattering) can be separated from a single light-field image, a task previously requiring multiple shots.
  • Exploiting Light-Field Angular Information: The paper highlights how the inherent angular sampling of light-field data can be used to differentiate between reflectance components based on their scattering properties and lobe widths.
  • Practicality and Accessibility: By modifying off-the-shelf light-field cameras, the proposed methods offer a more practical and accessible solution compared to complex, custom-built light stages or multi-shot acquisition systems.
  • Robustness to Illumination: The FPLF system's ability to work under uncontrolled, partially polarized illumination (like sunlight) significantly expands the applicability of single-shot reflectance separation to real-world outdoor scenarios.
  • Layered Normal Maps: The ability to derive separate normal maps for shallow and deep scattering components opens up new possibilities for detailed surface appearance editing and rendering.

Key Results

  • Diffuse-Specular Separation: The TPLF camera successfully separates diffuse and specular components in a single shot, achieving results comparable to existing multi-shot techniques.
  • Layered Reflectance Separation: The method demonstrates effective separation of specular-only, single scattering, shallow scattering, and deep scattering albedos and normals. Qualitative comparisons with a multi-shot technique (Ghosh et al. [8]) show comparable results, with the single-shot method offering significant advantages in speed and simplicity.
  • Performance under Various Conditions: The separation quality is shown to be robust to changes in camera-subject distance and illumination intensity.
  • Applications in Rendering and Editing: The separated albedos and normals are demonstrated to be useful for hybrid normal rendering and for appearance editing, allowing for intuitive control over skin appearance by adjusting the contribution of different scattering layers.

Limitations and Future Work

  • Resolution Trade-off: As with all light-field cameras, there is an inherent resolution trade-off. While the Lytro Illum camera offers higher resolution, the authors note that their custom microlens sampling and color demosaicing pipeline leads to sub-optimal color reproduction.
  • Dynamic Range: The Lytro Red Hot camera used for TPLF imaging has a restricted dynamic range, which can affect acquisition quality with point light sources. This is expected to improve with advancements in HDR light-field cameras.
  • Polarization Alignment: The TPLF camera requires the incident illumination to be polarized either vertically or horizontally to match the polarizer orientation. The FPLF camera overcomes this limitation.
  • Threshold Setting: The intensity thresholds used for layered reflectance separation are empirically set. Further research is needed to relate these thresholds to physiological characteristics of skin layers.
  • Translucency Estimation: The current method separates scattering albedos and normals but does not directly estimate translucency parameters (like diffuse mean free path), which are crucial for full subsurface scattering simulations. The authors suggest that existing methods could be adapted to estimate these parameters from their acquired data.
  • Future Applications: The authors envision applications in cosmetics, automotive, and cultural heritage for layered material analysis and rendering. They also suggest extending the approach to estimate more complete reflectance information (BRDF/BSSRDF) and to integrate with full subsurface scattering rendering pipelines.

Summary updated

BibTeX
@article{2018polarized,
  author = {Jaewon Kim (김재완) and Jaewon Kim and Abhijeet Ghosh},
  title = {{Polarized Light Field Imaging for Single-Shot Reflectance Separation}},
  journal = {Sensors},
  year = {2018},
  doi = {10.3390/s18113803},
}
Figure