Decomposing Multispectral Face Images into Diffuse and Specular Shading and Biophysical Parameters

Sarah Alotaibi, William A. P. Smith

TL;DR

This paper introduces a novel biophysical and dichromatic reflectance model to decompose multispectral face images into diffuse/specular shading and biophysical skin parameters (melanin, hemoglobin). This allows for high-quality, single-image decomposition without complex lighting, enabling intuitive editing and accurate reconstruction.

What Problem Does This Paper Address?

Understanding and manipulating the appearance of human faces is crucial for computer vision and graphics. A key challenge is decomposing a face image into its underlying physical components: diffuse shading (how light scatters within the skin), specular shading (surface reflections), and biophysical parameters that define skin color (like melanin and hemoglobin concentrations). Previous methods often relied on complex, controlled lighting setups or specialized equipment, limiting their applicability. This paper aims to address this by developing a method that can perform this decomposition from a single, uncontrolled multispectral image, making it more practical for real-world applications.

Methods

The core of this work is a novel biophysical and dichromatic reflectance model.

  • Dichromatic Model: This model assumes that the light we see from a surface is a combination of diffuse (subsurface scattered) and specular (surface reflected) components. The diffuse component is affected by the material's spectral reflectance, while the specular component is not.
  • Biophysical Skin Model: Instead of a generic spectral reflectance, the authors propose a model for skin that is parameterized by only two spatially varying biophysical parameters: melanin concentration (responsible for short-wavelength absorption) and hemoglobin concentration (responsible for green/blue wavelength absorption). These parameters are constrained to plausible ranges observed in healthy skin.
  • Model Fitting: For each pixel in a multispectral image, the model fitting process aims to find the best values for diffuse shading, specular shading, melanin, and hemoglobin. This is formulated as a nonlinear least squares problem, solved using a trust-region-reflective algorithm. To handle the constraints (e.g., parameters must be positive), they reparameterize the unknowns using the logistic function, turning it into an unconstrained optimization problem.
  • Editing: Once the decomposition is done, the authors enable intuitive editing of the parameter maps (e.g., changing melanin levels to simulate tanning). They then recompose the scene radiance and blend it with the original image using a learned skin segmentation mask to ensure realistic integration.
  • Rendering: A spectral radiance vector is converted to an sRGB image by integrating over wavelength, considering illumination, spectral reflectance, and camera spectral sensitivity.

Datasets

  • ISET Database: Used for quantitative evaluation. The authors computed best-fitting parameters for skin-labeled pixels and then evaluated the mean relative absolute error in reflectance spectra.
  • ImageVal Consulting Hyperspectral Scene Data: This dataset, containing high-resolution faces with spectral information, was used for qualitative results and editing demonstrations.
  • Manually Selected Skin Regions: For training the skin segmentation network, the authors manually selected skin and non-skin regions from the ImageVal dataset (1.1M skin pixels, 750k non-skin pixels).

Key Insights and Ideas

  • Information Sufficiency: A surprising conclusion is that a multispectral image contains sufficient information to well-posedly decompose skin appearance into its physical components, provided a biophysical model is used as a constraint.
  • Biophysical Constraint Power: The biophysical model, by limiting the possible spectral reflectances to those physically achievable by skin, significantly constrains the problem and enables accurate decomposition from a single image.
  • Intuitive Editing: Decomposing into semantically meaningful parameters (melanin, hemoglobin, diffuse/specular shading) allows for intuitive and plausible editing of skin appearance, such as simulating tanning or removing specular highlights.
  • Separation of Shading and Albedo: The method effectively separates shading effects from the underlying biophysical parameters, meaning that changes in lighting or surface detail don't incorrectly alter the estimated melanin or hemoglobin maps.

Key Results

  • Quantitative Accuracy: The method achieved a mean relative absolute error of 7.4% in reflectance spectra over pixels from the ISET database.
  • Qualitative Results: The paper presents visually convincing decompositions of multispectral face images, showing distinct maps for specular shading, diffuse shading, melanin, and hemoglobin. These maps are shown to accurately reflect known skin properties (e.g., lips and flushed cheeks in hemoglobin maps).
  • Editing Efficacy: Demonstrations show successful editing, including making skin lighter/darker by adjusting melanin, removing freckles, simulating tanning, and removing specularity.
  • Comparison to ICA: A comparison with an Independent Component Analysis (ICA) based method ([22]) shows that while ICA maps have some correspondence, the proposed physically meaningful decomposition is more plausible.

Limitations and Future Work

  • Four-Channel Data: The authors suggest that their model could potentially be fitted to four-channel data (e.g., RGB + NIR), and it would be interesting to see if similar results could be achieved.
  • Statistical Models: The inverse-rendered results could be used to learn statistical models of the variation in intrinsic biophysical face parameters.
  • Skin Segmentation: While a skin segmentation network is trained, the paper doesn't delve deeply into its architecture or performance beyond stating its purpose for blending edits.
  • Uncontrolled Lighting: While the method works with uncontrolled lighting, the paper assumes the illuminant's SPD is known for model fitting. The impact of unknown or varying illuminants is not explicitly discussed as a limitation.

Summary updated

Abstract

We propose a novel biophysical and dichromatic reflectance model that efficiently characterises spectral skin reflectance. We show how to fit the model to multispectral face images enabling high quality estimation of diffuse and specular shading as well as biophysical parameter maps (melanin and haemoglobin). Our method works from a single image without requiring complex controlled lighting setups yet provides quantitatively accurate reconstructions and qualitatively convincing decomposition and editing.
BibTeX
@inproceedings{alotaibi2019decomposing,
  author = {Sarah Alotaibi and William A. P. Smith},
  title = {{Decomposing Multispectral Face Images into Diffuse and Specular Shading and Biophysical Parameters}},
  booktitle = {ICIP},
  year = {2019},
  url = {https://openreview.net/forum?id=2ViuDSkOJ1},
}
Figure