Hair color digitization through imaging and deep inverse graphics

Robin Kips, Panagiotis-Alexandros Bokaris, Matthieu Perrot, Pietro Gori, Isabelle Bloch

TL;DR

This paper introduces a novel method for automatically digitizing the color appearance of physical hair samples. By combining a controlled imaging setup with a deep inverse graphics model and a path-tracing renderer, the system can capture hair color and synthesize realistic images of hair with that color, opening doors for applications in virtual reality and hair dye development.

What Problem Does This Paper Address?

Reproducing realistic hair color in computer graphics is notoriously difficult. The complex interplay of hair fiber geometry and light scattering makes manual tuning of rendering parameters a tedious and often inexact process. While existing methods focus on capturing hair shape, there's a significant gap in automating the capture of hair color appearance from physical samples. This is crucial for applications like virtual try-on for hair dyes or creating realistic digital hair assets for games and animation, where precise color matching is essential. Current inverse graphics approaches, which typically rely on differentiable renderers, are not suitable for hair rendering due to the complexity of path tracing.

Methods

The core of this work lies in a novel pipeline that bridges the gap between physical hair samples and digital rendering without relying on differentiable rendering.

  • Controlled Imaging Setup: A dedicated setup with a fixed camera and controlled lighting is used to capture images of hair swatches. This ensures consistent acquisition conditions.
  • Deep Inverse Graphics Encoder: Instead of traditional differentiable rendering, the authors employ a self-supervised deep inverse graphics model. This encoder is trained to predict rendering parameters directly from an image. The key innovation here is training the model using a loss function defined in the graphics parameter space rather than the image space, circumventing the need for a differentiable renderer.
  • Path-Tracing Renderer: A physically-based path-tracing renderer is used to generate synthetic hair images. This renderer takes hair parameters (color, melanin concentration, etc.) and scene parameters (camera, lighting, shape) as input.
  • Hair Digitization Pipeline: At inference time, a real hair swatch image is fed into the trained inverse graphics encoder to estimate its color parameters. These parameters are then used with the path-tracing renderer to synthesize new images of hair with the captured color, allowing for variations in hairstyle and lighting.

Datasets

  • Training Data: The authors generated a dataset of 5000 synthetic hair images using their path-tracing renderer. These images were created with randomly sampled hair parameters to ensure good generalization, even to less common colors.
  • Evaluation Data: For quantitative evaluation, a set of 300 original synthetic hair swatch images were generated with random parameters. These were then processed through the digitization pipeline, and the reconstructed images were compared to the originals.
  • Real-World Data: Images of real hair swatches were captured using their custom imaging setup for qualitative evaluation.

Key Insights and Ideas

  • Graphics Parameter Space Loss: The most significant insight is the successful application of a self-supervised learning approach using a loss function in the graphics parameter space. This bypasses the limitations of differentiable rendering for complex phenomena like hair scattering, making inverse graphics applicable to path-traced scenes.
  • Scalability for Hair Color: By focusing solely on hair color and using small hair swatches, the method achieves a more scalable approach compared to full hair capture systems. This is particularly beneficial for applications like hair dye development where testing on individual strands is practical.
  • Decoupling Color from Shape: The approach explicitly separates hair color digitization from hair shape estimation. This allows for flexibility, as hair shape can be controlled and modified independently in the rendering environment, enabling diverse hairstyles from a single color capture.

Key Results

  • Qualitative: The method accurately captures hair appearance across a wide variety of hair colors, as demonstrated by qualitative results on both synthetic and real hair swatch images. The fine color variations within shades of brown were successfully reproduced.
  • Quantitative: Synthetic experiments showed low reconstruction errors across standard metrics (L1, MSSIM, LPIPS), with mean values of 7.08 ± 11.79 for L1, 0.20 ± 0.04 for MSSIM, and 0.12 ± 0.19 for LPIPS. These results suggest the model is effective at reconstructing hair color parameters.

Limitations and Future Work

  • Limitations: The authors acknowledge that their model cannot reproduce the natural color variation that can exist within individual hair fibers. This is attributed to the current parameterization, which doesn't account for heterogeneity at the fiber level. Additionally, capturing effects like varying specular reflections (hair shine) or damage is outside the scope of this paper.
  • Future Work: Promising directions include incorporating color heterogeneity of hair fibers and modeling varying specular properties. This would likely involve extending the renderer and the inverse graphics model to handle more complex material properties and potentially require more sophisticated imaging techniques.

Summary updated

BibTeX
@inproceedings{kips2022hair,
  author = {Robin Kips and Panagiotis-Alexandros Bokaris and Matthieu Perrot and Pietro Gori and Isabelle Bloch},
  title = {{Hair color digitization through imaging and deep inverse graphics}},
  booktitle = {Image Processing: Algorithms and Systems},
  year = {2022},
  doi = {10.2352/ei.2022.34.10.ipas-383},
}

Loading related content...

Figure