Ear-to-ear Capture of Facial Intrinsics

Alassane Seck, William A. P. Smith, Arnaud Dessein, Bernard Tiddeman, Hannah Dee, Abhishek DuttaarXiv 2016 · OpenReview

TL;DR

This paper introduces a practical, hybrid geometric-photometric approach to capture complete, ear-to-ear 3D face models with intrinsic textures (diffuse and specular albedo) without requiring complex geometric calibration. By capturing a subject in multiple poses using a single photometric camera alongside a sparse multi-view setup, the method robustly stitches surface normals and textures into a highly detailed, fully relightable face model.

What Problem Does This Paper Address?

Creating high-fidelity, photorealistic 3D human faces requires capturing both accurate geometry and intrinsic surface reflectance properties (like diffuse and specular albedo). Existing methods generally fall into two camps:

  1. Photometric methods (e.g., lightstages): Provide pixel-level surface normals and material properties but are traditionally restricted to a single viewpoint because the polarizing filters required to separate diffuse and specular reflections are highly view-dependent. This leaves the sides of the face and ears poorly resolved or completely occluded.
  2. Geometric methods (e.g., multiview stereo): Can capture full ear-to-ear geometry but fail to extract relightable intrinsic textures, providing only a static "baked-in" texture map. Furthermore, they often require meticulous, fragile geometric calibration.

This paper bridges the gap by enabling full ear-to-ear capture of both shape and intrinsic textures using consumer-grade hardware and zero prior geometric calibration.

Methods

The authors propose a multi-step hybrid pipeline that combines uncalibrated multiview stereo (MVS) with spherical gradient photometric stereo:

  1. Multi-Pose Photometric Capture: Instead of attempting to capture multiple photometric views simultaneously (which breaks polarization calibration), the subject is captured in three distinct poses (frontal, left profile, right profile). The system uses a single photometric camera inside a lightstage alongside 7 auxiliary MVS cameras, yielding 24 total viewpoints across the three poses.
  2. Photometric Alignment: Because the lighting sequences are captured in series, subtle sub-pixel subject motion can cause blurring. The authors introduce a novel alignment technique: they convert images to an illumination-independent color space (combining HSV hue and normalized RGB) to isolate intrinsic color from shading changes. This provides a robust initialization for optical flow alignment.
  3. Per-View Bias Removal: Photometric normals often suffer from low-frequency bias due to discrete LED positioning and light attenuation. The authors use a coarse base mesh generated via uncalibrated MVS (using commercial software like Agisoft Photoscan) to correct these low-frequency errors for each view independently.
  4. Seamless Poisson Stitching: To merge the data, the mesh is segmented into overlapping patches. Textures are blended using a screened Poisson equation. For surface normals, rather than stitching in the 2D normal map domain (which violates unit-length constraints and surface integrability), the authors stitch directly in the mesh domain. They linearize the problem using a Direct Linear Transformation (DLT) inspired approach, effectively embossing the high-frequency photometric details directly onto the 3D mesh vertices.

Datasets

The authors did not use standard public datasets. Instead, they validated their approach using a custom-built lightstage consisting of 41 ultra-bright white LEDs mounted on a 1.8-meter geodesic dome. The setup utilized a Nikon D200 as the photometric camera (equipped with an optoelectronic polarizing filter) and Canon 7D cameras for the multiview setup. They evaluated their pipeline on a diverse set of real-world subjects varying in age, ethnicity, gender, and facial hair.

Key Insights and Ideas

  • Bypassing Hardware Limits via Poses: Accepting the view-dependency of polarization and moving the subject rather than the hardware allows full-face intrinsic capture without complex, custom multi-view polarization optics.
  • Color Space for Optical Flow: Standard optical flow assumes brightness constancy, which fails completely under changing gradient illumination. Transforming the images to a hue + normalized RGB space effectively strips out the shading, leaving a stable target for alignment.
  • Mesh-Domain Normal Stitching: Treating normal blending as a Laplacian mesh editing problem solved via DLT linearization elegantly avoids the non-convex optimization problems associated with forcing 2D normal maps to maintain unit length.
  • Exploiting Least Viewing Angles: By selecting patches based on the smallest viewing angle, the pipeline naturally discards data from grazing angles, completely bypassing the need to mathematically correct for complex "Fresnel gain" artifacts at occluding boundaries.

Key Results

The pipeline successfully produces high-resolution, ear-to-ear 3D face models that can be realistically relit from any angle.

  • Texture Stitching: The patch-based Poisson blending eliminates visible seams and patch boundary artifacts while maintaining sharp details, avoiding the blurring typical of naive averaging methods.
  • Geometry Refinement: Specular surface normals were shown to yield significantly higher surface detail than diffuse normals, as they are unaffected by subsurface scattering.

Limitations and Future Work

The primary limitation is that the multi-pose approach requires the subject to change positions sequentially, with the entire capture process taking around 9 seconds (three sequences of 3 seconds each). This makes the current pipeline unsuitable for dynamic expressions or performance capture.

To address this, the authors suggest future work could augment the setup with additional photometric cameras assigned to independent groups of polarized lights, allowing simultaneous multiview photometric capture without sacrificing exact diffuse/specular separation.

Summary updated

Abstract

We present a practical approach to capturing ear-to-ear face models comprising both 3D meshes and intrinsic textures (i.e. diffuse and specular albedo). Our approach is a hybrid of geometric and photometric methods and requires no geometric calibration. Photometric measurements made in a lightstage are used to estimate view dependent high resolution normal maps. We overcome the problem of having a single photometric viewpoint by capturing in multiple poses. We use uncalibrated multiview stereo to estimate a coarse base mesh to which the photometric views are registered. We propose a novel approach to robustly stitching surface normal and intrinsic texture data into a seamless, complete and highly detailed face model. The resulting relightable models provide photorealistic renderings in any view.
BibTeX
@misc{seck2016ear,
  author = {Alassane Seck and William A. P. Smith and Arnaud Dessein and Bernard Tiddeman and Hannah Dee and Abhishek Dutta},
  title = {{Ear-to-ear Capture of Facial Intrinsics}},
  year = {2016},
  eprint = {1609.02368},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1609.02368},
}
Figure