TL;DR
This paper introduces a practical, hybrid geometric-photometric approach to capture complete, ear-to-ear 3D face models with intrinsic textures (diffuse and specular albedo) without requiring complex geometric calibration. By capturing a subject in multiple poses using a single photometric camera alongside a sparse multi-view setup, the method robustly stitches surface normals and textures into a highly detailed, fully relightable face model.
What Problem Does This Paper Address?
Creating high-fidelity, photorealistic 3D human faces requires capturing both accurate geometry and intrinsic surface reflectance properties (like diffuse and specular albedo). Existing methods generally fall into two camps:
- Photometric methods (e.g., lightstages): Provide pixel-level surface normals and material properties but are traditionally restricted to a single viewpoint because the polarizing filters required to separate diffuse and specular reflections are highly view-dependent. This leaves the sides of the face and ears poorly resolved or completely occluded.
- Geometric methods (e.g., multiview stereo): Can capture full ear-to-ear geometry but fail to extract relightable intrinsic textures, providing only a static "baked-in" texture map. Furthermore, they often require meticulous, fragile geometric calibration.
This paper bridges the gap by enabling full ear-to-ear capture of both shape and intrinsic textures using consumer-grade hardware and zero prior geometric calibration.
Methods
The authors propose a multi-step hybrid pipeline that combines uncalibrated multiview stereo (MVS) with spherical gradient photometric stereo:
- Multi-Pose Photometric Capture: Instead of attempting to capture multiple photometric views simultaneously (which breaks polarization calibration), the subject is captured in three distinct poses (frontal, left profile, right profile). The system uses a single photometric camera inside a lightstage alongside 7 auxiliary MVS cameras, yielding 24 total viewpoints across the three poses.
- Photometric Alignment: Because the lighting sequences are captured in series, subtle sub-pixel subject motion can cause blurring. The authors introduce a novel alignment technique: they convert images to an illumination-independent color space (combining HSV hue and normalized RGB) to isolate intrinsic color from shading changes. This provides a robust initialization for optical flow alignment.
- Per-View Bias Removal: Photometric normals often suffer from low-frequency bias due to discrete LED positioning and light attenuation. The authors use a coarse base mesh generated via uncalibrated MVS (using commercial software like Agisoft Photoscan) to correct these low-frequency errors for each view independently.
- Seamless Poisson Stitching: To merge the data, the mesh is segmented into overlapping patches. Textures are blended using a screened Poisson equation. For surface normals, rather than stitching in the 2D normal map domain (which violates unit-length constraints and surface integrability), the authors stitch directly in the mesh domain. They linearize the problem using a Direct Linear Transformation (DLT) inspired approach, effectively embossing the high-frequency photometric details directly onto the 3D mesh vertices.
Datasets
The authors did not use standard public datasets. Instead, they validated their approach using a custom-built lightstage consisting of 41 ultra-bright white LEDs mounted on a 1.8-meter geodesic dome. The setup utilized a Nikon D200 as the photometric camera (equipped with an optoelectronic polarizing filter) and Canon 7D cameras for the multiview setup. They evaluated their pipeline on a diverse set of real-world subjects varying in age, ethnicity, gender, and facial hair.
Key Insights and Ideas
- Bypassing Hardware Limits via Poses: Accepting the view-dependency of polarization and moving the subject rather than the hardware allows full-face intrinsic capture without complex, custom multi-view polarization optics.
- Color Space for Optical Flow: Standard optical flow assumes brightness constancy, which fails completely under changing gradient illumination. Transforming the images to a hue + normalized RGB space effectively strips out the shading, leaving a stable target for alignment.
- Mesh-Domain Normal Stitching: Treating normal blending as a Laplacian mesh editing problem solved via DLT linearization elegantly avoids the non-convex optimization problems associated with forcing 2D normal maps to maintain unit length.
- Exploiting Least Viewing Angles: By selecting patches based on the smallest viewing angle, the pipeline naturally discards data from grazing angles, completely bypassing the need to mathematically correct for complex "Fresnel gain" artifacts at occluding boundaries.
Key Results
The pipeline successfully produces high-resolution, ear-to-ear 3D face models that can be realistically relit from any angle.
- Texture Stitching: The patch-based Poisson blending eliminates visible seams and patch boundary artifacts while maintaining sharp details, avoiding the blurring typical of naive averaging methods.
- Geometry Refinement: Specular surface normals were shown to yield significantly higher surface detail than diffuse normals, as they are unaffected by subsurface scattering.
Limitations and Future Work
The primary limitation is that the multi-pose approach requires the subject to change positions sequentially, with the entire capture process taking around 9 seconds (three sequences of 3 seconds each). This makes the current pipeline unsuitable for dynamic expressions or performance capture.
To address this, the authors suggest future work could augment the setup with additional photometric cameras assigned to independent groups of polarized lights, allowing simultaneous multiview photometric capture without sacrificing exact diffuse/specular separation.
Summary updated