Abstract
High-fidelity dynamic 3D reconstruction of human faces remains a significant challenge in computer vision, particularly for capturing fine-scale details like skin pores and wrinkles during motion. This paper presents Photogeometric Scene Flow (PGSF), a framework that integrates Photometric Stereo (PS), Multiview Stereo (MVS), and Optical Flow (OF) into a unified optimization process. By coupling these traditionally separate modalities, PGSF overcomes the limitations of individual methods—such as the spatial smoothing of MVS and the drift of PS—to produce high-resolution geometry and appearance simultaneously.
Introduction
Capturing the micro-details of a moving human face is a long-standing goal for the visual effects and gaming industries. Traditionally, researchers have relied on three distinct tools: Multiview Stereo (MVS) for absolute depth, Photometric Stereo (PS) for micro-geometry, and Optical Flow (OF) for motion tracking. However, each has inherent flaws: MVS often regularizes away fine details, PS suffers from low-frequency drift and struggles with dynamic subjects, and OF assumes brightness constancy—a condition violated by the changing lighting required for PS.
The proposed Photogeometric Scene Flow (PGSF) framework addresses these "siloed" limitations by treating geometry, motion, and appearance as a single, synergistic optimization problem. Instead of a sequential pipeline where errors propagate, PGSF leverages the strengths of each modality to regularize the others.
Methodology
The PGSF framework relies on a specialized hardware setup and a novel coupled optimization algorithm designed to handle high-speed motion.
Hardware and Data Acquisition
The researchers utilized a custom rig consisting of two high-speed cameras and nine colored LED clusters (Red, Green, Blue). Lighting is multiplexed in both time and color across a three-frame sliding window. This configuration ensures that every surface patch is observed under sufficient lighting directions to solve the photometric stereo equations without requiring a static light stage.
Coupled Optimization
The core innovation lies in the mutual reinforcement of PS, MVS, and OF:
- PS-Guided MVS: Rather than using generic spatial smoothing terms that blur the 3D model, surface gradients from PS act as a data-driven guide for MVS. This allows for continuous, high-detail depth estimates that preserve pores and wrinkles.
- Geometry-Based Relighting for OF: To solve the "changing light" problem in Optical Flow, the system uses the estimated geometry to "relight" frames. By cross-shading images to appear as if they were captured under consistent lighting, pixel tracking becomes significantly more accurate.
The optimization employs a coarse-to-fine Gaussian pyramid and block coordinate descent to manage large-scale deformations while refining micro-scale surface details.
Experimental Results
The method was validated using synthetic datasets and real-world 4K video captured at 60Hz. Synthetic tests demonstrated that PGSF significantly reduces errors in motion and surface normal estimation, particularly in high-deformation areas like the mouth.
Qualitatively, the system successfully reconstructed temporal variations in RGB albedo, capturing subtle physiological changes such as blushing. Unlike standard MVS, which often produces "lumpy" or overly smooth surfaces, PGSF results show distinct pores and sharp edges around the eyes and lips that align perfectly with the source video.
Conclusion and Future Work
Photogeometric Scene Flow demonstrates that coupling photometric and geometric constraints leads to superior 3D reconstruction of dynamic scenes. While the current implementation assumes Lambertian reflectance and requires specific hardware, it provides a robust foundation for future work involving complex reflectance models and unconstrained lighting environments.
Summary updated