High-Fidelity Face Tracking for AR/VR via Deep Lighting Adaptation

Lele Chen, Chen Cao, Fernando De la Torre, Jason M. Saragih, Chenliang Xu, Yaser Sheikh

Abstract

3D video avatars can empower virtual communications by providing compression, privacy, entertainment, and a sense of presence in AR/VR. Best 3D photo-realistic AR/VR avatars driven by video, that can minimize uncanny effects, rely on person-specific models. However, existing person-specific photo-realistic 3D models are not robust to lighting, hence their results typically miss subtle facial behaviors and cause artifacts in the avatar. This paper addresses previous limitations by learning a deep learning lighting model, that in combination with a high-quality 3D face tracking algorithm, provides a method for subtle and robust facial motion transfer from a regular video to a 3D photo-realistic avatar. Extensive experimental validation and comparisons to other state-of-the-art methods demonstrate the effectiveness of the proposed framework in real-world scenarios with variability in pose, expression, and illumination. Our project page can be found at https://www.cs.rochester.edu/ cxu22/r/wild-avatar/.

Introduction

Creating photorealistic 3D avatars that avoid the "uncanny valley" is a significant challenge for immersive communication. While person-specific (PS) models trained in controlled environments like light stages offer high fidelity, they are often fragile when deployed in real-world settings. Mismatches between studio training data and unpredictable consumer lighting—such as dim rooms or harsh desk lamps—frequently lead to tracking failures, loss of subtle expressions (e.g., lip pursing), and visual artifacts. This paper presents a framework to bridge this gap, enabling robust, high-fidelity facial motion transfer from standard smartphone video to 3D avatars.

Methodology

The proposed approach extends the Deep Appearance Model (DAM), which utilizes a Variational Auto-Encoder (VAE) to represent geometry and texture. The core innovation is a three-stage "In-the-Wild" registration pipeline designed to decouple rigid motion, facial expressions, and environmental lighting:

  1. Initialization: Coarse head pose and expression estimation using standard facial landmarks.
  2. Deep Lighting Adaptation: A physics-based lighting model employing gain and bias maps is introduced. Unlike traditional Spherical Harmonics, which are often too smooth to capture complex lighting, these maps capture high-frequency details like hard shadows and specular highlights. The model is fine-tuned using a small set of "key frames" from the target environment.
  3. High-Fidelity Tracking: Optimization of face parameters by minimizing the photometric difference between the rendered avatar and video frames, guided by dense optical flow to ensure the 3D mesh aligns perfectly with the user's skin.
Overview of the proposed framework, showing the captured image, estimated normal map, fully-lit avatar, and the final relit avatar.

Experimental Setup

The models were initially trained on a large-scale dataset captured in a calibrated light-stage rig with 460 white LEDs. For real-world validation, an "In-the-Wild" test set was created using the frontal camera of an iPhone, featuring 10 subjects performing diverse expressions under various natural lighting conditions. The researchers found that as few as 48 reference frames were sufficient to adapt the lighting model to a new environment.

Results and Discussion

The framework demonstrates superior performance over state-of-the-art methods such as MGCNet and I2ZNet. Quantitatively, it achieved a significantly lower $L_2$ reconstruction error (12.59 vs. 20.49 for the nearest competitor). Identity preservation, measured via Cosine Similarity (CSIM) from face recognition networks, reached 0.871, far exceeding generic morphable models which scored as low as 0.305.

Relit avatars under different viewpoints demonstrate robust 3D tracking and lighting adaptation.

Qualitatively, the system captures subtle nuances such as lip shapes and precise gaze direction, even in challenging low-light or profile views. By successfully separating lighting effects from facial geometry, the model avoids misinterpreting shadows as changes in facial shape.

Conclusion and Future Work

While the current system assumes static environmental lighting and requires person-specific pre-training, it represents a major step toward scalable, photorealistic AR/VR communication. Future work will focus on "online adaptation" to handle dynamic lighting and optimizing the computational load for real-time mobile applications.

Summary updated

Abstract

3D video avatars can empower virtual communications by providing compression, privacy, entertainment, and a sense of presence in AR/VR. Best 3D photo-realistic AR/VR avatars driven by video, that can minimize uncanny effects, rely on person-specific models. However, existing person-specific photo-realistic 3D models are not robust to lighting, hence their results typically miss subtle facial behaviors and cause artifacts in the avatar. This is a major drawback for the scalability of these models in communication systems (e.g., Messenger, Skype, FaceTime) and AR/VR. This paper addresses previous limitations by learning a deep learning lighting model, that in combination with a high-quality 3D face tracking algorithm, provides a method for subtle and robust facial motion transfer from a regular video to a 3D photo-realistic avatar. Extensive experimental validation and comparisons to other state-of-the-art methods demonstrate the effectiveness of the proposed framework in real-world scenarios with variability in pose, expression, and illumination. Our project page can be found at https://www.cs.rochester.edu/ cxu22/r/wild-avatar/.
BibTeX
@inproceedings{chen2021high,
  author = {Lele Chen and Chen Cao and Fernando De la Torre and Jason M. Saragih and Chenliang Xu and Yaser Sheikh},
  title = {{High-Fidelity Face Tracking for AR/VR via Deep Lighting Adaptation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=CwKy1Pt25-f},
}
Figure