High-fidelity Face Tracking for AR/VR via Deep Lighting Adaptation

Lele Chen, Chen Cao, Fernando De la Torre, Jason M. Saragih, Chenliang Xu, Yaser Sheikh

Abstract

3D video avatars can empower virtual communications by providing compression, privacy, entertainment, and a sense of presence in AR/VR. Best 3D photo-realistic AR/VR avatars driven by video, that can minimize uncanny effects, rely on person-specific models. However, existing person-specific photo-realistic 3D models are not robust to lighting, hence their results typically miss subtle facial behaviors and cause artifacts in the avatar. This paper addresses previous limitations by learning a deep learning lighting model, that in combination with a high-quality 3D face tracking algorithm, provides a method for subtle and robust facial motion transfer from a regular video to a 3D photo-realistic avatar. Extensive experimental validation and comparisons to other state-of-the-art methods demonstrate the effectiveness of the proposed framework in real-world scenarios with variability in pose, expression, and illumination.

Introduction

Creating a digital double that avoids the "uncanny valley" is a significant challenge, particularly when moving from controlled studio environments to "in-the-wild" settings using a single camera. Current state-of-the-art photo-realistic avatars often rely on person-specific models trained in specialized light stages. However, these models are notoriously brittle; they expect the specific lighting conditions of the lab. When driven by standard smartphone video, the mismatch in illumination often causes tracking failures, leading to artifacts, distorted expressions, and a loss of identity. This lack of robustness is a major hurdle for the scalability of AR/VR communication tools like Messenger or FaceTime.

Methodology

The proposed framework builds upon the Deep Appearance Model (DAM), a VAE-based architecture that represents a person's face through latent codes for geometry and texture. To enable real-world application, the researchers introduced a Deep Lighting Adaptation framework consisting of two primary components:

  1. Physics-Inspired Lighting Model: Instead of full photon simulation, the model uses a "gain and bias" approach. A lighting transfer module predicts gain ($g$) and bias ($b$) maps applied to the "fully-lit" studio texture. This allows the model to simulate complex effects like shadows and specular highlights without a calibrated physical environment.
  2. Three-Step Registration Pipeline:
    • Initialization: Standard facial landmarks provide a rough estimate of head pose and expression.
  3. Lighting Model Adaptation: The system uses a small subset of frames from the target environment to fine-tune the lighting model's weights, bridging the domain gap between the lab and the user's specific room.
  4. Face Tracking: Once calibrated, the system performs per-frame optimization using dense optical flow to ensure the avatar’s lips and gaze match the input video.
The lighting adaptation pipeline: from captured image to relit avatar.

Results and Discussion

The study utilized high-end Light-Stage data (40 cameras, 460 LEDs) for ground truth and "In-the-Wild" iPhone video for real-world testing. The proposed method significantly outperformed existing state-of-the-art models like MGCNet and I2ZNet across several metrics:

  • Visual Fidelity: The method maintains subject identity and captures sharp expressions (e.g., open mouths or squints) where other models produce distorted results.
  • Quantitative Accuracy: The method achieved an $L_2$ reconstruction error of 12.59, nearly half the error of MGCNet (24.56).
  • Identity Preservation: Using Cosine Similarity (CSIM) to measure identity mismatch, the model scored 0.871, indicating the reconstructed avatar looks much more like the actual person compared to competitors.
The relit avatar rendered from multiple viewpoints, demonstrating high-fidelity 3D reconstruction.

A key insight of this work is that decoupling motion and light prevents the optimizer from confusing shadows with facial movement. This allows for the capture of subtle lip shapes and gaze directions that traditional methods often miss.

Conclusion

While the current model assumes static lighting and requires person-specific pre-training, it represents a major step toward robust, high-fidelity avatars for consumer devices. Future work will explore online adaptation methods to handle dynamic lighting changes in real-time and move toward more universal models that reduce the need for person-specific studio training.

Summary updated

Abstract

3D video avatars can empower virtual communications by providing compression, privacy, entertainment, and a sense of presence in AR/VR. Best 3D photo-realistic AR/VR avatars driven by video, that can minimize uncanny effects, rely on person-specific models. However, existing person-specific photo-realistic 3D models are not robust to lighting, hence their results typically miss subtle facial behaviors and cause artifacts in the avatar. This is a major drawback for the scalability of these models in communication systems (e.g., Messenger, Skype, FaceTime) and AR/VR. This paper addresses previous limitations by learning a deep learning lighting model, that in combination with a high-quality 3D face tracking algorithm, provides a method for subtle and robust facial motion transfer from a regular video to a 3D photo-realistic avatar. Extensive experimental validation and comparisons to other state-of-the-art methods demonstrate the effectiveness of the proposed framework in real-world scenarios with variability in pose, expression, and illumination. Please visit https://www.youtube.com/watch?v=dtz1LgZR8cc for more results. Our project page can be found at https://www.cs.rochester.edu/u/lchen63.
BibTeX
@inproceedings{chen2021high,
  author = {Lele Chen and Chen Cao and Fernando De la Torre and Jason M. Saragih and Chenliang Xu and Yaser Sheikh},
  title = {{High-fidelity Face Tracking for AR/VR via Deep Lighting Adaptation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=CwKy1Pt25-f},
}
Figure