Self-Supervised Adaptation of High-Fidelity Face Models for Monocular Performance Tracking

Jae Shin Yoon, Takaaki Shiratori, Shoou-I Yu, Hyun Soo Park

TL;DR

This paper presents a method to animate high-fidelity 3D face models using just a single commodity camera (like a smartphone) in unconstrained, everyday environments. By combining a novel image-to-latent-code network (I2ZNet) with a self-supervised domain adaptation strategy based on temporal texture consistency, the approach bridges the gap between controlled laboratory data and real-world video without requiring any target-domain labels.

What Problem Does This Paper Address?

High-fidelity 3D face models, such as Deep Appearance Models (DAMs), can generate incredibly realistic avatars with pore-level details. However, deploying them in consumer applications faces two major hurdles:

  1. Modality Mismatch: These advanced models typically require specialized inputs—like tracked 3D meshes and unwrapped textures—rather than standard 2D video frames.
  2. Domain Mismatch: They are trained on pristine data captured in highly controlled laboratory environments with multi-camera arrays. When exposed to "in-the-wild" conditions (uncontrolled lighting, low resolution, complex backgrounds, and varying camera sensors), the correlation between facial geometry and appearance breaks down, causing tracking to fail or jitter.

This paper aims to democratize high-fidelity face tracking by enabling these lab-trained models to be driven by a single, monocular camera in any environment.

Methods

The authors tackle the problem in two stages: handling the input modality and adapting to the new environment.

1. I2ZNet (Image-to-z Network)

To bypass the need for 3D meshes and unwrapped textures, the authors designed I2ZNet. This network takes a single 2D image and directly regresses both the latent shape/appearance code ($z$) of the pre-trained face model and the 3D rigid head pose ($H$).

Illustration of the I2ZNet architecture, which combines VGGNet and HourglassNet features to regress the latent code and head pose.

As shown in Figure 2, I2ZNet utilizes a two-stream feature extraction process:

  • VGGNet extracts domain-invariant perceptual features.
  • HourglassNet captures local geometric details and facial landmarks. Features from different network depths are combined using skip connections and compressed before being mapped to the latent space via fully connected layers.

2. Self-Supervised Domain Adaptation

To handle new environments without manual labeling, the authors introduce a self-supervised fine-tuning process driven by three distinct loss functions:

  • Consecutive Frame Texture Consistency ($L_{CFTC}$): The core innovation. It operates on the assumption that a person's facial appearance (texture) shouldn't change drastically between two consecutive video frames. By projecting the predicted 3D geometry back onto the images, the system extracts "observed" textures and minimizes their difference over time.
  • Model-to-Observation Texture Consistency ($L_{MOTC}$): This ensures the avatar's generated texture matches the actual video frame. To handle lighting and white-balance shifts, they introduce a simple, learnable 1x1 convolutional layer that acts as a global color-correction matrix.
  • Facial Landmark Reprojection Consistency ($L_{FLRC}$): This anchors the 3D face structure by minimizing the distance between projected 3D model landmarks and 2D landmarks predicted by an off-the-shelf detector.
Overview of the self-supervised domain adaptation process using consecutive frames.

Datasets

  • Source Domain (Lab): The initial I2ZNet is trained using the multi-view image sequences originally used to build the identity's DAM, providing ground-truth latent codes ($z_{gt}$) and head poses ($H_{gt}$).
  • Target Domain (In-the-Wild): The authors collected a new evaluation dataset consisting of 1920×1080 resolution videos of four different subjects captured in diverse indoor and outdoor settings.
  • Annotation Strategy: For evaluation, they manually annotated 150 frames per subject with 5 personalized landmarks (such as specific moles or pimples). This avoids evaluation bias, as these unique points are entirely separate from the generic facial landmarks (like eye corners) used during network optimization.

Key Insights and Ideas

  • Dense Texture as a Supervisor: Instead of relying only on sparse landmarks or optical flow, the method uses every visible pixel in the unwrapped texture map across frames (Figure 4). This dense matching provides a significantly richer gradient source for domain adaptation.
  • Bypassing Explicit Lighting Models: Rather than trying to mathematically approximate complex real-world lighting (e.g., using spherical harmonics, which often fail with harsh shadows), the consecutive frame consistency constraint implicitly forces the geometry to be accurate without needing to model the environment explicitly.
  • Lightweight Testing: During the actual testing/inference phase, the texture unwrapping and landmark detection steps are discarded. The network functions as a simple, fast feed-forward pass (Figure 5).

Key Results

The method was compared against state-of-the-art baselines, including HPEN, 3DDFA, and PRNet.

  • Temporal Stability: The self-supervised adaptation improved the temporal stability of the tracking by 22% relative to the unadapted baseline model.
  • Jitter Reduction: On a smoothness metric where 1.0 is a perfect straight-line trajectory across three frames, the proposed method achieved a score of 1.14, significantly outperforming the baselines which hovered around 1.50 (Figure 6).
  • Qualitative Fidelity: The adapted model accurately captured complex expressions (like closed mouths when baselines erroneously predicted open mouths) and successfully corrected color discrepancies caused by ambient lighting (Figure 7).
Temporal stability graph showing that the proposed method achieves lower (more stable) scores across video frames.

Limitations and Future Work

  • Dependency on Pre-existing Models: The pipeline assumes that a high-fidelity, person-specific DAM has already been captured and trained in a lab setting beforehand. It cannot generate a high-fidelity tracker for an entirely unseen identity from scratch.
  • Global vs. Local Color Correction: The 1x1 convolution used for color correction applies globally. Consequently, it struggles with complex, highly localized lighting variations, such as strong specular highlights or distinct shadows cast across only one side of the face.

Summary updated

Abstract

Improvements in data-capture and face modeling techniques have enabled us to create high-fidelity realistic face models. However, driving these realistic face models requires special input data, e.g. 3D meshes and unwrapped textures. Also, these face models expect clean input data taken under controlled lab environments, which is very different from data collected in the wild. All these constraints make it challenging to use the high-fidelity models in tracking for commodity cameras. In this paper, we propose a self-supervised domain adaptation approach to enable the animation of high-fidelity face models from a commodity camera. Our approach first circumvents the requirement for special input data by training a new network that can directly drive a face model just from a single 2D image. Then, we overcome the domain mismatch between lab and uncontrolled environments by performing self-supervised domain adaptation based on "consecutive frame texture consistency" based on the assumption that the appearance of the face is consistent over consecutive frames, avoiding the necessity of modeling the new environment such as lighting or background. Experiments show that we are able to drive a high-fidelity face model to perform complex facial motion from a cellphone camera without requiring any labeled data from the new domain.
BibTeX
@inproceedings{yoon2019self,
  author = {Jae Shin Yoon and Takaaki Shiratori and Shoou-I Yu and Hyun Soo Park},
  title = {{Self-Supervised Adaptation of High-Fidelity Face Models for Monocular Performance Tracking}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  doi = {10.1109/cvpr.2019.00473},
}
Figure