Blendshapes GHUM: Real-time Monocular Facial Blendshape Prediction

Ivan Grishchenko, Geng Yan, Eduard Gabriel Bazavan, Andrei Zanfir, Nikolai Chinaev, Karthik Raveendran, Matthias Grundmann, Cristian SminchisescuarXiv 2023 · DOI · arXiv

Overview

Google researchers have introduced Blendshapes GHUM, a highly efficient machine learning pipeline that transforms a standard smartphone camera feed into 52 expressive facial blendshape coefficients in real-time. By using facial landmarks as an intermediate representation and training on a massive dataset of 3D scans, the system achieves high-fidelity motion capture at over 30 FPS on mobile devices.

The Challenge of Mobile Facial Motion Capture

Facial motion capture (MoCap) is the backbone of modern animation, allowing digital avatars to "mimic" human actors. While Hollywood uses expensive marker-based systems or multi-camera rigs, bringing this technology to mobile devices for applications like virtual avatars or telepresence is difficult.

The challenge is twofold:

  1. Data Scarcity: It is incredibly difficult for humans to manually annotate "blendshape coefficients" (the mathematical weights for expressions like "jaw drop" or "eye blink") directly from a 2D image.
  2. Computational Constraints: Predicting complex 3D facial geometry directly from pixels is computationally expensive for mobile processors.

This research solves these issues by creating an automated, "annotation-free" pipeline that translates 2D video into the industry-standard 52 blendshapes used by major animation platforms.

Methodology: From Pixels to Blendshapes

The authors break the problem into an offline "teacher" phase for data generation and an online "student" phase for real-time inference. The real-time pipeline is designed for maximum efficiency on mobile hardware.

Real-time inference pipeline: An RGB image is processed into a face mesh, which then feeds into an MLP-Mixer to predict 52 blendshape coefficients.

The Landmark Foundation

First, a lightweight MobileNetV2 model detects 478 2D facial landmarks. These landmarks act as a simplified "skeleton" of the face, making the subsequent mathematical mapping much easier.

The Real-time Model

The core of the system is a tiny MLP-Mixer architecture. This model doesn't look at the original image; it only looks at 146 specific landmark coordinates (contours of the eyes, lips, and face). It then predicts the 52 blendshape coefficients and a 6D rotation matrix for head pose. This separation makes the system robust to lighting changes and different camera hardware.

Data Generation and Training

To overcome the lack of labeled data, the researchers developed an offline pipeline using high-fidelity 3D scans:

  • 3D Scanning: They recorded 6,000 individuals in a specialized lab performing 40 predefined expressions.
  • Deformation Transfer: They used Affine Deformation Transfer to map a canonical 3D face template onto these scans.
  • Optimization: An L-BFGS optimization algorithm found the exact blendshape coefficients that best fit each 3D scan.
  • Synthetic Augmentation: By sampling random combinations of blendshapes, they generated 2 million pairs of facial landmarks and ground-truth coefficients to train the real-time model.

Key Results and Performance

The system demonstrates that high-quality facial MoCap is possible on consumer-grade mobile devices without specialized hardware like depth sensors.

Visual results showing the model's ability to capture subtle expressions and map them to a 3D avatar in real-time.
  • Speed: The entire pipeline is incredibly fast. Landmark detection takes ~8ms, and blendshape prediction takes just 1.2ms on a Pixel 6 phone.
  • Accuracy: The real-time model achieved a Mean Normalized Error (MNE) of 3.88%, which is impressively close to the human rater cross-validation error of 2.33%.
  • User Validation: In a qualitative study, human raters agreed that the model successfully captured the intended expression in 96% of test videos.

Future Directions

While the system is highly effective, it currently relies on a fixed blendshape basis. Future iterations could include explicit prediction of the neutral mesh, allowing the system to better capture the unique facial structure of every individual user. The researchers aim to open-source these models to benefit the wider research and developer community.

Summary updated

Abstract

We present Blendshapes GHUM, an on-device ML pipeline that predicts 52 facial blendshape coefficients at 30+ FPS on modern mobile phones, from a single monocular RGB image and enables facial motion capture applications like virtual avatars. Our main contributions are: i) an annotation-free offline method for obtaining blendshape coefficients from real-world human scans, ii) a lightweight real-time model that predicts blendshape coefficients based on facial landmarks.
BibTeX
@misc{grishchenko2023blendshapes,
  author = {Ivan Grishchenko and Geng Yan and Eduard Gabriel Bazavan and Andrei Zanfir and Nikolai Chinaev and Karthik Raveendran and Matthias Grundmann and Cristian Sminchisescu},
  title = {{Blendshapes GHUM: Real-time Monocular Facial Blendshape Prediction}},
  year = {2023},
  eprint = {2309.05782},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.05782},
}
Figure