Capturing the Unseen: Vision-Free Facial Motion Capture Using Inertial Measurement Units

Youjia Wang, Yiwen Wu, Ruiqian Li, Hengan Zhou, Hongyang Lin, Yingwenqi Jiang, Yingsheng Zhu, Guanpeng Long, Jingya Wang, Lan Xu, Jingyi Yu, Yu, Jingyi

Abstract

We present Capturing the Unseen (CAPUS), a novel facial motion capture (MoCap) technique that operates without visual signals. CAPUS leverages miniaturized Inertial Measurement Units (IMUs) as a new sensing modality for facial motion capture. While IMUs have become essential in full-body MoCap for their portability and independence from environmental conditions, their application in facial MoCap remains underexplored. We address this by customizing micro-IMUs, small enough to be placed on the face, and strategically positioning them in alignment with key facial muscles to capture expression dynamics. CAPUS introduces the first facial IMU dataset, encompassing both IMU and visual signals from participants engaged in diverse activities such as multilingual speech, facial expressions, and emotionally intoned auditions. We train a Transformer Diffusion-based neural network to infer Blendshape parameters directly from IMU data. Our experimental results demonstrate that CAPUS reliably captures facial motion in conditions where visual-based methods struggle, including facial occlusions, rapid movements, and low-light environments. Additionally, by eliminating the need for visual inputs, CAPUS offers enhanced privacy protection, making it a robust solution for vision-free facial MoCap.

Introduction

Standard facial motion capture (MoCap) is currently dominated by vision-based solutions like Apple’s ARKit or deep-learning models like DECA. While effective, these systems have a significant "line-of-sight" limitation. They fail if the user turns away from the camera, enters a low-light environment, or covers their face (e.g., while eating, drinking, or wearing a helmet). Furthermore, recording continuous video of a user’s face raises substantial privacy concerns.

Overview of the CAPUS system, including custom hardware units, facial placement, and 3D reconstruction.

While IMUs have revolutionized full-body MoCap, applying them to the face is difficult because facial movements are subtle. Traditional IMUs are far too bulky and heavy to be placed on delicate facial muscles without distorting expressions. CAPUS overcomes these hurdles with custom hardware and a generative AI pipeline.

Methodology

Custom Micro-IMU Hardware

The researchers fabricated custom "face units" measuring just 0.6 $cm^2$ and weighing only 0.3 grams—about 2.7% the weight of a commercial Xsens sensor. This was achieved by separating the sensing module (attached to the face) from the bulkier data transmission module.

Size comparison of the custom Face Unit against a quarter and commercial sensors, with detailed board layouts.

Anatomical Placement and Model

Instead of random placement, the team strategically positioned 11 sensors over key muscle groups (e.g., the zygomaticus for smiling). An auxiliary IMU is placed behind the ear to act as a "ground truth" for head rotation, allowing the system to subtract global head movement from local facial expressions.

To process the data, the authors used a Transformer Diffusion-based network. The model takes raw IMU signals (acceleration and orientation) as a condition and iteratively denoises a signal into a set of 53 Blendshape parameters, the standard coefficients used to animate 3D avatars. This generative approach produces more stable and realistic expressions than simple regression.

Datasets and Results

Because no such data existed, the authors created the first-ever facial IMU dataset featuring 20 participants. It includes synchronized streams of raw IMU signals, RGB video, audio, and ARKit blendshape parameters across multilingual speech and various facial expressions.

Key results include:

  • Accuracy: CAPUS achieved a Per Vertex Error (PVE) of 0.075 mm, comparable to state-of-the-art vision-based methods.
  • Robustness: In qualitative tests involving heavy occlusion (e.g., eating an apple), CAPUS successfully reconstructed mouth movements that caused vision-based models to fail completely.
  • Real-world Dynamics: The team found that models trained on simulated IMU data performed poorly, highlighting the importance of real-world physical dynamics and noise captured by their custom sensors.

Conclusion

While CAPUS is a breakthrough, future work is needed to improve wearability and move toward completely wireless "patch" systems. Nonetheless, by removing the need for visual inputs, CAPUS offers a robust, privacy-preserving solution for high-fidelity facial MoCap in any environment.

Summary updated

Abstract

We present Capturing the Unseen (CAPUS), a novel facial motion capture (MoCap) technique that operates without visual signals. CAPUS leverages miniaturized Inertial Measurement Units (IMUs) as a new sensing modality for facial motion capture. While IMUs have become essential in full-body MoCap for their portability and independence from environmental conditions, their application in facial MoCap remains underexplored. We address this by customizing micro-IMUs, small enough to be placed on the face, and strategically positioning them in alignment with key facial muscles to capture expression dynamics. CAPUS introduces the first facial IMU dataset, encompassing both IMU and visual signals from participants engaged in diverse activities such as multilingual speech, facial expressions, and emotionally intoned auditions. We train a Transformer Diffusion-based neural network to infer Blendshape parameters directly from IMU data. Our experimental results demonstrate that CAPUS reliably captures facial motion in conditions where visual-based methods struggle, including facial occlusions, rapid movements, and low-light environments. Additionally, by eliminating the need for visual inputs, CAPUS offers enhanced privacy protection, making it a robust solution for vision-free facial MoCap.
BibTeX
@inproceedings{wang2025capturing,
  author = {Youjia Wang and Yiwen Wu and Ruiqian Li and Hengan Zhou and Hongyang Lin and Yingwenqi Jiang and Yingsheng Zhu and Guanpeng Long and Jingya Wang and Lan Xu and Jingyi Yu and Yu, Jingyi},
  title = {{Capturing the Unseen: Vision-Free Facial Motion Capture Using Inertial Measurement Units}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i8.32873},
}
Figure