Abstract
We present Capturing the Unseen (CAPUS), a novel facial motion capture (MoCap) technique that operates without visual signals. CAPUS leverages miniaturized Inertial Measurement Units (IMUs) as a new sensing modality for facial motion capture. While IMUs have become essential in full-body MoCap for their portability and independence from environmental conditions, their application in facial MoCap remains underexplored. We address this by customizing micro-IMUs, small enough to be placed on the face, and strategically positioning them in alignment with key facial muscles to capture expression dynamics. CAPUS introduces the first facial IMU dataset, encompassing both IMU and visual signals from participants engaged in diverse activities such as multilingual speech, facial expressions, and emotionally intoned auditions. We train a Transformer Diffusion-based neural network to infer Blendshape parameters directly from IMU data. Our experimental results demonstrate that CAPUS reliably captures facial motion in conditions where visual-based methods struggle, including facial occlusions, rapid movements, and low-light environments. Additionally, by eliminating the need for visual inputs, CAPUS offers enhanced privacy protection, making it a robust solution for vision-free facial MoCap.
Introduction
Standard facial motion capture (MoCap) is currently dominated by vision-based solutions like Apple’s ARKit or deep-learning models like DECA. While effective, these systems have a significant "line-of-sight" limitation. They fail if the user turns away from the camera, enters a low-light environment, or covers their face (e.g., while eating, drinking, or wearing a helmet). Furthermore, recording continuous video of a user’s face raises substantial privacy concerns.
While IMUs have revolutionized full-body MoCap, applying them to the face is difficult because facial movements are subtle. Traditional IMUs are far too bulky and heavy to be placed on delicate facial muscles without distorting expressions. CAPUS overcomes these hurdles with custom hardware and a generative AI pipeline.
Methodology
Custom Micro-IMU Hardware
The researchers fabricated custom "face units" measuring just 0.6 $cm^2$ and weighing only 0.3 grams—about 2.7% the weight of a commercial Xsens sensor. This was achieved by separating the sensing module (attached to the face) from the bulkier data transmission module.
Anatomical Placement and Model
Instead of random placement, the team strategically positioned 11 sensors over key muscle groups (e.g., the zygomaticus for smiling). An auxiliary IMU is placed behind the ear to act as a "ground truth" for head rotation, allowing the system to subtract global head movement from local facial expressions.
To process the data, the authors used a Transformer Diffusion-based network. The model takes raw IMU signals (acceleration and orientation) as a condition and iteratively denoises a signal into a set of 53 Blendshape parameters, the standard coefficients used to animate 3D avatars. This generative approach produces more stable and realistic expressions than simple regression.
Datasets and Results
Because no such data existed, the authors created the first-ever facial IMU dataset featuring 20 participants. It includes synchronized streams of raw IMU signals, RGB video, audio, and ARKit blendshape parameters across multilingual speech and various facial expressions.
Key results include:
- Accuracy: CAPUS achieved a Per Vertex Error (PVE) of 0.075 mm, comparable to state-of-the-art vision-based methods.
- Robustness: In qualitative tests involving heavy occlusion (e.g., eating an apple), CAPUS successfully reconstructed mouth movements that caused vision-based models to fail completely.
- Real-world Dynamics: The team found that models trained on simulated IMU data performed poorly, highlighting the importance of real-world physical dynamics and noise captured by their custom sensors.
Conclusion
While CAPUS is a breakthrough, future work is needed to improve wearability and move toward completely wireless "patch" systems. Nonetheless, by removing the need for visual inputs, CAPUS offers a robust, privacy-preserving solution for high-fidelity facial MoCap in any environment.
Summary updated