TL;DR
This paper introduces CAPUS, a novel vision-free facial motion capture system that uses miniaturized Inertial Measurement Units (IMUs) placed on the face. By developing custom micro-IMUs and a Transformer Diffusion network, CAPUS can accurately reconstruct facial expressions without relying on cameras, offering enhanced privacy and robustness in challenging conditions.
What Problem Does This Paper Address?
Traditional facial motion capture (MoCap) systems heavily rely on visual data from cameras (RGB, RGBD). While effective, these methods have significant limitations: they require users to constantly face the camera, are vulnerable to occlusions (like eating or wearing masks), motion blur, and poor lighting conditions. Furthermore, the reliance on visual data raises privacy concerns. Full-body MoCap has seen success with IMUs due to their portability and independence from environmental factors, but their application to the more nuanced and subtle movements of the face has been underexplored. This paper aims to bridge this gap by developing a robust, privacy-preserving, and vision-free facial MoCap solution using IMUs.
Methods
The core of CAPUS lies in its innovative use of miniaturized IMUs for facial motion capture. The authors designed custom micro-IMUs, significantly smaller and lighter than existing commercial options, to be comfortably worn on the face without impeding natural movements. These IMUs are strategically placed to align with key facial muscles responsible for expression dynamics.
For data processing, they developed a Transformer Diffusion-based neural network. This network takes the IMU signals (acceleration and orientation) as input and learns to infer facial blendshape parameters, which are a standard way to represent facial expressions in 3D. The Transformer architecture is well-suited for capturing temporal dependencies in the IMU data, while the diffusion model allows for robust inference even with noisy or incomplete signals. The system also incorporates an auxiliary IMU placed behind the ear to account for head rotation, disentangling it from facial expression movements.
Datasets
The paper introduces the first facial IMU dataset, named IMU-ARKit. This dataset is multi-modal, containing:
- Synchronized IMU signals from 11 IMUs placed on the face.
- Visual data (RGB video sequences).
- ARKit parameters (Apple's standard for facial tracking).
- Audio signals.
- Subject emotion labels.
- Transcripts of spoken content.
This dataset was collected from 20 participants performing diverse activities, including multilingual speech, various facial expressions, and emotionally intoned auditions. This rich dataset is crucial for training and evaluating the IMU-based facial motion capture model.
Key Insights and Ideas
- IMUs are viable for facial MoCap: The paper demonstrates that IMUs, previously underutilized for facial capture, can effectively capture the subtle dynamics of facial expressions, rivaling vision-based methods.
- Strategic IMU placement is key: Aligning IMUs with specific facial muscle groups, rather than arbitrary locations, significantly improves the accuracy of expression reconstruction.
- Vision-free MoCap offers significant advantages: The absence of cameras provides enhanced privacy, allows for capture in occluded or low-light environments, and offers greater freedom of movement for the user.
- Transformer Diffusion is effective for IMU-to-Blendshape mapping: This neural network architecture proves adept at translating noisy IMU signals into smooth and accurate facial blendshape parameters.
- Simulated datasets are insufficient for facial IMU MoCap: Unlike full-body IMU MoCap, training facial capture models solely on simulated data derived from ARKit parameters does not yield satisfactory results, highlighting the need for real-world, multi-modal datasets like IMU-ARKit.
Key Results
- CAPUS achieves comparable visual quality to state-of-the-art vision-based methods like 3DDFA_V2 and DECA, particularly excelling in the eye region and capturing eyebrow movements better than 3DDFA_V2.
- Quantitative evaluations show CAPUS achieving a Point-to-Vertex Error (PVE) of 0.075 ± 0.055 mm and a Point-to-Vertex Error on Landmarks (PVE_LMK) of 0.126 ± 0.091 mm, which are competitive with or better than ablation studies.
- The system demonstrates robust performance in challenging scenarios such as occlusions (e.g., eating an apple) and rapid movements, where visual methods would typically fail.
- The custom-designed IMUs are significantly smaller (0.6 cm², 0.3 grams) and lighter (2.7% of an Xsens IMU) than commercial alternatives, making them practical for facial wear.
Limitations and Future Work
- Comfort and Wiring: While miniaturized, the current prototype still has some wiring and may present comfort challenges for prolonged use. Future work could focus on further reducing the size and improving the ergonomics of the IMU units and their connections.
- Sensor Size: Although significantly reduced, the IMUs are still noticeable on the face. Further miniaturization is desirable for seamless integration.
- Generalization to Extreme Expressions: While CAPUS handles challenging expressions well, the paper notes that some extreme distortions might still be difficult to perfectly recover, suggesting the need for more diverse training data or more sophisticated network architectures.
- Calibration and Synchronization: While addressed, ensuring perfect synchronization and calibration across all IMUs in real-world, dynamic scenarios remains an ongoing challenge.
The authors believe that IMU-based facial MoCap has the potential to become mainstream, offering a compelling alternative to vision-based methods, especially in applications requiring privacy, freedom of movement, or operation in challenging environments.
Summary updated