AUGlasses: Continuous Action Unit based Facial Reconstruction with Low-power IMUs on Smart Glasses

Yanrong Li, Tengxiang Zhang, Xin Zeng, Yuntao Wang, Haotian Zhang, Yiqiang ChenarXiv 2024 · DOI · arXiv

Introduction

Facial reconstruction is a cornerstone of immersive experiences in Augmented and Virtual Reality (AR/VR), enabling the creation of realistic digital avatars for social interaction. However, current facial tracking technologies face a significant "trilemma" involving power consumption, privacy, and form factor. Traditional camera-based systems (using RGB or IR sensors) offer high accuracy but are often bulky, power-intensive, and raise substantial privacy concerns. Alternative methods, such as millimeter-wave radar or acoustic sensing, frequently struggle with high energy demands or insufficient granularity for subtle expressions.

To address these challenges, we present AUGlasses, a novel wearable system that reconstructs 3D facial expressions in real-time using low-power Inertial Measurement Units (IMUs) embedded in smart glasses. By sensing micro-deformations of the skin at the temples, AUGlasses provides a privacy-preserving, energy-efficient solution suitable for all-day use.

Sensing Principle: The Temple as a Hub

A core insight of this work is that the temporalis muscle area serves as a physiological "hub" for both upper and lower facial movements. Muscle contractions during expressions like smiling or frowning cause subtle but detectable skin deformations at the temples. By placing sensors at these strategic locations, AUGlasses can infer movements of the mouth, eyes, and brows without requiring sensors directly on those features.

Heat map of facial muscle changes and corresponding IMU signal patterns.

As shown in the muscle activity heat map, significant changes occur near the temple area during various facial actions. The system captures these "micro-movements" using two tiny IMUs housed in bio-compatible, elastic silicone gel structures designed to maintain consistent skin contact.

Methodology

The AUGlasses pipeline consists of hardware sensing, artifact removal, and a deep learning-based reconstruction model.

Hardware and Signal Processing

The system employs two skin-contact IMUs at the temples and a third reference IMU on the glasses' frame. To isolate facial expressions from global head movements (e.g., walking or turning), we apply an affine transformation method. This compares signals from the temple sensors against the frame-mounted reference to filter out motion noise.

Deep Learning Architecture

The processed signals are fed into a specialized neural network:

  1. Feature Extraction: 1D-CNNs extract local temporal features from short IMU signal patches.
  2. Prefix-conditioned Transformer: To avoid the "exposure bias" common in auto-regressive models—where small errors accumulate over time—we developed a prefix-conditioned sequence forecasting strategy. This model utilizes a window of historical Action Unit (AU) data to predict future intensities, ensuring stable and smooth reconstruction.
  3. 3D Rendering: Predicted AU intensities, based on the Facial Action Coding System (FACS), are mapped to 3D blendshapes in Unity to animate a digital avatar.

Experimental Evaluation

We evaluated AUGlasses through a user study involving 13 participants. Ground truth AU intensities were obtained using the OpenFace vision-based toolkit. The study covered both stationary (sitting) and mobile (walking) scenarios to test robustness.

Real-time 3D facial reconstruction results compared with ground truth video frames.

Key Results

  • Accuracy: The system achieved a cross-user Mean Absolute Error (MAE) of 0.187 on a 0–5 intensity scale for AUs. The 3D reconstruction showed a cross-user MAE of 1.93 mm and a Normalized Mean Error (NME) of 2.75%, performing comparably to state-of-the-art vision-based solutions.
  • Efficiency: Total system power consumption is remarkably low at 49.95 mW. This is nearly two orders of magnitude lower than camera-based wearables, which can consume upwards of 4,000 mW.
  • Battery Life: With a standard 350mAh battery, AUGlasses can provide nearly 26 hours of continuous facial tracking.

Discussion and Future Work

While AUGlasses demonstrates high fidelity and efficiency, some limitations remain. Environmental factors like sweat can affect the silicone gel's grip, leading to sensor drift. Additionally, eye blinks currently cause signal spikes that require better filtering. Future work will focus on integrating dedicated blink detection and automating the calibration process to enhance the user experience.

By prioritizing Action Units over individual landmarks, AUGlasses achieves superior generalizability across different users, paving the way for practical, all-day facial reconstruction in next-generation smart glasses.

Summary updated

Abstract

Recent advancements in augmented reality (AR) have enabled the use of various sensors on smart glasses for applications like facial reconstruction, which is vital to improve AR experiences for virtual social activities. However, the size and power constraints of smart glasses demand a miniature and low-power sensing solution. AUGlasses achieves unobtrusive low-power facial reconstruction by placing inertial measurement units (IMU) against the temporal area on the face to capture the skin deformations, which are caused by facial muscle movements. These IMU signals, along with historical data on facial action units (AUs), are processed by a transformer-based deep learning model to estimate AU intensities in real-time, which are then used for facial reconstruction. Our results show that AUGlasses accurately predicts the strength (0-5 scale) of 14 key AUs with a cross-user mean absolute error (MAE) of 0.187 (STD = 0.025) and achieves facial reconstruction with a cross-user MAE of 1.93 mm (STD = 0.353). We also integrated various preprocessing and training techniques to ensure robust performance for continuous sensing. Micro-benchmark tests indicate that our system consistently performs accurate continuous facial reconstruction with a fine-tuned cross-user model, achieving an AU MAE of 0.35.
BibTeX
@misc{li2024auglasses,
  author = {Yanrong Li and Tengxiang Zhang and Xin Zeng and Yuntao Wang and Haotian Zhang and Yiqiang Chen},
  title = {{AUGlasses: Continuous Action Unit based Facial Reconstruction with Low-power IMUs on Smart Glasses}},
  year = {2024},
  eprint = {2405.13289},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.13289},
}
Figure