Introduction
Facial reconstruction is a cornerstone of immersive experiences in Augmented and Virtual Reality (AR/VR), enabling the creation of realistic digital avatars for social interaction. However, current facial tracking technologies face a significant "trilemma" involving power consumption, privacy, and form factor. Traditional camera-based systems (using RGB or IR sensors) offer high accuracy but are often bulky, power-intensive, and raise substantial privacy concerns. Alternative methods, such as millimeter-wave radar or acoustic sensing, frequently struggle with high energy demands or insufficient granularity for subtle expressions.
To address these challenges, we present AUGlasses, a novel wearable system that reconstructs 3D facial expressions in real-time using low-power Inertial Measurement Units (IMUs) embedded in smart glasses. By sensing micro-deformations of the skin at the temples, AUGlasses provides a privacy-preserving, energy-efficient solution suitable for all-day use.
Sensing Principle: The Temple as a Hub
A core insight of this work is that the temporalis muscle area serves as a physiological "hub" for both upper and lower facial movements. Muscle contractions during expressions like smiling or frowning cause subtle but detectable skin deformations at the temples. By placing sensors at these strategic locations, AUGlasses can infer movements of the mouth, eyes, and brows without requiring sensors directly on those features.
As shown in the muscle activity heat map, significant changes occur near the temple area during various facial actions. The system captures these "micro-movements" using two tiny IMUs housed in bio-compatible, elastic silicone gel structures designed to maintain consistent skin contact.
Methodology
The AUGlasses pipeline consists of hardware sensing, artifact removal, and a deep learning-based reconstruction model.
Hardware and Signal Processing
The system employs two skin-contact IMUs at the temples and a third reference IMU on the glasses' frame. To isolate facial expressions from global head movements (e.g., walking or turning), we apply an affine transformation method. This compares signals from the temple sensors against the frame-mounted reference to filter out motion noise.
Deep Learning Architecture
The processed signals are fed into a specialized neural network:
- Feature Extraction: 1D-CNNs extract local temporal features from short IMU signal patches.
- Prefix-conditioned Transformer: To avoid the "exposure bias" common in auto-regressive models—where small errors accumulate over time—we developed a prefix-conditioned sequence forecasting strategy. This model utilizes a window of historical Action Unit (AU) data to predict future intensities, ensuring stable and smooth reconstruction.
- 3D Rendering: Predicted AU intensities, based on the Facial Action Coding System (FACS), are mapped to 3D blendshapes in Unity to animate a digital avatar.
Experimental Evaluation
We evaluated AUGlasses through a user study involving 13 participants. Ground truth AU intensities were obtained using the OpenFace vision-based toolkit. The study covered both stationary (sitting) and mobile (walking) scenarios to test robustness.
Key Results
- Accuracy: The system achieved a cross-user Mean Absolute Error (MAE) of 0.187 on a 0–5 intensity scale for AUs. The 3D reconstruction showed a cross-user MAE of 1.93 mm and a Normalized Mean Error (NME) of 2.75%, performing comparably to state-of-the-art vision-based solutions.
- Efficiency: Total system power consumption is remarkably low at 49.95 mW. This is nearly two orders of magnitude lower than camera-based wearables, which can consume upwards of 4,000 mW.
- Battery Life: With a standard 350mAh battery, AUGlasses can provide nearly 26 hours of continuous facial tracking.
Discussion and Future Work
While AUGlasses demonstrates high fidelity and efficiency, some limitations remain. Environmental factors like sweat can affect the silicone gel's grip, leading to sensor drift. Additionally, eye blinks currently cause signal spikes that require better filtering. Future work will focus on integrating dedicated blink detection and automating the calibration process to enhance the user experience.
By prioritizing Action Units over individual landmarks, AUGlasses achieves superior generalizability across different users, paving the way for practical, all-day facial reconstruction in next-generation smart glasses.
Summary updated