Abstract
This paper presents a machine learning pipeline for facial expression recognition (FER) in Virtual Reality (VR) environments using surface electromyography (sEMG). Traditional camera-based FER is limited by headset occlusion, whereas sEMG captures muscle activity directly. Utilizing a VR mask with seven sensors, we classified five expressions (smile, frown, eyebrow raise, squeezed eyes, and neutral). Our results demonstrate that participant-wise standardization and temporal smoothing via Hidden Markov Models (HMM) significantly improve performance, achieving an F1-macro score of 0.86 in static conditions and 0.82 during active head movements.
Introduction
Understanding user emotional states in VR is critical for applications such as mental health monitoring, social interaction, and gaming. However, the physical form factor of VR headsets obstructs the upper face, rendering standard optical facial recognition ineffective. While sEMG offers a viable alternative by measuring electrical activity from facial muscles, it is susceptible to noise from head movements and variations in expression intensity. This study evaluates a multi-sensor sEMG approach to robustly classify expressions under realistic, "noisy" conditions.
Methodology
The study employed the emteqPRO™ VR mask insert, which integrates seven sEMG sensors targeting key muscle groups: the zygomaticus (smiling), frontalis (eyebrow raising), orbicularis (eye squeezing), and corrugator (frowning).
The technical pipeline consisted of signal preprocessing (Hampel filtering and spectrum interpolation), extraction of 238 time-domain and statistical features, and classification using Random Forest and XGBoost models. A Hidden Markov Model (HMM) was implemented as a post-processing step to smooth predictions over time. Crucially, the researchers found that participant-wise standardization—scaling data based on an individual's specific signal range—was essential to account for physiological differences and sensor contact variance.
Experimental Design
Data was collected from 30 participants across two tasks:
- Task A (Static): Posed expressions with varying durations and intensities.
- Task B (Dynamic): Expressions performed while executing specific head movements (up, down, left, right).
The dataset was inherently imbalanced, with the "neutral" state comprising over 55% of the samples. To address this, resampling techniques such as SMOTE (Synthetic Minority Over-sampling Technique) were evaluated.
Results
The integration of standardization, oversampling, and temporal smoothing yielded the highest performance. For static expressions (Task A), the Random Forest model achieved an accuracy of 89.48% and an F1-macro score of 0.86.
When head movements were introduced (Task B), performance remained robust, with XGBoost achieving an F1-macro score of 0.82. The study also noted that training on data without movement and testing on data with movement led to a 3% accuracy drop, emphasizing the need for motion-inclusive training sets for real-world VR applications.
Discussion and Conclusion
Several challenges were identified, notably the "muscle overlap" between frowning and squeezing eyes due to the proximity of the corrugator and frontalis muscles. Additionally, low-intensity "micro-smiles" were often indistinguishable from the neutral state. Despite these hurdles, the system's resilience to head movement suggests it is suitable for active VR scenarios. Future work will explore end-to-end deep learning to automate noise filtering and further personalize models to individual users.
Summary updated