Preprint: Towards Smart Glasses for Facial Expression Recognition Using OMG and Machine Learning

Ivana Kiprijanovska, Simon Stankoski, M. John Broulidakis, James Archer, Mohsen Fatoorechi, Martin Gjoreski, Charles Nduka, Hristijan Gjoreski

Abstract

This study evaluates the use of novel optomyography (OMG) based smart glasses, OCOsense™, for the monitoring and recognition of facial expressions. Experiments were conducted on data gathered from 27 young adult participants performing expressions including smiling, frowning, raising eyebrows, and squeezing eyes. The statistical analysis demonstrated that OMG sensors capture distinct variations in cheek and brow movements with high accuracy. A machine learning model trained on this data achieved an overall accuracy of 93% (0.90 F1-score) using leave-one-subject-out cross-validation, maintaining robustness across variations in expression intensity, head movement, and glasses positioning.

Introduction: The Challenge of Facial Expression Recognition

Facial expression recognition (FER) is a cornerstone of affective computing, with significant implications for mental health monitoring and human-computer interaction. However, current sensing methods face a "Goldilocks" problem. Camera-based systems raise privacy concerns and struggle in low light or during head movement. Conversely, Electromyography (EMG) is highly accurate but requires intrusive electrodes attached to the face. This paper investigates whether Optomyography (OMG)—which uses optical sensors to measure the mechanical movement of the skin from a distance—can bridge this gap by being non-contact, privacy-sensitive, and wearable.

Methods and Hardware

The researchers evaluated the OCOsense™ glasses, which integrate six OMG sensors positioned to monitor key facial muscles: the frontalis (forehead/eyebrows), zygomaticus (cheeks/smiling), and corrugator (between eyebrows).

The OCOsense™ glasses (left) and the corresponding facial muscle groups targeted by the OMG sensors (right).

The technical approach involved sophisticated signal processing to remove linear drift and calculate 3D skin movement magnitudes. A Random Forest model was trained on 162 statistical features extracted from 18 sensor streams. To ensure the model could generalize to new users, the team utilized a "leave-one-subject-out" (LOSO) cross-validation technique on a dataset of 27 healthy volunteers.

Experimental Design

Participants performed expressions across three distinct tasks to simulate real-world conditions:

  • Intensity and Duration: Variations between low/high intensity and short/long durations.
  • Head Movement: Expressions performed while moving the head (up, down, left, right).
  • Glasses Fit: Testing three different physical positions on the nose bridge (low, medium, high).

Results and Discussion: Muscle Fingerprints

A key insight from the study is the identification of distinct "muscle fingerprints" for specific expressions. The OMG sensors, which boast a high resolution down to ±4.0μm, allowed the team to quantify these differences statistically.

Box plots showing the distribution of cheek (left) and brow (right) movement across different facial expressions, highlighting the distinct activation patterns.

As shown in the data, smiles and squeezed eyes are primarily "cheek-heavy" movements, while frowns and eyebrow raises are "brow-heavy." The system achieved impressive quantitative benchmarks:

  • Overall Accuracy: 93%.
  • Movement Immunity: Performance remained stable at 92% accuracy even during active head movement, suggesting the glasses are viable for "in-the-wild" use.
  • Fit Sensitivity: The study found that a "high" fit on the bridge of the nose yielded the best results for upper-face expressions like frowns.

Conclusion and Future Work

The OCOsense™ glasses demonstrate that OMG is a robust technology for wearable FER. While this study focused on voluntary, posed expressions, it lays the groundwork for detecting spontaneous emotional reactions. Future research will focus on expanding demographic diversity and validating the glasses as a clinical tool for longitudinal monitoring of mood disorders, where facial expression patterns serve as critical clinical markers.

Summary updated

Abstract

Abstract This study aimed to evaluate the use of novel optomyography (OMG) based smart glasses, OCOsense™, for the monitoring and recognition of facial expressions. Experiments were conducted on data gathered from 27 young adult participants, who performed facial expressions varying in intensity, duration, and head movement. The facial expressions included smiling, frowning, raising the eyebrows, and squeezing the eyes. The statistical analysis demonstrated that: (i) OCO™ sensors based on the principles of OMG can capture distinct variations in cheek and brow movements with a high degree of accuracy and specificity; (ii) Head movement does not have a significant impact on how well these facial expressions are detected. The collected data were also used to train a machine learning model to recognise the four facial expressions and when the face enters a neutral state. We evaluated this model in conditions intended to simulate real-world use, including variations in expression intensity, head movement and glasses position relative to the face. The model demonstrated an overall accuracy of 93% (0.90 f1-score) – evaluated using a leave-one-subject-out cross-validation technique.
BibTeX
@inproceedings{kiprijanovska2023preprint,
  author = {Ivana Kiprijanovska and Simon Stankoski and M. John Broulidakis and James Archer and Mohsen Fatoorechi and Martin Gjoreski and Charles Nduka and Hristijan Gjoreski},
  title = {{Preprint: Towards Smart Glasses for Facial Expression Recognition Using OMG and Machine Learning}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41598-023-43135-5},
}
Figure