WiFE: WiFi and Vision based Intelligent Facial-Gesture Emotion Recognition

Yu Gu, Xiang Zhang, Zhi Liu, Fuji Ren

TL;DR

This paper introduces WiFE, a novel system for emotion recognition that combines facial expressions captured via vision with body gestures detected using commodity WiFi signals. By leveraging the complementary strengths of these two modalities and addressing challenges in WiFi sensitivity and data fusion, WiFE achieves significantly higher accuracy than single-modality approaches, paving the way for more robust and non-intrusive emotion understanding.

What Problem Does This Paper Address?

Human emotions are complex and rarely expressed through a single channel. While current emotion recognition research often relies on single modalities like facial expressions, this approach can be insufficient as people express emotions differently, sometimes emphasizing body language over facial cues. Existing multi-modal systems often struggle with data synchronization and the correlation between different modalities. Furthermore, many gesture recognition systems require contact or invasive sensors. This paper tackles the challenge of creating a more comprehensive and non-intrusive emotion recognition system by combining facial expressions with body gestures, specifically by using ubiquitous WiFi signals for gesture detection, thus overcoming the limitations of line-of-sight cameras and contact-based sensors.

Methods

The WiFE system employs a hybrid approach, integrating two key modalities:

  • Vision-based Facial Expression Recognition: Standard deep learning techniques are used. Faces are first detected and aligned using MTCNN. Then, three variants of DenseNet (121, 169, 201) are used to extract static features from video frames, and a VGG-LSTM network is employed to capture temporal dynamics.
  • WiFi-based Gesture Recognition: Instead of traditional sensors, the system leverages Channel State Information (CSI) from commodity WiFi signals. To enhance the sensitivity of WiFi signals to subtle human movements, the authors propose a method based on the Rician K factor theory. This involves strategically placing a lead plate to reduce the direct signal component, thereby amplifying the impact of multipath reflections caused by gestures.
  • Bi-modal Fusion: To handle the large volume and heterogeneous nature of data from both modalities, a score-level fusion strategy is adopted. This involves combining the outputs of individual classifiers (SVM for CSI, DenseNet-SVM and VGG-LSTM for vision) using another SVM to produce the final emotion prediction.

Datasets

  • Vision-CSI Emotion Database (VCED): This is a newly introduced, first-of-its-kind dataset specifically built for this research. It contains 1750 video clips and corresponding CSI sequences, capturing seven universal emotions (Angry, Disgust, Fear, Happy, Neutral, Sad, Surprise) from ten volunteers. The dataset aims to provide a rich source for studying the interplay between facial expressions and body gestures.
  • FER2013: This dataset was used for pre-training the neural networks for vision-based emotion recognition.

Key Insights and Ideas

  • WiFi for Gesture Recognition: The paper demonstrates that commodity WiFi signals, often overlooked for gesture sensing, can be effectively utilized for emotion-related body movements. This opens up possibilities for truly contactless and ubiquitous sensing.
  • Rician K Factor for Sensitivity Enhancement: The theoretical grounding in Rician K factor theory provides a principled way to improve the sensitivity of WiFi CSI to gestures by manipulating signal path components. This is a novel application of this theory in the context of gesture sensing.
  • Complementary Modalities: The research highlights that facial expressions and body gestures, while correlated, offer distinct and complementary information for emotion recognition. Combining them significantly boosts accuracy, especially for emotions that are less distinctly expressed facially (e.g., neutral).
  • Addressing Data Heterogeneity: The proposed CNN-RNN architecture and score-level fusion are effective in processing and integrating the diverse and non-synchronous data streams from vision and WiFi.

Key Results

The WiFE system achieved a remarkable 83.24% overall recognition accuracy for seven emotions. This significantly outperforms:

  • Gesture-only (CSI-SVM) approach: 66.48% accuracy.
  • Facial-only (vision-based) approaches: Densent121 (64.57%), Densent169 (64%), Densent201 (61.14%), and VGG-LSTM (66.67%).

The fusion method also showed a notable reduction in misclassification rates compared to single-modality systems. For instance, the CSI-only method had a 0% misclassification rate for angry to sad, while vision-based methods struggled with this. Conversely, vision-based methods performed better on "disgust" and "happy" than CSI-only.

Limitations and Future Work

  • Dataset Size and Diversity: While VCED is a novel contribution, it was collected from only ten volunteers. Future work could involve expanding the dataset with more participants, diverse demographics, and a wider range of emotional expressions and contexts.
  • Real-world Deployment Challenges: The paper focuses on controlled experimental settings. Real-world environments with more complex WiFi interference and dynamic human activity might pose additional challenges.
  • Fine-grained Emotion Recognition: The current system recognizes seven universal emotions. Future research could explore finer-grained emotional states or more nuanced expressions.
  • Computational Efficiency: While the paper uses deep learning, further optimization for real-time, low-power deployment on edge devices could be explored.

Summary updated

Abstract

Emotion is an essential part of Artificial Intelligence (AI) and human mental health. Current emotion recognition research mainly focuses on single modality (e.g., facial expression), while human emotion expressions are multi-modal in nature. In this paper, we propose a hybrid emotion recognition system leveraging two emotion-rich and tightly-coupled modalities, i.e., facial expression and body gesture. However, unbiased and fine-grained facial expression and gesture recognition remain a major problem. To this end, unlike our rivals relying on contact or even invasive sensors, we explore the commodity WiFi signal for device-free and contactless gesture recognition, while adopting a vision-based facial expression. However, there exist two design challenges, i.e., how to improve the sensitivity of WiFi signals and how to process the large-volume, heterogeneous, and non-synchronous data contributed by the two-modalities. For the former, we propose a signal sensitivity enhancement method based on the Rician K factor theory; for the latter, we combine CNN and RNN to mine the high-level features of bi-modal data, and perform a score-level fusion for fine-grained recognition. To evaluate the proposed method, we build a first-of-its-kind Vision-CSI Emotion Database (VCED) and conduct extensive experiments. Empirical results show the superiority of the bi-modality by achieving 83.24\% recognition accuracy for seven emotions, as compared with 66.48% and 66.67% recognition accuracy by gesture-only based solution and facial-only based solution, respectively. The VCED database download link is https://github.com/purpleleaves007/WIFE-Dataset.
BibTeX
@misc{gu2020wife,
  author = {Yu Gu and Xiang Zhang and Zhi Liu and Fuji Ren},
  title = {{WiFE: WiFi and Vision based Intelligent Facial-Gesture Emotion Recognition}},
  year = {2020},
  eprint = {2004.09889},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.09889},
}
Figure