Facial Micro-Expression Recognition Enhanced by Score Fusion and a Hybrid Model from Convolutional LSTM and Vision Transformer

Yufeng Zheng, Erik Blasch

TL;DR

This paper introduces a novel hybrid deep learning model that combines Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) networks, and Vision Transformers (ViTs) for enhanced facial micro-expression recognition. By fusing the outputs of multiple models, they achieve state-of-the-art performance, significantly improving accuracy and demonstrating the potential for real-time emotion understanding in human-machine interactions.

What Problem Does This Paper Address?

Facial expressions are a fundamental aspect of human communication, conveying emotions and intentions. Beyond the common "macro-expressions," there exist "micro-expressions" – fleeting, involuntary facial movements that reveal genuine emotions, often when individuals attempt to mask them. Recognizing these micro-expressions is crucial for machines to accurately understand human emotional states, enabling more nuanced and effective human-machine interactions. However, micro-expressions are challenging to detect due to their short duration and subtle nature, making them difficult even for trained human observers. Existing automated methods struggle with capturing the complex spatio-temporal dynamics of these expressions. This paper aims to address this challenge by developing a robust and accurate system for micro-expression recognition.

Methods

The core of this research lies in a hybrid deep learning architecture designed to capture both spatial and temporal features from facial videos. The proposed model integrates three key components:

  • Convolutional Neural Networks (CNNs): Used for extracting local spatial features from individual image frames, essentially understanding the "what" within a small region of the face.
  • Long Short-Term Memory (LSTM) Networks: Employed to summarize temporal dynamics across a sequence of frames, capturing how facial features change over time.
  • Vision Transformers (ViTs): Utilized to model sparse spatial relationships across image blocks and between frames, allowing the model to understand broader contextual relationships within the video.

The authors also introduce a score fusion technique. Instead of relying on a single model's prediction, they combine the outputs (predicted classes or probabilities) from multiple individual models. This fusion is performed using established machine learning classifiers like Support-Vector Machines (SVM) and Random Forests (RF) to leverage the collective intelligence of the ensemble.

Datasets

The study utilizes two publicly available datasets for training and evaluation:

  • MMEW (Micro-Expression and Macro-Expression Warehouse) dataset: Contains spontaneous micro-expressions collected from participants watching emotional video clips while maintaining a neutral expression. It includes 300 micro-expression samples with high-resolution images.
  • CASME II (Chinese Academy of Sciences Micro-expression II) dataset: A well-controlled dataset with high-intensity illumination, also featuring spontaneous micro-expressions elicited by emotional videos. It comprises 247 image sequences.

The paper notes that all facial images are preprocessed by resizing, standardization, and intensity scaling.

Key Insights and Ideas

  • Hybrid Architecture Synergy: The combination of CNNs, LSTMs, and ViTs proves highly effective. CNNs excel at local feature extraction, LSTMs capture temporal flow, and ViTs model long-range spatial dependencies. Together, they provide a more comprehensive understanding of micro-expressions than any single component alone.
  • The Power of Score Fusion: Aggregating predictions from multiple models significantly boosts recognition performance. This highlights that different models capture complementary information, and their combined output is more robust than individual predictions.
  • Real-time Potential: The proposed hybrid model, even with its complexity, demonstrates promisingly fast inference times, suggesting its feasibility for real-time applications.
  • Micro-expressions as a Window to True Emotion: The paper reinforces the idea that micro-expressions are involuntary and thus more indicative of genuine emotions, making their recognition a critical step towards more empathetic AI.

Key Results

The proposed Hybrid_FM6 model, which combines CNN, ConvLSTM, and ViViT components, achieved the highest accuracy on the MMEW dataset, reaching 0.8853. This is a substantial improvement over previous literature-reported accuracies (e.g., 0.6940).

When combined with score fusion (specifically using Random Forest with predicted class features), the accuracy on MMEW further increased to an impressive 0.9684. For the CASME II dataset, the Hybrid_FM5 model achieved the best single-model accuracy of 0.6565, and score fusion with Random Forest pushed this to 0.9112.

The paper also notes that certain expressions like "Anger" are easier to recognize, while "Fear" and "Sadness" are more challenging. The inference time for the Hybrid_FM6 model is approximately 20 milliseconds per sample (14 frames), making it suitable for real-time applications.

Limitations and Future Work

  • Dataset Size: While using established datasets, the authors acknowledge that larger and more diverse datasets could further improve model generalization.
  • Computational Cost: Although the hybrid model is efficient for inference, training deep learning models can still be computationally intensive.
  • Real-world Complexity: The current experiments are conducted on controlled datasets. Future work could explore performance in more challenging, unconstrained real-world scenarios with varying lighting, head poses, and occlusions.
  • Multimodal Fusion: The paper suggests that combining facial micro-expression recognition with other modalities (e.g., audio, physiological signals) could lead to even more comprehensive emotion understanding.
  • Hardware Acceleration: Leveraging GPUs or multiple GPUs could further reduce inference latency for real-time applications.

Summary updated

Abstract

In the billions of faces that are shaped by thousands of different cultures and ethnicities, one thing remains universal: the way emotions are expressed. To take the next step in human–machine interactions, a machine (e.g., a humanoid robot) must be able to clarify facial emotions. Allowing systems to recognize micro-expressions affords the machine a deeper dive into a person’s true feelings, which will take human emotion into account while making optimal decisions. For instance, these machines will be able to detect dangerous situations, alert caregivers to challenges, and provide appropriate responses. Micro-expressions are involuntary and transient facial expressions capable of revealing genuine emotions. We propose a new hybrid neural network (NN) model capable of micro-expression recognition in real-time applications. Several NN models are first compared in this study. Then, a hybrid NN model is created by combining a convolutional neural network (CNN), a recurrent neural network (RNN, e.g., long short-term memory (LSTM)), and a vision transformer. The CNN can extract spatial features (within a neighborhood of an image), whereas the LSTM can summarize temporal features. In addition, a transformer with an attention mechanism can capture sparse spatial relations residing in an image or between frames in a video clip. The inputs of the model are short facial videos, while the outputs are the micro-expressions recognized from the videos. The NN models are trained and tested with publicly available facial micro-expression datasets to recognize different micro-expressions (e.g., happiness, fear, anger, surprise, disgust, sadness). Score fusion and improvement metrics are also presented in our experiments. The results of our proposed models are compared with that of literature-reported methods tested on the same datasets. The proposed hybrid model performs the best, where score fusion can dramatically increase recognition performance.
BibTeX
@inproceedings{zheng2023facial,
  author = {Yufeng Zheng and Erik Blasch},
  title = {{Facial Micro-Expression Recognition Enhanced by Score Fusion and a Hybrid Model from Convolutional LSTM and Vision Transformer}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=U_I-01OdTyDM},
}
Figure