Two-Level Spatio-Temporal Feature Fused Two-Stream Network for Micro-Expression Recognition

Zebiao Wang, Mingyu Yang, Qingbin Jiao, Liang Xu, Bing Han, Yuhang Li, Xin TanSensors 2024 · DOI

TL;DR

This paper introduces a novel "Two-Level Spatio-Temporal Feature Fused Two-Stream Network" (TFT) for micro-expression recognition. By effectively fusing multi-level spatial and temporal features from dedicated encoders (ResNet for spatial, Swin Transformer for temporal), the TFT achieves state-of-the-art performance on challenging datasets, outperforming previous methods by a significant margin.

What Problem Does This Paper Address?

Micro-expressions (MEs) are fleeting, involuntary facial movements that reveal true emotions, making them crucial for understanding human affect. However, their short duration and subtle nature pose a significant challenge for automated recognition. Existing deep learning approaches, particularly those using Vision Transformers (ViTs), often involve computationally intensive processes and struggle to capture the fine-grained, multi-level spatio-temporal information essential for accurate ME detection. Traditional two-stream architectures also fall short by only utilizing the deepest features for classification, neglecting valuable information from earlier network stages.

Methods

The proposed TFT architecture tackles these challenges with a two-stream design:

  • Spatial Encoder (Modified ResNet): This stream focuses on extracting static texture features from the apex frame of a micro-expression. A modified ResNet architecture is used, optimized for computational efficiency and adapted to align with the temporal encoder's stages.
  • Temporal Encoder (Swin Transformer): This stream captures the dynamic facial muscle movements by processing optical flow information. The Swin Transformer is chosen for its efficient, hierarchical approach to self-attention, which balances local and global feature learning while significantly reducing computational complexity compared to standard ViTs.
  • Two-Level Feature Fusion: A novel fusion algorithm is introduced to integrate multi-level spatio-temporal features from both streams. This algorithm leverages the similarity in output feature shapes across different stages of the encoders, allowing for a more comprehensive capture of subtle ME details.
  • Temporal Aggregation (Weighted Average Operator): Recognizing that the apex frame holds the most salient information, a weighted average operator is employed to aggregate temporal features. This operator assigns higher weights to frames closer to the apex, effectively capturing the most informative parts of the ME sequence.
  • Transfer Learning: To overcome the scarcity of ME data, the model utilizes transfer learning, pre-training both encoders on the CK+ dataset before fine-tuning on ME datasets. This approach ensures consistency in input and preprocessing, leading to better adaptation of learned features.

Datasets

The proposed method was evaluated on three widely used spontaneous micro-expression datasets:

  • SMIC-HS (Spontaneous Micro-Expression Corpus - High Speed): A dataset of high-speed spontaneous micro-expressions.
  • CASME II (Chinese Academy of Sciences Micro-Expression II): Another prominent dataset for spontaneous micro-expression recognition.
  • SAMM (Spontaneous Actions and Micro-Movement database): A dataset containing spontaneous micro-movements.

The evaluation was conducted using a combined corpus from these datasets, relabeled into three classes (negative, positive, and surprise) for a comprehensive assessment.

Key Insights and Ideas

  • Multi-level Fusion is Crucial: The paper strongly emphasizes that simply fusing features from the deepest layers of a two-stream network is insufficient for ME recognition. Integrating features from multiple stages is vital for capturing the subtle, fine-grained information characteristic of MEs.
  • Swin Transformer's Efficiency: The adoption of the Swin Transformer for temporal feature extraction is a key innovation, offering a significant reduction in computational cost compared to ViTs while maintaining or improving performance. This is attributed to its shifted window mechanism.
  • Weighted Temporal Aggregation Matters: The simple mean operator is outperformed by a weighted average operator that prioritizes apex frames. This highlights the importance of understanding the temporal dynamics of MEs and assigning importance accordingly.
  • Transfer Learning for Data Scarcity: The success of transfer learning from macro-expression datasets (CK+) demonstrates a practical strategy to mitigate the challenge of limited ME data.

Key Results

The proposed TFT architecture achieved state-of-the-art results across the evaluated datasets. Notably:

  • It achieved an Unweighted Average Recall (UAR) exceeding 0.905 on CASME II, placing it among the top-performing frameworks in the literature.
  • The feature fusion algorithm improved both the F1 score and UAR by approximately 4% compared to simpler fusion methods.
  • The TFT demonstrated significantly shorter inference times (1.423s for four video samples) compared to a previous state-of-the-art method (SLSTT-LSTM, 2.363s).

Limitations and Future Work

The authors acknowledge a few limitations:

  • Cumbersome Training Process: The pre-training and fine-tuning stages require careful management of input formats and optical flow extraction methods to maintain consistency.
  • Reliance on Apex Frames: Like many existing ME recognition methods, the proposed approach relies on apex frames. The integration of apex frame detection directly into the recognition network is suggested as a future research direction.
  • Real-World Gap: While evaluated on established datasets, the authors note that these datasets are collected under strict experimental conditions, and a gap exists between these controlled environments and real-world scenarios. Future work could explore more robust methods for real-world applications.

The paper suggests exploring more efficient training strategies and investigating methods that better integrate apex frame detection with the recognition network.

Summary updated

Abstract

Micro-expressions, which are spontaneous and difficult to suppress, reveal a person's true emotions. They are characterized by short duration and low intensity, making the task of micro-expression recognition challenging in the field of emotion computing. In recent years, deep learning-based feature extraction and fusion techniques have been widely used for micro-expression recognition, particularly methods based on Vision Transformer that have gained popularity. However, the Vision Transformer-based architecture used in micro-expression recognition involves a significant amount of invalid computation. Additionally, in the traditional two-stream architecture, although separate streams are combined through late fusion, only the output features from the deepest level of the network are utilized for classification, thus limiting the network's ability to capture subtle details due to the lack of fine-grained information. To address these issues, we propose a new two-level spatio-temporal feature fused with a two-stream architecture. This architecture includes a spatial encoder (modified ResNet) for learning texture features of the face, a temporal encoder (Swin Transformer) for learning facial muscle motor features, a feature fusion algorithm for integrating multi-level spatio-temporal features, a classification head, and a weighted average operator for temporal aggregation. The two-stream architecture has the advantage of extracting richer features compared to the single-stream architecture, leading to improved performance. The shifted window scheme of Swin Transformer restricts self-attention computation to non-overlapping local windows and allows cross-window connections, significantly improving the performance and reducing the computation compared to Vision Transformer. Moreover, the modified ResNet is computationally less intensive. Our proposed feature fusion algorithm leverages the similarity in output feature shapes at each stage of the two streams, enabling the effective fusion of multi-level spatio-temporal features. This algorithm results in an improvement of approximately 4% in both the F1 score and the UAR. Comprehensive evaluations conducted on three widely used spontaneous micro-expression datasets (SMIC-HS, CASME II, and SAMM) consistently demonstrate the superiority of our approach over comparative methods. Notably, our approach achieves a UAR exceeding 0.905 on CASME II, making it one of the few frameworks in the published micro-expression recognition literature to achieve such high performance.
BibTeX
@inproceedings{wang2024two,
  author = {Zebiao Wang and Mingyu Yang and Qingbin Jiao and Liang Xu and Bing Han and Yuhang Li and Xin Tan},
  title = {{Two-Level Spatio-Temporal Feature Fused Two-Stream Network for Micro-Expression Recognition}},
  booktitle = {Sensors},
  year = {2024},
  doi = {10.3390/s24051574},
}
Figure