TL;DR
This paper introduces a novel multimodal learning framework that combines micro-expressions (MEs) and physiological signals (PS) to improve the accuracy of latent emotion recognition. By developing specialized feature extraction and attention-guided fusion mechanisms, the approach significantly outperforms existing methods, offering a more robust way to understand genuine human emotions.
What Problem Does This Paper Address?
Accurately recognizing human emotions is crucial for effective communication and human-computer interaction. While facial expressions, particularly subtle micro-expressions (MEs), are a primary indicator, individuals can consciously control them to mask their true feelings. Physiological signals (PS), such as heart rate and skin response, are involuntary and harder to suppress, offering a more authentic window into emotional states. However, relying solely on either MEs or PS has limitations. Existing multimodal approaches often struggle with effectively fusing these diverse data types, particularly when dealing with the noise inherent in PS or the fleeting nature of MEs. This paper aims to bridge this gap by proposing a unified framework that leverages the strengths of both MEs and PS for more reliable latent emotion recognition.
Methods
The proposed framework consists of three main components:
- ME Feature Extraction Branch: This branch processes color and depth information from ME video clips. It uses pre-trained VGG-Face and VGG-16 networks as backbones for feature extraction. Depth information is used to guide spatial features from each frame.
- PS Feature Extraction Branch: A novel 1D separable and mixable depthwise inception CNN is designed to extract features from physiological signals (EDA, ECG, PPG). This network uses depthwise convolutions and inception blocks to capture complex patterns at multiple scales within the 1D signal data. Before feeding into this network, PS signals undergo wavelet denoising to mitigate noise.
- Guided Attention Fusion Module: This module integrates features from both ME and PS branches. It employs a standard normal distribution weighted feature fusion method for ME features, giving more weight to frames closer to the apex of the micro-expression. For the overall multimodal fusion, a depth/physiology guided attention module is used. This module leverages attention mechanisms (specifically scaled dot-product attention) to allow ME features (color and depth) and PS features to guide each other, leading to a more informed and synergistic representation.
Datasets
The study utilizes Part C of the CAS(ME)³ corpus. This dataset is notable for its multimodality, including RGB images, depth information, voice, and physiological signals (EDA, ECG, PPG) collected under ecologically valid conditions (a mock crime scenario). The authors specifically select ME samples with less than 15 frames (500ms) to focus on the brief nature of micro-expressions.
Key Insights and Ideas
- Complementary Nature of MEs and PS: The core idea is that involuntary physiological signals can validate or reveal emotions masked by controllable micro-expressions, leading to more authentic emotion recognition.
- Importance of PS Preprocessing: Wavelet denoising is crucial for effectively utilizing physiological signals, which are often noisy.
- Specialized PS Network: A dedicated depthwise inception CNN is necessary for effectively extracting features from 1D physiological signals, outperforming simpler approaches.
- Temporal Weighting for MEs: Not all frames in an ME sequence are equally informative. Assigning higher weights to frames closer to the apex of the expression (using a standard normal distribution) improves feature fusion.
- Attention-Guided Multimodal Fusion: Using attention mechanisms where one modality's features guide the learning of another's leads to significantly better performance than simple concatenation or uniform fusion.
Key Results
The proposed multimodal framework achieved significant improvements over existing methods.
- Overall Performance: The paper reports an accuracy of 0.738 with an F1-score of 0.586 and UAR of 0.563 for the "Colour + Depth + PS" combination, which substantially outperforms the benchmark method by Li et al. (0.230 Acc, 0.244 UF1, 0.260 UAR for "Colour + PS").
- Feature Fusion Impact: The standardized normal distribution weighted fusion for ME features improved accuracy from 0.610 to 0.640 compared to a uniform distribution.
- Attention Module Effectiveness: The guided attention module outperformed simple concatenation for multimodal fusion, leading to an accuracy of 0.738 (ME + PS) compared to 0.701 for concatenation.
Limitations and Future Work
- Depth Information Noise: The authors acknowledge that depth information can be noisy, which is particularly problematic for subtle MEs. They suggest that future research could focus on better denoising techniques for depth data.
- Generalizability: While the CAS(ME)³ dataset is valuable, the performance on other datasets or in real-world, uncontrolled environments might differ.
- Computational Cost: The proposed framework involves multiple complex modules (depthwise inception CNN, attention mechanisms), which could lead to higher computational demands.
Future work could explore more advanced denoising techniques for both depth and physiological signals, investigate the transferability of the model to different datasets, and optimize the architecture for real-time applications.
Summary updated