Abstract
Micro-expressions (MEs) are involuntary, brief facial movements that reveal suppressed emotional states. Due to their low intensity and short duration, automated recognition is significantly challenging. This paper introduces a dual-channel deep learning pipeline that integrates facial graph representation learning with Action Unit (AU) fusion. By utilizing motion magnification and preserving spatial relationships through Depthwise Convolutions and Transformer Encoders, the model captures subtle geometric changes. Simultaneously, a Graph Convolutional Network (GCN) incorporates biological prior knowledge of AU co-occurrences. The proposed method achieves state-of-the-art results on major benchmarks, including an accuracy of 80.80% on CASME II and 82.39% on SAMM.
Introduction
Micro-expressions typically last between 0.04 and 0.2 seconds, making them nearly imperceptible to the human eye. Traditional computer vision methods often rely on texture-based features, which are susceptible to lighting variations, or geometric features that lack the adaptability of deep learning. Furthermore, standard deep learning models often flatten facial data into one-dimensional vectors, destroying critical spatial relationships. This research addresses these limitations by proposing a graph-based representation that preserves facial geometry and fuses it with the underlying muscle activation logic defined by Facial Action Units (AUs).
Methodology
The proposed architecture consists of three primary components: facial graph representation, AU learning, and feature fusion.
Facial Graph Representation Learning
To capture subtle movements, the system first employs a motion magnification network (MagNet) on the onset and apex frames. The model extracts 30 node patches (7x7 pixels) centered on key landmarks around the eyebrows and mouth. Unlike traditional methods that flatten these patches, this approach uses Depthwise Convolution (DConv) to treat each patch as a separate channel, preserving internal spatial features. Subsequently, a Transformer Encoder (ETran) models the global relationships (edges) between these nodes, allowing the network to understand how disparate facial movements correlate.
Action Unit Learning and Fusion
The second channel, the AUGCN model, utilizes a Graph Convolutional Network to learn relationships between 9 specific AUs. This model uses word embeddings for AU nodes and an adjacency matrix derived from statistical co-occurrence in human expressions. Finally, the AUFusion module performs a dot product between the facial graph features and the AU features, ensuring the classification is informed by both raw visual geometry and biological muscle movement rules.
Experimental Results
The model was evaluated on the CASME II, SAMM, and SMIC datasets. It demonstrated superior performance compared to existing state-of-the-art (SOTA) methods:
- CASME II: Achieved 80.80% accuracy, significantly outperforming the previous Graph-TCN benchmark of 73.60%.
- SAMM: Achieved 82.39% accuracy, setting a new high bar for deep learning methods on this multi-ethnic dataset.
- Composite Evaluation: On the MEGC 2019 protocol (CDE), the model reached an Unweighted F1-score (UF1) of 0.7914, proving its robustness across different recording conditions.
Conclusion
By combining spatial-preserving graph learning with Action Unit prior knowledge, this research significantly advances the accuracy of micro-expression recognition. The use of Depthwise Convolution for node learning was identified as a key factor in reducing spatial information loss. Future work will explore expanding the graph to include more facial regions and optimizing the architecture for real-time computational efficiency.
Summary updated