HMRM: A Hybrid Motion and Region-Fused Mamba Network for Micro-Expression Recognition

Zhe Guo, Yi Liu, Ray Luo, Jiayi Liu, Wei Lan

TL;DR

This paper introduces HMRM, a novel deep learning framework for micro-expression recognition (MER) that effectively balances accuracy and computational efficiency. By combining enhanced motion analysis with a Mamba-based encoder for efficient temporal modeling, HMRM achieves state-of-the-art results on benchmark datasets, making it suitable for real-world applications.

What Problem Does This Paper Address?

Micro-expressions (MEs) are fleeting, subtle facial movements that reveal genuine emotions, making them valuable for applications like emotion understanding, human-computer interaction, and security. However, recognizing these MEs is challenging due to their brief duration, low intensity, and fine-grained nature. Existing methods often struggle to capture these subtle spatiotemporal dynamics, especially under limited data and computational resources, hindering their deployment in practical systems. This paper aims to bridge this gap by developing a model that is both accurate and efficient for MER.

Methods

The proposed HMRM framework consists of three key components:

  • Hybrid Motion Feature Augmentation (HMFA): This module enhances motion representation. It uses a GRU-Attention Optical Flow Estimation (GRU-AOFE) mechanism to capture detailed motion patterns and a MotionMix strategy to synthesize new optical flow samples by swapping facial regions (eyes, mouth) between frames. This increases motion diversity and improves generalization, especially in data-scarce scenarios.
  • Grained Mamba Encoder: Built upon the Mamba architecture, this component efficiently models long-range temporal dependencies in a lightweight manner. It processes the optical flow maps by partitioning them into patches and using a bidirectional selective State Space Model (SSM) to capture spatiotemporal dynamics.
  • Regions Feature Fusion Strategy (RFFS): This strategy enhances the representation of localized facial dynamics. It divides the face into coarse regions (periocular and perioral zones) and further subdivides them into finer patches. Features from these regions are then fused using a multi-head attention mechanism, promoting cross-granularity and cross-regional interaction.

The paper also introduces a Multi-Scale Weighted Cross-Entropy (MS-WCE) loss function to adaptively emphasize discriminative features across different spatial scales, improving convergence and robustness.

Datasets

The authors evaluate HMRM on three widely used micro-expression recognition benchmark datasets:

  • CASME II: Contains 247 spontaneous ME sequences with a high frame rate (200 fps).
  • SMIC-HS: Includes 164 samples collected at 100 fps, captured under a "punishment threat" paradigm.
  • SAMM: Offers 159 high-resolution samples (2040 × 1088) from a diverse population.

They also create a composite dataset by merging these three to assess generalization across heterogeneous sources.

Key Insights and Ideas

  • Motion Augmentation is Crucial for Generalization: The MotionMix strategy, by synthesizing new motion patterns, significantly boosts performance, especially for underrepresented emotion classes like "Surprise." This highlights the importance of data diversity in MER.
  • Mamba's Efficiency for Temporal Modeling: The Grained Mamba Encoder proves effective in capturing long-range temporal dependencies without the quadratic complexity of Transformers, making the model lightweight and efficient.
  • Region-Based Fusion Captures Subtle Dynamics: Dividing the face into specific regions and fusing features from these areas helps the model focus on critical facial movements and improves robustness against noise and occlusion.
  • Attention in Optical Flow Estimation Matters: The GRU-AOFE module, with its integrated attention mechanism, outperforms traditional optical flow methods by better focusing on relevant facial regions and suppressing noise.

Key Results

HMRM achieves state-of-the-art (SOTA) performance on most evaluation metrics across the benchmark datasets.

  • On the composite dataset, HMRM achieves a Unweighted F1 score (UF1) of 0.8788 and an Unweighted Average Recall (UAR) of 0.8906, surpassing previous SOTA.
  • On the SAMM dataset, HMRM shows a significant improvement, achieving a UF1 of 0.8909 and UAR of 0.9017.
  • The model also demonstrates a favorable trade-off between model size and accuracy, reducing parameter count by nearly 50% compared to models with similar performance.
  • Ablation studies confirm the significant contributions of each proposed module (GRU-AOFE, MotionMix, RFFS) to the overall performance.

Limitations and Future Work

  • Dataset Imbalance: MER datasets inherently suffer from severe class imbalance, which can bias model optimization and evaluation. While HMRM's strategy helps, achieving fully balanced performance remains a challenge.
  • Limited Dataset Scale: The small size of MER datasets restricts the learning of high-capacity models and can hinder robustness.
  • Potential Dataset Bias: Existing benchmarks are often recorded under controlled conditions, limiting the generalizability of models to real-world scenarios with occlusions, varying illumination, and cluttered scenes.
  • Computational Trade-offs: While HMRM is designed to be lightweight, the hybrid motion augmentation module adds computation. Further pruning or quantization might be needed for edge device deployment.

Future work will focus on:

  • Optimizing feature extraction for low-resolution videos.
  • Investigating super-resolution and resolution-invariant learning.
  • Exploring self-supervised and domain adaptation methods.
  • Enhancing annotation consistency and developing uncertainty-aware learning.
  • Developing more comprehensive dynamic modeling strategies that integrate global and pose-invariant facial dynamics.
  • Optimizing for hardware-friendly deployment through model compression and adaptive inference.

Summary updated

Abstract

Micro-expression recognition (MER), as an important branch of intelligent visual sensing, enables the analysis of subtle facial movements for applications in emotion understanding, human-computer interaction and security monitoring. However, existing methods struggle to capture fine-grained spatiotemporal dynamics under limited data and computational resources, making them difficult to deploy in real-world sensing systems. To address this limitation, we propose HMRM, a hybrid motion and region-fused Mamba network designed for efficient and accurate MER. HMRM enhances motion representation through a hybrid feature augmentation module that integrates gated recurrent unit (GRU)-attention optical flow estimation with a regional MotionMix enhancement strategy to increase motion diversity. Furthermore, it employs a grained Mamba encoder to achieve lightweight and effective long-range temporal modeling. Additionally, a regions feature fusion strategy is introduced to strengthen the representation of localized expression dynamics. Experiments on multiple MER benchmark datasets demonstrate that HMRM achieves state-of-the-art performance with strong generalization and low computational cost, highlighting its potential for integration into compact, real-time visual sensing and emotion analysis systems.
BibTeX
@inproceedings{guo2025hmrm,
  author = {Zhe Guo and Yi Liu and Ray Luo and Jiayi Liu and Wei Lan},
  title = {{HMRM: A Hybrid Motion and Region-Fused Mamba Network for Micro-Expression Recognition}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25247672},
}
Figure