SelfME: Self-Supervised Motion Learning for Micro-Expression Recognition

Xinqi Fan, Xueli Chen, Mingjie Jiang, Ali Raza Shahid, Hong YanCVPR 2023 · DOI

Introduction

Micro-expressions (MEs) are the involuntary "tells" of the human face—brief movements lasting less than half a second that reveal genuine emotions. Due to their subtlety and speed, they are notoriously difficult for both humans and standard computer vision models to detect.

Traditionally, high-performing models for Micro-Expression Recognition (MER) have relied on handcrafted "optical flow" algorithms (like TV-L1) to map pixel movement between frames. However, these methods are computationally expensive and often fail to capture the specific nuances required for deep learning. SelfME introduces a novel framework that moves MER toward an end-to-end, self-supervised approach, learning to extract motion features automatically while leveraging the natural symmetry of spontaneous facial movements.

The SelfME Framework

The SelfME framework operates in two distinct stages: a self-supervised motion learning phase followed by a classification phase utilizing a Symmetric Contrastive Vision Transformer (SCViT).

The SelfME architecture consists of a self-supervised Motion Learning Stage and a Classification Stage using a Vision Transformer.

Motion Learning Stage

In this stage, the model is trained to "reconstruct" a target frame (the apex or peak of the expression) by warping a source frame (the onset or start). The system identifies motion-related keypoints and computes local displacements. To ensure the resulting motion is smooth and continuous, the authors employ a B-spline transformation. This maintains diffeomorphism, preventing the warped image from "tearing" or "folding," which is crucial for preserving the integrity of facial structures.

Classification Stage and Symmetric Contrastive Learning

Once the motion features are extracted, they are fed into a Vision Transformer (ViT). The core innovation here is the Symmetric Contrastive (SC) loss. Because spontaneous expressions are generally symmetric, the researchers introduced a mechanism to force the model to learn similar representations for the left and right sides of the face. This helps the model ignore noise—such as asymmetrical lighting or minor head tilts—and focus on the actual emotional muscle movements.

The Symmetric Contrastive mechanism regroups patches from the left and right sides of the face to enforce symmetry in learned representations.

Key Insights and Results

  • Symmetry as a Superpower: Enforcing facial symmetry via SC loss provided a larger performance boost than the motion learning itself, acting as a powerful inductive bias for the Transformer.
  • Ditching Handcrafted Flow: By learning motion from scratch, the model becomes more flexible, focusing specifically on the eyes, eyebrows, and mouth—the primary regions for micro-expressions.
  • State-of-the-Art Performance: SelfME set a new benchmark on the CASME II dataset, achieving an Unweighted F1-score (UF1) of 0.9078 and an Unweighted Average Recall (UAR) of 0.9290. This significantly outperformed previous state-of-the-art methods like FeatRef.
  • Ablation Success: Removing the Symmetric Contrastive component caused the UF1 to drop from approximately 90% to 86%, confirming its vital role in the system's accuracy.

Limitations and Future Work

While SelfME represents a significant leap forward, it faces challenges with grayscale datasets (like SAMM), where the lack of color information can destabilize motion extraction. Additionally, the model currently requires manual identification of onset and apex frames.

Future research aims to evolve SelfME into a unified pipeline capable of both spotting (finding the expression in a continuous video stream) and recognition (classifying the emotion) simultaneously, while also improving robustness against varying head poses.

Summary updated

BibTeX
@inproceedings{fan2023selfme,
  author = {Xinqi Fan and Xueli Chen and Mingjie Jiang and Ali Raza Shahid and Hong Yan},
  title = {{SelfME: Self-Supervised Motion Learning for Micro-Expression Recognition}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01329},
}
Figure