CMNet: Contrastive Magnification Network for Micro-Expression Recognition

Mengting Wei, Xingxun Jiang, Wenming Zheng, Yuan Zong, Cheng Lu, Jiateng LiuAAAI 2023 · DOI

CMNet: Contrastive Magnification Network for Micro-Expression Recognition

Micro-expressions (MEs) are involuntary, fleeting facial movements that last less than 0.5 seconds. Because they are so subtle, they are incredibly difficult for standard computer vision models to detect. Existing solutions often struggle with two major hurdles: feature contamination, where facial identity (texture) interferes with movement (intensity) features, and temporal disorder, where magnification strategies ignore the natural physical progression of an expression.

CMNet (Contrastive Magnification Network) is a novel framework designed to overcome these issues by isolating and magnifying subtle facial movements through contrastive learning and statistical temporal calibration.

Methodology

CMNet employs a dual-branch architecture to process micro-expression video clips. The primary goal is to decouple the transient intensity of muscle movements from the static identity of the face.

The overall architecture of CMNet, featuring dual branches for intensity magnification and facial representation extraction.

Intensity Distillation (ID) Loss

The framework uses contrastive learning to isolate intensity. By treating two augmented versions of the same frame as a positive pair and different frames from the same video as negative pairs, the model is forced to focus solely on differences in muscle movement. Since the person's identity remains constant throughout the video, the contrastive objective effectively "cancels out" static facial textures. To further improve learning, a probabilistic sampling strategy selects "harder" negatives—frames with slight intensity differences—forcing the model to learn finer boundaries.

Temporal Calibration via Wrst Loss

Micro-expressions follow a specific physical trend: starting at "onset," peaking at "apex," and fading at "offset." To ensure the model respects this flow, the authors build an Intensity Tendency Prototype, modeling intensity as a series of Gaussian distributions following an onset-apex-offset curve.

The intensity tendency prototype (a) and its application in modeling the natural temporal flow of micro-expressions (b).

Because "intensity" is a subjective and unlabelled metric, CMNet uses the Wilcoxon Rank Sum Test (Wrst) loss. This non-parametric statistical test ensures the rank order of the extracted features matches the natural progression of the expression prototype. This approach is more flexible than standard regression, as it focuses on the relative timing of movement rather than absolute numerical values.

Experimental Results

The model was evaluated on three standard public micro-expression benchmarks using the Leave-One-Subject-Out (LOSO) cross-validation protocol:

  • SAMM: Achieved 78.68% Accuracy and 77.19% F1-score, significantly outperforming previous methods like MTMNet.
  • SMIC-HS: Achieved 79.27% Accuracy and 80.11% F1-score, a major jump over the previous best (TSCNN at ~72%).
  • CASME II: Performed competitively with 78.05% Accuracy.

Ablation studies confirmed that the Intensity Distillation (ID) branch is the primary driver of performance, while the Wrst calibration provides the final refinement needed to achieve state-of-the-art results.

Discussion and Future Work

CMNet demonstrates that ranking-based temporal calibration is superior to rigid regression for unlabelled intensity features. However, the current model assumes a single, clear apex frame, which may not perfectly fit complex or "flickering" micro-expressions. Future research could explore more advanced temporal encoding, such as Transformers, to further improve the capture of these fleeting movements.

Summary updated

BibTeX
@inproceedings{wei2023cmnet,
  author = {Mengting Wei and Xingxun Jiang and Wenming Zheng and Yuan Zong and Cheng Lu and Jiateng Liu},
  title = {{CMNet: Contrastive Magnification Network for Micro-Expression Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i1.25083},
}
Figure