Video-Based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms

Xianye Ben, Yi Ren, Junping Zhang, Su-Jing Wang, Kidiyo Kpalma, Weixiao Meng, Yong-Jin LiuIEEE TPAMI 2022 · DOI

Introduction

Facial micro-expressions (MEs) are involuntary, transient facial movements that occur when an individual attempts to suppress a genuine emotion. Unlike macro-expressions, which are easily observable and last several seconds, MEs typically last less than 0.5 seconds. This brevity and subtlety make them a "needle in a haystack" for both human observers and computer vision systems. While trained professionals recognize MEs with only ~47% accuracy, automated analysis offers the potential for high-stakes applications in security, clinical psychology, and lie detection. This paper surveys the current landscape of video-based ME analysis, bridging neuropsychological theory with state-of-the-art algorithmic approaches.

Methodology: Spotting and Recognition

The analysis of micro-expressions is generally divided into two primary tasks: spotting and recognition.

Spotting Algorithms

Spotting involves identifying the temporal boundaries of an expression within a video stream—specifically the onset (start), apex (peak), and offset (end). Because pixel movements are extremely slight, spotting algorithms must distinguish between genuine emotional leaks and noise such as head tilts or eye blinks. Techniques like Main Directional Maximal Difference (MDMD) track the magnitude of optical flow to detect these subtle shifts.

Temporal sequence of a micro-expression, highlighting the subtle onset, apex, and offset phases.

Recognition Algorithms

Once an ME is spotted, recognition algorithms classify it into specific emotion categories (e.g., happiness, disgust, or surprise). The field has transitioned from hand-crafted features, such as Local Binary Patterns from Three Orthogonal Planes (LBP-TOP), to deep learning architectures. A significant breakthrough is the use of Transfer Learning CNNs (TLCNN), which leverage features learned from large-scale macro-expression datasets to overcome the chronic scarcity of micro-expression data.

The Micro-and-Macro Expression Warehouse (MMEW)

A major hurdle in the field has been the lack of paired data—instances where the same individual exhibits both macro and micro versions of the same emotion. To address this, the authors introduce the MMEW dataset.

MMEW consists of 300 micro-expression and 900 macro-expression samples captured at 1920×1080 resolution. This high spatial resolution is critical; it allows models to capture the "incisive" muscle contractions that are often lost in lower-resolution benchmarks like CASME II.

A high-resolution macro-expression sample from the MMEW dataset.

Evaluation and Results

By re-evaluating representative algorithms on a unified platform, the survey provides a rigorous comparison of current techniques:

  • Spotting Performance: The MDMD algorithm proved most effective on the CAS(ME)² dataset, demonstrating that tracking optical flow directionality is superior to texture-based methods for temporal localization.
  • Recognition Accuracy: Deep learning models currently set the state-of-the-art. The TLCNN achieved accuracies of 69.4% on MMEW and 73.5% on SAMM, confirming that macro-to-micro transfer learning is a powerful strategy for small-sample problems.
  • Efficiency: Despite the dominance of deep learning, hand-crafted features like Mean Directional Mean Optical Flow (MDMO) remain relevant due to their computational efficiency and robustness against overfitting.

Future Directions and Challenges

The field faces several "frontier" challenges as it moves toward real-world deployment. Privacy is a paramount concern, suggesting a need for Federated Learning approaches. Furthermore, most current research relies on lab-controlled environments; "in the wild" performance remains a significant gap. Future work must also focus on explainability—ensuring that high-stakes decisions made by these models are transparent and grounded in observable facial muscle movements (Action Units). Finally, the use of Generative Adversarial Networks (GANs) to synthesize realistic ME data offers a promising path to solving the data scarcity problem.

Summary updated

BibTeX
@article{ben2022video,
  author = {Xianye Ben and Yi Ren and Junping Zhang and Su-Jing Wang and Kidiyo Kpalma and Weixiao Meng and Yong-Jin Liu},
  title = {{Video-Based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  doi = {10.1109/tpami.2021.3067464},
}
Figure