Revealing the Invisible With Model and Data Shrinking for Composite-Database Micro-Expression Recognition

Zhaoqiang Xia, Wei Peng, Huai-Qian Khor, Xiaoyi Feng, Guoying Zhao

TL;DR

In the world of Micro-expression Recognition (MER), bigger isn't always better. This paper demonstrates that "shrinking" both model depth and input resolution actually improves performance on composite (mixed) datasets by preventing deep networks from overfitting to domain-specific noise. By combining a shallow Recurrent Convolutional Network (RCN) with three novel parameter-free modules, the authors achieved state-of-the-art results on the challenging MEGC2019 benchmark.

What Problem Does This Paper Address?

Micro-expressions are fleeting, involuntary facial movements that reveal genuine emotions. Recognizing them is incredibly difficult because they are subtle and short-lived. While deep learning has mastered "macro-expressions" (normal smiles or frowns), it struggles with MER, especially in composite-database scenarios.

Most MER research happens on small, controlled individual datasets. However, real-world applications require models to work across different cameras, lighting conditions, and ethnicities (domain shift). When you mix these datasets together, traditional deep models like ResNet or DenseNet often fail. They are so powerful that they "learn" the specific noise and characteristics of each dataset (the domain) rather than the universal, subtle muscle movements of the micro-expression itself.

Methods

The authors propose a strategy of Model and Data Shrinking. Instead of going deeper, they go shallower and use lower-resolution data to force the model to focus on the most essential features.

  1. The Backbone (RCN): They use a Recurrent Convolutional Network. The "recurrent" part isn't for temporal sequences in the traditional sense, but rather a way to perform multiple convolutions by "unfolding" a single layer with shared parameters. This keeps the model shallow and the parameter count low.
  2. Input Representation: They use Optical Flow Maps (horizontal and vertical velocity plus "optical strain") derived from the onset and apex frames of a video.
  3. Three Parameter-Free Modules: To boost representation power without adding learnable parameters (which would risk overfitting), they developed:
  4. Wide Expansion (RCN-W): Uses dilated convolutions to capture a larger receptive field.
  5. Shortcut Connection (RCN-S): Adds residual-style connections within the recurrent layers to improve gradient flow.
  6. Attention Unit (RCN-A): A parameter-free soft attention mechanism inspired by Class Activation Mapping (CAM) that helps the model focus on relevant facial regions (like the forehead or mouth).
  7. Neural Architecture Search (NAS): They used a gradient-based NAS strategy to automatically find the best way to combine these modules into a final robust architecture (RCN-F).

Datasets

The primary benchmark is the MEGC2019 (Micro-Expression Grand Challenge 2019) dataset. This is a composite dataset created by merging three existing ones:

  • SMIC: 164 samples, 100 fps.
  • CASME II: 145 samples, 200 fps.
  • SAMM: 133 samples, 200 fps.

The evaluation uses a Leave-One-Subject-Out (LOSO) protocol, which is the gold standard for ensuring a model generalizes to new people.

Key Insights and Ideas

  • The "Shrinking" Paradox: In a surprising "aha moment," the authors found that as input resolution increased beyond $100 \times 100$ or model depth increased, performance on composite datasets dropped significantly. High resolution introduces high-frequency noise that deep models mistake for emotional cues.
  • Domain Noise vs. Signal: Deep models are "distracted" by domain shifts. By shrinking the learning complexity, the authors effectively created a "bottleneck" that filters out domain-specific noise (like lighting differences between datasets) while letting the micro-expression signal pass through.
  • Parameter-Free is Key: Adding more layers usually helps in computer vision, but in MER—where data is scarce—it leads to overfitting. The success of their parameter-free modules proves you can increase a model's "expressiveness" without making it harder to train.

Key Results

The proposed RCN models consistently outperformed much larger, standard architectures on the MEGC2019 composite task:

  • Superiority over SOTA: The RCN-A model achieved an Unweighted Average Recall (UAR) of 0.7190, significantly beating ResNet18 (0.5629) and DenseNet121 (0.3414).
  • Robustness: While standard deep models saw their performance tank when moving from individual to composite datasets, the RCN variants remained stable.
  • Efficiency: Because the models are "shrunk," they are incredibly fast. Training and testing for all 68 subjects took only about 1.1 hours on a single GPU at $60 \times 60$ resolution, compared to much longer times for deeper networks.

Limitations and Future Work

  • Pre-processing Dependency: The method relies heavily on the quality of the Optical Flow Map and the accurate detection of "apex" frames. If the pre-processing fails to capture the motion correctly, the model cannot recover.
  • NAS Search Space: The Neural Architecture Search was limited to a specific set of combinations of the three proposed modules. A wider search space might yield even more efficient "shrunk" models.
  • Future Direction: The authors suggest that exploring more sophisticated ways to handle the temporal dynamics of micro-expressions (beyond just onset-to-apex flow) while maintaining low model complexity is a promising next step.

Summary updated

Abstract

Composite-database micro-expression recognition is attracting increasing attention as it is more practical for real-world applications. Though the composite database provides more sample diversity for learning good representation models, the important subtle dynamics are prone to disappearing in the domain shift such that the models greatly degrade their performance, especially for deep models. In this article, we analyze the influence of learning complexity, including input complexity and model complexity, and discover that the lower-resolution input data and shallower-architecture model are helpful to ease the degradation of deep models in composite-database task. Based on this, we propose a recurrent convolutional network (RCN) to explore the shallower-architecture and lower-resolution input data, shrinking model and input complexities simultaneously. Furthermore, we develop three parameter-free modules (i.e., wide expansion, shortcut connection and attention unit) to integrate with RCN without increasing any learnable parameters. These three modules can enhance the representation ability in various perspectives while preserving not-very-deep architecture for lower-resolution data. Besides, three modules can further be combined by an automatic strategy (a neural architecture search strategy) and the searched architecture becomes more robust. Extensive experiments on the MEGC2019 dataset (composited of existing SMIC, CASME II and SAMM datasets) have verified the influence of learning complexity and shown that RCNs with three modules and the searched combination outperform the state-of-the-art approaches.
BibTeX
@article{xia2021revealing,
  author = {Zhaoqiang Xia and Wei Peng and Huai-Qian Khor and Xiaoyi Feng and Guoying Zhao},
  title = {{Revealing the Invisible With Model and Data Shrinking for Composite-Database Micro-Expression Recognition}},
  journal = {IEEE Transactions on Image Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=yw0xrjCtEi},
}
Figure