MIMAMO Net: Integrating Micro- and Macro-motion for Video Emotion Recognition

Didan Deng, Zhaokang Chen, Yuqian Zhou, Bertram Shi

TL;DR

MIMAMO Net improves video emotion recognition by explicitly integrating micro-motion (subtle, fast) and macro-motion (sustained, slow) features through a two-stream recurrent architecture. Its standout innovation is replacing traditional optical flow with interframe phase differences, resulting in a model that is significantly more robust to illumination changes and 10x faster to compute than standard motion-based approaches.

What Problem Does This Paper Address?

Human emotions are expressed through a complex mix of spatial and temporal cues. Psychologists distinguish between micro-expressions—involuntary facial movements lasting a fraction of a second—and macro-expressions, which are more sustained.

Existing deep learning models for video emotion recognition often struggle to capture both effectively. Architectures like CNN-RNNs excel at sustained "macro" changes but often miss subtle "micro" flickers. Conversely, 3D CNNs (C3D) can capture both but are difficult to initialize with pretrained knowledge. Furthermore, most temporal models rely on optical flow, which is computationally expensive and notoriously sensitive to lighting changes—a major hurdle for "in-the-wild" videos where illumination is rarely constant.

Methods

The authors propose MIMAMO Net (Micro-Macro-Motion Net), a two-stage architecture designed to partition the learning task:

  1. Two-Stream CNN (Snippet Level): This stage extracts features from short video snippets.
  2. Spatial Stream: A pretrained ResNet50 extracts static facial features (macro-appearance).
  3. Temporal Stream: Instead of optical flow, this stream uses interframe phase differences derived from a Complex Steerable Pyramid (see Figure 1). Phase shifts in this pyramid are proportional to local displacements, allowing the network to capture subtle micro-motions.
  4. Recurrent Network (Video Level): The concatenated features from both streams are fed into a bidirectional Gated Recurrent Unit (GRU). This stage models the long-term temporal correlations, effectively capturing the "macro-motion" of evolving expressions over seconds.
Figure 1: Complex Steerable Pyramid: A diagram for a 2-scale, 2-orientation Complex Steerable Pyramid (Portilla and Simoncelli 2000). The highpass residual and the low- pass residual are omitted here because we do not include them in our method. Low-pass filters and oriented band-pass filters are shown in frequency domain.

The full architecture is visualized in Figure 2, showing how the two streams converge before the GRU stage.

Figure 2: MIMAMO Net: Our proposed model consists of two stages. The first stage is a Two-Stream Convolutional Neural Network. The second stage is a Gated Recurrent Unit network. The figure shows the case when the length of the input to the temporal stream is 13 frames, i.e., the number of phase difference frames is 12. We treat images from different frames and orientations as belonging to different channels (C). (a) and (b) denote the temporal stream and the spatial stream respectively.

Datasets

The model was evaluated on two challenging "in-the-wild" benchmarks:

  • OMG-Emotion Behavior Dataset: YouTube clips labeled at the utterance level for Arousal (how active) and Valence (how positive/negative).
  • Aff-Wild Dataset: A large-scale dataset of YouTube videos with frame-level Arousal and Valence annotations.

Key Insights and Ideas

  • Phase vs. Flow: The core "aha" moment is that phase differences from a steerable pyramid are superior to optical flow for this task. Because they don't rely on the "brightness constancy constraint," they are naturally invariant to global illumination shifts.
  • The 0.4s Sweet Spot: Through ablation studies (Table 3 in the paper), the authors found that a temporal window of 12 frames (~0.4 seconds) was optimal for the micro-motion stream. This aligns perfectly with the psychological definition of micro-expressions, validating their architectural choices.
  • Motion Matters for Arousal: The temporal stream provided much larger gains for predicting arousal than valence. This makes intuitive sense: arousal is closely tied to the intensity and speed of movement (e.g., a sudden gasp vs. a slow smile).
  • Feature Dynamics: Visualization (Figure 4) shows that temporal stream features change rapidly, capturing "flickers," while GRU features evolve smoothly, capturing the broader emotional arc.

Key Results

MIMAMO Net achieved state-of-the-art (SOTA) results on both benchmarks, using the Concordance Correlation Coefficient (CCC) as the primary metric:

  • OMG Dataset: It outperformed the previous SOTA visual-only model by 54.5% in arousal and 21.0% in valence. Remarkably, it even beat several bimodal models that used both audio and video.
  • Aff-Wild Dataset: It achieved a 21.1% improvement in arousal over the SOTA CNN-RNN baseline.
  • Robustness: In experiments where video brightness was randomly jittered, the performance of optical-flow-based models crashed, while the phase-difference-based MIMAMO Net remained stable (see Figure 6).
  • Efficiency: Computing the Complex Steerable Pyramid was found to be 10 times faster than a standard GPU implementation of optical flow.
Figure 6: Comparison of robustness to illumination changes. The phase-difference approach (bottom) remains stable as lighting variability (beta) increases, whereas optical flow (top) degrades significantly.

Limitations and Future Work

The authors acknowledge a few areas for expansion:

  • Rigid Motion: The current pipeline uses face alignment to remove head movements. However, head tilts and nods are often expressive; future versions might benefit from including this "discarded" motion.
  • Modality Gap: While the model performs exceptionally well for a visual-only system, it does not yet incorporate audio. Given that audio is a primary channel for emotion, integrating it into the MIMAMO framework is a logical next step.
  • Micro-expression Labels: The model learns to use micro-motions implicitly, but it isn't trained to explicitly identify specific micro-expressions due to a lack of large-scale labeled datasets for that specific task.

Summary updated

Abstract

Spatial-temporal feature learning is of vital importance for video emotion recognition. Previous deep network structures often focused on macro-motion which extends over long time scales, e.g., on the order of seconds. We believe integrating structures capturing information about both micro- and macro-motion will benefit emotion prediction, because human perceive both micro- and macro-expressions. In this paper, we propose to combine micro- and macro-motion features to improve video emotion recognition with a two-stream recurrent network, named MIMAMO (Micro-Macro-Motion) Net. Specifically, smaller and shorter micro-motions are analyzed by a two-stream network, while larger and more sustained macro-motions can be well captured by a subsequent recurrent network. Assigning specific interpretations to the roles of different parts of the network enables us to make choice of parameters based on prior knowledge: choices that turn out to be optimal. One of the important innovations in our model is the use of interframe phase differences rather than optical flow as input to the temporal stream. Compared with the optical flow, phase differences require less computation and are more robust to illumination changes. Our proposed network achieves state of the art performance on two video emotion datasets, the OMG emotion dataset and the Aff-Wild dataset. The most significant gains are for arousal prediction, for which motion information is intuitively more informative. Source code is available at https://github.com/wtomin/MIMAMO-Net.
BibTeX
@inproceedings{deng2020mimamo,
  author = {Didan Deng and Zhaokang Chen and Yuqian Zhou and Bertram Shi},
  title = {{MIMAMO Net: Integrating Micro- and Macro-motion for Video Emotion Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=uBhtOiWu5g},
}
Figure