TL;DR
MIMAMO Net improves video emotion recognition by explicitly integrating micro-motion (subtle, fast) and macro-motion (sustained, slow) features through a two-stream recurrent architecture. Its standout innovation is replacing traditional optical flow with interframe phase differences, resulting in a model that is significantly more robust to illumination changes and 10x faster to compute than standard motion-based approaches.
What Problem Does This Paper Address?
Human emotions are expressed through a complex mix of spatial and temporal cues. Psychologists distinguish between micro-expressions—involuntary facial movements lasting a fraction of a second—and macro-expressions, which are more sustained.
Existing deep learning models for video emotion recognition often struggle to capture both effectively. Architectures like CNN-RNNs excel at sustained "macro" changes but often miss subtle "micro" flickers. Conversely, 3D CNNs (C3D) can capture both but are difficult to initialize with pretrained knowledge. Furthermore, most temporal models rely on optical flow, which is computationally expensive and notoriously sensitive to lighting changes—a major hurdle for "in-the-wild" videos where illumination is rarely constant.
Methods
The authors propose MIMAMO Net (Micro-Macro-Motion Net), a two-stage architecture designed to partition the learning task:
- Two-Stream CNN (Snippet Level): This stage extracts features from short video snippets.
- Spatial Stream: A pretrained ResNet50 extracts static facial features (macro-appearance).
- Temporal Stream: Instead of optical flow, this stream uses interframe phase differences derived from a Complex Steerable Pyramid (see Figure 1). Phase shifts in this pyramid are proportional to local displacements, allowing the network to capture subtle micro-motions.
- Recurrent Network (Video Level): The concatenated features from both streams are fed into a bidirectional Gated Recurrent Unit (GRU). This stage models the long-term temporal correlations, effectively capturing the "macro-motion" of evolving expressions over seconds.
The full architecture is visualized in Figure 2, showing how the two streams converge before the GRU stage.
Datasets
The model was evaluated on two challenging "in-the-wild" benchmarks:
- OMG-Emotion Behavior Dataset: YouTube clips labeled at the utterance level for Arousal (how active) and Valence (how positive/negative).
- Aff-Wild Dataset: A large-scale dataset of YouTube videos with frame-level Arousal and Valence annotations.
Key Insights and Ideas
- Phase vs. Flow: The core "aha" moment is that phase differences from a steerable pyramid are superior to optical flow for this task. Because they don't rely on the "brightness constancy constraint," they are naturally invariant to global illumination shifts.
- The 0.4s Sweet Spot: Through ablation studies (Table 3 in the paper), the authors found that a temporal window of 12 frames (~0.4 seconds) was optimal for the micro-motion stream. This aligns perfectly with the psychological definition of micro-expressions, validating their architectural choices.
- Motion Matters for Arousal: The temporal stream provided much larger gains for predicting arousal than valence. This makes intuitive sense: arousal is closely tied to the intensity and speed of movement (e.g., a sudden gasp vs. a slow smile).
- Feature Dynamics: Visualization (Figure 4) shows that temporal stream features change rapidly, capturing "flickers," while GRU features evolve smoothly, capturing the broader emotional arc.
Key Results
MIMAMO Net achieved state-of-the-art (SOTA) results on both benchmarks, using the Concordance Correlation Coefficient (CCC) as the primary metric:
- OMG Dataset: It outperformed the previous SOTA visual-only model by 54.5% in arousal and 21.0% in valence. Remarkably, it even beat several bimodal models that used both audio and video.
- Aff-Wild Dataset: It achieved a 21.1% improvement in arousal over the SOTA CNN-RNN baseline.
- Robustness: In experiments where video brightness was randomly jittered, the performance of optical-flow-based models crashed, while the phase-difference-based MIMAMO Net remained stable (see Figure 6).
- Efficiency: Computing the Complex Steerable Pyramid was found to be 10 times faster than a standard GPU implementation of optical flow.
Limitations and Future Work
The authors acknowledge a few areas for expansion:
- Rigid Motion: The current pipeline uses face alignment to remove head movements. However, head tilts and nods are often expressive; future versions might benefit from including this "discarded" motion.
- Modality Gap: While the model performs exceptionally well for a visual-only system, it does not yet incorporate audio. Given that audio is a primary channel for emotion, integrating it into the MIMAMO framework is a logical next step.
- Micro-expression Labels: The model learns to use micro-motions implicitly, but it isn't trained to explicitly identify specific micro-expressions due to a lack of large-scale labeled datasets for that specific task.
Summary updated