Mode Variational LSTM Robust to Unseen Modes of Variation: Application to Facial Expression Recognition

Wissam J. Baddar, Yong Man Ro

Abstract

Recurrent neural networks, particularly LSTMs, are widely used for spatio-temporal feature encoding in video. However, they often retain information related to mode variations (e.g., identity, pose, illumination) that are irrelevant to tasks like facial expression recognition. This paper introduces the Mode Variational LSTM (MV-LSTM), which incorporates an additional cell state and gating functionality to explicitly model and mitigate these biases. By feeding a static sequence into this "Bias Cell," the network learns to isolate environmental noise from the dynamic expression signal. Evaluations on the Oulu-CASIA, AFEW, and a newly collected multi-modal dataset demonstrate that the MV-LSTM significantly improves robustness and classification accuracy compared to standard LSTM architectures.

Introduction: The Problem of Mode Variation Bias

In facial expression recognition (FER), the goal is to extract features that represent the movement of facial muscles over time. However, standard LSTMs are prone to capturing redundant information. If a subject is recorded from a side profile or under harsh lighting, the LSTM's hidden state often becomes "contaminated" with these static environmental factors. This lack of robustness leads to poor generalization when the model encounters a new subject or a different camera angle—a phenomenon known as mode variation bias. The LSTM's forget mechanism is often insufficient to discard these irrelevant features, which can negatively impact classification performance.

The Mode Variational LSTM Architecture

The researchers from KAIST propose a structural modification to the standard LSTM to disentangle the expression signal from environmental noise. The MV-LSTM utilizes a dual-cell structure:

  1. The Dynamics Cell: Functions like a standard LSTM, processing the original video sequence to capture the temporal dynamics of the expression.
  2. The Bias Cell: Specifically designed to encode the "mode of variation." This cell is fed a static sequence generated by replicating the first frame of the input sequence (typically a neutral expression) $N$ times.

Cross-Cell Peephole Connections

The innovation lies in how these cells interact. The architecture introduces cross-cell peephole connections, allowing the gates of the Dynamics Cell to "peep" into the state of the Bias Cell. By understanding the specific bias (e.g., "this person is lit from the left"), the Dynamics Cell can more effectively use its forget gate to discard redundant information in the actual video sequence. The two cells are synchronized through a shared output gate to produce a robust, "de-biased" feature vector.

Key Insights: Bias Convergence

The core premise of the MV-LSTM is that an LSTM, when fed a static image repeatedly, will eventually converge to a feature vector that represents the static characteristics of that image—its "bias."

Visualization of feature convergence for different modes of variation (appearance, illumination, and pose) using static sequences.

As shown in the figure above, different modes of variation (appearance, illumination, and pose) converge to distinct spatio-temporal feature signatures. The MV-LSTM leverages this behavior to isolate these factors without requiring explicit labels for pose or lighting during training. This allows the model to generalize to "unseen" variations by learning the structure of the noise rather than just the labels.

Experimental Evaluation

The MV-LSTM was evaluated across three major datasets to test its generalization and robustness:

  • Oulu-CASIA: In controlled lab settings, the MV-LSTM achieved 85.18% accuracy, a significant jump from the standard LSTM's 78.21%.
  • AFEW (Acting Facial Expressions in the Wild): In this challenging real-world dataset, the model reached 51.44% accuracy, outperforming standard RNN/LSTM baselines by over 11%.
  • KAIST Face MPMI: A new dataset introduced by the authors featuring 104 subjects across 13 poses and 4 illumination conditions. The MV-LSTM demonstrated remarkable stability, maintaining high recognition rates even at extreme 60-degree head turns, where traditional models showed steep performance degradation.
  • Cross-Race Generalization: In tests where the model was trained on one race and tested on another, the MV-LSTM improved accuracy by roughly 5-6% over the baseline, proving its ability to ignore identity-specific features.

Discussion and Conclusion

The Mode Variational LSTM offers a robust solution for spatio-temporal feature encoding by explicitly addressing the problem of environmental and identity-based bias. While the method currently relies on a "neutral" reference frame, its ability to disentangle features without explicit labels is a significant advancement. This architectural approach holds promise for other video analysis tasks where background noise and subject-specific variations hinder model generalization, such as action recognition or medical imaging analysis.

Summary updated

Abstract

Spatio-temporal feature encoding is essential for encoding the dynamics in video sequences. Recurrent neural networks, particularly long short-term memory (LSTM) units, have been popular as an efficient tool for encoding spatio-temporal features in sequences. In this work, we investigate the effect of mode variations on the encoded spatio-temporal features using LSTMs. We show that the LSTM retains information related to the mode variation in the sequence, which is irrelevant to the task at hand (e.g. classification facial expressions). Actually, the LSTM forget mechanism is not robust enough to mode variations and preserves information that could negatively affect the encoded spatio-temporal features. We propose the mode variational LSTM to encode spatio-temporal features robust to unseen modes of variation. The mode variational LSTM modifies the original LSTM structure by adding an additional cell state that focuses on encoding the mode variation in the input sequence. To efficiently regulate what features should be stored in the additional cell state, additional gating functionality is also introduced. The effectiveness of the proposed mode variational LSTM is verified using the facial expression recognition task. Comparative experiments on publicly available datasets verified that the proposed mode variational LSTM outperforms existing methods. Moreover, a new dynamic facial expression dataset with different modes of variation, including various modes like pose and illumination variations, was collected to comprehensively evaluate the proposed mode variational LSTM. Experimental results verified that the proposed mode variational LSTM encodes spatio-temporal features robust to unseen modes of variation.
BibTeX
@inproceedings{baddar2019mode,
  author = {Wissam J. Baddar and Yong Man Ro},
  title = {{Mode Variational LSTM Robust to Unseen Modes of Variation: Application to Facial Expression Recognition}},
  booktitle = {AAAI},
  year = {2019},
  url = {https://openreview.net/forum?id=Ss843yGld6B},
}

Loading related content...

Figure