Abstract
Recurrent neural networks, particularly LSTMs, are widely used for spatio-temporal feature encoding in video. However, they often retain information related to mode variations (e.g., identity, pose, illumination) that are irrelevant to tasks like facial expression recognition. This paper introduces the Mode Variational LSTM (MV-LSTM), which incorporates an additional cell state and gating functionality to explicitly model and mitigate these biases. By feeding a static sequence into this "Bias Cell," the network learns to isolate environmental noise from the dynamic expression signal. Evaluations on the Oulu-CASIA, AFEW, and a newly collected multi-modal dataset demonstrate that the MV-LSTM significantly improves robustness and classification accuracy compared to standard LSTM architectures.
Introduction: The Problem of Mode Variation Bias
In facial expression recognition (FER), the goal is to extract features that represent the movement of facial muscles over time. However, standard LSTMs are prone to capturing redundant information. If a subject is recorded from a side profile or under harsh lighting, the LSTM's hidden state often becomes "contaminated" with these static environmental factors. This lack of robustness leads to poor generalization when the model encounters a new subject or a different camera angle—a phenomenon known as mode variation bias. The LSTM's forget mechanism is often insufficient to discard these irrelevant features, which can negatively impact classification performance.
The Mode Variational LSTM Architecture
The researchers from KAIST propose a structural modification to the standard LSTM to disentangle the expression signal from environmental noise. The MV-LSTM utilizes a dual-cell structure:
- The Dynamics Cell: Functions like a standard LSTM, processing the original video sequence to capture the temporal dynamics of the expression.
- The Bias Cell: Specifically designed to encode the "mode of variation." This cell is fed a static sequence generated by replicating the first frame of the input sequence (typically a neutral expression) $N$ times.
Cross-Cell Peephole Connections
The innovation lies in how these cells interact. The architecture introduces cross-cell peephole connections, allowing the gates of the Dynamics Cell to "peep" into the state of the Bias Cell. By understanding the specific bias (e.g., "this person is lit from the left"), the Dynamics Cell can more effectively use its forget gate to discard redundant information in the actual video sequence. The two cells are synchronized through a shared output gate to produce a robust, "de-biased" feature vector.
Key Insights: Bias Convergence
The core premise of the MV-LSTM is that an LSTM, when fed a static image repeatedly, will eventually converge to a feature vector that represents the static characteristics of that image—its "bias."
As shown in the figure above, different modes of variation (appearance, illumination, and pose) converge to distinct spatio-temporal feature signatures. The MV-LSTM leverages this behavior to isolate these factors without requiring explicit labels for pose or lighting during training. This allows the model to generalize to "unseen" variations by learning the structure of the noise rather than just the labels.
Experimental Evaluation
The MV-LSTM was evaluated across three major datasets to test its generalization and robustness:
- Oulu-CASIA: In controlled lab settings, the MV-LSTM achieved 85.18% accuracy, a significant jump from the standard LSTM's 78.21%.
- AFEW (Acting Facial Expressions in the Wild): In this challenging real-world dataset, the model reached 51.44% accuracy, outperforming standard RNN/LSTM baselines by over 11%.
- KAIST Face MPMI: A new dataset introduced by the authors featuring 104 subjects across 13 poses and 4 illumination conditions. The MV-LSTM demonstrated remarkable stability, maintaining high recognition rates even at extreme 60-degree head turns, where traditional models showed steep performance degradation.
- Cross-Race Generalization: In tests where the model was trained on one race and tested on another, the MV-LSTM improved accuracy by roughly 5-6% over the baseline, proving its ability to ignore identity-specific features.
Discussion and Conclusion
The Mode Variational LSTM offers a robust solution for spatio-temporal feature encoding by explicitly addressing the problem of environmental and identity-based bias. While the method currently relies on a "neutral" reference frame, its ability to disentangle features without explicit labels is a significant advancement. This architectural approach holds promise for other video analysis tasks where background noise and subject-specific variations hinder model generalization, such as action recognition or medical imaging analysis.
Summary updated