Abstract
Spatio-temporal feature encoding is essential for encoding the dynamics in video sequences. Recurrent neural networks, particularly long short-term memory (LSTM) units, have been popular as an efficient tool for encoding spatio-temporal features in sequences. In this work, we investigate the effect of mode variations on the encoded spatio-temporal features using LSTMs. We show that the LSTM retains information related to the mode variation in the sequence, which is irrelevant to the task at hand (e.g. classification facial expressions). Actually, the LSTM forget mechanism is not robust enough to mode variations and preserves information that could negatively affect the encoded spatio-temporal features. We propose the mode variational LSTM to encode spatio-temporal features robust to unseen modes of variation. The mode variational LSTM modifies the original LSTM structure by adding an additional cell state that focuses on encoding the mode variation in the input sequence. To efficiently regulate what features should be stored in the additional cell state, additional gating functionality is also introduced. The effectiveness of the proposed mode variational LSTM is verified using the facial expression recognition task. Comparative experiments on publicly available datasets verified that the proposed mode variational LSTM outperforms existing methods. Moreover, a new dynamic facial expression dataset with different modes of variation, including various modes like pose and illumination variations, was collected to comprehensively evaluate the proposed mode variational LSTM. Experimental results verified that the proposed mode variational LSTM encodes spatio-temporal features robust to unseen modes of variation.
Introduction
Recognizing facial expressions in video sequences is a complex task because facial appearance is a "multifactor confluence." An expression is often confounded by "modes of variation" such as individual identity, head pose, and illumination conditions. While Long Short-Term Memory (LSTM) networks are widely used to capture temporal dynamics, they often fail to isolate the expression-related signal from these environmental biases.
The authors identify a significant flaw in standard LSTMs: the "forget gate" mechanism is often insufficient to discard irrelevant static information. Consequently, the encoded features become "leaky," retaining identity or lighting information that hinders generalization to unseen subjects or environments.
The Problem of LSTM Leakage
To demonstrate this leakage, the researchers conducted an experiment feeding a static sequence (replicating the first frame of a video) into a pre-trained LSTM. Theoretically, if the model only captured dynamics, the output should be neutral or constant. However, as shown in the figure below, the internal states converge to different values depending on the subject's identity and the lighting conditions.
This visual evidence confirms that standard LSTMs bake environmental bias into their memory, making them less robust to variations not seen during training.
Proposed Method: Mode Variational LSTM (MV-LSTM)
To address this, the Mode Variational LSTM (MV-LSTM) was developed. This architecture explicitly disentangles the dynamic expression from the static mode variations through several key innovations:
- Dual-Path Processing: The model processes the original video sequence (Dynamics Path) alongside a "static sequence" generated by replicating the first (neutral) frame (Bias Path).
- Additional Cell State: A dedicated cell state is introduced to specifically encode the mode variations found in the static sequence.
- Shared Output Gating: A novel gating mechanism regulates the output by considering both the dynamic and bias cell states. This allows the network to suppress features associated with the bias path from the final representation.
- Cross-Cell Peepholes: These allow the gates of the dynamic path to "peep" into the bias cell, facilitating more precise filtering of irrelevant noise.
Experimental Evaluation
The MV-LSTM was evaluated across three datasets:
- Oulu-CASIA: Focused on basic expressions under varying lighting.
- AFEW (Acting Facial Expressions in the Wild): Real-world movie clips with uncontrolled environments.
- KAIST Face MPMI: A new dataset introduced by the authors featuring 104 subjects across 13 poses and 4 illumination conditions to test robustness to extreme variations.
Key Results
The MV-LSTM consistently outperformed state-of-the-art methods:
- Accuracy Boost: On Oulu-CASIA, accuracy improved from 78.21% (standard LSTM) to 85.18%. On the challenging AFEW dataset, it reached 51.44%, a significant improvement over standard RNN baselines (~39.6%).
- Generalization: The model showed remarkable robustness in cross-racial tests and pose-invariance tests. When trained on front-facing faces and tested on side profiles, the MV-LSTM remained stable, whereas standard LSTM performance degraded sharply.
Conclusion
The MV-LSTM provides a robust framework for facial expression recognition by architecturally disentangling temporal dynamics from static environmental noise. By using the first frame as a neutral anchor, the model effectively learns to ignore identity and lighting biases. Future work may involve more sophisticated methods for selecting the "static" reference to handle videos that do not begin with a neutral expression, and extending this architecture to other domains like gesture or gait analysis.
Summary updated