Abstract
Spatio-temporal feature encoding is essential for capturing facial expression dynamics in video sequences. Most existing methods assume a fully segmented sequence is available at test time, causing delays in prediction. In interactive settings like affective agents, such delays are intolerable. This paper proposes a novel spatio-temporal feature learning method that enables "on-the-fly" prediction using partial sequences. By utilizing estimated expression intensity to generate dense labels, the model is trained with a novel objective function. Results show that the learned features robustly predict expressions from incomplete sequences, achieving state-of-the-art recognition rates on MMI and Oulu-CASIA datasets.
Introduction
Facial expression recognition (FER) is a cornerstone of affective computing. However, a significant bottleneck exists: the "waiting game." Most current FER systems are designed to analyze a pre-segmented video clip from start to finish, typically waiting for the "apex" frame—the moment of maximum emotional intensity—before making a prediction.
In real-world applications like social robotics or interactive AI agents, this delay is a dealbreaker. If a robot must wait seconds for a human to finish an expression before reacting, the interaction feels unnatural. The challenge lies in the fact that early frames are often subtle or nearly neutral, making "on-the-fly" prediction difficult without expensive, frame-by-frame manual labeling.
Methodology
The proposed framework combines a Convolutional Neural Network (CNN) for spatial feature extraction with a Long Short-Term Memory (LSTM) network for temporal dynamics. The core innovation is a training regime that uses three distinct objective terms to regulate spatio-temporal feature learning.
The three objective terms are:
- Classification Error ($E_1$): Ensures the final sequence prediction matches the overall label.
- Intensity Prediction Error ($E_2$): Frame-level labels are estimated by calculating the cosine similarity between the current frame and the apex frame. The model learns to predict this intensity at every step, providing "dense" supervision.
- Perceived Expression Clustering Error ($E_3$): Frames are grouped into "perceived neutral" and "perceived expressive" categories using K-means clustering. This helps the model distinguish the exact moment an expression becomes recognizable, reducing intra-class variation.
Experimental Setup
The model was evaluated on two benchmarks:
- MMI Dataset: 205 sequences across six basic expressions, supplemented with 38 simulated neutral sequences.
- Oulu-CASIA Dataset: 480 sequences captured under visible light.
- Data Augmentation: To prevent overfitting, extensive augmentation (flipping, rotation, translation, scaling) was used, resulting in 108 variations per sequence.
Results and Discussion
The proposed method outperformed state-of-the-art (SOTA) models on both datasets. On MMI, it achieved 78.96% accuracy (vs. 76.64% for standard CNN+LSTM). On Oulu-CASIA, it reached 82.86% accuracy (vs. 78.21% for CNN+LSTM).
The most significant advantage is early prediction. As shown in Figure 6, the model tracks expression intensity over time. At low intensity thresholds (e.g., 0.3 to 0.5), the proposed model maintains significantly higher accuracy than conventional LSTMs, proving its effectiveness for real-time, on-the-fly use. By explicitly clustering frames into neutral vs. expressive categories, the model learns to ignore initial noise and commit to a prediction much earlier in the sequence.
Conclusion
This research demonstrates that "on-the-fly" facial expression recognition is achievable by leveraging synthetic dense labels and intensity-based constraints. While the current training phase relies on identifying the "apex" frame, the resulting model significantly reduces prediction latency. Future work will focus on integrating this framework into live interactive environments to evaluate its impact on human-robot rapport.
Summary updated