Learning Spatio-Temporal Features With Partial Expression Sequences for On-the-Fly Prediction

Wissam J. Baddar, Yong Man Ro

Abstract

Spatio-temporal feature encoding is essential for capturing facial expression dynamics in video sequences. Most existing methods assume a fully segmented sequence is available at test time, causing delays in prediction. In interactive settings like affective agents, such delays are intolerable. This paper proposes a novel spatio-temporal feature learning method that enables "on-the-fly" prediction using partial sequences. By utilizing estimated expression intensity to generate dense labels, the model is trained with a novel objective function. Results show that the learned features robustly predict expressions from incomplete sequences, achieving state-of-the-art recognition rates on MMI and Oulu-CASIA datasets.

Introduction

Facial expression recognition (FER) is a cornerstone of affective computing. However, a significant bottleneck exists: the "waiting game." Most current FER systems are designed to analyze a pre-segmented video clip from start to finish, typically waiting for the "apex" frame—the moment of maximum emotional intensity—before making a prediction.

In real-world applications like social robotics or interactive AI agents, this delay is a dealbreaker. If a robot must wait seconds for a human to finish an expression before reacting, the interaction feels unnatural. The challenge lies in the fact that early frames are often subtle or nearly neutral, making "on-the-fly" prediction difficult without expensive, frame-by-frame manual labeling.

Methodology

The proposed framework combines a Convolutional Neural Network (CNN) for spatial feature extraction with a Long Short-Term Memory (LSTM) network for temporal dynamics. The core innovation is a training regime that uses three distinct objective terms to regulate spatio-temporal feature learning.

The proposed framework for spatio-temporal feature learning with expression intensity constraints.

The three objective terms are:

  1. Classification Error ($E_1$): Ensures the final sequence prediction matches the overall label.
  2. Intensity Prediction Error ($E_2$): Frame-level labels are estimated by calculating the cosine similarity between the current frame and the apex frame. The model learns to predict this intensity at every step, providing "dense" supervision.
  3. Perceived Expression Clustering Error ($E_3$): Frames are grouped into "perceived neutral" and "perceived expressive" categories using K-means clustering. This helps the model distinguish the exact moment an expression becomes recognizable, reducing intra-class variation.

Experimental Setup

The model was evaluated on two benchmarks:

  • MMI Dataset: 205 sequences across six basic expressions, supplemented with 38 simulated neutral sequences.
  • Oulu-CASIA Dataset: 480 sequences captured under visible light.
  • Data Augmentation: To prevent overfitting, extensive augmentation (flipping, rotation, translation, scaling) was used, resulting in 108 variations per sequence.

Results and Discussion

The proposed method outperformed state-of-the-art (SOTA) models on both datasets. On MMI, it achieved 78.96% accuracy (vs. 76.64% for standard CNN+LSTM). On Oulu-CASIA, it reached 82.86% accuracy (vs. 78.21% for CNN+LSTM).

Illustration of estimated expression intensity over a sequence, showing the transition from neutral to an expression class.

The most significant advantage is early prediction. As shown in Figure 6, the model tracks expression intensity over time. At low intensity thresholds (e.g., 0.3 to 0.5), the proposed model maintains significantly higher accuracy than conventional LSTMs, proving its effectiveness for real-time, on-the-fly use. By explicitly clustering frames into neutral vs. expressive categories, the model learns to ignore initial noise and commit to a prediction much earlier in the sequence.

Conclusion

This research demonstrates that "on-the-fly" facial expression recognition is achievable by leveraging synthetic dense labels and intensity-based constraints. While the current training phase relies on identifying the "apex" frame, the resulting model significantly reduces prediction latency. Future work will focus on integrating this framework into live interactive environments to evaluate its impact on human-robot rapport.

Summary updated

Abstract

Spatio-temporal feature encoding is essential for encoding facial expression dynamics in video sequences. At test time, most spatio-temporal encoding methods assume that a temporally segmented sequence is fed to a learned model, which could require the prediction to wait until the full sequence is available to an auxiliary task that performs the temporal segmentation. This causes a delay in predicting the expression. In an interactive setting, such as affective interactive agents, such delay in the prediction could not be tolerated. Therefore, training a model that can accurately predict the facial expression "on-the-fly" (as they are fed to the system) is essential. In this paper, we propose a new spatio-temporal feature learning method, which would allow prediction with partial sequences. As such, the prediction could be performed on-the-fly. The proposed method utilizes an estimated expression intensity to generate dense labels, which are used to regulate the prediction model training with a novel objective function. As results, the learned spatio-temporal features can robustly predict the expression with partial (incomplete) expression sequences, on-the-fly. Experimental results showed that the proposed method achieved higher recognition rates compared to the state-of-the-art methods on both datasets. More importantly, the results verified that the proposed method improved the prediction frames with partial expression sequence inputs.
BibTeX
@inproceedings{baddar2018learning,
  author = {Wissam J. Baddar and Yong Man Ro},
  title = {{Learning Spatio-Temporal Features With Partial Expression Sequences for On-the-Fly Prediction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=Sy-J6AguZS},
}
Figure