Learning Spatio-temporal Features with Partial Expression Sequences for on-the-Fly Prediction

Wissam J. Baddar, Yong Man Ro

Abstract

Spatio-temporal feature encoding is essential for encoding facial expression dynamics in video sequences. At test time, most spatio-temporal encoding methods assume that a temporally segmented sequence is fed to a learned model, which could require the prediction to wait until the full sequence is available to an auxiliary task that performs the temporal segmentation. This causes a delay in predicting the expression. In an interactive setting, such as affective interactive agents, such delay in the prediction could not be tolerated. Therefore, training a model that can accurately predict the facial expression "on-the-fly" (as they are fed to the system) is essential. In this paper, we propose a new spatio-temporal feature learning method, which would allow prediction with partial sequences. As such, the prediction could be performed on-the-fly. The proposed method utilizes an estimated expression intensity to generate dense labels, which are used to regulate the prediction model training with a novel objective function. As results, the learned spatio-temporal features can robustly predict the expression with partial (incomplete) expression sequences, on-the-fly. Experimental results showed that the proposed method achieved higher recognition rates compared to the state-of-the-art methods on both datasets. More importantly, the results verified that the proposed method improved the prediction frames with partial expression sequence inputs.

Introduction

Most state-of-the-art facial expression recognition (FER) systems are designed to process "temporally segmented" video clips. This requirement means they must observe the entire sequence—from a neutral state to the peak (apex) of the emotion—before making a classification. In real-world applications like affective computing or human-robot interaction, this creates a significant "prediction delay." If a social robot must wait for a human to finish a smile before responding, the interaction feels laggy and unnatural.

The challenge lies in the fact that early frames in an expression sequence are often subtle or nearly neutral, making them difficult to classify without the context of the full video. This research addresses this gap by proposing a model capable of making reliable predictions as soon as an expression begins to manifest, enabling "on-the-fly" recognition.

Proposed Methodology

The proposed framework combines spatial feature extraction with a regularized Long Short-Term Memory (LSTM) network designed to handle partial sequences.

Overview of the proposed framework for on-the-fly facial expression intensity estimation and spatio-temporal feature learning.

The approach follows three main stages:

  1. Spatial Feature Extraction: A pre-trained CNN extracts spatial features ($f_t$) from individual video frames.
  2. Expression Intensity Estimation: To provide frame-level supervision without manual labeling, the authors automatically estimate expression intensity. This is done by calculating the cosine similarity between the spatial features of the current frame and the "apex" (peak) frame of the sequence.
  3. Regularized LSTM Training: The LSTM is trained using a novel objective function with three components:
  4. $E_1$ (Classification Error): Standard cross-entropy loss at the end of the sequence.
  5. $E_2$ (Intensity Prediction Error): Forces the model to predict the estimated intensity at every frame, teaching it the dynamics of expression onset.
  6. $E_3$ (Perceived Clustering Error): Uses K-means clustering to group features into "perceived neutral" and "perceived expressive" clusters, helping distinguish subtle early expressions from neutral states.

Experimental Evaluation

The method was evaluated on two benchmarks: the MMI Dataset (205 sequences) and the Oulu-CASIA Dataset (480 sequences). Extensive data augmentation was used to prevent overfitting, resulting in 108 variations per sequence.

Key Results

The proposed method achieved state-of-the-art recognition rates: 78.96% on MMI and 82.86% on Oulu-CASIA, outperforming traditional 3D CNNs and standard CNN+LSTM architectures.

The most critical finding is the model's performance on partial sequences. As shown in the figure below, the inclusion of intensity prediction ($E_2$) and clustering ($E_3$) significantly boosts recognition rates when the expression intensity is low (e.g., between 0.3 and 0.7).

Partial expression prediction recognition rate across different intensity thresholds, demonstrating the impact of the $E_2$ and $E_3$ constraints.

By penalizing the model for failing to recognize intensity and expression clusters early on, the LSTM becomes more sensitive to subtle, non-apex frames. This effectively eliminates the need for auxiliary tasks like "expression transition detection" and allows for valid predictions much earlier in the video sequence.

Conclusion

This paper introduces a novel regularization strategy for LSTMs that leverages estimated expression intensity as a proxy for temporal progress. By training the model to recognize the "developing" stages of an emotion, the system achieves robust on-the-fly prediction. While current training requires knowledge of apex frames, future work may extend this to more complex, unsegmented "in-the-wild" video streams where expressions fluctuate naturally.

Summary updated

Abstract

Spatio-temporal feature encoding is essential for encoding facial expression dynamics in video sequences. At test time, most spatio-temporal encoding methods assume that a temporally segmented sequence is fed to a learned model, which could require the prediction to wait until the full sequence is available to an auxiliary task that performs the temporal segmentation. This causes a delay in predicting the expression. In an interactive setting, such as affective interactive agents, such delay in the prediction could not be tolerated. Therefore, training a model that can accurately predict the facial expression "on-the-fly" (as they are fed to the system) is essential. In this paper, we propose a new spatio-temporal feature learning method, which would allow prediction with partial sequences. As such, the prediction could be performed on-the-fly. The proposed method utilizes an estimated expression intensity to generate dense labels, which are used to regulate the prediction model training with a novel objective function. As results, the learned spatio-temporal features can robustly predict the expression with partial (incomplete) expression sequences, on-the-fly. Experimental results showed that the proposed method achieved higher recognition rates compared to the state-of-the-art methods on both datasets. More importantly, the results verified that the proposed method improved the prediction frames with partial expression sequence inputs.
BibTeX
@inproceedings{baddar2018learning,
  author = {Wissam J. Baddar and Yong Man Ro},
  title = {{Learning Spatio-temporal Features with Partial Expression Sequences for on-the-Fly Prediction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=Sy-J6AguZS},
}
Figure