Pairwise Conditional Random Forests for Facial Expression Recognition

Arnaud Dapogny, Kévin Bailly

TL;DR

This paper introduces Pairwise Conditional Random Forests (PCRF), a novel framework for facial expression recognition that leverages spatio-temporal information by analyzing pairs of images. PCRF significantly improves recognition accuracy by conditioning tree predictions on previous expression labels, outperforming standard Random Forests and existing state-of-the-art methods on several benchmarks.

What Problem Does This Paper Address?

Recognizing facial expressions from videos is challenging due to the dynamic nature of expressions, which involve complex spatio-temporal patterns. Existing methods often struggle to generalize to unseen facial variations and expression dynamics. While some approaches exploit temporal information, they can be computationally expensive or lack flexibility. This paper aims to bridge this gap by developing a more robust and efficient method for capturing and utilizing the temporal dynamics of facial expressions.

Methods

The core innovation is the Pairwise Conditional Random Forest (PCRF). Unlike traditional Random Forests that operate on individual frames, PCRF learns from pairs of images.

  • Heterogeneous Feature Templates: PCRF utilizes a rich set of features, including geometric (e.g., distances and angles between facial landmarks) and appearance-based (e.g., Histogram of Oriented Gradients - HoG) features. These are extracted from individual images.
  • Dynamic Features: Crucially, PCRF introduces derivative features that capture the change between two images in a pair. These dynamic features are essential for modeling temporal variations.
  • Conditional Forests: For training, PCRF creates collections of trees, where each collection is dedicated to transitions starting from a specific expression label (e.g., neutral to happy). This conditioning on the first frame's expression label helps reduce variability in the expression transition.
  • Temporal Averaging: During testing, for a given frame, PCRF forms pairs with previous frames. The predictions from the corresponding conditional trees are then averaged over time to produce a robust final prediction for the current frame. This temporal averaging allows the model to incorporate past observations.

Datasets

The authors evaluated their method on the following datasets:

  • Extended Cohn-Kanade (CK+) database: Contains acted facial expressions with no head pose variation.
  • BU-4DFE database: Features acted facial expressions with moderate head pose variations and lower intensity compared to CK+.
  • FG-NET FEED database: Includes spontaneous facial expressions with low intensity and moderate head pose variations.
  • BP4D database: Contains spontaneous facial expressions from subjects performing specific tasks.

They also performed cross-database evaluations, notably training on BU-4DFE and testing on FG-NET FEED and BP4D to assess generalization capabilities.

Key Insights and Ideas

  • Pairwise Analysis is Key: The paper demonstrates that analyzing pairs of images, rather than just individual frames, is crucial for capturing the dynamics of facial expressions effectively.
  • Conditioning on Previous State Matters: Conditioning the random forest trees on the expression label of the first image in a pair significantly reduces the variability of the learned transitions, leading to more accurate predictions. This is a key insight for modeling dynamic processes.
  • Dynamic Features Enhance Robustness: The introduction of derivative features that capture changes between image pairs provides a powerful way to model temporal evolution, improving robustness and decorrelation between trees.
  • Real-time Potential: The proposed method, even without extensive optimization, shows promising real-time performance, making it suitable for practical applications.

Key Results

  • Significant Accuracy Improvements: PCRF consistently outperformed static Random Forest models on all tested databases.
  • State-of-the-Art Performance: PCRF achieved state-of-the-art results on the BU-4DFE database for automatic FER using only 2D information.
  • Cross-Database Generalization: The conditional model significantly improved recognition rates on spontaneous FER tasks and even surpassed previous results on the BP4D database.
  • Efficiency: The method is shown to be computationally efficient, with profiling indicating real-time processing capabilities on standard hardware, especially when parallelized. For instance, evaluating a frame with 50 trees took approximately 4.8 ms.

Limitations and Future Work

  • Limited Temporal Window: While PCRF averages over a temporal window, the authors acknowledge that the "full model" might not fully capture very long-term dependencies.
  • Spontaneous Expression Challenges: The paper notes that spontaneous expressions can be more challenging due to variations in pace and offset. While PCRF shows improvement, further work might be needed for highly complex spontaneous scenarios.
  • Future Directions: The authors propose extending PCRF to handle pose and occlusion in "in the wild" scenarios. They also suggest exploring its application to other video analysis tasks like Facial Action Unit intensity prediction and gesture recognition.

Summary updated

Abstract

Facial expression can be seen as the dynamic variation of one’s appearance over time. Successful recognition thus involves finding representations of high-dimensional spatio-temporal patterns that can be generalized to unseen facial morphologies and variations of the expression dynamics. In this paper, we propose to learn Random Forests from hetero-geneous derivative features (e.g. facial fiducial point move-ments or texture variations) upon pairs of images. Those forests are conditioned on the expression label of the first frame to reduce the variability of the ongoing expression transitions. When testing on a specific frame of a video, pairs are created between this current frame and the pre-vious ones. Predictions for each previous frame are used to draw trees from Pairwise Conditional Random Forests (PCRF) whose pairwise outputs are averaged over time to produce robust estimates. As such, PCRF appears as a nat-ural extension of Random Forests to learn spatio-temporal patterns, that leads to significant improvements over stan-dard Random Forests as well as state-of-the-art approaches on several facial expression benchmarks. 1.
BibTeX
@inproceedings{dapogny2015pairwise,
  author = {Arnaud Dapogny and Kévin Bailly},
  title = {{Pairwise Conditional Random Forests for Facial Expression Recognition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  url = {https://openreview.net/forum?id=rkE-PZMdbr},
}
Figure