Multiple Local Curvature Gabor Binary Patterns for Facial Action Recognition

Anil Yüce, Nuri Murat Arar, Jean-Philippe ThiranHBU 2013 · DOI

TL;DR

This paper introduces a novel feature extraction method called Multiple Local Curvature Gabor Binary Patterns (LCGBP) for facial action unit (AU) detection. By combining curvature information with Gabor filters and Local Binary Patterns, their system achieves state-of-the-art accuracy on the Extended Cohn-Kanade (CK+) dataset, outperforming previous methods.

What Problem Does This Paper Address?

Recognizing facial expressions is crucial for human-computer interaction and has applications in areas like medical diagnosis. Facial expressions are composed of subtle muscle movements, defined as Action Units (AUs) by the Facial Action Coding System (FACS). Accurately detecting these AUs is challenging due to variations in illumination, pose, and the inherent complexity of facial movements. Existing methods, while improving, still struggle to capture the fine-grained details necessary for highly accurate AU detection, especially for subtle expressions. This paper aims to significantly boost AU detection accuracy by developing a more powerful and discriminative feature representation.

Methods

The core innovation is the Multiple Local Curvature Gabor Binary Patterns (LCGBP). This approach builds upon existing techniques:

  • Gabor Wavelets: These are standard for image analysis, known for their spatial locality and orientation selectivity.
  • Curvature Gabor (CG) Wavelets: The authors extend Gabor wavelets by incorporating a curvature parameter. This is motivated by the fact that facial features (like eyebrows and lips) and expression-induced wrinkles often have curved structures, which standard Gabor filters might not fully capture. They explore different curvature degrees (c) and Gaussian sizes (σ).
  • Local Binary Patterns (LBP): LBP is a robust texture descriptor that captures local image patterns.
  • LCGBP: The paper applies the LBP transform on top of the magnitude images produced by the Curvature Gabor wavelets. This means they are essentially describing the texture of the Gabor-filtered images, capturing both the curvature information from the filters and the local texture patterns.
  • Feature Extraction Strategy: For each video sequence, they extract features from both a neutral expression frame and a peak expression frame. The difference in these feature representations (specifically, the $\chi^2$ distance between their histograms) is used as the input to the classifier. This difference-based approach helps to isolate the changes due to facial expressions, reducing the impact of identity.
  • Feature Selection and Classification: Due to the high dimensionality of the extracted LCGBP features (over 200,000), a feature selection step is crucial. They employ the GentleBoost algorithm to select the most relevant features for each AU independently. The selected features are then used to train an Support Vector Machine (SVM) classifier (specifically, RBF kernels) for each AU.

Datasets

  • Extended Cohn-Kanade (CK+) database: This is the primary dataset used for training and evaluation. It contains videos of subjects performing facial expressions, with AUs manually coded. The authors use the first (neutral) and last (peak expression) frames of each sequence for their analysis.

Key Insights and Ideas

  • Curvature Matters: Incorporating curvature into Gabor filters significantly improves facial action unit detection compared to standard Gabor filters. This highlights the importance of modeling the inherent curved structures in facial textures and expression-induced deformations.
  • Multi-Scale and Multi-Curvature is Key: Using a combination of different Gaussian sizes (σ) and curvature degrees (c) for the Gabor filters leads to a more powerful and discriminative feature set. This suggests that no single filter configuration is optimal for all AUs; a diverse set is needed to capture the variety of facial movements.
  • Difference of Histograms is Effective: Using the difference in LCGBP histograms between neutral and expressive frames effectively isolates expression-related changes, making the system robust to identity variations.
  • Feature Selection is Essential: The sheer number of features generated by LCGBP necessitates a robust feature selection method like GentleBoost to avoid overfitting and computational burden, and to identify the most discriminative features for each specific AU.

Key Results

The proposed LCGBP framework achieves state-of-the-art performance on the Extended Cohn-Kanade (CK+) database.

  • Overall Accuracy: An average overall accuracy of 98.6% was achieved across 17 AUs using a leave-one-subject-out cross-validation.
  • F1 Score: An average F1 score of 96.3%.
  • Area Under ROC Curve (AUC): An average AUC of 99.7%.

These results are reported to be the highest achieved to date on this dataset for AU detection. The paper also shows that their method significantly outperforms previous works ([20], [16], [21]) and baseline LBP approaches.

Limitations and Future Work

  • Dataset Specificity: The authors acknowledge that while the CK+ database is widely used, it was recorded in relatively constrained situations (head pose, lighting). They suggest that further tests on datasets with greater variability in expressions are needed to assess the system's generalizability.
  • Future Work: The authors propose adding registration of facial texture to the system to improve robustness against unconstrained head poses. They also mention that the current system relies on identifying a "neutral" frame, and future work could explore methods to automatically identify such frames or handle expressions without a clear neutral baseline.

Summary updated

BibTeX
@inproceedings{yuce2013multiple,
  author = {Anil Yüce and Nuri Murat Arar and Jean-Philippe Thiran},
  title = {{Multiple Local Curvature Gabor Binary Patterns for Facial Action Recognition}},
  booktitle = {HBU},
  year = {2013},
  doi = {10.1007/978-3-319-02714-2\_12},
}
Figure