A Brief Review of Facial Emotion Recognition Based on Visual Information

ByoungChul Ko

TL;DR

This paper provides a comprehensive review of facial emotion recognition (FER) techniques, bridging the gap between traditional handcrafted feature methods and modern deep learning approaches. It highlights the evolution of FER, emphasizing the shift towards end-to-end learning with Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks, and discusses the critical role of publicly available datasets and evaluation metrics in advancing the field.

What Problem Does This Paper Address?

Facial expressions are a primary channel for human communication, conveying emotional states. The automatic recognition of these emotions (FER) is a significant area of research in computer vision and artificial intelligence, with applications ranging from human-computer interaction and virtual reality to advanced driver assistance systems. While FER has been studied for decades, the field faces challenges in accurately capturing the nuances of human emotion, especially spontaneous expressions, and in developing robust systems that generalize across different individuals and conditions. This review aims to provide a structured overview of the progress in FER, from early methods to state-of-the-art deep learning techniques, serving as a guide for both newcomers and experienced researchers.

Methods

The paper categorizes FER approaches into two main groups:

  • Conventional FER Approaches: These methods typically involve three steps: face and facial component detection, feature extraction (geometric, appearance, or hybrid), and expression classification using pre-trained classifiers like Support Vector Machines (SVMs), AdaBoost, or random forests. These methods rely on handcrafted features designed by experts.
  • Deep-Learning-Based FER Approaches: These methods leverage deep neural networks, particularly Convolutional Neural Networks (CNNs), to learn features directly from input images in an "end-to-end" fashion. This significantly reduces reliance on manual feature engineering. The review also highlights hybrid approaches that combine CNNs for spatial feature extraction from individual frames with Long Short-Term Memory (LSTM) networks to model temporal dynamics across video sequences. This CNN-LSTM architecture is particularly effective for capturing the sequential nature of facial expressions.

Datasets

The paper discusses several publicly available datasets crucial for FER research:

  • Extended Cohn-Kanade Dataset (CK+): Contains video sequences of posed and spontaneous emotions, with metadata for action units and prototypic emotions.
  • Compound Emotion (CE) Dataset: Features images of basic and compound emotions.
  • Denver Intensity of Spontaneous Facial Action Database (DISFA): Provides stereo video frames with manually scored AU intensities.
  • Binghamton University 3D Facial Expression (BU-3DFE): Offers annotated 3D facial expressions.
  • Japanese Female Facial Expressions (JAFFE): A dataset of posed facial emotions.
  • Extended Yale Face B (B+): A large dataset of facial images under various conditions.
  • MMI: Includes video sequences and still images annotated for AU presence.
  • Binghamton-Pittsburgh 3D Dynamic Spontaneous (BP4D-Spontanous): A 3D video database of spontaneous expressions.
  • Karolinska Directed Emotional Face (KDEF): Contains images of human emotional facial expressions from multiple angles.

The paper also mentions other sensors and their associated databases, such as NIR cameras and Kinect sensors, acknowledging the limitations of visible light imaging.

Key Insights and Ideas

  • Shift to End-to-End Learning: Deep learning, especially CNNs, has revolutionized FER by enabling models to learn features directly from raw image data, bypassing the need for manual feature engineering.
  • The Power of Spatio-Temporal Modeling: Recognizing that facial expressions evolve over time, hybrid CNN-LSTM models have emerged as a powerful approach, combining spatial feature extraction with temporal sequence modeling for improved accuracy.
  • Importance of Datasets and Benchmarks: The availability of diverse and well-annotated datasets is critical for training robust FER models and for enabling quantitative comparisons of different approaches.
  • 3D Data for Enhanced Analysis: While 2D analysis has limitations with pose and illumination variations, 3D facial expression analysis offers a more detailed examination of structural changes, leading to potentially higher performance.

Key Results

The paper presents a comparative analysis of conventional and deep-learning-based FER approaches using the MMI dataset. Deep-learning-based methods achieved an average accuracy of 72.65%, significantly outperforming conventional approaches with an average of 63.20%. Among deep learning methods, a hybrid CNN-LSTM approach achieved the highest accuracy of 78.61% by effectively learning spatio-temporal features.

Limitations and Future Work

The authors acknowledge several limitations of current deep-learning-based FER:

  • Data and Computational Demands: Training deep models requires large-scale, manually labeled datasets and significant computational resources (GPUs, CPUs), making them less suitable for resource-constrained mobile platforms.
  • Hyperparameter Tuning: Selecting appropriate hyperparameters for deep networks is complex and requires considerable expertise.
  • Lack of Theoretical Understanding: The underlying mechanisms of CNNs are not fully understood, making it difficult to predict their behavior or optimize them systematically.
  • Micro-Expression Recognition: Spontaneous and subtle micro-expressions remain a challenging area for FER.

Future work suggested includes developing more efficient deep learning algorithms to reduce computational burden, exploring new directions and applications for FER, and potentially combining various sensors with deep learning for enhanced performance, especially for spontaneous micro-expressions.

Summary updated

BibTeX
@inproceedings{ko2018brief,
  author = {ByoungChul Ko},
  title = {{A Brief Review of Facial Emotion Recognition Based on Visual Information}},
  booktitle = {Sensors},
  year = {2018},
  url = {https://openreview.net/forum?id=OJ1LzNo63R},
}
Figure