TL;DR
This paper introduces a novel Hierarchical Facial Expression Recognition (HL-FER) system designed to overcome challenges in accurately identifying human emotions from images. By employing a multi-stage approach that includes advanced pre-processing, robust feature extraction, and a hierarchical classification strategy, the system achieves a remarkable 98.7% weighted average recognition accuracy across multiple standard datasets.
What Problem Does This Paper Address?
Recognizing human facial expressions (FER) is a complex task due to several factors: varying lighting conditions in images, the need for accurate automatic face detection before feature extraction, and the inherent similarity between different expressions that makes distinguishing them difficult. Existing FER systems often struggle with these challenges, leading to reduced accuracy, especially when tested on diverse datasets. This paper aims to address these limitations by proposing a more robust and accurate hierarchical recognition scheme.
Methods
The proposed HL-FER system comprises three main modules:
- Pre-processing: This stage aims to enhance image quality and prepare them for subsequent processing. It includes:
- Global Histogram Equalization (GHE): Used to diminish illumination effects and artifacts, a novel application for FER according to the authors.
- Face Detection: A new scheme that combines gray-level and skin-tone-based methods with a Similarity Angle Measurement (SAM) approach to accurately locate and extract faces.
- Feature Extraction: This module extracts relevant information from the detected faces. The authors employ:
- Principal Component Analysis (PCA): To extract global features.
- Independent Component Analysis (ICA): To extract local features.
- Hierarchical Recognition: This is the core of the proposed system, involving a two-level classification strategy:
- Level 1 (Expression Category Recognition): Expressions are first grouped into three categories (Lips-Based, Lips-Eyes-Based, Lips-Eyes-Forehead-Based) based on the facial regions contributing most to the expression. Linear Discriminant Analysis (LDA) is applied to classify these categories, achieving 100% accuracy on the Cohn-Kanade dataset.
- Level 2 (Expression Recognition within Category): For each recognized category, a separate classifier (trained using LDA and Hidden Markov Models - HMMs) is used to identify the specific expression. This hierarchical approach helps to manage the high similarity between expressions by first separating them into broader groups.
Datasets
The HL-FER system was evaluated on three publicly available standard datasets:
- Cohn-Kanade (CK): Contains 5,400 expression frames from 100 subjects.
- Japanese Female Facial Expressions (JAFFE): Contains 205 expression frames from 10 Japanese female subjects.
- AT&T Dataset: Contains 240 facial frames from 40 subjects.
The paper also mentions experiments on the Yale database of facial expressions, with results provided in the appendix.
Key Insights and Ideas
- Hierarchical Classification is Crucial for Similar Expressions: The paper strongly supports the idea that breaking down the complex FER problem into hierarchical levels (categories first, then specific expressions) is key to overcoming the high similarity between different expressions. This approach effectively reduces the search space for each classifier.
- GHE as a Novel Pre-processing Step for FER: Applying Global Histogram Equalization for illumination correction in FER is presented as a novel contribution, showing effectiveness in improving image quality.
- Combined Face Detection for Robustness: The integration of gray-level and skin-tone-based methods with SAM for face detection demonstrates a robust approach that can handle variations in images.
- PCA and ICA Complement Each Other: The use of both PCA (global features) and ICA (local features) provides a comprehensive feature representation, contributing to better recognition accuracy.
Key Results
- High Overall Accuracy: The HL-FER system achieved a weighted average recognition accuracy of 98.7% across the three datasets.
- Dataset-Specific Performance: Individual dataset accuracies were also very high: 98.87% on Cohn-Kanade, 98.80% on JAFFE, and 98.50% on AT&T.
- Robustness Across Datasets: The system demonstrated consistent high performance across different datasets, indicating robustness despite variations in facial features (e.g., subjects wearing glasses in AT&T).
- Hierarchical Recognition's Impact: Experiments showed that removing the hierarchical recognition module significantly decreased accuracy, highlighting its importance.
- Face Detection Effectiveness: The proposed face detection algorithm achieved high detection rates (99% for CK and JAFFE, 98% for AT&T).
Limitations and Future Work
- Computational Cost: The HL-FER, while achieving high accuracy, has a higher computational cost compared to some existing methods due to its two-level classification and HMM usage.
- Real-World Scenarios: The current experiments were performed offline on standard datasets. The authors acknowledge that real-time implementation faces challenges from factors like clutter and varying face angles (non-frontal views), which could degrade performance.
- Smartphone Deployment: The complexity of the current HL-FER might be an issue for deployment on smartphones.
- Future Directions:
- Investigate lightweight classifiers (e.g., k-NN) for the first level to reduce complexity, while addressing their limitations.
- Develop robust segmentation techniques for real-time handling of clutter.
- Extend the system to handle non-frontal views and other real-life environmental factors.
- Improve efficiency while maintaining high accuracy for smartphone applications.
Summary updated