Facial Expression Recognition Using Visual Saliency and Deep Learning

Viraj Mavani, Shanmuganathan Raman, Krishna P. MiyapuramWorkshops @ ICCV 2017 · DOI · arXiv

Introduction

Automated Facial Expression Recognition (FER) is a critical component of human-computer interaction, yet it faces significant hurdles due to the high variability in human emotional expression. While traditional approaches often rely on the Facial Action Coding System (FACS) to identify specific muscle movements, these methods are computationally intensive. Modern deep learning techniques, particularly Convolutional Neural Networks (CNNs), have streamlined this process by automatically learning features. However, standard CNNs typically process images uniformly, whereas human perception naturally prioritizes specific regions like the eyes and mouth. This research investigates whether incorporating visual saliency—intensity maps representing areas of human attentional focus—can enhance the biological plausibility and generalization of FER models.

Methodology

The researchers developed a multi-stage pipeline to test the impact of visual attention on classification accuracy.

Preprocessing and Saliency Integration

The initial stage involved localizing and cropping faces using the Viola-Jones detector, followed by grayscale conversion and resizing to 256x256 pixels. To simulate human attention, the team utilized ML-Net, a deep multi-layer network, to generate visual saliency maps for each face. A novel "product image" was then created by multiplying the original cropped face by its corresponding saliency map. This process effectively masks the image, highlighting high-attention features while suppressing background noise and less relevant facial areas.

Model Architecture and Training

The study employed a fine-tuned AlexNet architecture, pre-trained on ImageNet. The final softmax layer was adapted to classify seven basic emotions: Angry, Fearful, Disgusted, Surprised, Happy, Sad, and Neutral. Training was conducted using Stochastic Gradient Descent (SGD) over 100 epochs with a linear learning rate decay.

Datasets

Two primary databases were used for evaluation:

  • Compound Facial Expressions of Emotion (CFEE): 1,610 images across 230 subjects.
  • Radboud Faces Database (RaFD): 1,407 images across 67 subjects, including multiple gaze directions.
Example of a 'Happy' expression from the dataset.

Results and Discussion

The performance of the models was evaluated through both intra-dataset and cross-dataset testing to measure generalization.

Classification Performance

Intra-dataset testing yielded high accuracy, with the model achieving 95.71% on RaFD and 74.79% on CFEE. In cross-dataset scenarios—training on CFEE and testing on the unseen RaFD environment—the model achieved 77.19% using raw images. When using the saliency-weighted "product images," the cross-dataset accuracy dropped to 65.39%. While lower than the raw image baseline, this result remains significantly above the chance level of 14.2%.

A key finding was the shift in how the model confused different emotions when saliency maps were applied. The saliency-based model frequently confused Angry, Neutral, and Sad expressions with Disgusted. This suggests that when the network is forced to focus on "human-like" attentional regions, these specific emotions share overlapping features in the attentional space that are not as prominent in raw pixel data.

Example of a 'Disgusted' expression, a category that became a focal point for model confusion in saliency-based tests.

Conclusion and Future Directions

While the direct application of saliency maps as a multiplicative mask led to a decrease in raw accuracy, it provided valuable insights into the relationship between human attention and machine learning feature extraction. The performance drop suggests that CNNs may still rely on "contextual" information from lower-saliency regions that the mask suppresses.

Future research will focus on integrating saliency as an additional input channel rather than a mask, allowing the model to retain global context while prioritizing key features. Additionally, comparing model confusion patterns directly against human error metrics will help determine if saliency-based models more accurately reflect biological emotion recognition processes.

Summary updated

BibTeX
@inproceedings{mavani2017facial,
  author = {Viraj Mavani and Shanmuganathan Raman and Krishna P. Miyapuram},
  title = {{Facial Expression Recognition Using Visual Saliency and Deep Learning}},
  booktitle = {Workshops @ ICCV},
  year = {2017},
  doi = {10.1109/iccvw.2017.327},
}
Figure