Authentic Emotion Mapping: Benchmarking Facial Expressions in Real News

Qixuan Zhang, Zhifeng Wang, Yang Liu, Zhenyue Qin, Kaihao Zhang, Sabrina B. Caldwell, Tom GedeonarXiv 2024 · DOI · arXiv

Introduction

Researchers have introduced a new benchmark for Facial Landmark Emotion Recognition (FLER) using a dataset of over 14,000 images extracted from real-world news broadcasts. By shifting the focus from resource-heavy RGB image processing to lightweight Graph Neural Networks (GNNs) that analyze the geometric relationships between facial points, this study provides a more efficient and privacy-preserving path for emotion AI in edge computing environments.

The Challenge of Real-World Emotion AI

Traditional facial emotion recognition (FER) typically relies on high-resolution RGB images. While accurate, this approach is resource-intensive, requiring significant computational power that is often unavailable on mobile or "edge" devices. Furthermore, processing full facial images raises significant privacy concerns, as raw pixel data can be used for unauthorized identification.

Facial Landmark Emotion Recognition (FLER) offers a solution by using only a set of discrete points (landmarks) on the face—such as the corners of the mouth or the edges of the eyebrows. However, the field has lacked a systematic benchmarking framework to compare how modern architectures, such as Graph Neural Networks and Transformers, perform on these landmarks, especially when dealing with the "messy" reality of non-laboratory data like news broadcasts.

Methodology: The Face as a Geometric Graph

The authors treat the face as a graph, where landmarks serve as nodes and the spatial relationships between them represent edges. This approach allows the model to focus on the underlying structure of an expression rather than the specific appearance of the individual.

Landmark extraction across diverse faces in the Real News Dataset illustrating the geometric approach to emotion recognition.

The research pipeline involves extracting faces using Dlib, identifying landmarks via MediaPipe, and feeding these geometric constructs into several state-of-the-art architectures:

  • Multilayer Perceptron (MLP): A standard baseline.
  • Graph Isomorphism Network (GIN): Focuses on the intrinsic topological structure of landmark patterns.
  • GraphSAGE: An inductive framework that learns to embed landmark features, allowing for generalization to unseen data.
  • GINFormer: A hybrid "Graph Transformer" that combines structural encodings with global attention mechanisms to capture complex interactions across the face.

The Real News Dataset

The paper introduces the Real News Dataset, which moves beyond lab-controlled environments to reflect the lighting, angles, and varied ethnicities found in global media. The dataset consists of 14,172 instances sourced from 318 news videos from international outlets like ABC, Al Jazeera, BBC, and CCTV.

Distribution of emotion categories in the Real News Dataset, highlighting the prevalence of Neutral and Sad expressions over Angry ones.

As shown in the distribution above, the dataset is categorized into five basic emotions: Neutral (27.1%), Sad (24.0%), Happy (23.9%), Fear (16.4%), and Angry (8.6%). This distribution reflects the professional yet often somber nature of news reporting.

Key Results and Benchmarking

The benchmark results demonstrate that while FLER is efficient, real-world emotion mapping remains a difficult task due to the subtlety of expressions in professional broadcasts.

Top Performers: * GINFormer achieved the highest average accuracy at 33.10%, closely followed by SAGE at 32.80%. * Both architectures significantly outperformed the standard MLP baseline (28.00%).

Class-Specific Insights: Interestingly, different models excelled at different emotions. SAGE dominated in "Happy" (49.50%) and "Neutral" categories, while GIN was significantly better at identifying "Angry" expressions (36.50%). Surprisingly, the simple MLP baseline performed best for "Sad" (40.50%), suggesting that the landmark configuration for sadness may be captured effectively by simpler linear relationships.

Conclusion and Future Directions

This research highlights the potential of landmark-based GNNs for privacy-friendly, efficient emotion AI. However, the relatively low overall accuracy compared to lab-based FER underscores the complexity of "in-the-wild" data.

Future work will need to address the data imbalance—particularly the scarcity of "Angry" expressions—and better integrate temporal dynamics. While this benchmark focuses on static landmark configurations, the timing and flow of facial movements are critical to human expression. By refining the trade-off between model weight and accuracy, these techniques can eventually be deployed on edge devices like smart home assistants or wearables to provide real-time, private emotional intelligence.

Summary updated

Abstract

In this paper, we present a novel benchmark for Emotion Recognition using facial landmarks extracted from realistic news videos. Traditional methods relying on RGB images are resource-intensive, whereas our approach with Facial Landmark Emotion Recognition (FLER) offers a simplified yet effective alternative. By leveraging Graph Neural Networks (GNNs) to analyze the geometric and spatial relationships of facial landmarks, our method enhances the understanding and accuracy of emotion recognition. We discuss the advancements and challenges in deep learning techniques for emotion recognition, particularly focusing on Graph Neural Networks (GNNs) and Transformers. Our experimental results demonstrate the viability and potential of our dataset as a benchmark, setting a new direction for future research in emotion recognition technologies. The codes and models are at: https://github.com/wangzhifengharrison/benchmark_real_news
BibTeX
@misc{zhang2024authentic,
  author = {Qixuan Zhang and Zhifeng Wang and Yang Liu and Zhenyue Qin and Kaihao Zhang and Sabrina B. Caldwell and Tom Gedeon},
  title = {{Authentic Emotion Mapping: Benchmarking Facial Expressions in Real News}},
  year = {2024},
  eprint = {2404.13493},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.13493},
}
Figure