FePh: An Annotated Facial Expression Dataset for the RWTH-PHOENIX-Weather 2014 Dataset

Marie Alaghband, Niloofar Yousefi, Ivan GaribayarXiv 2020 · arXiv

Introduction

In the field of Sign Language Recognition (SLR), research has historically prioritized hand shapes, orientations, and movements. However, sign language is a sophisticated "multi-channel" communication system where facial expressions are not merely supplementary but fundamental to grammatical structure and meaning. Facial cues can transform a statement into a question or differentiate between signs with identical manual components.

Despite their importance, existing facial expression datasets—often designed for general emotion detection—are frequently inadequate for sign language. Signers often "mouth" words to clarify meaning, and rapid head movements can lead to motion blur and varied poses that challenge standard algorithms. To address this gap, researchers have introduced FePh, the first large-scale annotated facial expression dataset specifically tailored for continuous sign language recognition.

Examples of signers from the FePh dataset showing varied facial expressions and hand positions.

Methodology

The FePh dataset consists of 3,359 manually annotated frames derived from the development set of the RWTH-PHOENIX-Weather 2014 corpus. A team of 12 annotators evaluated three primary metrics for each frame:

  1. Emotion: Categorized into seven basic emotions (sad, surprise, fear, angry, neutral, disgust, happy) or "None."
  2. Visibility: Assessment of whether the face was obscured by movement or hand gestures.
  3. Gender: Demographic data for statistical analysis.

A unique aspect of the methodology was the inclusion of dyads. Recognizing the complexity of human expression, annotators could select multiple emotions, capturing secondary and tertiary combinations like "surprise-fear." To minimize bias, automated facial recognition was used to crop faces, ensuring annotators focused solely on facial cues rather than accompanying hand gestures.

The emotional taxonomy used for annotation, including primary emotions and their potential combinations.

Dataset Context and Taxonomy

FePh is built upon the RWTH-PHOENIX-Weather 2014 dataset, which features professional interpreters signing German weather forecasts. This choice allows researchers to correlate new facial expression labels with existing hand-shape data from a well-established benchmark. The paper categorizes SLR resources into:

  • Hardware-based: Utilizing data gloves or Kinect sensors (e.g., UCI Australian Auslan), which offer precision but limited real-world applicability.
  • Vision-based: Utilizing standard cameras (e.g., ASL-LEX, Boston ASLLVD), which are more accessible for practical deployment.

Key Insights and Results

The research yielded several significant findings regarding the interplay between facial and manual signals:

  • The Impact of "Mouthing": Signers frequently mouth words, creating "noise" for traditional emotion detectors as speech-related mouth shapes can mimic expressions like anger or surprise.
  • Emotional Complexity: While the system allowed for complex dyads, the researchers found that no frame required more than two combined emotions, suggesting a linguistic limit to facial signal complexity.
  • Hand-Face Correlation: A meaningful association was discovered between specific hand shapes (e.g., "pincet" or "index") and specific emotions (e.g., "fear" or "surprise"), confirming the synchronized nature of multi-channel signing.

The final dataset includes 3,359 annotated frames. Approximately 90% of these express the seven basic emotions, while 10% represent complex dyads or the "None" category.

Limitations and Future Directions

While the release of over 3,000 annotated frames provides a robust foundation for statistical analysis, the authors note that larger datasets may be required for training state-of-the-art deep learning models from scratch. Future research should aim to expand the subject pool beyond weather reporting, test model performance on "in-the-wild" sign language videos, and integrate these findings into truly multi-modal SLR systems that process hands and faces simultaneously.

Summary updated

Abstract

Facial expressions are important parts of both gesture and sign language recognition systems. Despite the recent advances in both fields, annotated facial expression datasets in the context of sign language are still scarce resources. In this manuscript, we introduce an annotated sequenced facial expression dataset in the context of sign language, comprising over $3000$ facial images extracted from the daily news and weather forecast of the public tv-station PHOENIX. Unlike the majority of currently existing facial expression datasets, FePh provides sequenced semi-blurry facial images with different head poses, orientations, and movements. In addition, in the majority of images, identities are mouthing the words, which makes the data more challenging. To annotate this dataset we consider primary, secondary, and tertiary dyads of seven basic emotions of "sad", "surprise", "fear", "angry", "neutral", "disgust", and "happy". We also considered the "None" class if the image's facial expression could not be described by any of the aforementioned emotions. Although we provide FePh as a facial expression dataset of signers in sign language, it has a wider application in gesture recognition and Human Computer Interaction (HCI) systems.
BibTeX
@misc{alaghband2020feph,
  author = {Marie Alaghband and Niloofar Yousefi and Ivan Garibay},
  title = {{FePh: An Annotated Facial Expression Dataset for the RWTH-PHOENIX-Weather 2014 Dataset}},
  year = {2020},
  eprint = {2003.08759},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.08759},
}
Figure