Kinematic dataset of actors expressing emotions

Mingming Zhang, Lu Yu, Keye Zhang, Bixuan Du, Bin Zhan, Shaohua Chen, Xiuhao Jiang, Shuai Guo, Jiafeng Zhao, Yang Wang, Bin Wang, Shenglan Liu, Wenbo LuoScientific 2020 · DOI

TL;DR

This paper introduces the largest publicly available dataset of human body kinematics capturing six basic emotions and neutral expressions. This comprehensive dataset, collected using a portable motion capture system, aims to bridge the gap in understanding how emotions are encoded in full-body movements, paving the way for advancements in fields like computer vision and affective computing.

What Problem Does This Paper Address?

While human body language is a rich source of emotional information, research has historically focused more on facial expressions and vocal cues. There's a significant lack of comprehensive, publicly accessible datasets that capture the nuances of full-body emotional expression through kinematics. This deficit hinders research into how emotions are encoded in body movements, limiting our ability to develop accurate emotion recognition systems that go beyond just the face. Existing datasets are often limited in scope, lacking raw kinematic data, or focusing only on specific body parts or a narrow range of scenarios.

Methods

The researchers employed a portable wireless motion capture system (Noitom Perception Neuron) equipped with 17 wearable sensors. These sensors were strategically placed on actors' bodies, covering key anatomical points from head to toe. The system captured data at a sampling rate of 125 Hz, recording the position and rotation of 72 anatomical nodes. To ensure natural and varied performances, actors were guided by standardized instructions and a set of 70 daily event scenarios designed to elicit specific emotions. A pilot study was conducted to select the top five most recognizable scenarios for each emotion, ensuring ecological validity. For each emotion and neutral expression, actors performed for approximately six seconds. The collected raw data was exported into the BioVision Hierarchy (BVH) format, a standard for motion capture data.

Datasets

The paper introduces a new dataset: the "Kinematic dataset of actors expressing emotions." This dataset comprises 1402 recorded trials, each capturing the full-body kinematic data of an actor expressing one of six basic emotions (happiness, sadness, anger, fear, disgust, surprise) or a neutral expression. The data includes the position and rotation of 72 anatomical nodes at 125 Hz. The dataset is available via PhysioNet under a Restricted Health Data License, requiring users to sign an agreement before download.

Key Insights and Ideas

  • Body Language is Crucial for Emotion: The paper reinforces the idea that body movements are not just supplementary but can be as informative as facial expressions for recognizing emotions, especially in complex or intense situations.
  • Need for Comprehensive Kinematic Data: The scarcity of large-scale, raw kinematic datasets for emotional expression is a major bottleneck in the field, and this work directly addresses that.
  • Ecological Validity Matters: By using a variety of daily event scenarios and allowing for some free performance, the dataset aims to capture more naturalistic emotional expressions than highly controlled, artificial scenarios.
  • Potential for Cross-Disciplinary Impact: The dataset is positioned to benefit not only computer vision and affective computing but also fields like social neuroscience, psychiatry, and biometrics.

Key Results

The primary result is the creation and release of the "Kinematic dataset of actors expressing emotions," which the authors claim is the largest of its kind. The dataset contains 1402 trials, with each trial providing detailed kinematic information (position and rotation of 72 nodes) at 125 Hz. The data was collected from 22 semi-professional actors (half male) performing standardized scenarios for six basic emotions and a neutral state. The paper also includes technical validation demonstrating good calibration quality of the motion capture system, evidenced by the centralized and consistent spatial positions of the actors' mass centers across recordings.

Limitations and Future Work

The authors acknowledge that while they aimed for consistent trial durations (around six seconds), some trials ended up being shorter or longer due to variations in human time perception or operator error. They also note that differences in scenarios within the same emotion might introduce "noise," as the specific object of an emotion (e.g., a proximal dog vs. a speeding car for anger) can influence the body's expression. Future work could involve further examining the relationship between scenario-induced movements and subjective emotional experience, as well as addressing trial duration inconsistencies. The dataset's availability is also subject to a data use agreement, which might be a limitation for some users.

Summary updated

Abstract

Human body movements can convey a variety of emotions and even create advantages in some special life situations. However, how emotion is encoded in body movements has remained unclear. One reason is that there is a lack of public human body kinematic dataset regarding the expressing of various emotions. Therefore, we aimed to produce a comprehensive dataset to assist in recognizing cues from all parts of the body that indicate six basic emotions (happiness, sadness, anger, fear, disgust, surprise) and neutral expression. The present dataset was created using a portable wireless motion capture system. Twenty-two semi-professional actors (half male) completed performances according to the standardized guidance and preferred daily events. A total of 1402 recordings at 125 Hz were collected, consisting of the position and rotation data of 72 anatomical nodes. To our knowledge, this is now the largest emotional kinematic dataset of the human body. We hope this dataset will contribute to multiple fields of research and practice, including social neuroscience, psychiatry, computer vision, and biometric and information forensics.
BibTeX
@inproceedings{zhang2020kinematic,
  author = {Mingming Zhang and Lu Yu and Keye Zhang and Bixuan Du and Bin Zhan and Shaohua Chen and Xiuhao Jiang and Shuai Guo and Jiafeng Zhao and Yang Wang and Bin Wang and Shenglan Liu and Wenbo Luo},
  title = {{Kinematic dataset of actors expressing emotions}},
  booktitle = {Scientific},
  year = {2020},
  doi = {10.1038/s41597-020-00635-7},
}
Figure