Fusing Body Posture With Facial Expressions for Joint Recognition of Affect in Child-Robot Interaction

Panagiotis Paraskevas Filntisis, Niki Efthymiou, Petros Koutras, Gerasimos Potamianos, Petros MaragosIEEE RA-L 2019 · DOI · OpenReview

TL;DR

This paper introduces a deep learning method that fuses body posture and facial expressions for more robust emotion recognition, particularly in the challenging context of child-robot interaction. By leveraging hierarchical multi-label annotations and a novel network architecture, they achieve significant improvements over facial-only approaches, demonstrating the crucial role of body language in understanding emotions.

What Problem Does This Paper Address?

Recognizing human emotions is vital for social robots to establish rapport and provide effective interactions, especially with children. Traditional methods often rely heavily on facial expressions, which can be unreliable or unobservable in real-world scenarios, particularly with spontaneous, unscripted behavior. Children, in particular, may not always present clear facial cues, and their interactions are inherently unpredictable. This paper addresses the gap in robust, multi-modal emotion recognition by proposing a system that integrates body posture information alongside facial expressions, aiming to overcome the limitations of facial-only approaches in dynamic and challenging environments like child-robot interaction.

Methods

The core of the proposed method is a deep neural network (DNN) architecture designed for hierarchical multi-label training (HMT). This architecture features separate branches for processing facial expressions and body posture.

  • Facial Expression Branch: This branch uses a pre-trained Residual Network (ResNet-50) on facial images to extract feature vectors. Temporal max pooling is applied to capture the peak intensity of facial expressions across video frames.
  • Body Posture Branch: This branch utilizes 2D pose estimation (e.g., from OpenPose) to extract skeleton joint information from video frames. Temporal average pooling is employed here to capture the general patterns of body movement over time, rather than just peak moments.
  • Fusion: The outputs from the facial and body branches are concatenated and fed into a further network layer. This fused representation is used for whole-body emotion recognition.
  • Hierarchical Multi-Label Training: The system is trained using hierarchical labels that distinguish between the overall emotion (whole body), the emotion conveyed by the face, and the emotion conveyed by the body. This allows for multi-stage losses, providing more granular supervision during training. The final loss function combines losses from the facial branch, body branch, whole-body branch, and a final fused prediction.

Datasets

The authors utilized two datasets for their experiments:

  • GEMEP (GEneva Multimodal Emotion Portrayals) database: This dataset contains acted emotional expressions from adults. It was used for exploratory analysis of the different branches of their HMT architecture.
  • BabyRobot Emotion Database (BRED): This is a newly collected dataset specifically designed for child-robot interaction. It includes multimodal recordings of children (ages 6-12) interacting with robots, featuring both acted and spontaneous emotional expressions. BRED includes hierarchical annotations for face, body, and whole-body emotions.

Key Insights and Ideas

  • Body Language is Crucial for Emotion Recognition: The paper strongly emphasizes that body posture is not just a supplementary cue but a primary modality for understanding emotions, especially in contexts where facial expressions are ambiguous or absent.
  • Hierarchical Labels Improve Robustness: Using hierarchical labels (whole body, face, body) allows the model to learn more nuanced representations and provides better supervision, leading to improved performance.
  • Children's Emotions are Complex: The BRED dataset highlights the challenges of emotion recognition in children, who may express emotions differently and less predictably than adults, underscoring the need for multi-modal approaches.
  • Fusion Outperforms Modality-Specific Models: The core finding is that combining facial and body information consistently leads to better emotion recognition accuracy than relying on either modality alone.

Key Results

  • The proposed HMT method significantly improved emotion recognition accuracy compared to facial-only baselines.
  • On the BRED dataset, the HMT-4 model (joint training with all four losses) achieved the highest balanced F1-score of 0.70 and balanced accuracy of 0.72 for whole-body emotion recognition.
  • The body branch alone showed lower accuracy than the face branch, but its fusion with facial information was critical for overall performance.
  • For certain emotions like "pride," the body branch was more effective than the face branch, illustrating the complementary nature of the modalities.
  • The BRED dataset revealed that children often rely on body expressions for emotions like "fear" and "anger" where facial cues might be less pronounced.

Limitations and Future Work

  • Dataset Imbalance: The BRED dataset exhibits significant class imbalance, particularly for body labels, which the authors addressed by using balanced cross-entropy loss and reporting balanced metrics.
  • Limited Scope of Body Expression: While the paper uses 2D pose, future work could explore 3D pose for richer spatial information.
  • Computational Cost: Deep learning models can be computationally intensive, which might be a consideration for real-time robot deployment.
  • Future Directions: The authors suggest further research into more complex fusion strategies, exploring other modalities (like audio), and creating larger, more diverse datasets for child-robot interaction. They also highlight the need for more sophisticated body pose representations beyond simple joint coordinates.

Summary updated

Abstract

In this letter, we address the problem of multi-cue affect recognition in challenging scenarios such as child–robot interaction. Toward this goal we propose a method for automatic recognition of affect that leverages body expressions alongside facial ones, as opposed to traditional methods that typically focus only on the latter. Our deep-learning based method uses hierarchical multi-label annotations and multi-stage losses, can be trained both jointly and separately, and offers us computational models for both individual modalities, as well as for the whole body emotion. We evaluate our method on a challenging child–robot interaction database of emotional expressions collected by us, as well as on the GEneva multimodal emotion portrayal public database of acted emotions by adults, and show that the proposed method achieves significantly better results than facial-only expression baselines.
BibTeX
@article{filntisis2019fusing,
  author = {Panagiotis Paraskevas Filntisis and Niki Efthymiou and Petros Koutras and Gerasimos Potamianos and Petros Maragos},
  title = {{Fusing Body Posture With Facial Expressions for Joint Recognition of Affect in Child-Robot Interaction}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2019},
  doi = {10.1109/lra.2019.2930434},
}
Figure