Modeling User Empathy Elicited by a Robot Storyteller

Leena Mathur, Micol Spitale, Hao Xi, Jieyun Li, Maja J. Mataric

Abstract

Virtual and robotic agents capable of perceiving human empathy have the potential to participate in engaging and meaningful human-machine interactions that support human well-being. Prior research in computational empathy has focused on designing empathic agents that use verbal and nonverbal behaviors to simulate empathy and attempt to elicit empathic responses from humans. The challenge of developing agents with the ability to automatically perceive elicited empathy in humans remains largely unexplored. Our paper presents the first approach to modeling user empathy elicited during interactions with a robotic agent. We collected a new dataset from the novel interaction context of participants listening to a robot storyteller (46 participants, 6.9 hours of video). After each storytelling interaction, participants answered a questionnaire that assessed their level of elicited empathy during the interaction with the robot. We conducted experiments with 8 classical machine learning models and 2 deep learning models (long short-term memory networks and temporal convolutional networks) to detect empathy by leveraging patterns in participants' visual behaviors while they were listening to the robot storyteller. Our highest-performing approach, based on XGBoost, achieved an accuracy of 69% and AUC of 72% when detecting empathy in videos. We contribute insights regarding modeling approaches and visual features for automated empathy detection. Our research informs and motivates future development of empathy perception models that can be leveraged by virtual and robotic agents during human-machine interactions.

Introduction

In the field of Human-Robot Interaction (HRI), significant effort has been dedicated to teaching robots how to express empathy through programmed gestures and scripts. However, the reciprocal ability—perception—is largely missing. For a robot to be truly effective in therapeutic or educational settings, such as interacting with children on the Autism Spectrum (ASD) or the elderly, it must be able to sense whether its human partner is emotionally engaged.

Conceptual diagram of the robot storyteller perceiving user empathy based on visual features.

This research addresses the challenge of automatically perceiving "elicited empathy"—the internal emotional state a human feels in response to a robot. Unlike simple sentiment analysis (e.g., detecting a smile), empathy detection requires identifying nuanced non-verbal cues that signal a deep social and emotional connection.

Methods and Dataset

The researchers framed empathy detection as a binary classification problem, distinguishing between "empathic" and "less-empathic" states based on user self-reports.

Data Collection

The study utilized a novel dataset involving 46 university students who listened to a tabletop robot (QT) tell three different stories. This resulted in 6.9 hours of video data. Ground truth was established using a validated 8-item empathy questionnaire administered after each story, with the median score serving as the threshold for classification.

Experimental setup showing the robot (QT) and the visual feature extraction process using facial landmarks.

Feature Extraction and Modeling

Using the OpenFace 2.2.0 toolkit, the team extracted 709 raw visual features, including eye gaze, 17 Facial Action Units (FAUs), and head pose. For classical machine learning, these were aggregated into fixed-length vectors using statistical descriptors.

The study compared 8 classical models (e.g., SVM, Random Forest) against deep learning architectures like Long Short-Term Memory (LSTM) and Temporal Convolutional Networks (TCN). Interestingly, XGBoost emerged as the top performer, likely because deep learning models often require larger datasets than are typically available in high-quality, lab-controlled HRI studies.

Key Results and Insights

  • Model Performance: The XGBoost model achieved a 69% accuracy and an Area Under the Curve (AUC) of 72%, significantly outperforming the 50% chance baseline.
  • The "Dimpler" Effect: The most predictive feature was FAU 14 (the "Dimpler" muscle). Empathic listeners showed a mean intensity of 0.23 in this region, compared to 0.11 for less-empathic listeners. This suggests that empathy in a storytelling context is closely tied to subtle, expressive mouth movements.
  • Expressivity as a Proxy: 19 of the top 25 predictive features were Facial Action Units. This confirms that users who feel a stronger emotional connection tend to "mirror" the narrator or react with more active facial expressions, even when they are primarily in a listening role.

Conclusion and Future Work

This study provides a foundational approach for automated empathy perception. While successful, the authors note that future models could be improved by incorporating audio (prosody) and physiological signals (e.g., heart rate). Additionally, moving from a stationary robot to a mobile one could reveal whether "gaze following" serves as a stronger signal for empathy. The ultimate goal is to generalize these models to diverse populations, such as children with ASD, where automated empathy perception could significantly enhance clinical outcomes.

Summary updated

Abstract

Virtual and robotic agents capable of perceiving human empathy have the potential to participate in engaging and meaningful human-machine interactions that support human well-being. Prior research in computational empathy has focused on designing empathic agents that use verbal and nonverbal behaviors to simulate empathy and attempt to elicit empathic responses from humans. The challenge of developing agents with the ability to automatically perceive elicited empathy in humans remains largely unexplored. Our paper presents the first approach to modeling user empathy elicited during interactions with a robotic agent. We collected a new dataset from the novel interaction context of participants listening to a robot storyteller (46 participants, 6.9 hours of video). After each storytelling interaction, participants answered a questionnaire that assessed their level of elicited empathy during the interaction with the robot. We conducted experiments with 8 classical machine learning models and 2 deep learning models (long short-term memory networks and temporal convolutional networks) to detect empathy by leveraging patterns in participants' visual behaviors while they were listening to the robot storyteller. Our highest-performing approach, based on XGBoost, achieved an accuracy of 69% and AUC of 72% when detecting empathy in videos. We contribute insights regarding modeling approaches and visual features for automated empathy detection. Our research informs and motivates future development of empathy perception models that can be leveraged by virtual and robotic agents during human-machine interactions.
BibTeX
@inproceedings{mathur2021modeling,
  author = {Leena Mathur and Micol Spitale and Hao Xi and Jieyun Li and Maja J. Mataric},
  title = {{Modeling User Empathy Elicited by a Robot Storyteller}},
  booktitle = {ACII},
  year = {2021},
  url = {https://openreview.net/forum?id=G3cjfrYfov},
}
Figure