Emotive Response to a Hybrid-Face Robot and Translation to Consumer Social Robots

Maitreyee Wairagkar, Maria R. Lima, Daniel Bazo, Richard Craig, Hugo Weissbart, Appolinaire C. Etoundi, Tobias Reichenbach, Prashant Iyenger, Sneh Vaswani, Christopher James, Payam M. Barnaghi, Chris Melhuish, Ravi VaidyanathanarXiv 2020 · OpenReview

TL;DR

This paper introduces a "hybrid-face" robot design that combines a 3D-printed faceplate with a digital display to convey emotions effectively and naturally. They validated this approach by measuring human physiological responses (EEG) and subjective feedback, finding it comparable to human-human emotional communication, and successfully translated this concept into a commercial product, Miko.

What Problem Does This Paper Address?

Creating social robots that can effectively interact with humans requires them to convey and understand emotions. However, building fully actuated, human-like robotic faces is complex, expensive, and can sometimes lead to the "uncanny valley" effect, where a near-human appearance elicits discomfort. This paper addresses the challenge of creating robots that can express emotions in a way that is both understandable and evokes a natural human response, without the need for overly complex mechanical designs. They aim to bridge the gap between research prototypes and commercially viable social robots that can foster trust and enhance human-robot interaction.

Methods

The core innovation lies in the "hybrid-face" design, which uses a static 3D-printed faceplate for structure and depth, combined with a digital display for dynamic facial expressions. This allows for a simplified yet expressive face.

Key methodological contributions include:

  • Hybrid-Face Design: A novel combination of a 3D-printed structure and a digital display to create a sense of depth and allow for dynamic expressions.
  • Affect Space Modeling: Development of both a categorical and a three-dimensional affect space (arousal, valence, stance) to represent robot emotions. This allows for a systematic way to generate and control facial expressions.
  • Pupil Dilation Integration: Incorporating pupil dilation as an additional degree of freedom for conveying emotive states, a feature often overlooked.
  • Neurophysiological Validation (EEG): Using Event-Related Potentials (ERPs), specifically the N170 component (a well-established marker for face processing), to objectively measure human subconscious responses to the robot's expressions. This allows for a direct comparison to responses evoked by human faces.
  • Behavioral Validation: Conducting forced-choice emotion recognition experiments and interviews to gather subjective feedback on the robot's expressiveness.
  • Commercial Translation: Adapting the hybrid-face concept into a simplified, commercially viable IoT robot called Miko.

Datasets

The paper does not explicitly mention using pre-existing datasets for training. Instead, it relies on:

  • Human Participants: Data was collected from 19 healthy participants for the hybrid-face robot experiments and from two groups (15 and 10 participants) for the Miko robot experiments.
  • Japanese Female Facial Expression database [53]: Used in Appendix I for comparison of N170 ERP response to human faces.

Key Insights and Ideas

  • Simplified Abstraction Works: The research strongly suggests that a simplified, hybrid-face approach can effectively convey emotions and evoke comparable human responses to more complex, fully actuated faces. This is a significant finding for the cost-effective development of social robots.
  • Depth Matters: The inclusion of a 3D-printed faceplate, providing a sense of depth, was found to be important for creating a more natural and engaging affective experience compared to a purely digital display on a flat screen.
  • EEG as a Powerful Tool: The study demonstrates the utility of EEG, particularly the N170 component, as a robust, objective measure for quantifying subconscious human responses to robotic emotional expressions, showing similarity to responses to human faces.
  • Pupil Dilation's Potential (and Uncertainty): While pupil dilation was introduced as an additional expressive feature, the results were inconclusive regarding its impact on emotion recognition. This highlights an area for future exploration.
  • Commercial Viability Achieved: The successful translation of the hybrid-face concept into the Miko robot, which is currently in use, validates the practical applicability and market potential of their approach.

Key Results

  • Emotion Recognition: The hybrid-face robot achieved recognition rates above 80% for its main expressions. The commercial Miko robot achieved over 90% expression recognition in interviews.
  • Neurophysiological Response: Human EEG responses to the hybrid-face robot's emotions were comparable to those evoked by actual human faces, as measured by the face-specific N170 ERP. This similarity was also observed for Miko.
  • Pupil Dilation: While not definitively conclusive, the study provided estimations of pupil dilation for different emotions, suggesting potential relationships that could be explored further.
  • Miko's Performance: Miko's simplified facial features (eyes only) were sufficient to convey emotions effectively, with behavioral and physiological experiments showing comparable results to the more complex hybrid-face robot.

Limitations and Future Work

  • Pupil Dilation Inconclusiveness: The paper acknowledges that the impact of pupil dilation on emotion recognition is currently inconclusive and requires further investigation. The current digital eye color also presented a challenge for clear pupil dilation perception.
  • Animation and Realism Tuning: The paper suggests that animation and realism features in the hybrid-face robot could be fine-tuned to improve expression recognition rates.
  • Mouth and Eyebrows for Specific Emotions: The authors hypothesize that the loss of mouth and eyebrows in Miko might contribute to slightly lower recognition rates for stern and disgust compared to the hybrid-face robot.
  • Miko II Development: Emotix has since released Miko II, a robot with greater autonomy, building on the findings of this research. Future work will involve implementing Miko II for accelerated learning in children and as a support tool for the elderly.
  • Quantifying Depth's Effect: While depth was deemed useful, further work is needed to quantify its precise effect on affective experience.
  • Miko's Conversational Ability: The study focused on facial expressions, and the impact of Miko's conversational abilities on affective engagement was not investigated.

Summary updated

Abstract

We introduce the conceptual formulation, design, fabrication, control and commercial translation with IoT connection of a hybrid-face social robot and validation of human emotional response to its affective interactions. The hybrid-face robot integrates a 3D printed faceplate and a digital display to simplify conveyance of complex facial movements while providing the impression of three-dimensional depth for natural interaction. We map the space of potential emotions of the robot to specific facial feature parameters and characterise the recognisability of the humanoid hybrid-face robot's archetypal facial expressions. We introduce pupil dilation as an additional degree of freedom for conveyance of emotive states. Human interaction experiments demonstrate the ability to effectively convey emotion from the hybrid-robot face to human observers by mapping their neurophysiological electroencephalography (EEG) response to perceived emotional information and through interviews. Results show main hybrid-face robotic expressions can be discriminated with recognition rates above 80% and invoke human emotive response similar to that of actual human faces as measured by the face-specific N170 event-related potentials in EEG. The hybrid-face robot concept has been modified, implemented, and released in the commercial IoT robotic platform Miko (My Companion), an affective robot with facial and conversational features currently in use for human-robot interaction in children by Emotix Inc. We demonstrate that human EEG responses to Miko emotions are comparative to neurophysiological responses for actual human facial recognition. Finally, interviews show above 90% expression recognition rates in our commercial robot. We conclude that simplified hybrid-face abstraction conveys emotions effectively and enhances human-robot interaction.
BibTeX
@misc{wairagkar2020emotive,
  author = {Maitreyee Wairagkar and Maria R. Lima and Daniel Bazo and Richard Craig and Hugo Weissbart and Appolinaire C. Etoundi and Tobias Reichenbach and Prashant Iyenger and Sneh Vaswani and Christopher James and Payam M. Barnaghi and Chris Melhuish and Ravi Vaidyanathan},
  title = {{Emotive Response to a Hybrid-Face Robot and Translation to Consumer Social Robots}},
  year = {2020},
  eprint = {2012.04511},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.04511},
}
Figure