UGotMe: An Embodied System for Affective Human-Robot Interaction

Peizhen Li, Longbing Cao, Xiao-Ming Wu, Xiaohan Yu, Runze Yang

Introduction

Achieving emotional intelligence in humanoid robots is a cornerstone of natural human-robot interaction (HRI). While deep learning models have excelled at emotion recognition in controlled, static datasets, real-world "in the wild" scenarios present significant hurdles. Humanoid robots must operate in dynamic, multiparty environments where environmental noise and computational latency often degrade the user experience. This paper introduces UGotMe, an embodied system designed to overcome these challenges by integrating physical robot movements with a novel multimodal emotion recognition model.

The Challenge of Multiparty Interaction

Interacting in a room with multiple people introduces two primary "embodiment challenges." First, environmental noise complicates visual processing; a robot's camera captures everyone in the field of view, making it difficult to distinguish the active speaker from laughing bystanders or moving background objects. Second, real-time latency is critical for natural flow. High-resolution video processing and large model inference often cause delays that break the illusion of social presence.

UGotMe distinguishes between active speakers and inactive bystanders in complex environments.

Methodology: Active Denoising and VL2E

The UGotMe system employs a two-pronged approach to address these issues, splitting tasks between the robot hardware and an on-edge local server.

Active Denoising Strategies

Rather than relying purely on software filters, UGotMe utilizes the robot's physical body as a pre-processing tool. The system uses microphones to determine the Direction of Sound Arrival (DoSA), prompting the robot to physically adjust its head and eye cameras to center the active speaker. This "active face extraction" ensures the most relevant visual data is prioritized. Furthermore, the system implements Person-Specific Neutral Normalization, calculating "delta images" from a baseline neutral frame to account for individual variations in resting facial expressions.

System Architecture and the VL2E Model

At the heart of the system is the Vision-Language to Emotion (VL2E) model. This multimodal framework processes visual and textual data in parallel:

  • Visual Modality: An InceptionResnetv1 backbone extracts features from facial sequences.
  • Textual Modality: A SimCSE model encodes dialogue, utilizing prompt learning (e.g., "[speaker] feels [mask]") to capture contextual nuances.
  • Multimodal Fusion: A Crossmodal Transformer merges these signals to predict one of seven basic emotions (neutral, surprise, fear, sadness, joy, disgust, or anger).
The UGotMe pipeline: from multimodal feature extraction to real-time emotional mirroring.

To minimize latency, the system utilizes the ZMQ library for high-speed data transfer, streaming raw RGB data in a dedicated thread between the robot and the server.

Experimental Results

The system was evaluated using both benchmark datasets and real-world deployment.

Benchmark Performance

On the MELD (Multimodal Emotion Lines Dataset), which features complex multiparty dialogues from the TV show Friends, the VL2E model achieved a weighted average F1 score of 67.29. This outperformed several established baselines, including FacialMMT (66.58) and DialogueRNN (57.03).

Real-World Validation

UGotMe was deployed on the Ameca humanoid robot for live tests with human volunteers. In these scenarios, the system achieved an emotion response accuracy of 77.29%, a significant improvement over the baseline TelME system (50.63%), which was frequently distracted by inactive bystanders. Human participants rated the interaction highly, providing a user experience score of 7.89/10.

Discussion and Future Work

A key insight from this research is the necessity of multimodal context; removing the textual modality caused the F1 score to drop significantly, highlighting that visual cues alone are often insufficient for interpreting complex human dialogue.

Currently, UGotMe implements "parallel empathy," where the robot mirrors the user's emotion. Future research aims to transition toward reactive empathy, where the robot provides supportive responses (e.g., comforting a sad user). Additionally, the authors plan to incorporate acoustic prosody (tone and pitch) and develop more robust methods for estimating a person's baseline neutral state, as the current system assumes the initial frame is always neutral.

Conclusion

UGotMe demonstrates that by leveraging a robot's physical embodiment as a filter and employing high-speed multimodal fusion, humanoid robots can achieve a new level of emotional intelligence in complex, real-world environments.

Summary updated

BibTeX
@inproceedings{li2025ugotme,
  author = {Peizhen Li and Longbing Cao and Xiao-Ming Wu and Xiaohan Yu and Runze Yang},
  title = {{UGotMe: An Embodied System for Affective Human-Robot Interaction}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2025},
  doi = {10.1109/icra55743.2025.11128819},
}
Figure