Introduction
Achieving emotional intelligence in humanoid robots is a cornerstone of natural human-robot interaction (HRI). While deep learning models have excelled at emotion recognition in controlled, static datasets, real-world "in the wild" scenarios present significant hurdles. Humanoid robots must operate in dynamic, multiparty environments where environmental noise and computational latency often degrade the user experience. This paper introduces UGotMe, an embodied system designed to overcome these challenges by integrating physical robot movements with a novel multimodal emotion recognition model.
The Challenge of Multiparty Interaction
Interacting in a room with multiple people introduces two primary "embodiment challenges." First, environmental noise complicates visual processing; a robot's camera captures everyone in the field of view, making it difficult to distinguish the active speaker from laughing bystanders or moving background objects. Second, real-time latency is critical for natural flow. High-resolution video processing and large model inference often cause delays that break the illusion of social presence.
Methodology: Active Denoising and VL2E
The UGotMe system employs a two-pronged approach to address these issues, splitting tasks between the robot hardware and an on-edge local server.
Active Denoising Strategies
Rather than relying purely on software filters, UGotMe utilizes the robot's physical body as a pre-processing tool. The system uses microphones to determine the Direction of Sound Arrival (DoSA), prompting the robot to physically adjust its head and eye cameras to center the active speaker. This "active face extraction" ensures the most relevant visual data is prioritized. Furthermore, the system implements Person-Specific Neutral Normalization, calculating "delta images" from a baseline neutral frame to account for individual variations in resting facial expressions.
System Architecture and the VL2E Model
At the heart of the system is the Vision-Language to Emotion (VL2E) model. This multimodal framework processes visual and textual data in parallel:
- Visual Modality: An InceptionResnetv1 backbone extracts features from facial sequences.
- Textual Modality: A SimCSE model encodes dialogue, utilizing prompt learning (e.g., "[speaker] feels [mask]") to capture contextual nuances.
- Multimodal Fusion: A Crossmodal Transformer merges these signals to predict one of seven basic emotions (neutral, surprise, fear, sadness, joy, disgust, or anger).
To minimize latency, the system utilizes the ZMQ library for high-speed data transfer, streaming raw RGB data in a dedicated thread between the robot and the server.
Experimental Results
The system was evaluated using both benchmark datasets and real-world deployment.
Benchmark Performance
On the MELD (Multimodal Emotion Lines Dataset), which features complex multiparty dialogues from the TV show Friends, the VL2E model achieved a weighted average F1 score of 67.29. This outperformed several established baselines, including FacialMMT (66.58) and DialogueRNN (57.03).
Real-World Validation
UGotMe was deployed on the Ameca humanoid robot for live tests with human volunteers. In these scenarios, the system achieved an emotion response accuracy of 77.29%, a significant improvement over the baseline TelME system (50.63%), which was frequently distracted by inactive bystanders. Human participants rated the interaction highly, providing a user experience score of 7.89/10.
Discussion and Future Work
A key insight from this research is the necessity of multimodal context; removing the textual modality caused the F1 score to drop significantly, highlighting that visual cues alone are often insufficient for interpreting complex human dialogue.
Currently, UGotMe implements "parallel empathy," where the robot mirrors the user's emotion. Future research aims to transition toward reactive empathy, where the robot provides supportive responses (e.g., comforting a sad user). Additionally, the authors plan to incorporate acoustic prosody (tone and pitch) and develop more robust methods for estimating a person's baseline neutral state, as the current system assumes the initial frame is always neutral.
Conclusion
UGotMe demonstrates that by leveraging a robot's physical embodiment as a filter and employing high-speed multimodal fusion, humanoid robots can achieve a new level of emotional intelligence in complex, real-world environments.
Summary updated