TL;DR
This paper introduces a novel system that enables intelligent robots to generate realistic body gestures during conversations, mimicking human non-verbal communication. By adapting sequence-to-sequence models to process both verbal and non-verbal cues, the system synthesizes natural-looking gestures for both listening and speaking phases, demonstrated on a virtual avatar and the Pepper robot.
What Problem Does This Paper Address?
Current intelligent robots, while capable of verbal communication, often lack natural non-verbal expressiveness. This can make human-robot interactions feel unnatural and less engaging. Existing approaches to robot gestures are often limited to pre-defined movements or focus solely on the speaking phase. This paper aims to bridge this gap by developing a system that can synthesize realistic body gestures for robots during both listening and speaking phases of a conversation, enhancing the robot's ability to comprehend and express itself in a human-like manner.
Methods
The core of the system relies on a sequence-to-sequence (seq2seq) architecture, a powerful framework for mapping input sequences to output sequences, commonly used in machine translation. The authors adapt this for two distinct tasks:
- Listening Model: This model takes the speaker's verbal input (text) and their body gestures (keypoints) as input and generates the listener's non-verbal feedback gestures.
- Speaking Model: This model takes only the speaker's verbal response (text) as input and generates body gestures to accompany the speech.
Before feeding data into the seq2seq models, a gesture parsing module is crucial:
- Keypoint Extraction: 2D keypoints of the upper body are extracted from video frames using AlphaPose.
- 3D Transformation: These 2D keypoints are then transformed into 3D coordinates.
- Rotation and Normalization: To handle variations in camera perspective and body scale, a novel vector normalization method is employed. This method represents keypoints as vectors relative to adjacent ones and normalizes their lengths, focusing on directional changes rather than absolute positions. This is a key innovation over simpler individual or global normalization techniques.
Finally, a motion synthesis module reconstructs the predicted keypoint sequences into actual body movements for a virtual avatar and the Pepper robot.
Datasets
The system is trained on a substantial dataset of human-human conversations collected from:
- The Ellen Show (2263 videos)
- The Tonight Show (1978 videos)
These talk show videos were segmented into clips based on audio signals, and speaker roles were identified to isolate individual conversational turns. The collected data consists of clip-text pairs with associated speaker information.
Key Insights and Ideas
- The importance of non-verbal cues during listening: The paper highlights that humans provide non-verbal feedback even when listening, a crucial aspect for natural conversation that is often overlooked.
- Vector normalization for robust gesture representation: The authors found that traditional coordinate-based normalization methods struggled with scale variations. Their proposed vector normalization, which focuses on relative joint movements, proved more effective for capturing essential body dynamics.
- Persona matters: An experiment swapping input and ground truth sequences revealed that the "persona" (i.e., whether the model learns to act as the host or the guest) significantly impacts performance, underscoring the importance of consistent training data.
- Elbow-wrist connection is challenging: The analysis of individual keypoint vector accuracy revealed that the connection between the elbow and wrist showed the largest deviation, likely due to the frequent and complex movements of the lower arm during speech.
Key Results
- Improved gesture synthesis: The proposed system, particularly with vector normalization, achieved significantly better results in terms of both Mean Squared Error (MSE) and cosine similarity compared to individual and global normalization methods. For instance, vector normalization achieved a cosine similarity of 0.992 for the listening model on the Ellen Show dataset, compared to 0.308 for individual normalization.
- Realistic motion synthesis: The system successfully generated synthesized body gestures on both a virtual avatar and the Pepper robot, demonstrating its practical applicability in creating more engaging human-robot interactions. Figure 10 showcases examples of synthesized gestures for both listening and speaking phases.
Limitations and Future Work
- Limitations:
- The current system focuses on upper-body keypoints, as lower-body visibility and movement are often limited in conversational videos.
- The paper acknowledges that the vector connecting elbow and wrist is the most challenging to predict accurately due to its dynamic nature.
- The persona of the speaker/listener in the training data has a significant impact on the model's performance.
- Future Work:
- The authors aim to investigate a real-time conversation model based on their proposed system.
- Further exploration into incorporating lower-body gestures could enhance realism.
- Addressing the persona dependency could lead to more generalized models.
Summary updated