TL;DR
This paper introduces XBG, a novel end-to-end imitation learning system that enables humanoid robots to autonomously perform human-robot interaction (HRI) tasks. By fusing exteroceptive (visual) and proprioceptive (robot state) data, XBG learns complex behaviors from teleoperated demonstrations, paving the way for more natural and collaborative human-robot partnerships.
What Problem Does This Paper Address?
Humanoid robots are increasingly expected to operate in human-centric environments, requiring them to not only perform tasks but also to interact and collaborate seamlessly with people. This necessitates robots that can understand human intentions, adapt to changing contexts, and take initiative when appropriate. However, developing such autonomous HRI capabilities is challenging due to the complexity of robot control, the need for rich environmental perception, and the difficulty of collecting diverse, high-quality demonstration data. Existing approaches often focus on specific tasks or rely on simpler robot platforms, leaving a gap in end-to-end learning for whole-body autonomous behavior in realistic HRI scenarios.
Methods
XBG employs a multimodal, end-to-end imitation learning architecture. The core of the system is a neural network composed of four main components:
- Exteroceptive Component: This module processes visual (RGB) and depth camera data. The RGB branch uses a pre-trained YOLOv8 backbone for feature extraction, while the depth branch uses convolutional layers. These features are then fused.
- Proprioceptive Component: This module takes in the robot's own state information, including upper-body joint positions, locomotion velocities, and motor currents from arm joints. This provides the robot with self-awareness.
- Modal Fusion Component: A mid-fusion strategy combines the exteroceptive and proprioceptive features into a single tensor representing the robot's understanding of its environment and its own state at a given moment.
- Temporal Component: A Long Short-Term Memory (LSTM) network processes sequences of these fused features over a time window (1.6 seconds during training). This allows the robot to learn temporal dependencies and understand the dynamics of interactions.
The output of the network is the desired joint positions for the upper body and locomotion velocities for the lower body, predicting these actions 30ms ahead. The system is trained using data collected via teleoperation.
Datasets
The paper introduces a new dataset collected from teleoperating the ergoCub humanoid robot in various Human-Robot Interaction (HRI) scenarios. This dataset comprises 100 minutes of demonstrations covering:
- Handwaving
- Handshaking
- Payload reception
- Walking
- Walking with a payload
The data includes synchronized RGB and depth images at 30Hz, along with robot state information (joint positions, velocities, etc.) at 100Hz. For training, this data is down-sampled to 10Hz and processed into 1.6-second time windows.
Key Insights and Ideas
- Multimodal Fusion for HRI: Combining visual perception (what the robot sees) with its own internal state (proprioception) is crucial for robots to understand and participate in complex HRI tasks.
- Temporal Context is Key: Learning from sequences of observations, rather than single snapshots, significantly improves the robot's ability to perform HRI behaviors, mirroring how humans process interactions over time.
- Teleoperation as a Data Source: Using teleoperation to collect demonstrations provides a practical and effective way to gather rich, diverse data for training complex robot behaviors in realistic scenarios.
- End-to-End Learning for Whole-Body Control: The XBG system demonstrates that imitation learning can be successfully applied to learn end-to-end policies for whole-body humanoid robot control in interactive settings.
Key Results
The XBG system was evaluated on the real ergoCub robot. An ablation study (Table I) showed that using both RGB and Depth data (RGBD) yielded the highest success rates across various tasks. Specifically, the full RGBD model achieved:
- Handwave: 100% success
- Handshake: 53% success
- Payload reception: 100% success
- Walking: 93% success
- Walking with payload: 90% success
A "behaviour blending" experiment, where the robot autonomously transitioned between 30 different interactions, demonstrated its generalization capabilities, successfully completing 70% of these varied scenarios.
Limitations and Future Work
The authors acknowledge that the proprioceptive signals (motor currents) can be noisy, leading to less smooth movements, which they addressed with filtering. While the system shows promising results, the paper suggests future work could include:
- Integrating Audio Modalities: Incorporating audio perception could further enhance the robot's understanding of human cues and context.
- Leveraging Large Language Models (LLMs): Exploring the potential of LLMs could enable more sophisticated reasoning and interaction capabilities for human-robot collaboration.
Summary updated