Robotic Imitation of Human Actions

Josua Spisak, Matthias Kerzel, Stefan Wermter

Abstract

Imitation learning allows robots to acquire new skills by observing humans, yet it is often hindered by the perspective gap and the correspondence problem. This paper introduces a multi-stage architecture that enables a humanoid robot to perform "one-shot" imitation of complex tasks from a single video demonstration. The system leverages a generative diffusion model for temporal action segmentation and an open-vocabulary object detector (ViLD) to translate visual input into a symbolic plan. Experimental results on the NICOL robot demonstrate the feasibility of this abstraction-based approach in cluttered environments.

Introduction

Robotic imitation of human actions is a cornerstone of intuitive human-robot interaction. However, two primary challenges persist: the perspective gap, where the robot's viewpoint differs from the human's, and the correspondence problem, arising from the physical differences between human and robotic bodies. Traditional methods often rely on massive datasets or direct teleoperation. This research proposes a pipeline that bypasses these issues by abstracting human intent into a symbolic sequence of goals, allowing the robot to calculate its own trajectories using its unique body schema.

Figure 4: The pipeline enables a robot to replicate a human's pick-and-place action from a single video demonstration.

Methods

The proposed architecture consists of four key stages:

  1. Temporal Action Segmentation: A generative diffusion model, utilizing an ASFormer backbone, analyzes video frames to generate action labels (e.g., "moving with object"). The diffusion process effectively denoises the label sequence, ensuring smooth transitions.
  2. Spatial Object Detection: The ViLD open-vocabulary detector identifies objects based on text prompts, providing flexibility beyond pre-defined categories.
  3. Symbolic Reasoning: The system synthesizes temporal and spatial data to create a high-level plan, such as Grasp(Object) -> Move(Target).
  4. Execution: An Inverse Kinematics (IK) solver translates the symbolic plan into specific joint movements for the NICOL robot.

Experimental Setup

The system was evaluated using a custom dataset featuring five items from the YCB object set, including a red bowl, spam can, and tomato soup can. The setup involved 120 videos of a human performing pick-and-place tasks in a cluttered environment to simulate real-world complexity.

Figure 5: The experimental workspace featuring various YCB objects used to test the open-vocabulary detection and imitation capabilities.

Results and Discussion

The diffusion-based segmentation model achieved 89.19% accuracy, significantly outperforming traditional classifiers in sequence stability. Object localization accuracy ranged from 73% to 81%. In physical trials, the robot successfully grasped objects in 69.44% of attempts and completed the entire end-to-end imitation task without error in 44% of cases. These results validate the use of symbolic abstraction to bridge the gap between human demonstrations and robotic execution.

Conclusion

While the current system demonstrates the viability of one-shot imitation, limitations remain in grasping robustness for varied geometries and handling occlusions. Future work will focus on integrating dynamic grasping poses and improving 3D coordinate mapping for more diverse environments.

Summary updated

Abstract

Imitation can allow us to quickly gain an understanding of a new task. Through a demonstration, we can gain direct knowledge about which actions need to be performed and which goals they have. In this paper, we introduce a new approach to imitation learning that tackles the challenges of a robot imitating a human, such as the change in perspective and body schema. Our approach can use a single human demonstration to abstract information about the demonstrated task, and use that information to generalise and replicate it. We facilitate this ability by a new integration of two state-of-the-art methods: a diffusion action segmentation model to abstract temporal information from the demonstration and an open vocabulary object detector for spatial information. Furthermore, we refine the abstracted information and use symbolic reasoning to create an action plan utilising inverse kinematics, to allow the robot to imitate the demonstrated action.
BibTeX
@inproceedings{spisak2024robotic,
  author = {Josua Spisak and Matthias Kerzel and Stefan Wermter},
  title = {{Robotic Imitation of Human Actions}},
  booktitle = {ICDL},
  year = {2024},
  url = {https://openreview.net/forum?id=BQRcqi4NTj},
}
Figure