Abstract
Imitation learning allows robots to acquire new skills by observing humans, yet it is often hindered by the perspective gap and the correspondence problem. This paper introduces a multi-stage architecture that enables a humanoid robot to perform "one-shot" imitation of complex tasks from a single video demonstration. The system leverages a generative diffusion model for temporal action segmentation and an open-vocabulary object detector (ViLD) to translate visual input into a symbolic plan. Experimental results on the NICOL robot demonstrate the feasibility of this abstraction-based approach in cluttered environments.
Introduction
Robotic imitation of human actions is a cornerstone of intuitive human-robot interaction. However, two primary challenges persist: the perspective gap, where the robot's viewpoint differs from the human's, and the correspondence problem, arising from the physical differences between human and robotic bodies. Traditional methods often rely on massive datasets or direct teleoperation. This research proposes a pipeline that bypasses these issues by abstracting human intent into a symbolic sequence of goals, allowing the robot to calculate its own trajectories using its unique body schema.
Methods
The proposed architecture consists of four key stages:
- Temporal Action Segmentation: A generative diffusion model, utilizing an ASFormer backbone, analyzes video frames to generate action labels (e.g., "moving with object"). The diffusion process effectively denoises the label sequence, ensuring smooth transitions.
- Spatial Object Detection: The ViLD open-vocabulary detector identifies objects based on text prompts, providing flexibility beyond pre-defined categories.
- Symbolic Reasoning: The system synthesizes temporal and spatial data to create a high-level plan, such as
Grasp(Object) -> Move(Target). - Execution: An Inverse Kinematics (IK) solver translates the symbolic plan into specific joint movements for the NICOL robot.
Experimental Setup
The system was evaluated using a custom dataset featuring five items from the YCB object set, including a red bowl, spam can, and tomato soup can. The setup involved 120 videos of a human performing pick-and-place tasks in a cluttered environment to simulate real-world complexity.
Results and Discussion
The diffusion-based segmentation model achieved 89.19% accuracy, significantly outperforming traditional classifiers in sequence stability. Object localization accuracy ranged from 73% to 81%. In physical trials, the robot successfully grasped objects in 69.44% of attempts and completed the entire end-to-end imitation task without error in 44% of cases. These results validate the use of symbolic abstraction to bridge the gap between human demonstrations and robotic execution.
Conclusion
While the current system demonstrates the viability of one-shot imitation, limitations remain in grasping robustness for varied geometries and handling occlusions. Future work will focus on integrating dynamic grasping poses and improving 3D coordinate mapping for more diverse environments.
Summary updated