Abstract
Learning complex manipulation tasks from a single human demonstration remains a significant challenge in robotics due to visual noise, the invisibility of physical forces, and the difficulty of generalizing across objects. We present Parameterized Symbolic Abstraction Graphs (PSAG), a framework that converts video demonstrations into symbolic graphs grounded in physics simulations. By optimizing trajectories within a digital twin, the system estimates required forces and torques, enabling successful transfer to real-world tasks such as cutting, pouring, and rolling. Our method significantly outperforms traditional imitation learning baselines, particularly in force-sensitive manipulation.
Introduction
Learning from Demonstration (LfD) is a long-standing goal in robotics, promising to scale autonomous skills by allowing robots to learn from human videos. However, three major hurdles persist: visual noise from irrelevant backgrounds, the physical invisibility of contact forces, and the challenge of generalizing skills to objects with different shapes or material properties. While traditional methods often attempt to mimic raw trajectories, they lack the precision and physical grounding required for complex interactions like slicing or pouring.
Methodology
The PSAG framework addresses these challenges through a multi-stage pipeline that bridges the gap between pixels and physical actions.
1. Parameterized Symbolic Abstraction Graph (PSAG) Construction
The system first processes the input video using off-the-shelf vision models (e.g., Grounding DINO, Segment Anything) to identify and segment relevant objects. It constructs a symbolic graph where nodes represent objects with 6DOF poses and edges define spatial and contact relationships. To ensure spatial accuracy, a fine-tuning process is applied to depth estimation models to maintain temporal consistency across frames.
2. Learning to Simulate
The core innovation lies in the "Learning to Simulate" phase. The PSAG is imported into a physics simulator using the Moving Least Squares Material Point Method (MLS-MPM). The system optimizes the robot's trajectory within this digital twin to match the goals observed in the video while adhering to physical constraints. This process allows the robot to "uncover" the invisible forces and torques necessary to complete the task.
3. PSAG-to-Real Transfer
To generalize to new objects, the framework utilizes Non-uniform Normalized Object Coordinate Space (NUNOCS), mapping learned trajectories onto objects of varying scales and shapes. A hybrid motion-force controller then executes the optimized plan on a physical robot arm.
Experiments and Results
The framework was evaluated on five challenging tasks: cutting an avocado, cutting vegetables, pouring liquid, rolling dough, and slicing pizza. These tasks involve deformable materials and require precise force application. Human demonstrations were sourced from YouTube, and 3D models were retrieved from the Objaverse-XL dataset to build simulation environments.
| Task | Baselines (Avg. Success) | Ours (PSAG) |
|---|---|---|
| Cutting Avocado | 0% - 10% | 75% |
| Cutting Vegetable | 0% - 65% | 75% |
| Pouring Liquid | 5% - 80% | 80% |
| Rolling Dough | 0% - 25% | 70% |
| Slicing Pizza | 0% - 60% | 70% |
The results demonstrate that PSAG is particularly robust in force-sensitive tasks. While baselines like Interaction Warping or IRL might succeed in simple pouring, they largely fail in tasks like cutting an avocado, where understanding the interaction with the internal pit is crucial.
Discussion and Conclusion
The success of PSAG underscores the importance of force as a "missing link" in imitation learning. By using simulation as an intermediate teacher, the robot can ground visual data in physical reality, effectively filtering out visual noise.
However, limitations remain. The pipeline's performance is dependent on the accuracy of initial vision modules. Furthermore, manual effort is still required for hyperparameter tuning of simulation costs and fine-tuning material properties like viscosity. Future work will focus on automating simulation-to-real alignment and enhancing the robustness of the underlying perception system.
Summary updated