TL;DR
This paper introduces a geometry-supervised framework that enables robots to perform tasks they have never seen before by "borrowing" knowledge from other robots. By using calibrated forward kinematics to map joint-space actions into a shared tool-space, the authors create a universal action interface that allows a visual policy to generalize across different robot embodiments and skills in a zero-shot manner.
What Problem Does This Paper Address?
In the world of robotics, data is often "siloed." One dataset might show a Franka arm stacking bowls, while another shows a UR5 opening a microwave. Because these robots have different joint configurations (kinematic chains), the raw joint-level commands from one cannot be directly used by the other.
The challenge is compositional generalization: if Robot A knows Skill 1 and Robot B knows Skill 2, can we get Robot B to perform Skill 1 without collecting new demonstrations? Standard methods that pool data often struggle because joint labels remain tied to specific robots. This paper tackles the "missing edges" in the robot-task graph, aiming for zero-shot transfer where a robot executes a familiar skill it has never personally demonstrated.
Methods
The authors propose a two-stage training framework centered on a shared action interface:
- Action Interface Pretraining: They train a Variational Autoencoder (VAE) where a shared encoder maps any robot's joint actions into a 32-dimensional latent space ($z$). Crucially, they use cross-robot geometric supervision. If Robot A performs a task, its tool pose (calculated via Forward Kinematics, or FK) serves as the ground truth target for Robot B's decoder. Robot B's decoder is trained to produce joint angles that result in that same tool pose, provided the target is reachable.
- Visual Policy Learning: With the action interface frozen, they train a visual-language policy (using backbones like $\pi_{0.5}$ or Diffusion-based models) to predict actions in the shared latent space $z$. Because the latent space is grounded in universal tool-space targets, the policy can be trained on all available data and then executed through any robot's native decoder.
A key technical detail is the use of native reconstruction to anchor decoders to their demonstrated configurations, preventing the "ambiguity" that arises when multiple joint configurations can reach the same tool pose.
Datasets
The researchers evaluated their method using two primary simulation environments:
- RoboTwin: A benchmark featuring heterogeneous robots (ALOHA, ARX-X5, Franka, UR5, Piper) and diverse manipulation tasks (e.g., shoe placement, bread in skillet, microwave opening).
- ManiSkill3: Used specifically for a cube-pushing task to test portability across different simulators.
The training sets were intentionally designed to leave specific robot-task combinations unobserved to test zero-shot capabilities.
Key Insights and Ideas
- Tool-Space is the Universal Language: While joint angles are embodiment-specific, the effect of an action (the tool's position and orientation) is universal. Calibrated geometry (FK) provides the "Rosetta Stone" needed to translate skills between robots.
- Geometry as Supervision, Not Just Representation: Unlike methods that simply use End-Effector (EEF) coordinates as input, this method uses geometry to supervise the learning of native joint decoders. This allows the robot to retain its specific kinematic "style" while following a universal command.
- Native Reconstruction is Essential: Simply training on tool poses isn't enough because robots are redundant (many ways to reach one point). Reconstructing the original joint demonstrations helps the model learn a "prior" for how each specific robot naturally moves.
- Reachability Filtering: You can't supervise a robot to reach a target it physically can't. The framework uses numerical Inverse Kinematics (IK) as a "witness" to select only valid targets for cross-robot supervision.
Key Results
The proposed "Full" pipeline significantly outperformed baseline native joint prediction on held-out (unseen) robot-task combinations:
- RoboTwin Performance: On the $\pi_{0.5}$ backbone, the method improved held-out success rates from 17.0% to 33.6%. For the FastWAM backbone, the jump was even more dramatic: from 9.7% to 29.8%.
- ManiSkill3 Transfer: In the ManiSkill3 pushing task, the framework achieved 28.6%–43.5% held-out success, whereas native joint policies virtually failed, reaching only 3.1%–11.6%.
- Speedup: The learned native decoders provided a 1.71× median speedup in command generation compared to using a standard numerical IK solver at runtime.
Limitations and Future Work
- Calibration Dependence: The method relies on accurate kinematic descriptions and installation transforms. Errors in calibration would propagate through the FK supervision.
- Reachability Constraints: The transfer is limited to the intersection of the robots' workspaces. If a task requires a reach that Robot B cannot physically achieve, the transfer will fail.
- Dynamic Tasks: The current study focuses on quasi-static manipulation. Future work could explore how these geometric priors handle high-speed, dynamic tasks where joint velocities and torques become more critical than just end-effector poses.
- Real-World Transfer: While the results in simulation are robust, the authors suggest that testing on physical hardware with real-world sensing noise is a logical next step.
Summary updated