Zero-Shot Compositional Generalization for Embodied Manipulation

Baicheng Li, Wenkang Qin, Youqi Pan, Dongxiao Mao, Yukun Zhou, Chuanfu Shen, Dong Wu, Shunkai Zhou, Zike Yan, Wei Sui, Hongbin ZhaarXiv 2026 · arXiv

TL;DR

This paper introduces a geometry-supervised framework that enables robots to perform tasks they have never seen before by "borrowing" knowledge from other robots. By using calibrated forward kinematics to map joint-space actions into a shared tool-space, the authors create a universal action interface that allows a visual policy to generalize across different robot embodiments and skills in a zero-shot manner.

What Problem Does This Paper Address?

In the world of robotics, data is often "siloed." One dataset might show a Franka arm stacking bowls, while another shows a UR5 opening a microwave. Because these robots have different joint configurations (kinematic chains), the raw joint-level commands from one cannot be directly used by the other.

The challenge is compositional generalization: if Robot A knows Skill 1 and Robot B knows Skill 2, can we get Robot B to perform Skill 1 without collecting new demonstrations? Standard methods that pool data often struggle because joint labels remain tied to specific robots. This paper tackles the "missing edges" in the robot-task graph, aiming for zero-shot transfer where a robot executes a familiar skill it has never personally demonstrated.

Figure 1. Known robots and skills, unseen combinations. Three illustrated training pairings (blue labels) leave six unseen robot–task combinations (orange outlined labels). Rows show Franka Panda, UR5 and Piper. Columns show three skills. Generated conceptual images illustrate the setting.

Methods

The authors propose a two-stage training framework centered on a shared action interface:

  1. Action Interface Pretraining: They train a Variational Autoencoder (VAE) where a shared encoder maps any robot's joint actions into a 32-dimensional latent space ($z$). Crucially, they use cross-robot geometric supervision. If Robot A performs a task, its tool pose (calculated via Forward Kinematics, or FK) serves as the ground truth target for Robot B's decoder. Robot B's decoder is trained to produce joint angles that result in that same tool pose, provided the target is reachable.
  2. Visual Policy Learning: With the action interface frozen, they train a visual-language policy (using backbones like $\pi_{0.5}$ or Diffusion-based models) to predict actions in the shared latent space $z$. Because the latent space is grounded in universal tool-space targets, the policy can be trained on all available data and then executed through any robot's native decoder.

A key technical detail is the use of native reconstruction to anchor decoders to their demonstrated configurations, preventing the "ambiguity" that arises when multiple joint configurations can reach the same tool pose.

Figure 2 shows the supervision path: (A) aligning actions through geometry and (B) training the visual policy.

Datasets

The researchers evaluated their method using two primary simulation environments:

  • RoboTwin: A benchmark featuring heterogeneous robots (ALOHA, ARX-X5, Franka, UR5, Piper) and diverse manipulation tasks (e.g., shoe placement, bread in skillet, microwave opening).
  • ManiSkill3: Used specifically for a cube-pushing task to test portability across different simulators.

The training sets were intentionally designed to leave specific robot-task combinations unobserved to test zero-shot capabilities.

Key Insights and Ideas

  • Tool-Space is the Universal Language: While joint angles are embodiment-specific, the effect of an action (the tool's position and orientation) is universal. Calibrated geometry (FK) provides the "Rosetta Stone" needed to translate skills between robots.
  • Geometry as Supervision, Not Just Representation: Unlike methods that simply use End-Effector (EEF) coordinates as input, this method uses geometry to supervise the learning of native joint decoders. This allows the robot to retain its specific kinematic "style" while following a universal command.
  • Native Reconstruction is Essential: Simply training on tool poses isn't enough because robots are redundant (many ways to reach one point). Reconstructing the original joint demonstrations helps the model learn a "prior" for how each specific robot naturally moves.
  • Reachability Filtering: You can't supervise a robot to reach a target it physically can't. The framework uses numerical Inverse Kinematics (IK) as a "witness" to select only valid targets for cross-robot supervision.

Key Results

The proposed "Full" pipeline significantly outperformed baseline native joint prediction on held-out (unseen) robot-task combinations:

  • RoboTwin Performance: On the $\pi_{0.5}$ backbone, the method improved held-out success rates from 17.0% to 33.6%. For the FastWAM backbone, the jump was even more dramatic: from 9.7% to 29.8%.
  • ManiSkill3 Transfer: In the ManiSkill3 pushing task, the framework achieved 28.6%–43.5% held-out success, whereas native joint policies virtually failed, reaching only 3.1%–11.6%.
  • Speedup: The learned native decoders provided a 1.71× median speedup in command generation compared to using a standard numerical IK solver at runtime.
Figure 6 shows successful zero-shot executions on held-out combinations in both RoboTwin and ManiSkill3.

Limitations and Future Work

  • Calibration Dependence: The method relies on accurate kinematic descriptions and installation transforms. Errors in calibration would propagate through the FK supervision.
  • Reachability Constraints: The transfer is limited to the intersection of the robots' workspaces. If a task requires a reach that Robot B cannot physically achieve, the transfer will fail.
  • Dynamic Tasks: The current study focuses on quasi-static manipulation. Future work could explore how these geometric priors handle high-speed, dynamic tasks where joint velocities and torques become more critical than just end-effector poses.
  • Real-World Transfer: While the results in simulation are robust, the authors suggest that testing on physical hardware with real-world sensing noise is a logical next step.

Summary updated

Abstract

Robot demonstrations are collected on different embodiments, leaving many robot-skill combinations unobserved. Pooling these data shares visual knowledge, while joint labels remain tied to the demonstrating kinematic chain. We propose a geometry-supervised training framework for zero-shot composition of familiar robots and skills. The central insight is that a demonstrated command provides a tool-space target for every robot that can realize it. Calibrated forward kinematics supplies this supervision to foreign native action interfaces without their joint labels for that skill. Native reconstruction retains demonstrated configurations, and positive reachability checks select targets for cross-robot supervision. A visual policy learns to predict shared actions and executes through the target robot's native decoder. During policy learning, one prediction receives geometric supervision through both source and foreign native realizations. Matched supervision controls, repeated policy training and evaluations across three backbones and two simulators demonstrate improved transfer to missing robot-task combinations.
BibTeX
@misc{li2026zero,
  author = {Baicheng Li and Wenkang Qin and Youqi Pan and Dongxiao Mao and Yukun Zhou and Chuanfu Shen and Dong Wu and Shunkai Zhou and Zike Yan and Wei Sui and Hongbin Zha},
  title = {{Zero-Shot Compositional Generalization for Embodied Manipulation}},
  year = {2026},
  eprint = {2611.22797},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22797},
}
Figure