Safety Filtering in the Shared Latent Space of Cross-Embodiment Diffusion Policies

Robots increasingly learn from demonstrations collected on other bodies, such as human hands, dexterous robot hands, or different arms. One way to pool such data is to map every body's actions into a shared "latent action space", a compact learned coordinate system where semantically equivalent motions from different robots land in the same place. A diffusion policy (a generator that turns random noise into actions by repeated denoising) is then trained in that space, and a small per-robot decoder turns latent actions into joint commands. The weak point is deployment. The target robot usually has tighter joint limits, less reach, or weaker motors than the bodies that produced the data, so many latent actions decode into commands it cannot safely execute. The standard fix wraps the policy in a safety filter that minimally edits each unsafe command. Recent work shows this fix backfires for diffusion policies: the edit pushes the robot into states the policy never saw, and the policy then falls apart.

We propose to move the safety correction from the robot's joint space into the shared latent space. Physical constraints are pulled back through the target robot's decoder, and the projection is solved there. The corrected action is then still a latent action the policy recognizes, because some embodiment in the training data produced something like it. When no nearby latent correction exists, the system re-samples a different behavior mode from the diffusion policy. Only when all modes fail does it fall back to braking along the current path, which preserves a hard safety guarantee. The central prediction is that this beats both reactive joint-space filtering and sample-and-reject, and that the margin grows as the target robot's feasible set shrinks relative to the demonstrators'. That is, the benefit should scale with the size of the embodiment gap.

Core Claim

The closest existing pieces each solve part of this problem. Latent Action Diffusion for Cross-Embodiment Manipulation supplies the representation. It learns a 16-dimensional shared action space with contrastive encoders across a Franka gripper and two multi-fingered hands, and reports that co-training raises average success by 13.4% and block stacking by 25.3%. Its decoders have no safety mechanism, and decoded commands can violate joint limits or collide.

From Demonstrations to Safe Deployment: Path-Consistent Safety Filtering for Diffusion Policies, called PACS, motivates the proposal. It shows that reactive control barrier function filters steer diffusion policies off their data and lose up to 68% task success. Its remedy is to keep the policy's spatial path and only slow down along it. That remedy deadlocks when an obstacle permanently blocks the path, and PACS only considers a single arm.

SafeDiffuser and SafeFlowMatcher project generated trajectories onto safe sets during or after generative sampling. Both work in physical coordinates with analytic obstacles, on a single embodiment, and in open-loop planning.

Fisher-Preserving Guidance keeps guidance updates on the data manifold. Its manifold is statistical, and its safety notion is a proxy rather than a hard constraint.

No prior work combines a hard physical constraint check with a correction that lives in a cross-embodiment latent space. The claim is specific: for a fixed latent policy, the location of the minimal-intervention projection determines whether a safety edit causes the off-distribution collapse PACS documented. Projecting in the latent space plus mode re-sampling should recover steering ability without that collapse.

Research Program

Policy. We train a cross-embodiment latent diffusion policy in the style of Latent Action Diffusion. It outputs latent action chunks $z$ covering 16 future steps, executes 8, and then replans, as in Diffusion Policy. The target embodiment $e$ has a differentiable decoder $D_e$ mapping $z$ to a joint-command chunk.

Constraints. Constraints are written as functions $h_j(D_e(z)) \ge 0$ over the whole chunk, in the multi-step spirit of discrete-time barrier formulations (Safety-Critical MPC with Discrete-Time CBF). They include:

  • joint position and velocity limits;
  • a torque proxy from a rigid-body model;
  • self-collision and obstacle clearance computed by forward kinematics and signed distance.

Latent projection. Given the policy's proposed chunk $z_0$, the filter solves

$$ \min_{z} \; \|z - z_0\|_W^2 \quad \text{s.t.} \quad h_j(D_e(z_0)) + \nabla_z h_j(D_e(z_0))^\top (z - z_0) \ge \delta_j \;\; \forall j, \qquad \|z - z_0\| \le \rho . $$

Here $W$ is a weighting matrix, estimated from the training latents, that makes moves along high-density latent directions cheap and moves off them expensive. The term $\delta_j > 0$ is a margin that absorbs linearization error. The radius $\rho$ is a trust region that keeps the solution close to what the policy proposed. The problem is a convex quadratic program in 16 variables, so it should solve in well under a millisecond.

Exact check and fallbacks. The linearization carries no guarantee, so the decoded result is always re-checked exactly in physical space with set-based reachability, as in PACS.

  • If the check fails, or the quadratic program is infeasible, the policy draws new noise seeds. This selects a different behavior mode, for example going around the other side of an obstacle. Projection is retried on the new samples.
  • If every sample in the budget fails, the robot brakes along its current path using PACS-style braking. This last stage is what carries the safety guarantee. The latent stages exist to keep that stage rare.

Analysis. For every intervention we record how far the post-correction states and latents lie from the training support. We measure this as nearest-neighbor distance in latent space and as the policy's denoising error on its next input. This tests directly whether latent projection avoids the catch-22.

Embodiment-gap dial. We vary the target robot's joint limits, reach, and torque caps in controlled steps. This lets us chart intervention rate, re-sample rate, and success as a function of how much of the demonstration support the target body can actually execute.

Baselines

Unfiltered policy. This is the naive reference: the latent diffusion policy run unfiltered on the target robot. PACS reports that an unfiltered diffusion policy violated constraints in 56% of timesteps in its physical interaction tasks. We expect a similar pattern here, and worse under tightened limits.

Reactive joint-space filter. The policy's latent is decoded, then a standard barrier-function quadratic program edits the joint commands. This is the filter PACS reports losing up to 68% task success to in simulation. It shares our decoder, constraints, and solver, so the only difference is where the projection happens. This baseline doubles as the key ablation isolating the proposal's main ingredient.

PACS. This is the strongest existing method for filtering closed-loop diffusion policies: reachability-based braking along the decoded path. It reports 0% violations and 80% task success on real interaction tasks. We run it with the same reachability module, the same policy checkpoint, and identical obstacle scenarios. To beat it we must match its zero-violation rate and exceed its success rate on scenes with persistent blockages, without losing on scenes with only transient obstacles.

Best-of-N rejection sampling. We draw N latent chunks, decode them, and keep the safe one with the lowest intervention, falling back to braking if none is safe. This mirrors the rejection component of Risk-Guided Diffusion and the multi-sample filtering in Fisher-Preserving Guidance. We found no reported result on Lacuna for best-of-N rejection on cross-embodiment manipulation. It is matched to our method by wall-clock compute per control step, not by sample count, so N is set larger for the baseline.

Latent classifier guidance. Gradients of the constraint penalty are added during denoising in latent space. We tune it on a held-out scene set with the same budget as our margins.

Ablations. Two further ablations remove mode re-sampling and remove the density weighting $W$.

Why not just…?

Why not just sample many actions and reject the unsafe ones? This is the real threat, and the experiment is designed around it. When the target robot can execute most of the demonstrated behavior, a modest N will almost always contain a safe sample, and we expect best-of-N to roughly tie our method. Our prediction is that as the embodiment-gap dial tightens, the fraction of raw samples that decode feasibly collapses. Rejection then needs rapidly growing N. Latent projection, by contrast, can repair near-feasible samples that rejection would discard. If compute-matched best-of-N stays within noise of our method across the dial, we will report that the projection adds nothing and the contribution reduces to mode re-sampling plus braking.

Why not just filter in joint space after decoding, as usual? PACS's evidence is that this edit drives diffusion policies off their data and costs up to 68% success. Our out-of-support measurements test whether the latent edit avoids that drift. The comparison is clean because both filters share decoder, constraints, and solver.

Why not just use PACS braking? Braking is safe and keeps the policy on its data. It cannot route around an obstacle that stays put, and PACS itself names this deadlock. Our persistent-blockage scenes measure exactly this case. Braking remains the final fallback in our system, so we inherit its guarantee rather than compete with it.

Why not just guide the denoising with safety gradients? SafeDiffuser reports that classifier guidance drops Maze2D task score from about 1.5 to about 0.38. Risk-Guided Diffusion reports that guidance either still collides or freezes, depending on its weight. We include latent guidance as a baseline anyway, because guidance in a 16-dimensional latent space might behave differently than guidance in raw trajectory space.

Why not just fine-tune or train an RL refiner per robot, like CE-Nav? CE-Nav adapts a frozen flow expert to each body with PPO and reaches 0.8575 mean success on Go2. It uses only soft penalties, so it offers no hard guarantee. Per-robot fine-tuning is complementary to our approach: a fine-tuned policy would simply trigger fewer interventions. We will report a fine-tuned-on-target variant under our filter to show the two compose.

Proposed Experiments

Simulation. We build a robomimic-style manipulation suite (Lift, Can, Square) with Panda demonstrations plus at least one simulated multi-fingered hand. Deployment is on a held-out arm whose limits are tightened in five steps along the embodiment-gap dial.

Obstacle scenes. Each task gets two scene types. Transient obstacles cross the path and then leave, which is PACS's favorable regime. Persistent obstacles block the demonstrated route, so success requires switching modes.

Metrics. We report task success, timestep violation rate, deadlock rate, intervention magnitude, out-of-support distance after interventions, and control-step latency. The latency target is at least the 10 Hz rate Diffusion Policy reports for the policy itself, with the filter adding under 1 ms.

Hardware. We reproduce the block-stacking and pick-and-place tasks from Latent Action Diffusion on a Franka gripper, adding static and human-hand obstacles, and transfer a policy co-trained with hand data.

Decisive checks. There are three:

  1. Our system must match PACS's zero-violation rate.
  2. It must exceed PACS on persistent-blockage success.
  3. It must exceed compute-matched best-of-N, with the gap widening as limits tighten.

If the out-of-support distance after latent projection is not clearly below that of joint-space projection, the core mechanism is wrong regardless of success numbers.