Robots that learn from demonstrations by other bodies, such as human hands, differently sized arms, or a gripper instead of a stick, run into a mismatch that is partly about timing. A human flicks a light switch in half a second. A heavy industrial arm needs two seconds for the same motion. Yet most modern robot policies produce actions in fixed "chunks", blocks of a constant number of control steps (say 16 or 30) that are executed before the model is asked again. A fixed chunk ties what the robot does to how long it has been doing it. When the demonstrator and the robot move at different speeds, the same chunk boundary falls in a different phase of the task for each body: mid-reach for one, mid-grasp for the other. This note proposes that chunk boundaries be placed at object events, the moments when the manipulated object's situation changes: contact begins, contact ends, a drawer reaches its limit, debris crosses into a goal zone. These events look the same whichever body produced them. Within each chunk, the policy predicts a timing-free path that each embodiment then runs at its own speed.
In two sentences, the idea is this. The model is asked for new actions whenever the object's contact state is about to change, not every N steps. Human videos and robot data therefore line up event-by-event without time warping, and the timing of the motion itself is learned separately for each robot. The payoff we test is that a single policy trained on mixed-speed demonstrators transfers to a slower or faster robot without the drop in success that fixed chunks suffer, and at lower inference cost than methods that re-query the model at every step to decide when to replan.
What already exists and where this sits
Two literatures meet here. The first concerns rigid temporal segmentation. Work in sequential recommendation has repeatedly found that fixed time slices misrepresent bursty behaviour. Sequential Recommendation with Dual Side Neighbor-based Collaborative Relation Modeling shows static slices are too coarse for fast intent shifts and too fine for slow trends. Location-based activity recognition shows that a duration threshold cannot tell a meaningful stop from a traffic delay. These papers motivate the proposal rather than supply a method: duration is a poor proxy for meaning, and a robot slowed by its actuators is the traffic-delayed traveller.
In robot control the same problem appears as fixed action chunks, and three recent methods relax it within a single robot. Mixture of Horizons in Action Chunking (MoH) predicts chunks of several lengths in parallel and commits to a short prefix when those predictions disagree. Improving Generative Behavior Cloning via Self-Guidance and Adaptive Chunking (SGAC) replans when a freshly sampled chunk's cosine similarity to the executing plan drops, which requires inference at every step. Real-Time Robot Execution with Masked Action Chunking (REMAC) keeps the horizon fixed and trains the model to tolerate delayed prefixes. All three decide boundaries from the model's own action predictions, not from the task's physical state, and none is evaluated across embodiments. Our proposal borrows their premise that commitment length should vary. It replaces the trigger with one that is shared across bodies.
The second literature concerns the embodiment gap. XIRL and GraphIRL learn embodiment-invariant progress signals with temporal cycle-consistency, which pairs frames from two videos that sit at the same stage of the task. Both assume a single monotone progress curve and do not segment the task into phases. Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstration shows that tracking object pose alone removes morphology bias. We reuse its object-centric view, and its FoundationPose/SAM 2 pipeline, as the source of events from human video. Its reward, however, tracks the object trajectory continuously and in time rather than by phase.
The closest work is XSkill, which clusters human and robot clips into a fixed number $K$ of shared skill prototypes and aligns them at run time so the robot can proceed at its own pace. The proposal differs from XSkill in three concrete ways. Its boundaries come from observable object-state changes rather than learned clusters, so the number of segments per task is not a tuned hyperparameter. Each segment is also the policy's execution horizon, not a conditioning label for a clock-chunked diffusion policy. Finally, timing is modelled explicitly and per embodiment instead of being absorbed by alignment.
AnyBody constrains the design. Morphology-token transformers interpolate within known limb structures but fall to near zero on novel arms, and all embodiments there run on a shared fixed-frequency clock. We do not claim to fix morphological extrapolation. We claim to remove one confound, the timing mismatch, that current benchmarks never isolate.
Core Claim
Consider a demonstration from any embodiment $e$. It can be cut at object events $t_0 < t_1 < \dots < t_M$, where $M$ varies by episode. The sequence of object states at those cuts is shared across embodiments, even though the elapsed times $t_{k+1}-t_k$ are not. We therefore parameterise the policy's $k$-th chunk as a path over a normalised phase $s \in [0,1]$ and add a predicted duration:
$$a_t = \pi_\theta\!\left(s = \tfrac{t - t_k}{\hat\tau_{e,k}} \,\middle|\, o_{t_k}, z_e\right), \qquad \hat\tau_{e,k} = g_\phi(o_{t_k}, z_e).$$
Here $a_t$ is the command at time $t$, and $o_{t_k}$ is the observation at the segment start. The embodiment descriptor is $z_e$: a learned embedding, or joint velocity and torque limits when available. The policy network $\pi_\theta$ outputs the timing-free path, and $g_\phi$ is a small head that predicts how long embodiment $e$ will take for this segment. A separate lightweight event predictor, running on object state at every step, fires when the predicted object event occurs. It triggers the next query early if the event happens sooner than $\hat\tau$. If the event has not occurred by $1.5\hat\tau$, a timeout triggers a replan.
Human videos without actions still train the object-level event sequence and the phase-normalised end-effector path through retargeting. Robot data trains $g_\phi$ and the low-level path.
The falsifiable prediction is specific. Scale an embodiment's maximum velocity at test time by a factor of 0.5 or 2 relative to all training embodiments. Clock-chunked policies should lose success, because their chunk boundaries now fall in the wrong phase. The event-chunked policy should lose far less. It should also spend fewer model calls per episode than SGAC.
Research Program
The work has three parts.
The first builds the event labeller. In simulation it reads ground-truth contact and articulation state. For real human video it applies the object-pose pipeline of Human2Sim2Robot and marks events where the relative velocity and distance between hand and object cross a change-point, plus articulation joint limits. We will measure labeller agreement with simulator ground truth, since everything downstream depends on it.
The second trains the phase-parameterised diffusion policy with its duration head on mixed-embodiment data, with the event predictor trained jointly on object-state sequences.
The third is a controlled speed-mismatch protocol, which is the contribution we most expect others to reuse. Existing cross-embodiment benchmarks vary geometry but not dynamic bandwidth. We add velocity and acceleration scaling to each embodiment so timing mismatch can be studied separately from shape mismatch.
Proposed Experiments
We use four settings. The first is X-MAGICAL, the 2D sweeping task with three stick lengths and a gripper used by XIRL and GraphIRL, where events are debris-agent contacts and debris entering the goal zone. The second is the modified Franka Kitchen used by XSkill, with a sphere-agent demonstrator and a robot executor, where events are sub-task contacts and completions. The third is the AnyBody Push task across its procedural and industrial arms. The fourth is LIBERO-Long, as a single-embodiment check that event chunking does not cost in-domain performance. On each, we train on a subset of embodiments at their native speeds and test in three conditions: seen embodiments, the held-out embodiment, and every embodiment under 0.5× and 2× velocity scaling.
We report four quantities. Success rate is the main one. Model calls per episode and FLOPs measure inference cost. Boundary-phase consistency is the fraction of chunk boundaries that fall within a fixed window of a ground-truth object event. On the real robot we report completion rate.
A small real-world study follows XSkill's human-video-to-UR5 kitchen setup, with human demonstrations segmented by the video labeller.
Baselines
The naive reference is a diffusion policy with fixed 16-step chunks trained on the same mixed-embodiment data. It shows how much the timing mismatch costs when nothing is done.
The first strong competitor is XSkill, the most direct prior method for cross-embodiment multi-stage transfer. It reports 84.8% average success on unseen tasks in its modified Franka Kitchen, against 22.8% for a goal-conditioned diffusion baseline (XSkill). We run its released configuration with $K$ tuned on the same validation split we use for our own hyperparameters. Beating it means matching its native-speed success and holding up clearly better under velocity scaling, which XSkill's alignment transformer is meant to handle and which is therefore a fair test.
The second competitor is MoH, the strongest adaptive-horizon method. It reports 99% average LIBERO success with $\pi_{0.5}$ and up to 2.5× throughput over fixed-prefix execution (MoH). We reimplement its multi-horizon head on our backbone, so model size and data are identical.
The third is SGAC. It reports a 23.25% average improvement over vanilla diffusion policies on Robomimic and Kitchen and 16× fewer FLOPs than BID (SGAC). It is the natural cost comparison because it replans on detected change.
On the real robot, REMAC's 81.2% Grasp-Hard completion against 75.3% for real-time chunking (REMAC) indicates the gains latency-aware chunking gives. We include REMAC-style prefix masking as an orthogonal add-on rather than a rival.
No reported result was found on Lacuna for any of these methods under embodiment velocity scaling, or for MoH or SGAC across embodiments. Those numbers will be produced here under equal tuning budgets: the same number of hyperparameter trials, the same data, and matched parameter count.
Two ablations isolate the key ingredients. The first keeps the phase-parameterised policy but cuts chunks at fixed lengths equal to each task's mean segment length. The second keeps event boundaries but drops the duration head, executing each segment at a single shared speed.
Why not just…?
Why not just time-warp the demonstrations to the robot's speed and keep fixed chunks? A global speed factor or dynamic time warping fixes the average rate but not the phase-dependent one. Slow robots are slow mainly during large free-space motions, not during short contact phases. XIRL argues that trajectory alignment fails when embodiments traverse different intermediate states (XIRL). Still, a per-embodiment speed rescaling is cheap, so we include it as an extra baseline. If it closes most of the gap under uniform velocity scaling, we will narrow the claim to non-uniform bandwidth limits, such as acceleration caps that bite only on long reaches.
Why not just keep fixed chunks and replan whenever a contact sensor or proprioceptive threshold fires? This is the real threat, since it captures the event-trigger half of the idea with no learning. We run it directly: fixed 16-step chunks plus replanning on simulator contact changes. If it matches our method under speed scaling, then the benefit lies only in the trigger. The remaining contribution would be the training-time alignment of human and robot data, which we would then measure separately by removing human demonstrations. We expect the trigger alone to fall short because it reacts after the event rather than planning to it. It also gives no learning signal from action-free human video.
Why not just train MoH or SGAC on mixed-embodiment data? Their triggers come from agreement among the policy's own predictions, which says nothing about which embodiment is executing. Nothing in the file shows them transferring across bodies, and SGAC's per-step inference is costly. The MoH and SGAC baselines above answer this directly. If either matches us under speed scaling at similar cost, object events add nothing and we will say so.
Why not just condition a morphology-token transformer on embodiment and let it learn timing? AnyBody's near-zero extrapolation result (AnyBody) suggests implicit conditioning does not generalise beyond the seen range. That is exactly the 0.5× and 2× test conditions.
Decisive Checks
The proposal succeeds if three things hold. Under 0.5× and 2× velocity scaling, the event-chunked policy keeps most of its native-speed success while the fixed-chunk ablation and MoH do not. The fixed-length-segment ablation loses most of that robustness, showing that boundary placement rather than the phase parameterisation does the work. Model calls per episode sit well below SGAC's.
It fails if the contact-triggered fixed-chunk baseline matches it. It also fails if detected events from video agree poorly with ground truth, since real human video would then be out of reach. In that case the program would fall back to simulation-labelled events, and the real-robot claim would be dropped.
References
- Sequential Recommendation with Dual Side Neighbor-based Collaborative Relation Modeling
- Location-based activity recognition
- Mixture of Horizons in Action Chunking
- Improving Generative Behavior Cloning via Self-Guidance and Adaptive Chunking
- Real-Time Robot Execution with Masked Action Chunking
- XIRL: Cross-embodiment Inverse Reinforcement Learning
- Graph Inverse Reinforcement Learning from Diverse Videos
- Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstration
- XSkill: Cross Embodiment Skill Discovery
- AnyBody: A Benchmark Suite for Cross-Embodiment Manipulation