Reference-Frame Tokens for Bimanual Human-to-Robot Transfer

When people use two hands, the hands rarely do the same job. One hand holds, steadies, or positions the work: the bread, the jar, the jacket hem. The other does the precise part relative to it: slicing, unscrewing, zipping. Interface designers have used this asymmetry for decades. Robot learning mostly ignores it when it turns human demonstrations into robot behavior. Current methods either compress both hands into one flat vocabulary of motion "tokens" (short discrete codes, like words, that stand for snippets of movement), or they retarget each hand's continuous joint angles onto a robot whose hands, arms, and body differ from a person's.

We propose a tokenizer that writes bimanual motion the way the task is organized. A coarse, slow vocabulary describes where the reference hand puts the frame. A second vocabulary describes what the manipulating hand does, expressed in that moving frame rather than in world coordinates. A small continuous correction then restores the sub-millimeter precision that contact requires. Which hand plays which role is inferred over time, not fixed in advance. Our bet is that this split puts the quantization error where the task tolerates it, in the stabilizing hand. It also makes the manipulation tokens largely indifferent to the robot's arm and body. Human bimanual data could then be decoded onto different robot hands, including mixed pairs such as a gripper paired with a dexterous hand, by swapping only small per-hand decoders.

Core Claim

Let $T^{r}_t$ be the 6-D pose of the reference hand's wrist at time $t$ and $T^{m}_t$ that of the manipulating hand. We tokenize the reference trajectory $T^{r}_{1:H}$ with a small codebook at a low temporal rate. We tokenize the manipulating hand's relative motion

$$x^{m}_t = \big(T^{r}_t\big)^{-1} T^{m}_t, \quad q^{m}_t,$$

where $q^{m}_t$ are its finger joint targets, with a larger codebook at a higher rate. A per-step role variable $\rho_t \in \{\text{left-ref}, \text{right-ref}, \text{symmetric}\}$ decides which hand is which. The symmetric value covers genuinely synergistic phases such as carrying a tray.

The claim has three testable parts. First, under this factorization, reference-hand trajectories in human bimanual data compress into far fewer codes than either hand does in world frame. Second, relative manipulation tokens extracted from the same human clip retargeted onto different robot hands agree with each other more than world-frame tokens do. Third, a policy that generates these tokens and decodes them through embodiment-specific heads with a contact-aware continuous residual outperforms monolithic discrete and continuous policies when transferring human bimanual demonstrations to dexterous robot hands. The margin should be largest on mixed-morphology pairs and on tasks where the hands swap roles.

How This Sits Against Prior Work

The motivating evidence comes from two separate camps. On the interface side, giving the non-dominant hand the frame and the dominant hand the precise action reduces mode switching and preserves context. Käser et al. do this for magnified views and Preim et al. for 3D editing. Splitting positioning from triggering across hands removes cursor jitter entirely (Wang). On the robot side, BUDS assigns one arm to stabilize and the other to act, using continuous keypoint and recurrent behavior-cloning models. That split reaches 76.9% task success, against an unstructured joint policy 56% lower. BUDS fixes roles per task, works with identical parallel-jaw arms, and never crosses from human hands to robots. We keep its asymmetry, but we make roles inferred and swappable and move the representation into a discrete vocabulary that can be learned from human data.

BiRP is the closest conceptual precedent. It parameterizes one arm's trajectory relative to the other through Gaussian mixture models (soft clusterings of trajectory points), deployed on a humanoid. It stays continuous, is costly on dense trajectories, and maps human demonstrations straight to Cartesian targets without handling multi-finger hands. We reuse its relative-frame insight as the coordinate system of a learned codebook.

The method comes from discrete tokenization. Discrete Policy trains a vector-quantized autoencoder (an encoder whose outputs snap to the nearest entry in a learned codebook) over robot actions, then generates codes with latent diffusion. It reports over 65% on real bimanual UR5 tasks against 37.5% for Diffusion Policy. Its codebook is monolithic and trained only on in-domain robot data. Human-motion tokenizers show why discretization helps: it acts as a filter that removes tracking jitter, as in BEST, and it avoids averaged, frozen poses, as in TM2T and the categorical latents of MeshTalk. Tokens can also be generated by language-model-style sequence models, as in MotionGPT. None of these factor the two hands by function.

The constraining evidence comes from embodiment-gap work. DexMV shows that matching fingertips alone makes robot fingers clip through objects. DexMachina shows that kinematic retargeting fails on articulated bimanual tasks without reinforcement-learning-based "functional" retargeting. ManipTrans needs a continuous, physics-aware residual on top of hand imitation to keep objects from dropping. These results rule out a purely discrete action path for contact, which is why our design ends in a continuous residual. They also supply our data pipeline: DexManipNet and ManipTrans-style simulated retargets give paired human-and-robot versions of the same motion. AdaMorph decodes a continuous, morphology-agnostic intent into 12 humanoids via embodiment-conditioned normalization. We borrow its idea of small embodiment-specific decoders, but for bimanual hand-object manipulation rather than whole-body locomotion. TwinVLA and DECO both decouple bimanual control into components. TwinVLA uses symmetric twin arms with cross-attention; DECO separates sensing modalities. Neither gives one hand the job of defining the other's coordinate frame.

Research Program

Data. We first build paired data. Human bimanual clips come from ARCTIC and the human sources compiled into DexManipNet. Each clip is retargeted onto Shadow, Inspire, and Allegro hands using the ManipTrans residual pipeline. Only clips whose simulated rollout succeeds are kept, so every human clip has physically valid robot counterparts.

Role labels. We label roles weakly. For each window, the reference hand is the one whose wrist moves slower in the object's frame while in contact with a different part than the other hand. When the two hands move together rigidly, the window is labeled symmetric. These heuristic labels only initialize a role classifier. The tokenizer and the classifier are then trained jointly: the role that gives lower reconstruction error under the factored codebooks is the label, iterated in an expectation-maximization style. This lets roles swap mid-clip, for example on ARCTIC's articulated objects, where one hand anchors the base and the other opens the lid, then they trade.

Tokenizer. The reference codebook is small and runs at a low rate, because stabilizing motion should be slow and low-entropy. The manipulation codebook runs at a higher rate over relative wrist pose and finger targets.

Decoders and residual. Each embodiment gets its own decoder head per hand. Reference decoders map frame tokens to arm and wrist targets. Manipulation decoders map relative tokens to finger joints for that hand. A continuous residual policy, trained in simulation and conditioned on tokens, object pose, and contact distances in the ManipTrans style, corrects the decoded command. The residual is deliberately limited in magnitude, so the tokens must carry the motion and the residual only repairs contact.

Planner. On top sits an autoregressive transformer that generates role, reference, and manipulation tokens from object state and, where available, vision. It is trained on human-derived tokens. Robot data is used only to fit decoders and the residual.

Analysis. We measure codebook usage and per-role entropy. We also measure cross-embodiment token agreement: the fraction of timesteps where the same human clip, encoded from two different robot retargets, yields the same manipulation token.

Baselines

The naive reference point is direct kinematic replay. The human motion is retargeted with multi-point inverse kinematics in the DexMV style and played open-loop on each robot hand. It shows how much of the gap is pure geometry. No reported result found on Lacuna for open-loop kinematic replay on DexManipNet bimanual tasks. The nearest number is ManipTrans's standard reinforcement-learning baseline at 12.1% bimanual success.

The strongest transfer competitor is ManipTrans itself. It reports 39.5% success on complex bimanual DexManipNet tasks (ManipTrans). It is strongest because it shares our residual-on-imitation structure and solves the same human-to-dexterous-hand problem. We run it on the identical task split and hands with its published training budget per trajectory, and give our residual the same simulation steps. ManipTrans is a per-trajectory transfer method, not a policy that generalizes to new clips. We therefore report both settings: transfer of held-in clips, where beating 39.5% is the bar, and held-out clips, where ManipTrans must be retrained. In the held-out setting we count it as a ceiling that our single trained model should approach without per-clip training.

The strongest discrete competitor is Discrete Policy: one flat vector-quantized codebook over both hands' actions, decoded by latent diffusion. We train it on the same human-derived data, with the same total codebook size as our two codebooks combined and the same residual. Its reported number, over 65% on its real bimanual UR5 suite against 37.5% for Diffusion Policy, comes from a different setup and cannot be compared directly. We reproduce it on ours.

For continuous policies, we train Diffusion Policy and the DECO transformer on the same data. The bar comes from DECO-50, where DECO reports 72.25% vision-only, ACT 57.25%, and Diffusion Policy 51.25% (DECO).

BUDS-style fixed roles is an ablation that keeps our architecture but freezes the role assignment per task.

The ablations that isolate the key ingredient are the following:

  1. A monolithic codebook over relative-frame actions. This uses the same coordinate change as ours but a single shared vocabulary.
  2. Symmetric per-hand codebooks in world frame.
  3. Removing the residual.
  4. Removing the role variable.

Every method gets the same data, the same simulation budget for any residual or reinforcement-learning stage, and an equal hyperparameter search of a fixed number of trials. Beating a baseline means a success-rate difference that holds across three seeds with non-overlapping 95% bootstrap intervals over episodes.

Why not just…?

Why not just express both hands relative to each other and keep one codebook? This is the most serious threat. The relative coordinate change may carry most of the benefit, and Discrete Policy already shows a flat codebook works on bimanual robots. The monolithic-relative ablation tests this directly. Our prediction is that the two match on fixed-role, single-embodiment tasks. Factoring should win in three cases: when roles swap, when the two robot hands differ in morphology (a single codebook couples both hands' decoders), and as the task count grows, where Discrete Policy itself reports codebook complexity growing. If the monolithic-relative variant matches us in all three, we will report that the frame, not the factorization, is the contribution.

Why not just use a continuous diffusion policy on relative-frame actions? Continuous diffusion avoids quantization error, and DECO reaches 72.25% on dexterous bimanual data without tokens. Against it, Discrete Policy reports roughly a 26-point margin over Diffusion Policy on bimanual UR5 and shows entangled latents as tasks scale. In our setting, tokens are also what make human data reusable across decoders. We run Diffusion Policy and DECO with relative-frame inputs. If they match on held-out clips and on cross-embodiment transfer, the discrete layer is not earning its place.

Why not just retarget with inverse kinematics and add a residual, as ManipTrans does? That works per clip, but it needs retraining for every new trajectory and every new hand, and it has no planner that can compose new sequences. DexMachina and ManipTrans both show that pure kinematics fails on contact. The experiment that settles this is held-out clips and a held-out hand: we decode to an embodiment whose decoder is fitted on only a small fraction of clips, against ManipTrans retrained per clip.

Why not just track object flow and ignore the hands? Im2Flow2Act reaches 81% zero-shot on single-arm tasks this way. In two-handed dexterous grasps, however, the hands occlude the object, which is why DECO needed touch sensing for a 20-point gain. OKAMI also beat an object-only baseline by 58.3% by retargeting hand pose. We include an object-flow-conditioned variant of our planner without hand tokens to measure this on our tasks.

Proposed Experiments

The first check is a cheap analysis on ARCTIC before any policy training. If reference-hand trajectories do not need markedly fewer codes than manipulation trajectories at equal reconstruction error, the core premise fails and we stop. The second check measures cross-embodiment token agreement for relative versus world-frame manipulation tokens on clips retargeted to Shadow, Inspire, and Allegro. Relative tokens must agree substantially more often, or the transfer story collapses.

The main experiment trains on human-derived tokens and evaluates success on held-in and held-out DexManipNet bimanual tasks and on the DexMachina articulated-object benchmark. For DexMachina, no reported numeric success rate was found on Lacuna; the file states only qualitative gains on Waffleiron-300, so we report its method rerun on our split. Results are broken down by three task conditions: fixed-role, role-swapping (ARCTIC articulated objects), and mixed-morphology pairs (Allegro paired with Inspire).

Cross-embodiment breadth is tested on RoboTwin 2.0 across its five gripper embodiments, where the reference decoder is swapped per robot. No reported policy success number was found on Lacuna for a direct comparison; the file gives only relative improvements. A small real-world confirmation on DECO-50's Assembly and Waste Disposal subtasks compares against DECO's reported 72.25% vision-only figure.

The decisive result would be our method beating both the monolithic-relative ablation and DECO on role-swapping and mixed-morphology tasks while staying close on fixed-role ones. A uniform gain would point instead to a generic capacity effect, and a gain only on fixed-role tasks would mean the role machinery adds nothing.

References

  1. FingerGlass: efficient multiscale interaction on multitouch screens
  2. A 3d Puzzle for Learning Anatomy
  3. Enabling User Interactions with Video Contents
  4. Stabilize to Act: Learning to Coordinate for Bimanual Manipulation
  5. BiRP: Learning Robot Generalized Bimanual Coordination Using Relative Parameterization Method on Human Demonstration
  6. Discrete Policy: Learning Disentangled Action Space for Multi-Task Robotic Manipulation
  7. BEST: BERT Pre-training for Sign Language Recognition with Coupling Tokenization
  8. TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts
  9. MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement
  10. MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators
  11. DexMV: Imitation Learning for Dexterous Manipulation from Human Videos
  12. DexMachina: Functional Retargeting for Bimanual Dexterous Manipulation
  13. ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning
  14. AdaMorph: Unified Motion Retargeting via Embodiment-Aware Adaptive Transformers
  15. TwinVLA: Data-Efficient Bimanual Manipulation with Twin Single-Arm Vision-Language-Action Models
  16. DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter
  17. ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation
  18. Flow as the Cross-domain Manipulation Interface
  19. OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation
  20. RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation