Spending Compute Only Where Bodies Disagree: Geometry-Gated Joint Tokens for Real-Time Retargeting

Motion retargeting maps a movement recorded on one body onto another body, such as a human dancer onto a shorter robot or a cartoon character. Copying the source's joint angles works when the two bodies have similar proportions and the motion has no contacts. It fails where limb lengths differ along a chain that ends at a contact: feet slide, hands sink into hips, pinches miss. Learned retargeters fix these failures by running a transformer over every joint in every frame. A transformer here is a network that treats each joint-at-a-frame as a "token" and lets all tokens exchange information. Most of that work is spent on joints for which angle-copying was already correct.

We propose to route computation by body geometry. A cheap gate, computed from how much the source and target bone lengths disagree along each joint's chain and from whether that chain is near a contact, decides which joint tokens pass through the transformer. The remaining tokens keep the copied angle. The transformer's job becomes predicting corrections only where the bodies disagree. Our claim is that this gives a better accuracy-per-millisecond curve on embedded hardware than generic compression. We also expect it to produce smoother output from noisy real-world pose input, because unlike attention-based token selection, the gate does not change when the sensor noise does.

Core Claim

Retargeting error is concentrated in a small and predictable set of joint-frames. That set can be identified from known geometry, not from learned attention scores. The file supports this indirectly. It reports that parent-relative joint rotations already preserve joint bends regardless of bone lengths (Motion and Appearance Adaptation for Cross-domain Motion Transfer), and that the failures cluster at end-effectors and contacts (Contact-Aware Retargeting of Skinned Motion; Semantics2Hands; Retargeting Matters).

For joint $j$ at frame $t$, let $L_s(j)$ and $L_t(j)$ be the summed source and target bone lengths from $j$ to the tip of its chain. Let $c_{j,t}$ be a contact probability predicted by a small per-joint network. The gate score is

$$ s_{j,t} = \left|\log \frac{L_t(j)}{L_s(j)}\right| + \lambda\, c_{j,t}, $$

where $\lambda$ weights contact against proportion mismatch. The top-$k$ tokens per frame are activated, with $k$ set by the latency budget. Target joints that have no source counterpart, such as a tail or extra legs in a cross-topology transfer, get $s=\infty$ and are always active. The output rotation is

$$ \theta_{j,t} = \theta^{\text{copy}}_{j,t} + g_{j,t}\,\Delta_{j,t}, $$

where $\theta^{\text{copy}}$ is the parent-relative copied rotation, $g_{j,t}\in\{0,1\}$ is the gate, and $\Delta_{j,t}$ is the transformer's correction.

Idle tokens are not deleted. Following the idle-and-reactivate design of No Token Left Behind, which lets unselected image tokens skip attention and rejoin deeper layers, idle joints still serve as keys and values so that active joints can see the whole body. Only active joints are updated as queries.

Relation to Prior Work

Three works supply the method. Skeleton-Aware Motion Retargeting Using Masked Pose Modeling treats joints as tokens over a "super-skeleton" that contains the joints of all training topologies, and runs at 23 FPS. We reuse its tokenization. We differ in never processing all joints and in predicting corrections rather than full rotations. IdleViT supplies reversible idling. Its selection is driven by attention and was developed for image classification, and the file notes that token reduction is essentially untested on spatio-temporal structure. FQ-ViT supplies integer quantization of the softmax and layer normalization operations, which we apply on top of gating.

Other works motivate the proposal. HUMOS and the contact-aware retargeting paper show that body shape and contact are where naive transfer breaks. Both spend full model capacity everywhere, and the contact-aware method adds test-time optimization. Semantic-Aware Motion Encoding for Topology-Agnostic Character Animation handles differing skeletons with joint tags generated by a large language model. Its reported failure mode, implausible poses when the source motion exceeds what the target body can do, is one our copy-plus-correction structure should dampen, because corrections are bounded residuals and not free generation.

Two works constrain the design. Unified Locomotion Transformer shows that a single transformer can run at 50 Hz on a Jetson AGX Orin for one fixed quadruped. We adopt that board and that rate target but leave the control policy untouched. Adversarially Masking Synthetic to Mimic Real shows that noise injection helps synthetic-to-real transfer, so we train with sensor-like noise and dropped joints on the source stream.

Research Program

We will build the gated retargeter on the super-skeleton tokenization. It will be trained on Mixamo and AT-HumanML3D/AT-AniMo4D for cross-character and cross-topology transfer, and on LaFAN1 and AMASS for human-to-Unitree-G1 transfer. The training loss combines rotation and end-effector position terms, a foot-contact sliding penalty, and a mesh interpenetration term borrowed from the contact-aware retargeting work.

The gate uses no learned parameters beyond the contact head. Source bone lengths come from the first second of a noisy stream and are then held fixed, so the gate does not flicker with frame-to-frame noise.

The analysis has three parts. First, a budget sweep over $k$, measuring the accuracy-latency curve on a Jetson Orin in FP16 and in integer precision using the FQ-ViT operators. Second, an error-localization study: on the full-compute model, measure what fraction of total error comes from the top-$k$ geometric-score joints. This is the empirical test of the core claim. Third, a noise study that feeds edge-quality pose estimates and synthetic tracker noise into the retargeter and measures jerk.

For the robot setting, retargeted G1 motions are tracked in simulation with sensor noise and latency. This follows the Sim2Sim protocol of Retargeting Matters.

Baselines

The naive reference is parent-relative rotation copying with per-segment scaling and no learned correction. It runs essentially for free and sets the lower bound on accuracy and the upper bound on speed. Beating it on contact and penetration metrics is required, but it is not the goal.

The strongest learned competitor is full-compute masked pose modeling. On Mixamo it reports a Skeleton and Collisions Error of 0.134 against 0.216 for SAN, and runs at 23 FPS (Skeleton-Aware Motion Retargeting Using Masked Pose Modeling). We will retrain it on identical data with the same architecture width, the same number of training steps and the same tuning budget, and deploy it on the same Jetson with the same FP16 and integer pipeline. Beating it means matching its Skeleton and Collisions Error within 5% relative while running at least twice as fast. Equivalently, it means lower error at equal latency.

For cross-topology transfer, the reference is the SATA autoencoder, which reports a root error of 0.9583 and a joint-position error of 1.3570 on AT-HumanML3D (Semantic-Aware Motion Encoding for Topology-Agnostic Character Animation). We will compare under its reported protocol.

For mesh penetration, the reference is the contact-aware retargeter's interpenetration error of 0.81, against 1.32 for SAN+IK (Contact-Aware Retargeting of Skinned Motion). It uses test-time optimization, so we report it both with and without a latency cap.

For the G1 setting, the reference is GMR, an optimization-based inverse kinematics method that matches end-effector positions. The file reports only qualitative results for it: "near-perfect" Sim2Sim success on Kung Fu and Dance, while the PHC baseline drops to 0%. No reported numerical result was found on Lacuna for GMR position or rotation error. We will run it on the same 21 LaFAN1 sequences and report tracking success, position error and rotation error. Beating GMR is not the goal. The target is matching its tracking success while running online on the edge board.

Two compression baselines isolate the key ingredient. The first uses the same backbone with IdleViT's attention-driven token selection at an identical keep ratio. The second uses structured pruning via SAViT at identical FLOPs. A random-gate ablation at the same $k$ separates the benefit of geometric gating from the benefit of simply computing less.

Why not just…?

Why not just quantize the full transformer and skip gating? This is the most serious threat. A skeleton has only a few dozen joints, so attention may not dominate latency at all. The budget sweep will show directly whether FP16 or integer full-compute already meets 50 Hz on the Orin with acceptable error. If it does, the efficiency half of the claim fails. The proposal would then reduce to the error-localization and noise-smoothness findings, and we would report it that way. Gating is only worth keeping if it helps at long temporal windows or on the smaller Orin Nano, where FastPose-ViT found even FP16 transformers limited to about 13 FPS sequentially.

Why not just use inverse kinematics with per-segment scaling, as GMR does? For human-to-humanoid transfer this may be competitive, and the G1 comparison will say so. It has two known limits. It has no defined mapping across non-matching topologies (A high-order graph convolutional network for homomorphic and heterogeneous skeletal motion retargeting). And on skinned characters, SAN+IK leaves interpenetration at 1.32 against 0.81 for a geometry-aware learned method. The Mixamo and AT-AniMo4D experiments address both limits.

Why not just use IdleViT's attention-based selection? Attention scores depend on the input, so with noisy real-world poses the selected set changes from frame to frame. We expect that switching to appear as joint jitter, which is exactly the question Gap 3 raises. The matched-keep-ratio comparison, run on clean and noisy input and scored by jerk and foot sliding, settles this. If attention gating matches ours on both, the geometric prior adds nothing and we will say so.

Proposed Experiments and Decisive Checks

The first decisive check is error localization. In the full-compute model, the top 25% of joint-frames by geometric score should account for most of the correction magnitude and most of the residual error when corrections are removed. If they do not, the premise fails.

The second is the Pareto check. On the Jetson Orin, the gated model should dominate full-compute masked pose modeling, attention-based idling and SAViT pruning in Skeleton and Collisions Error versus latency on Mixamo, and in joint-position error versus latency on AT-HumanML3D.

The third is noise robustness. Under injected tracker noise and dropped joints, the gated model should show lower jerk than attention-gated idling at equal budget, with no loss in foot-sliding percentage. For reference, HUMOS reports 7.37% foot sliding against over 20% for its baselines on AMASS.

The fourth is the downstream check. Online-retargeted G1 motions, tracked in simulation with noise and latency, should reach tracking success comparable to offline GMR.

A single negative outcome is also informative. If the random gate at equal $k$ performs as well as the geometric gate, then retargeting is simply over-provisioned, and the right conclusion is a smaller model rather than a smarter router.