Motion retargeting takes a movement performed by one body, such as a human dancer, and reproduces it on a body with different proportions or even a different joint layout, such as a cartoon character, a robot arm or a quadruped. When no paired examples exist, the standard training signal comes from unpaired image-to-image translation. That signal is cycle consistency: translate the motion to the target body and back, and require that you recover what you started with. This rule quietly assumes the source motion was fully observed. Real captures from monocular video are not. Hands vanish behind objects, legs leave the frame, and a person turning away hides half their joints for seconds at a time. Applied naively, the cycle rule then asks the model to reproduce missing or garbage coordinates. It also punishes any plausible completion that happens to differ from the zeros a tracker emitted.
We propose to train the translator so that the round trip is scored only on what was actually observed. The backward translator, which maps target motion back to the source body, is a stochastic generator rather than a deterministic function. It can therefore express that many source motions are consistent with one retargeted motion. Where the source is occluded, the model fills in using the target domain's own sense of plausible motion, learned adversarially from complete target motions. The prediction we want to test is specific. Such an observation-masked, one-to-many cycle should beat both the current masked-token retargeter and the obvious "impute first, then retarget" pipeline. The gain should appear once occlusions of the limbs that actually drive the target last long, from tens of frames up to seconds. For brief occlusions it should buy nothing. Temporal scale enters as the axis along which that crossover is measured, and as a two-rate encoder needed to bridge long gaps.
Why this gap is real
Three findings in the literature motivate the work. First, unconstrained cycle-consistent translation fails badly on skeletons. Unsupervised Motion Retargeting for Human-Robot Imitation applies CycleGAN and UNIT with graph convolutions to human-to-robot retargeting on Mixamo. It reports 243 mm and 209 mm error against 79 mm for a training-free rotation copy, and concludes that cycle consistency alone under-determines motion content. Second, structure rescues unpaired retargeting. Skeleton-Aware Networks for Deep Motion Retargeting pools homeomorphic skeletons (skeletons that are the same graph up to bone subdivision) into a shared "primal skeleton" and reaches a cross-structural error of 2.25. Motion and Appearance Adaptation for Cross-domain Motion Transfer preserves relative joint angles and reaches FID 61.7 against 175.9 for a first-order motion model. Both papers assume clean keypoints. Third, masked modelling handles missing joints. Skeleton-Aware Motion Retargeting Using Masked Pose Modeling (MPM) tokenizes joints as quaternions inside a "super-skeleton," retargets across non-homeomorphic bodies with a skeleton-and-collision error of 0.134 against 0.216 for skeleton-aware networks, and tolerates up to 10 masked joints. A Unified Masked Autoencoder with Patchified Skeletons for Motion Synthesis (UNIMASK-M) completes and forecasts human motion under 20% joint occlusion, but only within one domain.
MPM is the closest existing work, and it constrains the proposal. It already shows that masking and non-homeomorphic retargeting coexist. Its reported robustness, however, concerns masked joints rather than long, structured occlusions such as a whole limb missing for many frames or entire dropped frames. It also does not change what the unpaired training objective rewards when inputs are incomplete. A second line of evidence constrains the design. Quantification of Occlusion Handling Capability of a 3D Human Pose Estimation Framework shows that explicit 0/1 visibility flags combined with a long temporal receptive field matter enormously. Its dilated temporal network holds 68.6 mm on Human3.6M with 16 of 17 joints missing, while VideoPose3D degrades past 418 mm. So visibility must be an input, and long context is not optional.
Core Claim
The cycle-consistency objective itself is the wrong object under occlusion, and two fixes are needed together. The first fix scores the reconstruction only on observed joint-frames. The second makes the backward direction one-to-many, which removes the pressure on the forward translator to smuggle unrecoverable source detail through the target motion. This smuggling is the "steganography" failure known from unpaired image translation under information asymmetry.
Concretely, let $x \in \mathbb{R}^{T \times J \times d}$ be a source motion of $T$ frames and $J$ joints with $d$-dimensional rotations, and let $m \in \{0,1\}^{T \times J}$ be its visibility mask. The forward translator $G$ maps the visible input $(x \odot m, m)$ to a complete target motion $y$. A backward conditional generator $p_\psi(x \mid y)$ outputs a distribution over source motions. The masked cycle loss is
$$ \mathcal{L}_{\text{cyc}} = -\,\mathbb{E}\Big[\textstyle\sum_{t,j} m_{t,j}\,\log p_\psi\big(x_{t,j} \mid G(x \odot m, m) + \epsilon\big)\Big], $$
where $\epsilon$ is small noise at the scale of target-domain capture jitter. It is added to destroy low-amplitude hidden codes in $y$. An adversarial loss on $y$ against complete target-domain motions supplies the prior that determines $y$ where $m=0$. We keep the structural invariants that prior work showed are necessary: MPM-style super-skeleton tokens and relative joint angles. The contribution is the objective, not a new backbone.
Research Program
We will build the translator on an MPM-style transformer over joint tokens with visibility flags appended. We will add a two-rate temporal encoder that pairs a fine, short-stride pathway with a coarse, long-stride pathway, in the spirit of the micro/macro split in Spatio-temporal action detection and localization using a hierarchical LSTM. This addresses a specific problem. A single-rate window either cannot see across a two-second limb occlusion, or it blurs the sub-second contact events on either side of it. This is the dilution-versus-blindspot trade-off described in Direction 3 of the literature. The backward generator is a lightweight conditional masked model that samples source completions. We then study the objective in three settings, each with a different amount of target ground truth: Mixamo human-to-human characters, Mixamo human-to-robot following the robot setup of the imitation paper above, and human-to-quadruped following MPM.
Two analyses are central. The first sweeps occlusion type and duration. Types are random joints, fixed body part and full-frame dropout, following the protocols of the quantification paper. Durations run from 1 to roughly 120 frames. We separately occlude "driving" limbs, which map onto target degrees of freedom, and "irrelevant" limbs, which do not. The second is a steganography probe. We perturb the translated motion with noise of increasing amplitude and measure how fast cycle reconstruction collapses. A model that hides information collapses at tiny amplitudes, while one that does not degrades smoothly.
Baselines
The naive reference point is rotation copy with visibility-aware linear interpolation. Occluded rotations are linearly interpolated from the nearest visible frames, then copied to the target where joints correspond. It costs nothing to train. Unsupervised Motion Retargeting for Human-Robot Imitation reports 79 mm for plain rotation copy on clean Mixamo human-to-robot, so any learned method that does not beat it on clean data is disqualified before occlusion is even considered. This baseline has no definition for non-homeomorphic targets, so it is reported only on the human-to-human and human-to-robot splits.
The strongest existing method is MPM (Skeleton-Aware Motion Retargeting Using Masked Pose Modeling). It is fed the same occlusion masks through its mask tokens and retrained on our splits with the same backbone size, training steps and hyperparameter search budget as our model. Its reported Mixamo skeleton-and-collision error is 0.134. Our first check is that our reimplementation reproduces a number near that on clean data. Beating MPM means lower normalized retargeting error on occlusion lengths above roughly one second, with no regression within noise on clean inputs.
The strongest pipeline competitor is impute-then-retarget. UNIMASK-M (A Unified Masked Autoencoder with Patchified Skeletons for Motion Synthesis) is trained on source-domain motion to complete occluded inputs, and clean-input MPM then retargets the result. It gets the same total parameter and compute budget as our single model. For completeness we also report skeleton-aware networks on the homeomorphic splits (cross-structural error 2.25, intra-structural 2.76 in Skeleton-Aware Networks for Deep Motion Retargeting), and deterministic CycleGAN, reported at 243 mm, as the lower anchor.
Four ablations isolate the key ingredients. The first replaces the masked cycle with a full cycle on zero-filled inputs. The second makes the backward generator deterministic. The third removes the noise $\epsilon$. The fourth collapses the two-rate encoder to a single rate with matched parameters. If the full-cycle and deterministic variants perform as well as the full model, the core claim is false.
Why not just…?
Why not just impute the missing joints first and then retarget clean motion? This is the real threat, and we treat it as a co-equal hypothesis. For short occlusions the two-stage pipeline should match us, because source-domain completion is nearly unambiguous there. The difference should appear for long occlusions of driving limbs. There, UNIMASK-M must commit to one completion using only human-motion statistics, and its errors then propagate through the retargeter. Our model resolves the ambiguity with the target domain's prior. It also ignores occlusion of limbs that the target never uses. The duration sweep, run separately on driving and irrelevant limbs, settles this. If the pipeline matches us at every duration, we will say so, and the contribution shrinks to the objective-level analysis and the steganography probe.
Why not just feed masks to MPM, which already handles occluded joints? MPM's reported robustness covers up to 10 masked joints. The file contains no reported result for long body-part or full-frame occlusions. More importantly, whatever its unpaired objective is, it is not scored per observed entry and one-to-many. The ablation that swaps our masked stochastic cycle into an otherwise identical MPM-style backbone isolates exactly this difference. If that ablation shows no gain, the objective is not what matters.
Why not just copy rotations and smooth over gaps? On clean, near-isomorphic skeletons rotation copy is hard to beat at 79 mm. The quantification paper, however, shows that temporal models without visibility guidance collapse under heavy occlusion, degrading past 418 mm. Linear filling cannot recover a limb that stays hidden for dozens of frames. Rotation copy also has no definition for non-homeomorphic targets. We report it anyway on every split where it applies, so readers can see where learning is actually needed.
Why not just use a standard CycleGAN on skeleton coordinates? The file already answers this: 243 mm on Mixamo, three times worse than rotation copy. We include it only to confirm that the steganography probe detects information hiding in a model known to be under-determined.
Proposed Experiments
We train all models unpaired on Mixamo character motions. Held-out same-motion pairs across characters serve only as evaluation ground truth, following the protocol of skeleton-aware networks and the robot-imitation paper. Occlusions are applied at test time with the three protocols above, over the duration grid, and separately on driving and irrelevant limbs. For real rather than synthetic occlusion, we lift monocular Human3.6M video with detected 2D keypoints, keep the detector's own visibility flags, and retarget to Mixamo characters. Without ground truth there, we report bone-length violation, foot skating and the target-domain discriminator score.
The metrics are height-normalized retargeting error and millimetre error on the observed and occluded portions separately, skeleton-and-collision error, and the collapse amplitude of the steganography probe. For quadrupeds we report MPM's plausibility measures and a small human preference study.
The decisive result is a crossover plot of retargeting error against occlusion duration for our model, MPM and impute-then-retarget. The claim holds if three things are true together. First, the curves agree within noise below about ten frames. Second, our model wins clearly beyond about one second of driving-limb occlusion. Third, the masked-cycle and stochastic-backward ablations each lose most of that win. The claim fails if the win is absent, or if it comes only from the two-rate encoder, in which case the finding is that temporal context rather than the objective matters.
References
- Unsupervised Motion Retargeting for Human-Robot Imitation
- Skeleton-aware networks for deep motion retargeting
- Motion and Appearance Adaptation for Cross-domain Motion Transfer
- Skeleton-Aware Motion Retargeting Using Masked Pose Modeling
- A Unified Masked Autoencoder with Patchified Skeletons for Motion Synthesis
- Quantification of Occlusion Handling Capability of a 3D Human Pose Estimation Framework
- Spatio-temporal action detection and localization using a hierarchical LSTM