When a robot hand copies a human grasp, a solver searches for joint angles that put the robot's fingers and palm where the human's were touching the object. This is called retargeting. For contact-rich motions the search problem usually has several distinct good answers. The robot's middle finger might take over the role of the human's ring finger. The thumb might oppose from above or below. The wrist might roll one way or the other. Each answer sits in its own valley of the objective, and a local solver started in the wrong valley finishes with fingers passing through the object or pinches that miss. Our proposal is to treat the set of good answers for each frame as a mixture with a small number of components. We train a fast network that outputs $K$ candidate starting points, one per valley, instead of one. A standard local solver then polishes each candidate, and a cheap sequence decoder picks a consistent valley over time.
The reason to expect this to matter comes from mixture theory. A network trained to output a single warm start, using squared error on targets that fall into several modes, learns roughly the average of those modes. That average can lie in no valley at all. Training a $K$-output proposer then runs into the classic multi-component difficulty: with three or more components, EM-style fitting develops spurious optima in which components merge. We borrow the known partial remedy of fitting more components than exist, and we measure empirically how valley coverage scales with $K$ across hands whose shapes differ increasingly from the human hand.
Core Claim
Frame-level retargeting of contact-rich human motion onto non-human-shaped hands is a multimodal inverse problem. This multimodality has three consequences, and none of the existing retargeting pipelines addresses them together. First, single-output warm starts fail systematically on frames where several modes are present. Second, a $K$-output proposer, refined locally and decoded over time, can match the quality of an expensive independent per-frame search at a fraction of its cost. Third, the proposer's own training is sensitive to local optima in the way mixture theory predicts, so over-parameterizing $K$ beyond the true number of modes improves coverage.
Concretely, for a human frame $h$ with contact targets, let $E_h(q)$ be the retargeting objective over robot joint configuration $q$. $E_h$ sums a contact-chart matching term, a penetration penalty and a pinch-correspondence term. The proposer $f_\theta(h) = (q_1, \dots, q_K)$ is trained on solutions found by an expensive search, using a winner-take-all loss:
$$\mathcal{L}(\theta) = \mathbb{E}_h \left[ \min_{k \le K} \| q_k(h) - q^\star_{m(h)} \|^2 \right],$$
Here $q^\star_{m(h)}$ ranges over the distinct low-energy solutions found for $h$, and a soft version of the minimum is applied with annealing. The minimum is what stops the proposer from averaging modes. It is also exactly an EM-like hard assignment, which is why component collapse is expected and has to be studied rather than assumed away.
Grounding in Prior Work
Three papers motivate the gap. Kinematic Motion Retargeting for Contact-Rich Anthropomorphic Manipulations matches dense contact distributions on local surface charts across non-isometric hands, including a three-fingered hand. It reports that sequential inverse kinematics falls into local minima and drifts. Its response is to solve every frame independently from a coarse root estimate and then smooth with B-splines. That approach is offline and relies on hand-drawn axial curves. We keep its objective but replace brute-force independent solving with learned multi-valley proposals.
Geometric Retargeting (GeoRT) trains a single network that maps human to robot poses at 1000 Hz and deliberately penalizes curvature so the mapping is "flat." A single-valued smooth map cannot represent a choice between modes. GeoRT therefore represents the most plausible failure case we predict, and it is also our speed reference.
Analyzing Key Objectives in Human-to-Robot Retargeting shows that dropping the pinch term drives screw-picking success to 0%. Precision contact terms are what make the objective sharply multimodal, so they cannot be smoothed away.
The method comes from two sources. The first is Deep Learning Warm Starts for Trajectory Optimization on the International Space Station, which trains a network to place a non-convex trajectory optimizer inside its basin. It predicts one warm start, and our contribution is to show when one is not enough. The second is the mixture literature. Bad Local Optima of EM for Gaussian Mixtures shows that spurious optima with merged components appear at $k \ge 3$. Benefits of over-parameterization with EM shows that fitting extra components can avoid such traps, although only in stylized settings without recovery guarantees. We test whether that remedy transfers to a learned proposer on a physical configuration space. Mixture-of-experts robot models such as ECo-MoE and Being-H0.5 route between experts per embodiment and fight expert collapse with diversity losses. Our mixture instead operates per frame over solution modes within one embodiment, a different axis.
Two results act as constraints. Provable accelerated gradient method for nonconvex low rank optimization finds that basins shrink under ill-conditioning. Redundant finger chains near singular configurations are ill-conditioned, so proposals must be accurate and not merely diverse. Fast Stochastic Algorithms for SVD and PCA explains why random starts in high dimensions waste most of their iterations, which bears on the multi-start baseline. We make no claim to provable basin entry. Theory of this kind, for example Alternating Minimization for Mixed Linear Regression, exists only for $k=2$ linear models. Our scaling study is empirical.
Research Program
The work has four stages.
Build the oracle. For every frame of the GRAB-derived demonstrations (GRAB), we run a large multi-start search with the contact-chart objective for each target hand: Allegro, LEAP, Shadow, and the three-fingered hand from the kinematic retargeting paper. We cluster the low-energy solutions into modes. This gives us, per hand and per frame, a count of distinct valleys and their energies.
Train the proposer. Proposers are trained with $K \in \{1, 2, 4, 8, 16\}$, conditioned on the human hand pose and on object geometry near the contacts.
Refine and decode. Each proposal is refined with a fixed number of Gauss-Newton or quasi-Newton steps. A Viterbi decoder then chooses one refined candidate per frame. It minimizes the sum of frame energies plus a switching cost on joint-space jumps, which prevents the "snapping" that sequential solvers produce.
Analyze. We measure mode-coverage rate against $K$, against the non-isometry of the hand, and against the true mode count. We also measure collapse, defined as proposal heads within a small distance of one another, with and without over-parameterization.
Baselines
The naive reference is sequential inverse kinematics warm-started from the previous frame's solution, run with the same objective and the same per-frame iteration budget. This is what practitioners do by default, and the kinematic retargeting paper reports that it drifts and gets trapped. Beating it by a wide margin is necessary but not interesting.
The strongest quality baseline is the independent per-frame solver of Kinematic Motion Retargeting, using its coarse root estimation and B-spline smoothing on the same 30 GRAB demonstrations across six objects. That paper reports hand-object and self-intersection volumes "typically less than 5% of the hand volume." We found no reported runtime number on Lacuna, only that it requires substantial offline computation, so we will time it on our hardware. Beating it means matching its intersection volume and contact error while running at least an order of magnitude faster.
The strongest speed baseline is GeoRT, retrained per hand with its five losses. It reports 1000 Hz, motion preservation of 0.94 against 0.73 for prior methods, C-space coverage of 90%, and 87.5% real-world grasp success. It carries no contact-chart objective, so we also evaluate it as a warm start refined by our local solver.
The fair-compute baseline is random multi-start: $K$ random or perturbed-previous-frame starts refined with the same step budget as our $K$ proposals and decoded by the same Viterbi step. This baseline is what isolates the value of learned proposals.
Two ablations isolate the key ingredients. The first is the $K=1$ proposer, a single amortized warm start as in the ISS work, with identical architecture, data and refinement. It tests the mode-averaging claim directly. The second removes the Viterbi decoder by taking the per-frame lowest-energy candidate, which tests whether temporal mode selection matters. All methods share the objective, the refinement solver and the tuning budget, and all learned methods train on the same oracle labels.
Why not just…?
Why not just warm-start from the previous frame? The kinematic retargeting paper tested this and abandoned it because of drift and local-minimum traps. The noisy tracking reported in the key-objectives analysis makes such frame jumps worse. We include it anyway, because on slow, unimodal segments it may be competitive. We will report results separately for frames where the oracle finds one mode and frames where it finds several.
Why not just run many random restarts? This is the real threat. With enough restarts it recovers every valley, and it requires no training. The counter-argument is efficiency. Ill-conditioning shrinks basins (Provable accelerated gradient method), and random high-dimensional starts spend their iterations on flat regions (Fast Stochastic Algorithms for SVD and PCA). We will plot oracle-matching rate against total refinement iterations for both methods. If random multi-start at equal compute matches the proposer, the learned component is unnecessary, and we will report that result.
Why not just train one good warm-start network, or use GeoRT directly? A single output trained with squared error on multimodal targets converges toward the average of the modes. The flatness GeoRT imposes deliberately makes the map single-valued and smooth. The $K=1$ ablation tests this. On multi-mode frames we predict that the averaged prediction has higher energy after refinement than the best of its neighboring modes. If the oracle shows that frames are almost always unimodal for these hands, this argument collapses and a single warm start suffices.
Why not just use a convex relaxation? The only convex lifting in the file for mixed regression (A Convex Formulation for Mixed Regression with Two Components) is restricted to two components and linear models. Penetration constraints and contact-chart constraints are neither linear nor few in number, so no available relaxation fits this problem.
Proposed Experiments and Decisive Checks
The first check can kill the idea. Across hands, what fraction of GRAB frames have two or more oracle modes whose energies lie within a small tolerance of the best? If that fraction is small for Allegro and LEAP but large for the three-fingered hand, the method matters mainly for strongly non-human hands, which is a narrower but honest claim.
The primary metrics are hand-object and self-intersection volume as a fraction of hand volume, contact-distribution matching error, the fraction of frames that reach within a small margin of the oracle's best energy, and wall-clock time per frame.
The second check is the scaling curve. Oracle-matching rate is plotted against $K$ for each hand. We expect the curve to saturate once $K$ reaches the typical mode count. We further expect over-parameterized $K$ with annealed assignment to show less head collapse than $K$ set equal to the mode count, as the EM over-parameterization result suggests.
The third check is equal-compute comparison against random multi-start, as described in the Baselines section.
The downstream check replays retargeted demonstrations for the anthropomorphic hand in the DexMV MuJoCo relocate and pour tasks (DexMV). DexMV reports 67–100% success on relocate from its own retargeting pipeline, and reports that the DAPG baseline reaches 27.2% on pour. We ask whether demonstrations produced by our pipeline preserve or improve imitation success, compared with demonstrations from sequential inverse kinematics and from GeoRT. The pipeline should run fast enough for online use, and it should remain at or below the kinematic retargeting paper's intersection levels. If both hold, a practical and measurable gap between accuracy and speed in dexterous retargeting would be closed.
References
- Kinematic Motion Retargeting for Contact-Rich Anthropomorphic Manipulations
- Geometric Retargeting: A Principled, Ultrafast Neural Hand Retargeting Algorithm
- Analyzing Key Objectives in Human-to-Robot Retargeting for Dexterous Manipulation
- Deep Learning Warm Starts for Trajectory Optimization on the International Space Station
- Local Maxima in the Likelihood of Gaussian Mixture Models: Structural Results and Algorithmic Consequences
- Benefits of over-parameterization with EM
- ECo-MoE: Embodiment-Conditioned Mixture of Experts Increases the Evolvability of Robots
- Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalization
- Provable accelerated gradient method for nonconvex low rank optimization
- Fast Stochastic Algorithms for SVD and PCA: Convergence Properties and Convexity
- Alternating Minimization for Mixed Linear Regression
- GRAB: A Dataset of Whole-Body Human Grasping of Objects
- A Convex Formulation for Mixed Regression with Two Components: Minimax Optimal Rates
- DexMV: Imitation Learning for Dexterous Manipulation from Human Videos