When the Hand Forgives: Planning Against Multi-Modal Forecasts with a Compliant Gripper

A robot catching an object from a human, or grasping something whose future position is uncertain, has to guess where the object will be. Trajectory predictors that output several possible futures can already make that guess. They are built to avoid the "averaging trap": a model trained to minimize squared error predicts the mean of all futures, which can be a path through an obstacle or a grasp at a point the object never reaches. Soft and underactuated grippers have a separate, physical property. Fingers made of silicone, springs, or tendon-coupled links fold around an object even when the hand arrives centimeters off target. One measured soft hand tolerates a misplacement region of 50–66 cm² (Robust proprioceptive grasping with a soft robot hand). This proposal argues that the two facts should be combined in the decision step, not just stacked. When the gripper's tolerance region is larger than the distance between two predicted futures, the robot should not pick one future. It should aim at the point whose tolerance region covers the most predicted probability. In that regime, aiming "between modes" is no longer a mistake.

The idea in two sentences: measure, for a given compliant hand, the set of object positions it can grasp from each commanded pose (its "capture funnel"). Then, rather than executing the most likely predicted future, choose the commanded pose and commit time that maximize the predicted probability falling inside that funnel. The payoff is earlier and more reliable commitment in dynamic tasks such as human-to-robot handover. Rigid hands cannot do this, because their funnel is too small to cover more than one mode.

Core Claim

Let $p(x \mid h)$ be a multi-modal predictor's distribution over the object pose $x$ at grasp time, given observed history $h$. Let $s(a, x) \in [0,1]$ be the probability that the hand, commanded to grasp pose $a$, succeeds when the object is actually at $x$. We call $s$ the capture funnel and estimate it once per hand in simulation and on hardware. The proposed action is

$$ a^* = \arg\max_a \; \mathbb{E}_{x \sim p(x \mid h)}\big[\, s(a, x) \,\big], $$

the pose that captures the most predicted probability. Standard practice instead takes $a = \text{mean of the top mode}$. The claim is that this hand-aware choice beats top-mode execution by a margin that grows with the ratio of funnel radius to mode separation. The margin should vanish for rigid grippers and should also vanish when modes sit farther apart than the funnel can reach. That gives a falsifiable prediction about where the benefit lives, not only whether it exists.

A second, optional ingredient follows from the same quantity. The predictor can be fine-tuned with the loss $-\log \mathbb{E}_{x \sim p}[s(a^*, x_{\text{true}})]$, which rewards the predictor only for distinctions the hand cannot absorb. This targets a known weakness of mixture models, which must fix the number of modes $K$ in advance (HOME). Under a hand-aware loss, modes closer together than the funnel radius do not need to be separated at all.

Relation to Prior Work

The closest work falls into three groups.

The first group motivates the gap. In the DexH2R handover benchmark, human motion forecasting and diffusion-based approach policies are run on a rigid ShadowHand. The policies struggle in the tighter-alignment "Hard Mode", which the file attributes to the rigid fingers' lack of positional tolerance. An LSTM with a mixture-density output, reported in From Virtual Demonstration to Real-World Manipulation Using LSTM and MDN, reaches 95% push-to-pose success in simulation but falls to 60% on a real Baxter. The authors blame unmodeled friction. Neither paper changes the end-effector, and neither feeds hand tolerance back into how a predicted mode is chosen.

The second group supplies evidence that compliance absorbs error. Consensus-Driven Uncertainty for Robotic Grasping based on RGB Perception finds tendon-driven underactuated hands more forgiving of pose-estimation errors than parallel jaws. It treats only static, single-shot grasps, so time and multi-modal futures are absent. Surprisingly Robust In-Hand Manipulation shows that soft contact compresses many initial poses into few outcomes without any sensing. We borrow its "funnel" notion and make it a quantity measured per hand.

The third group supplies the forecasting machinery. CVAE-based predictors such as Trajectron++ and Robot Trajectron handle multi-modal futures. Robot Trajectron applies prediction to shared control on a rigid Franka in free space. CoverNet recasts prediction as classification over a feasible trajectory set. Our decision step resembles CoverNet, except that the "set" is defined by what the hand can physically capture rather than by vehicle kinematics.

Contact-Aware Neural Dynamics forecasts object motion through contact with a diffusion model on a rigid tactile hand. Adaptive Compliance Policy predicts stiffness for active impedance control. Both put compliance or contact into the learned model. We instead keep the predictor hand-agnostic and put the hardware's physical tolerance into the decision rule.

Research Program

The work has three parts.

Funnel measurement. For three end-effectors, we measure $s(a,x)$ over a grid of offsets between the commanded grasp pose and the true object pose, crossed with object identity and approach velocity. The three hands are a rigid parallel-jaw gripper, a tendon-driven underactuated hand, and a soft pneumatic hand. We first do this in MuJoCo using the rigid and underactuated hand models of the consensus-uncertainty study. We then repeat a sparse subset on hardware and fit a small classifier on the combined data. Including velocity matters because handover objects are moving. The open question is whether simulated funnels for soft hands transfer, given the simulation gap noted for hyperelastic materials in Contact-Aware Neural Dynamics. We report the gap between simulated and real funnels explicitly, since it is a result in its own right.

Hand-aware decision rule. We train a multi-modal forecaster on DexH2R's human hand and object trajectories (4,282 trials, 56 objects). We use a Trajectron++-style CVAE, chosen for speed, and add a diffusion variant as a check. At each control step we compute $a^*$ by sampling poses from the predictor and scoring candidate commands. We also optimize the commit time: the robot commits once $\max_a \mathbb{E}[s]$ exceeds a threshold. This lets a forgiving hand commit before the human's intent fully resolves.

Hand-aware fine-tuning. We fine-tune the predictor with the capture loss above and test whether it needs fewer mixture components for equal capture rate.

Baselines

The naive reference point is a single-output regression predictor trained with squared error, executed on each hand. It shows how much of the gain comes from multi-modality at all. The file reports LSTM-MSE at 0% on virtual push-to-pose against 95% for LSTM-MDN (From Virtual Demonstration…). We expect it to fail on rigid hands and to be partly rescued by soft ones.

The strongest existing competitor for handover approach generation is 3D Diffusion Policy (DP3) as benchmarked in DexH2R. It is the method that best balanced success and safety in Hard Mode there. No reported success-rate number appears in the file: I found no reported result on Lacuna for DP3's numerical Hard Mode success. We therefore retrain DP3 ourselves on the same DexH2R split. It gets the same GPU budget as our predictor plus decision rule, so we cap DP3 near the file's reported 5 GPU days rather than giving it more. Both systems get the same hyperparameter-search budget, and DP3 runs on each hand. Beating it means higher handover success at equal or earlier commit time on the soft and underactuated hands, with no loss of the safety metric DexH2R defines.

The key ablation is top-mode execution: the same predictor on the same compliant hand, aiming at the most likely mode. This isolates the decision rule from the hardware. A second ablation is a heuristic hedge, which aims at the probability-weighted mean of the modes within a fixed radius. This tests whether the learned funnel $s$ matters or a hand-agnostic radius suffices.

Why not just…?

Why not just put a soft hand on an existing predictor and execute the top mode? This is the most serious threat. The hand alone may already absorb most of the error. The top-mode ablation answers it directly. If hand-aware selection gives no gain over top-mode execution once modes are closer together than the funnel radius, the decision rule is unnecessary and we will report that. The proposal's own geometry predicts the gain should appear only in that regime, so a null result elsewhere is expected rather than damning.

Why not just use a better predictor on a rigid hand? DexH2R's Hard Mode struggles suggest that even diffusion policies cannot match rigid-finger tolerances in dynamic handover. Better prediction also cannot resolve intent the human has not yet committed to. Our commit-time analysis tests this. If a rigid hand with DP3 matches a compliant hand at the same commit time, our claim fails.

Why not just use active impedance control on a rigid arm? Adaptive Compliance Policy shows active stiffness greatly helps contact tasks, at 96% against 14% for a stiff baseline on flipping. But it regulates force after contact. It does not widen the region of positions from which a grasp closes successfully, which is what covering two modes requires. We include a Franka with impedance control as a fourth "hand" in the funnel measurement. If its funnel is as wide as the soft hand's, the passive hardware is not needed and the decision rule still applies.

Why not just rely on open-loop funnels with no prediction? The RBO Hand 3 results depend on static, pre-authored keyframes. A handover target moves and changes. Running the funnel-only policy, which grasps at the current object pose with no forecast, shows how much prediction contributes on a moving target.

Proposed Experiments

The simulation study replays DexH2R object trajectories in MuJoCo against the three hands. We use a 2×3 design: rigid, underactuated, or soft hand, crossed with naive regression, top-mode, or hand-aware selection, plus DP3 and the heuristic hedge. The primary metric is grasp success. Secondary metrics are commit time before the human releases the object, and minimum average displacement error, reported to show that capture rate and displacement error can disagree.

The decisive check plots the gain of hand-aware over top-mode selection against the ratio of funnel radius to predicted mode separation. That separation is computed per trial from the predictor's components. The claim predicts near-zero gain for rigid hands at all ratios, and a rising gain for compliant hands once the ratio exceeds about one. A flat curve falsifies the idea.

A physical study with roughly ten participants handing over a subset of YCB objects repeats the top-mode versus hand-aware comparison on the underactuated and soft hands. Its purpose is to verify that simulated funnels predict real ones.

A smaller transfer check revisits the push-to-pose setting where an LSTM with mixture-density output dropped from 95% to 60% on hardware. It tests whether a series-elastic tool interface, of the kind Suh used to absorb positioning overshoot, combined with funnel-aware selection recovers part of that drop.

Finally, the fine-tuning experiment reports capture rate against the number of mixture components $K$, with and without the capture loss. The prediction is that hand-aware training reaches its plateau at smaller $K$ for compliant hands.