Robot hands with 16 to 30 motors can be steered in two ways today. Planners look ahead several steps and check that each imagined motion obeys physics, but they need exact object coordinates. Reactive policies map camera images straight to motor commands, but they never look ahead. The best dexterous planner in the literature, DexHandDiff, samples future hand and object trajectories with a diffusion model, a generator that learns to turn random noise into realistic samples step by step. It then throws out "ghost states," imagined futures in which the object moves even though no finger touches it. Its authors note that this filter needs ground-truth hand and object coordinates and leave vision as future work. Systems that work from raw images and touch, such as DECO, act reactively and make no feasibility check at all.
This note proposes a way to join the two. The question a ghost-state filter really asks is whether the fingers are touching the object while it moves. A fingertip touch sensor measures exactly that. Object coordinates are only an indirect route to the answer. So the planner keeps a learned, image-derived description of the object, a short learned code vector rather than a 3D pose. It takes the hand's state exactly from joint encoders. It replaces the coordinate-based contact critic with a contact predictor trained on recorded touch signals. Planning happens over this mixed space, and any imagined future in which the object code changes while predicted contact is near zero is penalized. Touch therefore provides free contact labels at training time, and the planner never needs object coordinates at test time.
Core Claim
A diffusion planner that sees the object only through pixels, and whose contact-consistency check is learned from touch, will recover most of the robustness to shifted goals that the ground-truth-state planner of DexDiffuser shows. It will also clearly beat reactive visuo-tactile policies when the test goal differs from the training goal. The new ingredient is narrow and testable: touch supervises a contact predictor inside the planner. Prior work instead uses touch as an extra policy input (DECO, ADEPT) or uses geometric coordinates as the contact check (DexHandDiff, the collision-aware corrector in DROP).
Research Program
The state the planner reasons over has two parts. The hand part $h_t$ is the measured joint configuration. It is exact and needs no learning, which follows the lesson of MoDex that the hand's own body should be modeled separately from the task. The object part $z_t$ is a low-dimensional code produced by an image encoder. We train the encoder to predict its own future codes given actions, in the spirit of SALE's learned state-action embeddings (For SALE), and not to reconstruct pixels. The Virtues of Laziness in Model-based RL shows that pixel-likelihood models spend their capacity on task-irrelevant detail, which motivates this choice. Self-prediction alone can collapse to trivial codes, so we add a small task-progress prediction head as an anchor and test the effect in an ablation.
A contact head $\hat c_t = \sigma(g(h_t, z_t))$ predicts the probability that each finger is touching the object. Here $g$ is a small network and $\sigma$ is the sigmoid. It is trained only against binarized touch-sensor readings from the demonstration data. Touch is never fed to the planner at test time except to update the current state. The planner is a diffusion model over short future chunks of hand joint deltas and object codes. Each sampled plan is scored by
$$ S(\tau) = \hat V(\tau) - \lambda \sum_{t} \lVert z_{t+1} - z_t \rVert \,\big(1 - \max_f \hat c_{t,f}\big), $$
In this score, $\tau$ is the sampled chunk, $\hat V$ is a learned estimate of task progress toward the commanded goal, $f$ indexes fingers, and $\lambda$ trades off progress against contact consistency. The second term is the touch-supervised replacement for DexHandDiff's coordinate-based dynamics critic. It penalizes object motion that happens without predicted contact. Actions are chunk-wise joint-space deltas. Demystifying Action Space Design shows that this format bounds error growth on 6-DoF arms and explicitly leaves high-DoF hands open, so our runs also test whether that finding carries over. The planner replans after each executed chunk, as in DexHandDiff, and keeps DexHandDiff's split between pre-contact and post-contact phases.
Baselines
The oracle reference is DexHandDiff run on ground-truth hand and object state, using its published configuration on the goal-shifted Adroit and Shadow Hand tasks. The file reports 59.2% average success on goal-shifted tasks against 29.5% for the next best baseline, and 70.7% averaged over all tasks (DexHandDiff). We do not expect to beat this number, since it uses information we withhold. Our success criterion is to close at least two-thirds of the gap between the reactive visual baseline and this oracle on goal-shifted tasks.
The naive reference point is a standard visuomotor Diffusion Policy trained on the same demonstrations, with images and proprioception as input. The file reports its success on 30° door opening falling to 16.7% against 70% for interaction-aware planning (DexDiffuser).
The strongest perception-side competitor is a DECO-style decoupled transformer that receives vision, proprioception and touch through its adapter (DECO). On DECO-50 it reaches 82.50% average success with touch and 72.25% without. It is the strongest system that consumes the same sensors as ours. We retrain it on our simulated data with the same demonstrations, the same encoder backbone size, and the same hyperparameter-search budget, measured in GPU-hours. Beating it means higher goal-shifted success at equal in-distribution success.
The strongest pipeline competitor estimates object pose from the same images, using a learned keypoint or pose regressor trained on the same frames with simulator labels, and feeds that estimate into the unmodified DexHandDiff. This is the reviewer's obvious alternative, and it gets its own discussion below.
Two ablations isolate the key ingredient. The first removes the contact term, setting $\lambda = 0$. The second keeps the term but trains $g$ against contacts inferred from the object code alone, with no touch labels. If the second ablation matches the full method, touch is not doing the work and the claim fails.
Why not just…?
Why not just estimate the object pose and run DexHandDiff on it? This is the most serious threat. In simulation, a pose regressor with clean renders may be accurate enough that the pipeline matches the oracle. DROP's experience suggests otherwise once fingers occlude the object. Its vision-based keypoints produced finger-object interpenetration, and removing its geometric corrector cost 33% of rotations (DROP). The comparison will be settled by stratifying test episodes according to the fraction of object pixels hidden by the hand. If the pose pipeline matches our method even at high occlusion, our contribution shrinks to "touch adds robustness only marginally," and we will report it that way. The method also needs no pose labels or object meshes, which matters for deformable or unseen objects. That advantage is real, but it is secondary to the occlusion test.
Why not just add touch to a reactive policy, as DECO and ADEPT do? Both show that touch helps reactive control. DECO gains 20 absolute points on contact-rich tasks, and ADEPT reaches 8/10 on asymmetric peg insertion. Neither looks ahead, and the goal-shift collapse of reactive diffusion policies (16.7% at 30° door) suggests that touch as an input does not fix extrapolation to new goals. The tactile-augmented DECO baseline under goal shift tests exactly this. If it closes the gap, planning is not needed.
Why not just run model-predictive control in the learned latent space? Continuous optimization in visual latent spaces drifts off the data and fails: 2.1% against 92.1% for a latent roadmap on hard box stacking (Enabling Visual Action Planning). We use a diffusion sampler, which stays near the demonstration distribution, and we include cross-entropy-method shooting over the same latent model as a further baseline to confirm this point in the dexterous setting.
Proposed Experiments
The primary testbed is the goal-shifted Adroit and Shadow Hand suite used by DexDiffuser and DexHandDiff (door, pen reorientation, hammer). We add rendered camera images and per-fingertip simulated touch readouts from the simulator. Metrics are success rate on in-distribution and shifted goals and the ghost-state rate, defined as the fraction of executed chunks in which the simulator registers object motion with no hand contact. We also report the L2 gap between planned and simulated states, which DexHandDiff halved relative to state-based diffusers, and planning frequency against DexHandDiff's reported 36 Hz.
The decisive check has three parts. Our full method must cut the ghost-state rate relative to the $\lambda = 0$ ablation. It must also beat the touch-label-free critic. Its goal-shifted success must exceed tactile DECO. A secondary check perturbs touch at test time with dropout or noise only, to confirm that the gain comes from training-time supervision and not from a hidden test-time crutch.
If hardware is available, the method is retrained on DECO-50 and evaluated on held-out object placements. The reported reference is DECO with the tactile adapter at 82.50%. For the asymmetric peg-insertion task of the Functional Manipulation Benchmark, the reference is ADEPT's 8/10. On the 16-DoF LEAP-hand cube reorientation task we compare to DROP's median of 30.5 rotations. That comparison is not apples-to-apples, because DROP runs an analytical simulator at test time and we do not. It still asks how much test-time physics can be replaced by learned, touch-checked latent planning.
References
- DexHandDiff: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulation
- DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter
- ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning
- DROP: Dexterous Reorientation via Online Planning
- MoDex: Planning High-Dimensional Dexterous Control via Learning Neural Hand Models
- For SALE: State-Action Representation Learning for Deep Reinforcement Learning
- The Virtues of Laziness in Model-based RL: A Unified Objective and Algorithms
- Demystifying Action Space Design for Robotic Manipulation Policies
- Enabling Visual Action Planning for Object Manipulation Through Latent Space Roadmap