A diffusion policy is a robot controller that produces actions by starting from random noise and cleaning it up over many small steps. It has become a default for imitation learning because it can represent several equally good ways of doing a task without averaging them into a bad compromise. Its cost is paid at every control cycle: tens of network evaluations for each action chunk. On a 24-joint hand or a two-handed humanoid that cost gets worse, and contact events happen faster than the policy can replan. This proposal argues that most of that denoising work is wasted in high-dimensional hands. Across 16 to 48 joints, the part of the action that is truly uncertain, meaning which grasp, which side, which finger gait, plausibly lives in a handful of dimensions. Once that choice is fixed, the rest of the joint motion is close to determined.
The proposal has two parts. First, it will measure how the number of denoising steps needed to match a full-step teacher grows as the joint count rises while the number of distinct strategies stays fixed. Second, it builds a controller that runs iterative sampling only in a small learned latent (a compressed code of the action chunk) and turns that code into full joint commands with a single forward pass. Because that last pass is cheap, it can be rerun at every control tick with fresh torque and touch readings while the sampled code is held fixed. If the hypothesis holds, sampling cost should track the dimension of the uncertain part rather than the joint count, and high-DoF policies could keep diffusion's ability to represent several strategies at the control rates contact-rich tasks need.
Core Claim
Write an action chunk as $a \in \mathbb{R}^{d \times H}$ ($d$ joints, $H$ timesteps) and split it as $a = D_\phi(z, o_t)$. Here $z \in \mathbb{R}^m$ with $m \ll d$ is a code sampled by a diffusion or flow model conditioned on observation $o$, and $D_\phi$ is a deterministic decoder conditioned on the latest observation $o_t$. The claim is that the number of network evaluations $K^*(\epsilon)$ needed to come within $\epsilon$ task success of a full-step teacher depends on $m$ and on the number of behavioral modes, not on $d$. Diffusing directly in joint space inherits a dependence on $d$ because the sampler has to resolve many strongly coupled, low-uncertainty directions. The testable prediction is concrete. In the full-joint policy, the steps needed to match the teacher should rise with $d$. In the latent-split policy, that curve should be flat once $m$ exceeds the dimension of the multimodal part.
Relation to Prior Work
Four lines of prior work bear on this claim.
The first line accelerates diffusion policies, and none of it has tested high-DoF action spaces. Diffusion Policy established action-chunk denoising and receding-horizon execution, but runs at roughly 10 Hz on 6–7 DoF arms. Consistency Policy distills the multi-step policy into a 1–3 step student that nearly matches its teacher on Robomimic. LightDP prunes transformer layers and adds consistency distillation, reaching a 93× speedup on Push-T. Both fast methods treat step reduction as independent of action structure, and both were evaluated only on low-DoF benchmarks. We supply the missing measurement across DoF, and we propose moving where sampling happens rather than only shrinking how much of it happens. Their distillation recipe is also a component we reuse inside the latent.
The second line is the closest existing work, and it motivates this proposal. PF-DAG already separates a primary mode, chosen from a discrete learned codebook by a small network, from a one-step continuous generator for the fine motion. It reports gains on Adroit and DexArt and proves that removing between-mode variance lowers error. Our proposal differs in three ways:
- The uncertain part is a continuous code of chosen dimension $m$ rather than a codebook of fixed size $K$, which the PF-DAG authors identify as a trade-off in their own method.
- Iterative sampling is kept where it is needed, in the latent, rather than being replaced by a classifier.
- The question is framed as a scaling law in $m$ versus $d$, which PF-DAG does not measure.
The third line supplies the architectural analogy. Würstchen cut text-to-image training compute about 8× by running diffusion on a compact semantic representation and decoding cheaply. The research file notes that no robot policy has applied this idea to joint-space actions. The hierarchical high-DoF controllers summarized there, which separate intent from execution, point the same way but do not frame the separation as a way to reduce sampling cost.
The fourth line constrains the design. Demystifying Action Space Design finds that joint-space control needs expressive generative models, and that chunk-wise deltas avoid the error accumulation of step-wise deltas. We therefore keep chunk-wise delta joint targets as the decoded output. Two papers show that touch and torque are necessary for high-DoF contact tasks:
- Visuomotor diffusion for in-hand manipulation reports that success fell from 85% to 30% on bottle unscrewing when joint effort was removed.
- DECO reports a 20-point gain on contact-rich subtasks from its tactile adapter.
Both of those systems use full multi-step sampling. This is why our decoder, not the sampler, consumes high-rate effort and touch signals. RFS shows that on a hand, changing the initial noise switches behavioral modes while output residuals only make local corrections. That supports the premise that mode choice lives in a small, steerable space. RFS itself leaves the multi-step sampler in the loop.
Research Program
We will build three things.
The first is a conditional action autoencoder. The encoder maps a demonstrated joint-space chunk to $z \in \mathbb{R}^m$. The decoder $D_\phi(z, o_t)$ reconstructs chunk-wise delta joint targets from $z$ plus current proprioception, effort and touch. Training uses a reconstruction loss with a small KL term that keeps $z$ well scaled for diffusion.
The second is a diffusion or flow sampler over $z$, conditioned on vision and proprioception. We will run it at its native step count and, separately, after consistency distillation following Consistency Policy.
The third is a two-rate execution scheme. $z$ is resampled at the chunk rate, around 10 Hz. The decoder is re-evaluated at every control tick, 50–100 Hz, with the newest sensor readings, so the finger commands react to contact changes without new sampling.
The analysis has three pieces.
- We sweep $m$ against $d$ to trace the steps-to-match curves.
- We run a residual-multimodality diagnostic. We fit mixture models to the decoder's reconstruction residuals given the true $z$. If the residual is effectively single-mode, the claim that the decoder only needs one pass is justified. If it is not, the data tell us $m$ is too small.
- We check whether PCA of actions (synergy directions) already captures the multimodal subspace. If it does, the learned encoder is unnecessary.
Baselines
The naive reference points are two. The first is an MSE behavior-cloning MLP with the same encoder and parameter budget; it sets the latency floor and the cost of averaging modes, which PF-DAG reports as the failure mode of vanilla behavior cloning. The second is the full-step joint-space Diffusion Policy teacher, which sets the success ceiling. On DECO-50, the DECO paper reports Diffusion Policy at 51.25% average success and its own vision-only DECO at 72.25% (DECO).
The strongest existing competitor on the latency-versus-dexterity question is PF-DAG. It reports 72% average success over 18 simulation tasks, including Adroit and DexArt, against 51% for both 3D Diffusion Policy and Flow Policy, at up to 40.9 FPS. We found no per-task Adroit numbers on Lacuna for any of these methods, so we will rerun PF-DAG ourselves on the identical task list. We will use its published codebook size and also sweep $K$ with the same tuning budget as our $m$ sweep.
The second strong competitor is joint-space consistency distillation, Consistency Policy applied directly to all $d$ joints. On Robomimic Tool Hang it reports 77% for a 3-step student against 79% for the 100-step teacher. We add LightDP-style layer pruning on top of it, because pruning is independent of our idea and could be stacked with it.
The fairness rules apply to every method:
- The same demonstrations are used for all methods.
- All methods share the same visual and proprioceptive encoder and are matched in total parameters.
- Each method gets the same number of hyperparameter trials and the same number of training GPU-hours, with teacher training counted for any method that needs a teacher.
- Latency is measured on the same desktop GPU and the same embedded GPU.
Beating a baseline means matching or exceeding its success rate at equal or fewer network evaluations per control tick. It also requires either higher mode coverage or lower wall-clock latency on the embedded device.
Two ablations isolate the key ingredient. The first freezes $m = d$, so the latent is as large as the joint space and the method collapses to latent diffusion without compression. The second replaces the learned encoder with PCA synergies. A third ablation disables two-rate decoding to isolate the gain from reacting to fresh contact.
Why not just…?
Why not just consistency-distill the full joint-space policy? This is the most serious threat. The file contains no evidence either way at 24–48 DoF. Every consistency result it reports comes from 6–7 DoF tasks, and the concern that deterministic few-step students lose multimodality is raised there only in general terms. If the steps-to-match curve for the distilled joint-space policy turns out to be flat in $d$, our central claim is false. In that case the contribution shrinks to the measurement itself plus two-rate decoding. The DoF-lifting experiment below is designed to settle this directly.
Why not just use PF-DAG's discrete mode selector? It already achieves one-step generation on dexterous hands. Its fixed codebook size, however, is an acknowledged limitation, and it gives up iterative sampling over the mode choice entirely. We will know whether continuous latent sampling matters if PF-DAG's coverage of demonstrated strategies saturates as $K$ grows while our coverage keeps rising with $m$. This should be most visible in tasks with a continuum of grasp angles, the kind RFS steers between. If PF-DAG matches us everywhere, we will report that.
Why not just run diffusion in PCA synergy space? This may work as well as a learned encoder, and that outcome would not refute the claim, because the claim is about where sampling compute goes, not which encoder is used. The PCA ablation tests it directly. We add the residual-multimodality diagnostic because PCA is linear and observation-independent and may leave mode information in the discarded directions. If it does not, PCA becomes the recommended recipe.
Why not just cut DDIM steps at test time? LightDP reports that reducing steps without distillation degrades performance. Consistency Policy reports naive DDIM at roughly 9× the latency of its 3-step student on laptop hardware. We still include few-step DDIM in the step sweep because it is free to run.
Proposed Experiments
The decisive experiment is a DoF-lifting study that separates joint count from the number of strategies. We take Push-T and Robomimic Square demonstrations, whose multimodality is known, and lift the action into $d \in \{2, 8, 16, 32, 64\}$ dimensions. The lift is a fixed random smooth nonlinear map plus correlated null-space coordinates with single-mode demonstrations. This mimics a redundant hand whose extra joints carry no extra decisions.
For each $d$ we record the number of network evaluations each method needs to reach within 2 points of the full-step teacher's success. The hypothesis predicts two curves. For full-joint diffusion and its distilled version the curve should rise with $d$. For the latent-split policy it should be flat once $m$ reaches the lifted task's intrinsic mode dimension.
We then test on real high-DoF benchmarks:
- Adroit (24-DoF Shadow Hand).
- DexArt, Allegro hand on Faucet, Bucket, Laptop and Toilet, reported separately for seen and unseen objects.
- Bi-DexHands, comparing one hand against two (24 versus 48 DoF). Demonstrations come from PPO experts trained with several seeds so the data contain multiple strategies.
- DECO-50, for bimanual humanoid data with touch.
Metrics are success rate, network evaluations and wall-clock latency per control tick, and mode coverage. Mode coverage is the fraction of demonstrated strategy clusters reproduced across 100 rollouts from matched initial states. We also count mode-bouncing, defined as strategy switches within an episode.
The contact-specific check runs on the DECO-50 contact-rich subtasks and on a simulated in-hand rotation task, with and without two-rate decoding. The prediction is that refreshing the decoder at 50–100 Hz with effort and touch readings recovers most of the benefit that DECO gets from its tactile adapter, at a fraction of the sampler cost.
The claim fails under either of two conditions:
- The distilled joint-space policy shows no growth in required steps with $d$.
- The residual diagnostic finds substantial multimodality left after conditioning on $z$ at every tested $m$.
We will treat either outcome as a reportable negative result.
References
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
- Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillation
- On-Device Diffusion Transformer Policy for Efficient Robot Manipulation
- Primary-Fine Decoupling for Action Generation in Robotic Imitation
- Abstract: Würstchen - An Efficient Architecture for Large-scale Text-to-image Diffusion Models
- Demystifying Action Space Design for Robotic Manipulation Policies
- Learning Dexterous In-Hand Manipulation with Multifingered Hands via Visuomotor Diffusion
- DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter
- RFS: Reinforcement learning with Residual flow steering for dexterous manipulation
- Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning