Hedging Against the Simulator: Fast Multi-Candidate Action Selection for Sim-to-Real Robot Policies

Robot policies trained in simulation often fail on real hardware for a predictable reason. The simulator gets some physics wrong, such as friction, contact stiffness, or how parts jam during insertion, and the policy's favourite move was learned under those wrong physics. Some recent policies already generate a pool of candidate motions and pick one. They pick the most typical candidate, the densest cluster in the pool. Under a simulator mismatch, that densest cluster is the behaviour most tuned to the simulator's mistakes. This proposal tests a simple alternative. The policy generates a diverse pool of candidate motions in a single fast forward pass. It executes the typical one first. It then moves to less typical alternatives when the robot's own sensors show that the executed motion is not doing what it commanded.

Two things have kept this from being tried. First, generating many candidates used to require slow, many-step diffusion sampling, which caps control rates near 10 Hz. Second, the fast one-step policies that fixed latency were often trained to collapse onto a single behaviour. The proposed work brings three pieces together. It uses one-step flow policies as the generator. It measures whether their candidate pools contain genuinely different behaviours, borrowing that diagnosis from text decoding. It adds a selection rule that reads execution feedback and needs no human corrections. The central empirical claim is a crossover. When the simulator matches reality, picking the densest candidate is best. As the physics gap grows, density-based selection degrades, while feedback-driven switching to non-modal candidates holds up. Plain repetition of the same or a jittered action does not hold up.

Core Claim

A diffusion policy is a generator that produces a short chunk of future robot actions by gradually denoising random noise. KDPE samples 100 such chunks in parallel and executes the one at the peak of a manifold-aware kernel density estimate. This raises RoboMimic average success from 84.9% to 88.2% and ToolHang from 62% to 76%. It also helps under visual shifts such as changed object colour. That result motivates this proposal and also constrains it.

It motivates the proposal because it shows that a candidate pool plus a cheap selector helps. It constrains the proposal because KDPE's evidence concerns appearance shifts and in-distribution outliers, where the dense mode really is the safe choice. Our claim is narrower and, we think, different. Under a dynamics gap, the dense mode inherits the simulator's errors. This echoes the finding that simulation-trained networks latch onto simulator artifacts (Jet substructure classification). It also echoes the finding of DEAL that soft weighting over a diverse population beats picking the single best member when the target dynamics are unmodeled. If the claim holds, the recoverable successes lie in the pool's minority modes, and the right selector must use evidence from the real robot rather than from the pool's own statistics.

A theme from text generation sets the ceiling on what this can achieve. A reranker can only select what the candidate pool contains (A Systematic Exploration of Diversity in Machine Translation). If the pool is redundant, no selection rule helps, so pool diversity is measured as a first-class quantity rather than assumed.

Research Program

Generator. We use a one-step average-velocity flow policy. These models learn to jump from noise to an action chunk in one network evaluation. Examples are OMP, which reports 82.3% average simulated success on Meta-World and Adroit, and the underlying MeanFlow objective. OMP's finite-difference derivative approximation is used to keep memory manageable.

We deliberately prefer this model class to OneDP. OneDP reaches 62.5 Hz by distilling with a reverse-KL objective, and its authors note that this objective is mode-seeking, so it is likely to give redundant pools. We will verify this rather than assume it.

N candidates (N between 8 and 64) are drawn as one batched forward pass from different noise seeds. They are then clustered into behavioural modes using the geodesic distances over position, rotation and gripper state that KDPE defines.

Diversity diagnosis. Text diffusion models show "delayed branching": parallel rollouts stay in lockstep early on (Single-Step Initialization). The fix there is to commit one random position at step zero. We ask whether continuous one-step action generators have an analogue, namely many seeds mapping to one mode. If they do, we test the analogue of that fix: the noise is stratified along the few directions of the latent space that most change the decoded chunk's terminal pose, estimated once per observation from the generator's Jacobian.

This component is conditional. If plain random seeds already yield as many distinct modes as stratified seeds, we drop it and report that finding. A Tale of Two Temperatures is the closest precedent for buying diversity at low sampling cost. It works on discrete text reasoning. Its warning that excessive randomness commits outputs before enough context is available is the reason stratification is confined to a few directions.

Selector. Clustering yields $K$ modes. Each mode $k$ has a pool density $\hat p_k$, from the same kernel estimate KDPE uses. Each mode also carries an execution residual $r_k$, defined as the running mean gap between commanded and achieved end-effector displacement. This residual is recorded whenever a chunk from that mode was executed in a nearby state within the current episode, and its default is zero. The selector executes

$$k^\star = \arg\max_k \; \log \hat p_k - \beta\, r_k ,$$

where $\beta \ge 0$ trades trust in the simulation-learned distribution against real-world evidence.

With $\beta = 0$ the rule reduces exactly to KDPE-style density selection. With $\beta > 0$, a dense mode that keeps stalling against unmodeled friction or jamming loses priority to an alternative. The residual comes from proprioception the robot already has. Unlike TRANSIC, no human teleoperated corrections are needed. Unlike NavDP, whose critic is trained on privileged simulator signals and therefore shares the simulator's blind spots, no learned critic is needed either. $\beta$ is tuned once on a held-out set of simulated dynamics shifts and then frozen for all other conditions.

Baselines

Naive reference. The do-nothing reference is the single-sample one-step policy (OMP-style, N = 1) executed open-loop. It gives a latency floor and a no-hedging success rate.

Strongest existing method. The strongest competitor is KDPE as published: a multi-step diffusion policy, 100 candidates, density-mode selection. It is the strongest because it is the only published method that already exploits a candidate pool for robustness, and it reports 88.2% average RoboMimic success at roughly 10–12 Hz. We run it with the same demonstrations, the same observation encoder size and the same tuning budget for its kernel bandwidths.

We compare in two ways. One comparison holds wall-clock control rate fixed, which forces KDPE to fewer candidates or steps. The other holds candidate count fixed, which lets KDPE run slower. We also run "KDPE on the fast generator": our one-step pool with $\beta = 0$. Beating KDPE means higher success under dynamics shift at matched or higher control rate, with no more than a two-point loss at zero shift.

Fast single-trajectory methods. OneDP reports 84.3% on RoboMimic at 62.5 Hz. STEP reports 20 ms latency on an edge GPU and a velocity-aware perturbation that frees stalled motions. These are the right competitors for the question "is a pool needed at all, or does a fast single motion plus a nudge suffice?" Each is run from the same demonstrations with its published recipe.

Ablations. The key ablations isolate each ingredient in turn. Setting $\beta = 0$ isolates the feedback term. Replacing selection by uniform random choice among modes isolates whether informed switching matters, as opposed to simply switching. Removing stratified noise isolates the diversity mechanism. An oracle that executes whichever pooled candidate actually succeeds gives the ceiling the pool permits.

Why not just…?

Why not just use domain randomization? Randomizing friction and masses during training is the standard answer. DEAL reports that it tends to produce conservative behaviour, and that it failed or drifted on a real Unitree Go2 where identified parameters succeeded. Randomization is not exclusive with our method, though. All generators in the main experiments are trained with moderate randomization. The question is whether hedging adds anything on top. If randomization alone closes the gap, the crossover experiment will show it as no difference between $\beta = 0$ and $\beta > 0$.

Why not just run N one-step samples and pick the densest, i.e. KDPE on a fast model? This is the most serious threat. It is exactly the $\beta = 0$ ablation. If it matches the full method under dynamics shift, the feedback selector is unnecessary and the contribution shrinks to "fast pools work." The experiment is designed so this outcome is clearly visible. We will report it as a negative result rather than tune around it.

Why not just retry the same motion with a perturbation, as STEP does? STEP's kick addresses stalls caused by small commanded actions. It cannot change which behaviour is attempted, for example approaching a peg from a different angle. If most real failures are stiction stalls, STEP will match us. We therefore stratify failures by type, separating stalls from wrong-approach jams, to see where each method wins.

Why not just learn a real-world residual from human corrections? TRANSIC reaches 85% on FurnitureBench's Screw task with 20–100 real trajectories, which is a strong result. It needs a human operator, however. We report it as an upper reference and ask how much of that gain feedback-driven hedging recovers with zero human data.

Proposed Experiments

Controlled study. The controlled study uses a simulation-to-simulation gap, so the gap can be dialled. Policies are trained on RoboMimic (Square, ToolHang), ManiSkill2 (Turn Faucet, Push Chair) and FurnitureSim insertion and screwing subtasks under nominal physics. They are then evaluated under graded shifts in friction, contact stiffness and object mass. We found no reported result on Lacuna for any of these methods under controlled dynamics shift, so all such numbers will be new.

Metrics. We report task success, control frequency, number of distinct modes per pool, oracle best-of-pool success, and the fraction of successes produced by non-modal clusters.

Decisive check. The decisive check is the shape of the success-versus-gap curves. The hypothesis predicts three things. First, the $\beta = 0$ and $\beta > 0$ curves coincide at zero gap. Second, they separate as the gap grows. Third, the separation is accounted for by executions of non-modal clusters. If instead oracle best-of-pool success also collapses at large gaps, the pool lacks the needed behaviours. In that case the diversity diagnosis, not the selector, becomes the main contribution.

Real-world test. The real-world test runs on a Franka Panda with FurnitureBench insertion and screwing subtasks. On these subtasks, offline policies typically complete only about 2 of 12 subtasks (FurnitureBench), and sim-to-real failures concentrate there. We compare the full method against OneDP, STEP, KDPE and the $\beta = 0$ ablation over matched trials, with TRANSIC's published Screw number as the human-assisted reference.