Tuning Generated Simulators From Noisy Human Comparisons of Real Robot Rollouts

Robots trained in simulation usually do worse on real hardware, and the standard fix is to adjust the simulator using real-world trials. Existing methods need either a clean numeric task score in the real world or precise measurement of real robot and object states. Many real tasks have neither: scooping food, bussing a table, or cleaning a pan has no built-in reward and no cheap state tracker. Meanwhile, foundation-model pipelines now build the simulator itself, choosing meshes, physical constants and task code, and they make many choices that nobody checks against reality. We propose to close this loop with the cheapest signal available on a real robot. A person watches two real rollouts and says which one did the task better, or that they cannot tell. Those noisy comparisons drive a search over the generator's settings: which assets to use, which physics parameters, and how much to randomize.

The core claim is that a few dozen noisy pairwise human judgments of real rollouts are enough to steer a generative simulation pipeline toward settings whose trained policies transfer. They should match a simulator-adaptation method that is given a true numeric real-world reward, and beat both binary success labels and automatic vision-language judges at equal human effort. Two ingredients make this work. The first is a likelihood that separates human perceptual noise from the randomness of physical rollouts. The second is a query rule that only asks humans to compare rollouts the model predicts they can actually tell apart.

What exists and what is missing

Three bodies of work supply the parts but never meet. Generative simulation systems remove manual asset authoring. RoboGen has GPT-4 propose tasks, retrieves Objaverse and PartNetMobility meshes, filters them with a vision-language model, and writes reward code. Gen2Sim lifts single images to meshes and asks GPT-4 for mass and friction. Both run open loop: RoboGen states it cannot verify intended behavior without manual inspection, and Gen2Sim's real transfer is a proof of concept with no adaptation from real discrepancies. These papers motivate the work. Their generator choices are exactly the unchecked knobs we propose to tune.

Simulator-adaptation methods supply the outer loop but constrain the feedback it can use. SimOpt matches simulated to real trajectories and needs clean real state estimates. AdaptSim argues that the simulator should be useful rather than realistic, and meta-learns how to move simulator parameters to raise real task reward. It assumes that reward is a deterministic number, and it works on hand-built Drake scenes. We keep AdaptSim's task-driven stance and drop its two assumptions: the feedback becomes ordinal and noisy, and the search space includes generative asset choices rather than only physics constants.

Preference learning supplies the noise modeling and query design. Wilson et al. treat comparisons as noisy observations of a latent quantity and cut query counts with active selection. Multi-dueling bandits with dependent arms places Gaussian-process priors over continuous arms. S-EPOA shows that labels degrade when humans must compare indistinguishable segments, and that querying well-separated behaviors fixes this. We reuse these techniques. The difference is that in all of these papers the compared items are simulated trajectories. Here they are real rollouts of policies trained under different simulators, so the "arm" is a simulator setting, not a policy.

The closest work is RobotArena ∞, which builds simulations from videos and gathers over 7,000 crowd preference pairs aggregated with a Bradley-Terry model. It uses those preferences only to rank fixed policies inside simulation. It neither adapts the simulator nor tests on hardware. Two papers act as warnings that shape the design. Causal confusion in preference-based reward learning shows that a reward model fit to preferences at 92.8% accuracy can drive Itch Scratching success from 97% to 1.3%. We therefore never train the policy against a learned preference reward. Preferences only choose among simulators, and policies always train on the generator's task code inside simulation. Kim et al. show that static reward-model accuracy barely predicts downstream quality. For this reason we judge the method only by real task outcomes, never by how well it predicts preferences.

Research Program

The generator is a RoboGen- or Gen2Sim-style pipeline exposed through a setting vector $\theta$. The vector covers which mesh source to use for each object (retrieved, or lifted from a photo of the real object), multipliers on the LLM-proposed mass and friction, domain-randomization widths for physics and textures, and the level of visual fidelity (stylized textures versus a Gaussian-splat background, as in PolaRiS). Including the last two knobs lets the loop settle, task by task, the open conflict between (CAD)²RL's "diversity beats fidelity" and PolaRiS's finding that visual mismatch dominates.

For each proposed $\theta$ we generate the simulation, train a policy in parallel simulation, and run it a few times on the real robot while recording video. We place a Gaussian-process prior on a latent transfer quality $u(\theta)$. Each real rollout of setting $\theta$ yields an outcome quality $q = u(\theta) + \eta$, where $\eta \sim \mathcal{N}(0, \sigma_r^2)$ captures physical rollout randomness. A human comparing rollouts $a$ and $b$ answers "a", "b", or "can't tell":

$$P(a \succ b) = \Phi\!\left(\frac{q_a - q_b - \delta}{\sigma_h}\right), \quad P(\text{tie}) = \Phi\!\left(\frac{\delta - (q_a - q_b)}{\sigma_h}\right) - \Phi\!\left(\frac{-\delta - (q_a - q_b)}{\sigma_h}\right),$$

Here $\Phi$ is the standard normal cumulative distribution, $\sigma_h$ is the person's perceptual noise, and $\delta$ is their indifference band. The tie option keeps uncertain answers from becoming coin flips and turns them into information that the two settings are close. Because each setting gets several rollouts, $\sigma_r$ and $\sigma_h$ can be identified separately: repeated rollouts of one setting expose physical randomness, while repeated judgments of one video pair expose human noise.

The next pair of settings to test maximizes the expected reduction in uncertainty about the best $\theta$. It is restricted to pairs whose predicted gap $|q_a - q_b|$ exceeds the estimated band $\delta$, which carries S-EPOA's distinguishability idea from skills to simulators. The prior mean can be warm-started by running the same loop against held-out simulator variants with simulated humans, in the spirit of AdaptSim's simulation-side meta-training.

Baselines

The naive reference is the generator's default output: LLM-proposed physics, retrieved meshes, and moderate randomization, trained once and deployed zero-shot, as RoboGen and Gen2Sim would do. A second reference is wide domain randomization over every knob with no feedback, in the style of (CAD)²RL. This tests whether feedback is needed at all, and SimOpt's warning about over-conservative policies predicts it will fall short on contact-rich tasks.

The strongest competitor is AdaptSim given a true numeric real reward on the same tasks. It is the only method that adapts the simulator for task performance rather than realism. We run it on the same generated simulators, with the same setting vector, the same budget of real rollouts, and the same simulator compute per setting. It serves as a near-upper reference: matching it with ordinal human feedback is the win condition. AdaptSim reports reaching peak bottle-pushing performance in 16 real trials versus 40 for direct training, and 60% success on sliced-cucumber scooping where baselines failed. Both figures come from AdaptSim. For the realism-matching alternative, SimOpt reports 90% real swing-peg-in-hole success after two iterations (SimOpt). We include it only on tasks where real states can be tracked, since it requires them.

A feedback-matched baseline gives the same humans the same viewing time, asks for binary success per rollout, and drives the same Gaussian-process search with a scalar likelihood. An automatic-judge baseline replaces humans with a vision-language model that compares the same real videos, analogous to RobotArena ∞'s automated scoring. No reported result was found on Lacuna for any preference- or judge-driven simulator adaptation, so these baselines produce new numbers rather than borrowed ones.

Three ablations isolate the key ingredients. The first uses a plain Bradley-Terry likelihood with no tie option and no separation of rollout noise. The second picks query pairs by uncertainty alone, without the distinguishability constraint. The third restricts the setting vector to physics multipliers, which tests whether tuning the generative asset and fidelity knobs matters beyond classical parameter tuning. Beating a baseline means higher final real success, or equal success with fewer real rollouts, with confidence intervals over at least three seeds and two human cohorts.

Why not just…?

Why not just use a vision-language model as the judge and drop humans entirely? This is the most serious threat. RobotArena ∞ reports that automated and human rankings agreed on which policies were stronger. The difference may lie in near-ties and subtle failures, such as a scoop that spills half its contents or a stack that holds but is crooked. The automatic-judge baseline settles this directly on identical videos. If it matches the human loop, the result still stands as "comparisons of real rollouts can drive generated-simulator tuning," and the human-noise model becomes a tool for calibrating the model judge rather than the headline.

Why not just ask humans for success or failure and run AdaptSim on those labels? Binary labels carry at most one bit per rollout and saturate once most rollouts partly succeed, which is exactly the graded regime of scooping and cleaning. A pairwise judgment with a tie option carries graded information. The feedback-matched baseline tests this at equal human time. If binary labels match comparisons, the preference likelihood is unnecessary and we will report that.

Why not just co-train on real data instead of tuning the simulator? PolaRiS mixes simulated and real data, but it needs real demonstrations, and it targets evaluation rather than repairing the generator. Our loop needs only rollouts of policies trained in simulation plus judgments. We include a co-training arm given real teleoperated demonstrations equal in robot time to our rollout budget. This shows whether comparison-driven tuning is a cheaper route to the same transfer.

Why not just randomize everything widely? This is cheap and sometimes enough, as (CAD)²RL shows for drone navigation. SimOpt and AdaptSim, however, both report failures on contact-rich tasks. The wide-randomization baseline measures where randomization suffices and where it does not.

Proposed Experiments

The first stage runs at scale without hardware. It uses AdaptSim's Drake tasks (bottle pushing and food scooping) and ManiSkill2 tasks, with a held-out perturbed simulator, including changed meshes, standing in for "real." Simulated human labelers follow the likelihood above, with noise swept up to the 30% label-flip level that S-EPOA used. This stage sizes the gains of each ablation and checks whether $\sigma_h$ and $\sigma_r$ are recoverable. ManiSkill2 anchors the scale of the gap: PickCube drops from 91% in simulation to 60% zero-shot on a real robot (ManiSkill2).

The second stage moves to a Franka arm on PolaRiS-style tasks (Food Bussing, Block Stacking, Pan Cleaning) plus a scooping task. Real budgets are 40 to 60 rollouts per method, and two cohorts of about ten people judge recorded video pairs. Success is scored afterward by a held-out rubric applied by separate raters blind to method.

Two checks decide the claim. First, at equal real rollouts, the human-comparison loop should come within the confidence interval of AdaptSim with true reward, and beat the binary-label and automatic-judge arms on graded tasks. Second, removing the tie option and the distinguishability constraint should measurably slow convergence under realistic human noise. If the first check fails, ordinal real-world feedback is too weak to tune generated simulators. If the second fails, the gains come from search over generator knobs alone, and that finding is also worth reporting. A secondary analysis records which fidelity and randomization settings the loop selects on each task, giving direct per-task evidence on the diversity-versus-photorealism question.