Failure Labels as Measurements: Tuning Domain Randomization from MoCap-Free Real Rollouts

Robots trained in simulation are usually made robust by domain randomization: during training, the simulator's physics and visuals (friction, mass, camera latency, lighting) are varied at random so the policy can survive the real world without any real-world training. Picking how much to vary each parameter is the hard part. Randomize too little and the policy breaks on real hardware. Randomize too broadly and it becomes timid. The methods that tune the randomization range from real data all compare simulated and real trajectories, and recording an accurate real trajectory has meant a motion-capture rig: ceiling cameras that track markers on the robot and objects. Outside the lab, that rig is gone, and onboard odometry drifts exactly when contact happens.

This note proposes a cheaper kind of real-world measurement. A failure-diagnosis vision-language model watches a real rollout through the robot's own camera and outputs a failure type: slip, misaligned approach, wrong rotation, or "my pose estimate drifted." The proposal treats that label as a noisy observation of the hidden physical parameters. It runs the same diagnoser on simulated rollouts drawn under candidate parameters, and it measures the diagnoser's confusion matrix in simulation, where the true failure type is known. Together these give a likelihood of each observed real failure label under each parameter setting. Bayesian inference over that likelihood then narrows the randomization distribution after a few dozen real attempts, with no motion capture. The claim is that a distribution over why the robot failed carries enough information about the reality gap to replace trajectory matching. It is far more informative than success rate alone, and far more reliable than drifting onboard state estimates.

Core Claim

The proposal uses the four research directions in different ways. Semantic failure feedback supplies the measurement. Removing external localization defines the regime and is the reason a new measurement is needed. Domain randomization is the object being tuned. Sample efficiency is the yardstick, measured as real rollouts needed to reach a target success rate.

Three lines of work motivate the claim. BayesSim infers a posterior over simulator parameters from summary statistics of a few real trajectories, using likelihood-free inference: it fits the posterior from simulated samples without writing down the likelihood. Active Domain Randomization trains a sampler that seeks parameters where a discriminator separates randomized rollouts from reference rollouts. EXI-Net adapts explicit and latent physics parameters online by backpropagating state-prediction error, and it reaches 90–100% real pushing success, but its authors state it depends on motion capture for object and end-effector poses. All three define the reality gap through numeric state discrepancy, so they inherit a dependence on accurate tracking.

On the measurement side, AHA and REFLECT show that failures can be classified and explained from video. AHA does this with a seven-mode taxonomy trained on procedurally perturbed simulation data (its FailGen generator), and it transfers zero-shot to a real UR5. REFLECT builds hierarchical multisensory summaries. Both use the diagnosis only to re-prompt a planner, not to correct the simulator. We reuse AHA's data-generation recipe and model as the diagnoser. We differ in where its output goes: into a posterior over physics parameters.

The file shows no prior work that feeds semantic failure diagnoses into domain randomization, which is Gap 2 in the compiled material. We also could not find any work that makes the diagnoser's own error rate an explicit term in the inference.

Research Program

The mechanism has one central equation. Let $\theta$ be the simulator parameters, let $m$ be the true failure mode of a rollout (including "success" and "estimation drift"), and let $y$ be the diagnoser's output label. The likelihood of observing label $y$ under $\theta$ is

$$p(y \mid \theta) = \sum_{m} C(y \mid m)\, p(m \mid \theta),$$

where $p(m \mid \theta)$ is the frequency of each true mode when the current policy runs in simulation under $\theta$, and $C(y \mid m)$ is the diagnoser's confusion matrix. Given $N$ real rollouts with labels $y_1,\dots,y_N$, the randomization distribution is updated toward $p(\theta \mid y_{1:N}) \propto p(\theta)\prod_i p(y_i \mid \theta)$. The policy is then retrained or fine-tuned under that narrower, shifted distribution, and the loop repeats.

Two design choices make this honest rather than circular. The first concerns how $p(m \mid \theta)$ is labelled. In simulation, true failure modes come from rule-based checks on privileged state, for example gripper-object relative velocity for slip and pose error at contact for misalignment. No language model is involved at this step. The second concerns where $C$ is estimated. It is measured in simulation under the current visual randomization, then corrected with a small hand-labelled set of real failures. The correction matters because AHA's real-world margin over GPT-4o is only 4.9% on RoboFail, so diagnoser noise is real and must be modelled rather than ignored.

Three parts would be built. The first is a diagnoser that sees only the onboard (wrist or head) camera and proprioception. It would be fine-tuned from AHA on FailGen-style perturbations rendered from the onboard viewpoint, plus a new "estimation drift" mode. Drift examples are generated by injecting odometry and camera-extrinsic errors in simulation, so the model learns to separate "the world did something unexpected" from "the robot's belief about where it is was wrong." This addresses Gap 1 in the file: current reasoners assume their camera geometry is ground truth.

The second part is the inference loop. We follow BayesSim's mixture-density posterior and replace its trajectory summary statistics with the label histogram. The likelihood above is used directly because $m$ is discrete and $p(m \mid \theta)$ is cheap to estimate by parallel simulation.

The third part is the policy stack. It uses an LSTM policy with dynamics randomization as in Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. Training uses an asymmetric critic that sees privileged state while the actor sees only onboard images, as in Asymmetric Actor Critic for Image-Based Robot Learning. The action space is an operational-space controller from robosuite. These components are reused unchanged, so any gain comes from the randomization distribution alone.

One prior result constrains the design. What Matters in Learning A Zero-Shot Sim-to-Real RL Policy for Quadrotor Control? found that blanket randomization over measurable quantities hurts, and that system identification plus selective randomization of sensitive parameters cut tracking error by more than 50%. We therefore identify directly measurable constants (link masses, arm geometry) by system identification. Inference runs only over parameters that cannot be measured without tracking: contact friction, object mass and center of mass, actuation latency, camera extrinsics, and image and IMU noise. That study also notes it still required motion capture for state feedback, which is the dependency we remove.

Baselines

The naive reference point is uniform domain randomization over the full parameter box, with no real-world update, using the same policy architecture and simulation budget. On the closest task, real Fetch puck pushing, Sim-to-Real Transfer of Robotic Control with Dynamics Randomization reports a 0.89 ± 0.06 success rate for its randomized LSTM policy. That policy had tracked state, so we expect our onboard-only reproduction to land lower and will report our own measured number. Beating this baseline means a higher real success rate at an equal simulation budget, after no more than 60 real rollouts spent on the update.

The success-only ablation is the most important comparison because it isolates the key ingredient. It runs the identical inference loop, but each real rollout contributes only a binary pass/fail observation, equivalent to collapsing $C$ and $m$ to two classes. It uses the same real rollouts, the same posterior model, and the same retraining compute. If it matches the full method, the semantic labels add nothing and the idea is wrong.

The strongest existing method is BayesSim driven by onboard state estimates. This is BayesSim's posterior inference with summary statistics computed from onboard visual-inertial odometry and visual object tracking, the realistic way to run it without motion capture. The file contains no reported result on Lacuna for BayesSim on these tasks, so we run it ourselves with the same real rollouts and with posterior hyperparameters tuned on the same sim-to-sim validation budget as ours. Its oracle variant uses motion-capture summary statistics. It serves as an upper reference, not a competitor. EXI-Net's 90–100% real pushing success with motion capture is a similar ceiling. Matching the oracle within its confidence interval while using no motion capture would be a strong result. Clearly beating the onboard-odometry version is the minimum bar.

We also run Active Domain Randomization with the same simulation compute. It tests whether a sim-only adversarial curriculum, which spends no real rollouts, already closes the gap. A final ablation removes the confusion-matrix correction by setting $C$ to the identity. This shows whether modelling diagnoser error is necessary. Without it, real-world mislabels such as slip called misalignment should bias the posterior.

Why not just…?

Why not just use success rate as the real-world signal? This is the most serious threat, and the success-only ablation tests it directly. The argument against it is about information. Many different parameter shifts lower success by the same amount: higher friction, more latency, or a miscalibrated camera can each cut success by a similar margin. A single success rate cannot tell them apart, so the posterior widens along every axis that lowers success. That pushes toward the over-conservatism the file documents. A failure-mode histogram separates these causes, because slip and misalignment respond to different parameters. The sim-to-sim experiment below measures this directly as parameter-recovery error versus number of rollouts. If the two curves coincide, we drop the semantic component.

Why not just feed onboard odometry into BayesSim or EXI-Net? Onboard estimates degrade exactly where the reality gap matters, at contact. Aerial Manipulation with Contact-Aware Onboard Perception and Hybrid Control reports contact-phase velocity RMSE of 0.0619 m/s for off-the-shelf OpenVINS, against 0.0121 m/s with contact-aware factors. Drift of that size makes trajectory summaries partly reflect estimator error rather than physics. Even so, a good onboard estimator on a fixed-base arm may be adequate, which is why we run it as the strongest baseline rather than dismiss it. We also run a hybrid that adds the label histogram to odometry statistics. If odometry alone wins, the honest conclusion is that semantic labels matter only on platforms where drift is severe. The secondary mobile-base experiment tests exactly that case.

Why not just randomize more broadly, or fine-tune the policy on real data? Broad randomization is the naive baseline. Prior evidence from Active Domain Randomization (high-variance uniform randomization on a physical Poppy arm) and from the quadrotor study (blanket randomization degrading agile tracking) suggests it trades peak performance for survival. Real-world reinforcement-learning fine-tuning needs reward measurement and many more interactions than the roughly 60 real attempts our method spends. We include a matched-budget comparison: behaviour-cloning fine-tuning on 60 real rollouts, keeping successful episodes, against our method. This shows whether spending the same rollouts on the simulator rather than the policy is the better use.

Proposed Experiments

The decisive first check is sim-to-sim, where ground truth is known. A held-out "real" simulator is created from robosuite and RLBench tasks (lift, pick-and-place, nut assembly, puck pushing). It has hidden parameter values and a shifted renderer with different textures and lighting, so the diagnoser faces a genuine visual gap. For each method we plot parameter-recovery error and post-update task success against the number of "real" rollouts, from 5 to 100. The prediction is that the label-based posterior concentrates faster than success-only inference, and faster than odometry-based BayesSim once drift is injected. The full method should be roughly tied with oracle-state BayesSim when no drift is present. Gains should be largest for parameters that produce distinctive failure modes (friction produces slip, latency and extrinsics produce misalignment). Gains should be smallest for parameters that only change timing. That pattern would confirm the mechanism, not just the headline number.

The diagnoser is validated separately. We measure accuracy on real RoboFail and on ManiSkill-Fail against AHA-13B and GPT-4o. The new drift mode gets its own precision and recall, checked against injected-drift ground truth in simulation. On hardware, a single arm with only a wrist camera runs puck pushing, pick-and-place, and peg insertion, with no motion capture available during training or update. A motion-capture rig is used only offline, to score the oracle reference. We report real success rate after zero, one, and three update rounds of 20 rollouts each, with 50 evaluation trials per condition and binomial confidence intervals.

A secondary experiment uses a mobile manipulator whose base moves between attempts, so onboard drift is unavoidable. This is where we expect the gap over odometry-driven inference to widen most.

The idea is falsified if any of the following holds: success-only inference matches the full method in the sim-to-sim recovery curves; odometry-based BayesSim matches it on the mobile platform; or removing the confusion-matrix correction does no harm. That last case would mean the labels are either perfect or irrelevant, and both readings undercut the proposal.