Robots are increasingly trained on human demonstrations of mixed quality, and a common remedy is to score each demonstrated action with a simulator or a learned value function and discard the ones that look bad. This remedy has a hidden flaw. The simulator is never an exact copy of the real world. A skilled operator who presses harder because the real table has more friction will look like a clumsy operator when the same action is replayed in a simulator with less friction. Current methods that estimate demonstrator skill have no way to tell these two cases apart, so they penalize exactly the demonstrations that encode real-world physics the simulator gets wrong.
We propose a simple rule. A demonstrated action may be overruled by the simulator only in places where the demonstrations themselves show that the simulator predicts reality well. Elsewhere, the robot keeps imitating the human. To make the rule concrete, we compress continuous robot actions into a small per-state vocabulary of discrete action codes. For each code we maintain a confidence interval on its value, and we widen that interval by a local estimate of how wrong the simulator is. We then eliminate a code only when it is provably worse than a rival even after accounting for simulator error. The key observation is that simulator error can be measured from the demonstrations without knowing whether the demonstrator was skilled. A demonstration shows where the world went after an action, and that is a fact about physics, not about the operator's judgment.
Core Claim
Expert-skill estimation methods assume that demonstrator and learner share the same physics. Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality (CAIL) learns per-trajectory confidence weights inside an adversarial imitation loop and states this assumption explicitly. Imitation Learning by Estimating Expertise of Demonstrators (ILEED) learns state-dependent demonstrator expertise embeddings and makes the same assumption. Sim-to-real methods make the mirror-image assumption. Reinforced Grounded Action Transformation for Sim-to-Real Transfer learns an action-rewriting policy so that simulated transitions match real ones. Provably Safe Sim-to-Real Transfer uses confidence bounds to locate a region where simulator and reality disagree. Both treat real-world data as clean evidence about dynamics, with no demonstrator error in it.
Our claim is that the two sources of discrepancy are separable in a specific, usable way. Transition residuals separate them: a demonstrated triple of state, action and next state, replayed in simulation, gives the simulator's local prediction error whether or not the action was a good one. Value comparisons are confounded, so that local error must be charged against any value-based judgment before the simulator is allowed to veto a demonstrated action.
The result is a policy that behaves like offline reinforcement learning where the simulator is locally certified and like imitation where it is not. We expect it to beat both pure imitation and simulator-grounded filtering in the regime of scarce, mixed-quality real demonstrations paired with a miscalibrated simulator.
Mechanism
We reuse the state-conditioned action quantizer from Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning (SAQ). SAQ is a vector-quantized autoencoder that maps each state's continuous actions to one of 16 to 128 codes. The authors used it to make conservative offline reinforcement learning exact. We use it for a different reason: a finite per-state candidate set is what makes elimination with confidence bounds well defined.
For state $s$ and code $k$, let $\hat Q(s,k)$ be a value estimate computed from simulator rollouts that start at $s$ and execute the decoded action. Let $c(s,k)$ be its statistical confidence width, obtained from rollout variance and a value ensemble. Let $b(s,k)$ be an upper bound on how far the simulated value can be from the real value.
Code $k$ is eliminated at $s$ when
$$\hat Q(s,k) + c(s,k) + b(s,k) \;<\; \max_j \big[\hat Q(s,j) - c(s,j) - b(s,j)\big].$$
This is the successive-elimination test of racing algorithms, with a bias penalty added in the spirit of Multi-Fidelity Best-Arm Identification. That paper prunes arms across biased simulators but requires the maximum bias to be supplied by the user. We estimate the bias instead of assuming it.
We set $b(s,k) = H\,V_{\max}\,\hat\epsilon(s,k)$. Here $H$ is the effective horizon, $V_{\max}$ is the value range, and $\hat\epsilon(s,k)$ is an upper-confidence estimate of the one-step transition error near $(s,k)$. The horizon-times-error form is the standard simulation-lemma bound on how per-step model error accumulates into value error.
We obtain $\hat\epsilon$ by replaying every demonstrated transition in the simulator and recording the next-state residual. We then extend these residuals to nearby state-code pairs with a Lipschitz envelope, borrowed from Certificate-Guided Pruning for Stochastic Lipschitz Optimization. That paper maintains upper envelopes under an unknown, online-estimated Lipschitz constant. We apply the same device to simulator error rather than to an objective. This replaces the Provably Safe paper's assumption that simulator error is either zero or bounded away from zero, which fails for continuous friction and contact.
Far from any demonstration, $\hat\epsilon$ is large, so nothing is pruned there and the demonstrated code survives by default. The final policy is trained by likelihood on demonstrated codes, restricted to the codes that survive at each state. Optionally, ILEED-style expertise weights are fit on the surviving behavior only, so that the expertise model sees suboptimality with dynamics mismatch already removed.
Research Program
The first stage builds a controlled testbed in which both factors can be dialed independently. We take Robomimic's human datasets for Lift, Square and RoboTurk Cans, which are already graded by proficiency into Worse, Okay and Better tiers. We treat the nominal MuJoCo environment in which they were collected as "reality," which makes the human actions optimal-ish for that physics.
We then build a deliberately miscalibrated copy to serve as "the simulator." It changes friction, object mass, joint damping and actuation latency, with shift strength as a sweep variable. This inverts RGAT's surrogate-real protocol so that the demonstrations live on the real side, which is the realistic case. We also inject synthetic demonstrator noise (pauses, overshoots, wrong grasp modes) at controlled rates, so that for every demonstrated action we know the ground truth. That ground truth tells us whether the action was a human error, a dynamics-compensating action that looks bad in simulation, or neither.
The second stage measures attribution directly. For each demonstrated action, we ask whether the method retains or rejects it. We report precision and recall of rejecting true errors, and separately the rate at which dynamics-compensating actions are wrongly rejected. We expect CAIL and ILEED run against the miscalibrated simulator to reject compensating actions at a high rate, and our method to reject them rarely.
A discrete sanity check comes from MiniGrid, used by ILEED, with unmodeled "windy cells" in the style of the Provably Safe gridworld. There the attribution problem can be solved exactly for comparison.
The third stage tests the certificate itself. We check that pruned codes are, in the true environment, actually worse than the retained best, and we measure how often the optimal code is wrongly eliminated as the Lipschitz constant is misestimated. We also compare against a variant with heavy-tailed rollout noise, since the file notes that elimination under non-sub-Gaussian noise can catastrophically prune the optimal option.
A final, smaller stage moves to hardware. Real teleoperated demonstrations on a physical arm are paired with its simulator, and success is reported against the same baselines. We treat this as confirmatory, not as the main evidence.
Baselines
The naive reference is behavioral cloning on all demonstrations through the same SAQ codebook, so that representation is held fixed. On unmodified Robomimic data, IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data reports behavioral cloning at 13.7% success on Lift and 0% on RoboTurk Cans. It reports IRIS itself at 81.3% on Lift and 28.3% on Cans from state inputs. IRIS is included as the strongest pure-offline method that stitches good fragments from fumbled demonstrations without using a simulator.
The strongest offline reinforcement learning competitor is SAQ-IQL and SAQ-CQL trained on the real demonstrations alone. SAQ reports approximately 44% success on Square for SAQ-CQL and 2 to 3 times the success of continuous counterparts on long-horizon Robomimic tasks. This baseline is the most dangerous one because it shares our action representation and needs no simulator. Beating it requires showing that simulator evaluation adds value once its errors are charged for.
The strongest expertise-estimation competitors are CAIL and ILEED, each given the miscalibrated simulator for any rollouts they need. The file has no isolated Square number for ILEED. It reports ILEED winning 21 of 23 settings, with up to 60% improvement under high noise.
The strongest grounded competitor is a pipeline. We first ground the simulator with an action transformer fit to the demonstration transitions, in the style of RGAT but offline, since our setting forbids online real interaction. We then run SAQ-IQL on simulator-augmented data. The file gives no numeric RGAT rewards. We also include test-time value re-ranking as in Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance. V-GPS samples candidate actions and picks the one with the highest learned value; it reports an average 82.8% relative improvement for Octo-small on real WidowX tasks.
The key ablation sets $b(s,k)=0$, giving plain confidence elimination with no bias charge. A second ablation replaces the state-local $\hat\epsilon$ with a single global bias constant, as IISE would use. If the full method does not beat the zero-bias ablation as simulator shift grows, the central claim is false.
All methods get the same demonstrations, the same simulator budget in rollout steps, the same codebook where applicable, and an equal hyperparameter search of 20 configurations each, over 5 seeds. Beating a baseline means higher success in "reality" at every shift strength above zero, with non-overlapping 95% intervals, while matching it within noise at zero shift.
Why not just…?
Why not just ground the simulator first and then run CAIL or offline reinforcement learning in it? This is the real threat, because a perfectly grounded simulator would dissolve the confound. The difficulty is that grounding is fit from the same few demonstrations and is least accurate exactly where data are sparse. Yet the pipeline uses the grounded model with full confidence everywhere, including states no demonstration visited. RGAT also relies on online real interaction, which the file flags as its main cost. Our grounded-pipeline baseline settles the question. If it matches us at high shift, the contribution reduces to "grounding plus filtering," and we would report that honestly.
Why not just use a pessimistic value ensemble and pick the action with the best lower bound? Ensemble disagreement measures statistical uncertainty about value under the simulator, not the simulator's own error. Ensemble members trained on the same wrong physics agree with each other and are confidently wrong. The zero-bias ablation, with an ensemble-based $c(s,k)$, is exactly this alternative, and the shift-strength sweep shows whether it degrades.
Why not just ignore the simulator and run offline reinforcement learning on the real demonstrations? When real demonstrations are plentiful and cover the task, this may well win, and SAQ's Square result shows it is strong. Our method should reduce to it when $\hat\epsilon$ is large everywhere. We therefore vary the number of real demonstrations and expect our advantage to appear only in the scarce-data regime. If no such regime exists, the simulator is not earning its place.
Why not just use domain randomization? Randomization makes the policy robust to a spread of physics but does nothing about operator errors in the data. It also tends toward conservative behavior on precise contact tasks such as Square. We include a randomized-simulator variant of SAQ-IQL as a secondary baseline, but we do not expect it to address attribution.
Decisive Checks
The idea stands or falls on three results in the controlled Robomimic testbed.
First, attribution. As simulator shift grows, CAIL and ILEED should increasingly reject dynamics-compensating actions, while our method's false-rejection rate stays flat. Its recall on injected human errors should hold where demonstrations are dense.
Second, success. Task success in the nominal environment should exceed the zero-bias ablation and the grounded pipeline at moderate-to-large shift, and should match SAQ-IQL on real data alone at zero shift.
Third, certificate validity. The rate at which the truly best code is eliminated should stay below the nominal confidence level when the Lipschitz constant is estimated online. Under deliberately heavy-tailed rollout noise, we expect this validity to break, and we will report it as the method's known failure mode rather than hide it.
References
- Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality
- Imitation Learning by Estimating Expertise of Demonstrators
- Reinforced Grounded Action Transformation for Sim-to-Real Transfer
- Provably Safe Sim-to-Real Transfer
- Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning
- Multi-Fidelity Best-Arm Identification
- Certificate-Guided Pruning for Stochastic Lipschitz Optimization
- IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data
- Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance