Real Failures as Permanent Vetoes on the Simulator

Robots trained in simulation often fail on real hardware because the simulator is wrong in a few specific places. Typical culprits are a sticky contact, an unmodeled actuator delay, or a soft object treated as rigid. Most fixes either make the simulator vaguer everywhere or try to correct it everywhere. Vagueness means randomizing physics so broadly that the policy becomes timid. Global correction means learning a correction to its dynamics from real data, and that spends real-robot time on regions where the simulator was already fine. This note proposes a third option. Treat sim-to-real calibration as a structured negotiation between two parties. The simulator side keeps proposing policy improvements backed by simulated rollouts. The real robot side may reject a proposal only by pointing to real evidence, usually a failure, that the simulator is wrong in a specific region of states and actions. Each such rejection permanently removes the simulator's authority in that region. From then on, a model fit to real data handles that region and the simulator handles everything else.

This borrows the main lesson of logic-based negotiation: you get a guarantee that the exchange ends if rejected claims can never be reasserted, so the set of live options only shrinks. That rule gives a sim-to-real loop with a bound on the number of correction rounds. The loop spends real samples only where the simulator has been shown to be wrong, and it uses failed real episodes as its main source of evidence instead of discarding them. The two-sentence version: real failures are vetoes that permanently strip the simulator of authority over the region where they happened. Because vetoes are never retracted and regions are counted at a fixed resolution, the correction loop provably ends, and the real data it needs grows with the size of the region where the simulator is wrong, not with the size of the whole task.

Core Claim

The claim is that a non-retractable, failure-triggered mismatch set gives two things that existing continuous sim-to-real methods lack. It gives a finite bound on adaptation rounds. It also gives a real-data cost tied to where the simulator is wrong. And it should do this without the instability that comes from correcting the whole simulator.

Two pieces of prior work bracket the claim. Provably Safe Sim-to-Real Transfer supplies the central object. It identifies a mismatch region $B$ of state-action pairs where the simulator deviates from reality, uses real data only on $B$, and proves that real sample complexity scales with $|B|$ rather than with the whole state-action space. However, it is restricted to finite tabular problems and assumes a "separable shift": every transition is either exactly right or wrong by at least a margin $\sigma_s$. Neither holds for a continuous robot. The negotiation literature supplies the termination mechanism. Tran Cao Son et al. get finite termination by forbidding repetition of refuted assumptions. Sefi Erlich et al. permanently strip rejected offers from the candidate set. Dongmo Zhang et al. forbid retracting a concession. Azzurra Ragone et al. show that repetition bans fail unless the language is bounded so that offers cannot become endlessly finer. That last point is the one that matters for continuous control: a refuted region must have a minimum size, or the opponent could refute ever-smaller slivers forever.

Put together, these yield a continuous analogue of the tabular mismatch-set algorithm. In place of the separability assumption, it uses a fixed refutation resolution $\epsilon$ measured in a learned latent space. In place of a global statistical test, it uses failure-triggered vetoes.

Mechanism

Let $z = \phi(s,a)$ be a low-dimensional embedding of a state-action pair. The embedding comes from the encoder of a real-time anomaly detector of the kind built in RAPT. RAPT trains a recurrent probabilistic model on nominal simulated trajectories and flags real transitions whose negative log-likelihood exceeds a statistical gate. Partition the reachable embedding space into cells of radius $\epsilon$, and let $B_k$ be the set of refuted cells after round $k$.

The policy is trained entirely in a hybrid model. The simulator's transitions are used outside $B_k$, and a local dynamics model $\hat P_{\text{real}}$, fit only on real transitions that land in $B_k$, is used inside it:

$$\hat P_k(s' \mid s,a) = \begin{cases} P_{\text{sim}}(s' \mid s,a) & \phi(s,a) \notin B_k \\ \hat P_{\text{real}}(s' \mid s,a) & \phi(s,a) \in B_k \end{cases}$$

Each round has two moves. First, the proposer (the simulator-backed learner) offers an updated policy trained in $\hat P_k$. Second, the opponent (the real robot) runs a small batch of real episodes. If no episode fails and no transition trips the anomaly gate outside $B_k$, the proposal is accepted. Otherwise the opponent must name at least one new cell where a failure or anomaly occurred. RAPT-style attribution localizes the cell by tracing backward in time to the transitions responsible for the deviation. That cell is added to $B_k$.

The protocol is asymmetric, in the spirit of the argumentation games of Giovanni Buraglio et al.. The opponent may never re-attack a cell already in $B_k$. The proposer may never again use simulator transitions there. The real model inside $B_k$ can keep improving, but the simulator's claim is never reinstated. Because $B_k$ grows by at least one cell per rejection and never shrinks, the number of rejected rounds is bounded:

$$K_{\text{reject}} \le N(\epsilon),$$

Here $N(\epsilon)$ is the number of $\epsilon$-cells covering the reachable embedding space. The bound holds simply because each rejection consumes a distinct cell. In practice the relevant quantity is the number of cells where the simulator is actually wrong, $|B^\star|$, and the real-data cost should scale with $|B^\star|$ times the samples needed to fit $\hat P_{\text{real}}$ per cell.

The guarantee is only meaningful if mismatch concentrates on a low-dimensional set, such as contact onsets, friction regimes, or actuator saturation. That is why the cells live in a learned latent space and not the raw state space. Measuring how $|B^\star|$ compares to $N(\epsilon)$ is itself one of the findings.

Learning from failure enters in two ways. Failed real episodes are the main source of refutations, so the information that imitation pipelines throw away becomes the only thing that changes the model. Failed episodes are also the main training data for $\hat P_{\text{real}}$ inside the refuted cells, which is consistent with the observation in Section 1.2 that failures carry information about reachable states and obstacles.

This design also respects a constraint from Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning. That paper finds that pushing failures directly into a large policy destabilizes it. Here, failures never update the policy directly. They update the world model the policy is trained in.

Research Program

The work has three parts.

The first part builds the protocol on top of an existing sim-to-real stack. It needs a policy learner (PPO with standard domain randomization as the starting point), a RAPT-style anomaly detector trained on nominal simulation, a backward-attribution step that maps a failure to embedding cells, and small local real-dynamics models, one ensemble per connected component of $B_k$.

The second part studies resolution. The cell size $\epsilon$ trades the termination bound against modeling fidelity, exactly as Azzurra Ragone et al. trade language expressiveness against finiteness. We will sweep $\epsilon$ and record rounds to acceptance, final real performance, and how closely $|B_K|$ matches the known ground-truth mismatch in surrogate tasks.

The third part tests whether non-retraction is the essential ingredient or a harmless convenience. It compares against a soft, retractable version in which the simulator's authority in each cell is a posterior weight that can recover.

Baselines

The naive reference point is zero-shot transfer of a policy trained with fixed, hand-chosen domain randomization, deployed with no real adaptation. It sets the floor and shows how large the gap is in each benchmark.

The strongest automated randomization competitor is a DrEureka-style procedure. It runs simulation stress tests to bound physics parameters and has a language model propose randomization ranges and reward terms. Because it changes the simulator globally without real-data correction, it is the right test of whether localized correction beats better-tuned vagueness. It will get the same number of real evaluation episodes for choosing among its proposals that our method gets for refutation. Beating it means higher real success at equal real episodes, or equal success with fewer.

The strongest dynamics-correction competitor is Reinforced Grounded Action Transformation (RGAT). It trains an action transformer between policy and simulator with reinforcement learning so that simulated transitions match real ones everywhere. It is the closest existing method on our main surrogate setting, because it too was evaluated on perturbed MuJoCo stand-ins for reality, and it is reported to be sensitive to its trust-region constraints. RGAT gets an equal hyperparameter-search budget, counted in tuning runs, and exactly the same real transitions. Beating it means better real return at matched real data, plus lower variance across its trust-region settings. We found no reported numerical result on Lacuna for RGAT on these tasks. The file reports RGAT's qualitative finding that classical GAT collapses with deep policies but gives no figure, so all RGAT numbers will be produced by our own runs.

The key ablations are four. A global residual-dynamics model is fit on all real data and applied everywhere, which tests locality. The soft, retractable version tests non-recantation. An all-anomaly trigger that ignores task failure tests whether failures specifically are the informative signal. A random-cell refutation control checks that attribution is doing real work. On a small discretized task we also run the tabular algorithm from Provably Safe Sim-to-Real Transfer directly, to check that our method recovers its mismatch set where its assumptions hold.

Why not just…?

Why not just learn a residual dynamics correction everywhere from real data? This is the most serious alternative, and it may win when mismatch is spread across the whole space, such as a uniform mass error. Our bet is that real gaps are concentrated, as in contact and friction regimes, and that a global correction both wastes data on correct regions and degrades the simulator where it was right. RGAT's reported fragility when correcting everywhere is indirect evidence for this. The global-residual ablation settles it. We will include one surrogate with deliberately diffuse mismatch and one with localized mismatch, and we will report the diffuse case honestly even if the global model wins there.

Why not just do Bayesian system identification and keep the posterior soft? Soft posteriors are principled. However, they give no bound on the number of rounds, and they let the simulator regain authority after one lucky real episode, which is exactly the cycling the negotiation literature guards against. If the soft, retractable ablation matches our method on sample cost and does not oscillate in practice, then non-recantation is unnecessary. In that case we would reframe the contribution around localized, failure-triggered correction alone.

Why not just tune domain randomization better, for example with a language model? DrEureka narrows randomization ranges from simulated stress tests but never consults real failures inside the loop. The file also notes that broad randomization makes policies too conservative for precise contact-rich tasks. The comparison on insertion-style SURREAL tasks will show whether better global ranges close the gap or whether localized real models are needed.

Why not just learn a failure-aware critic on real rollouts and keep the policy fixed? The Q-planning approach of Beyond Imitation uses failures well, but it cannot rescue a state where the base policy never proposes a good action. Correcting the training model lets the policy itself move outside its original support. A combined variant that adds a Q-critic on top of our corrected-model policy will show whether the two help each other.

Proposed Experiments

The first testbed follows RGAT's protocol: MuJoCo locomotion where the "real" environment has injected friction, mass, and actuator-latency perturbations. Some perturbations are localized, such as friction changes only on certain terrain or latency only at saturation, and others are diffuse. Because the true mismatch set is known, we can measure attribution precision and the ratio of $|B_K|$ to $|B^\star|$.

The second testbed uses manipulation tasks from SURREAL with perturbed contact parameters as the "real" environment. Contact gaps there are the motivating case.

The third and hardest testbed uses the soft-body simulator Elastica as "reality" while the proposer simulates a rigid approximation. This stresses whether low-dimensional concentration of mismatch survives a structurally wrong simulator.

Metrics are four: real episodes needed to reach a target success rate, final real success or return, number of protocol rounds against the $N(\epsilon)$ bound, and real failures incurred during adaptation. All are reported over at least five seeds, with real-data budgets matched across methods.

The decisive checks are three. Our method must beat both DrEureka-style randomization and RGAT in real episodes to target on the localized-mismatch tasks. The retractable ablation must show either more rounds or oscillating authority for the gap to count as caused by non-recantation. And measured real-data cost must track $|B^\star|$ rather than task size when we grow the state space while holding the perturbed region fixed. If the third check fails, the central scaling claim fails with it, whatever the headline success rates.