Safe Sim-to-Real Adaptation as Two-Rate Asynchronous Learning

A robot adapting a simulator-trained policy on real hardware learns from two very different streams of experience. The simulator produces transitions in bulk at almost no cost. The real robot produces a few transitions slowly, and some of them are dangerous to collect. Today's methods with safety guarantees handle this setting in batch. They wait for real episodes, re-estimate where the simulator is wrong, re-solve a planning problem, and repeat. This proposal instead treats the adaptation as one learner receiving asynchronous updates, meaning each entry of its value table is updated at its own irregular times. Simulator backups refresh the state-action pairs where the simulator has been shown to be accurate. Scarce real transitions refresh only the pairs where it has been shown to be wrong. The claim is that classical asynchronous convergence theory, read correctly, needs real visits only to the mismatched region, not to the whole state space. That removes an apparent conflict between convergence and safety. The same theory also supplies a computable measure of how out of date each safety estimate is, and the robot can use that measure to decide how much to explore on each step.

In two sentences: the simulator can stand in for the real world wherever it has been certified, so an asynchronous learner converges while the robot visits only the uncertified or mismatched region. The price of asynchrony is that estimates in rarely visited regions go stale, and we make exploration explicitly pessimistic in proportion to that staleness, so that constraint violations stay at zero during learning and not only at convergence.

Where the directions meet, and what is already done

The closest existing work is Provably Safe Sim-to-Real Transfer. It frames transfer as reward-free safe learning in a tabular constrained Markov decision process: an MDP whose task reward is paired with a separate cost that must stay under a budget. The method starts by treating the simulator as possibly wrong everywhere. It uses confidence bounds on real data to shrink a mismatch set $B$. It builds a hybrid model that uses real statistics on $B$ and simulator dynamics elsewhere, and it explores by mixing a known safe policy with an exploratory one, with the mixing weight calibrated to keep cumulative cost within budget. It proves that real-world sample complexity scales with $|B|$, not with the size of the full state-action space. On its 5x5 windy-gridworld benchmark it reports 0% unsafe iterations and 77,958 real samples. This paper supplies our problem setup and our main baseline.

What it does not do matters here. Its analysis assumes synchronous, episodic collection followed by a linear-program solve. It assumes the "$\sigma_s$-separable" condition, under which simulator errors are either exactly zero or larger than a known threshold. It is tabular, and the authors name function approximation as open. Our proposal keeps its hybrid-model idea and its safe-mixture exploration. It replaces the batch solve with an incremental, two-stream update whose convergence and during-learning safety we analyze with asynchronous stochastic approximation.

The asynchronous theory sets both the opportunity and the apparent obstacle. Convergence of Stochastic Iterative Dynamic Programming Algorithms proves convergence with probability one for asynchronous updates if every component is updated infinitely often with decaying step sizes. This condition looks incompatible with safe exploration, which deliberately avoids some states. The resolution is that the theorem constrains updates, not real-world visits. A simulator backup counts as an update. So in the hybrid scheme the covering requirement falls on the real stream only over $B$ and over pairs still awaiting certification.

Two related results are close but differ in what they cover. Convergence of Indirect Adaptive Asynchronous Value Iteration Algorithms proves convergence when the model improves online, but it has no constraint and no safety requirement during learning. A Unified Switching System Perspective and Convergence Analysis of Q-Learning Algorithms bounds asynchronous Q-learning trajectories between stable linear comparison systems. We reuse that style of argument to bound how inconsistent a stale entry can become.

Two further lines motivate the transient-safety problem. Breaking Safety Paradox with Feasible Dual Policy Iteration shows that safer policies generate fewer boundary samples and so starve the estimate of what is feasible. In our setting this means the safety estimates near hazards, which are exactly where the windy mismatch cells sit, are the ones updated least often. Primal-dual methods such as the natural policy gradient primal-dual method guarantee constraint satisfaction only at convergence, so they cannot stand in for our during-learning guarantee.

On the asynchronous side, Gradient-Aware Scheduling couples staleness with a curriculum over task difficulty and gives no constraint guarantee. Handling Delay in Real-Time Reinforcement Learning restores the Markov property under latency by augmenting the state, but it has no safety budget. Neither paper connects staleness to the certification of simulator accuracy.

Core Claim

We will build and analyze a single incremental learner for reward and cost values. Each update targets one state-action pair, and the next state comes from one of two sources. If the pair is currently certified accurate, the next state is drawn from the simulator. If the pair is in the current mismatch set $B_t$, a real transition is used. Certification and the membership of $B_t$ change while learning proceeds. We make three claims.

The first claim concerns convergence. The learner converges with probability one to the constrained optimum of the real system when real transitions cover only the eventually-stable $B$ and the not-yet-certified pairs reachable under the safe policy. We will also give a finite-time bound. Its dependence on the real data enters through $|B|$ and through the minimum real visitation rate on $B$ under the mixture policy.

The second claim concerns staleness. Each table entry carries a computable quantity that measures how inconsistent it may have become since its last update. For the cost table $C$ we define

$$\kappa_t(s,a) = \gamma \,\max_{t' \in [t-\tau_t(s,a),\,t]} \lVert C_{t'} - C_{t-\tau_t(s,a)} \rVert_\infty ,$$

where $\gamma$ is the discount factor and $\tau_t(s,a)$ is the number of global updates since the pair $(s,a)$ last received its own update. The bound comes from a simple fact. A Bellman target for $(s,a)$ depends on other entries only through a $\gamma$-contraction, so if the rest of the table has moved by at most some amount since $(s,a)$ was last touched, the target has moved by at most $\gamma$ times that amount.

The third claim concerns safe exploration. The pessimistic cost estimate used to set the exploration weight is the sum of three terms: the current cost estimate, the usual confidence width from real-sample counts, and $\kappa_t$. With the exploration weight set this way, the incremental learner keeps its guarantee of zero violations during learning even while estimates are stale and while states are being moved into $B$. Without the $\kappa_t$ term we expect violations in exactly one transient: when a state first joins $B$, its cost value still reflects the optimistic simulator, and that stale value keeps feeding into its neighbors' targets.

Research Program

The theoretical part has three pieces. First, we adapt the asynchronous stochastic-approximation proof to the case where the operator itself changes over time. A pair's update rule switches from a simulator backup to a real-sample backup when certification fails. If $B_t$ stabilizes after finitely many switches, which holds with high probability under separability, the problem reduces to the fixed-operator case after the last switch. Second, we derive the finite-time rate with the switching-system comparison argument, tracking the real visitation rate on $B$ separately from the simulator rate. Third, we prove that the staleness-inflated cost bound upper-bounds the true cost of the mixture policy at every step, and therefore keeps the constraint satisfied throughout learning.

The algorithmic part implements this learner in tabular form. It then extends the learner to linear function approximation, following the constant-step-size stability results in Convergent Combinations of Reinforcement Learning with Linear Function Approximation. That paper matters because it shows that an update at one state moves estimates at neighboring states. In our setting a real-data update inside $B$ can therefore corrupt certified simulator regions. We will measure this leakage directly, and we will test whether $\kappa_t$, computed on the weight vector rather than on table entries, still bounds it.

The empirical part runs a controlled sweep over three knobs that the batch method does not model. The first is delay: real transitions reach the learner with fixed or random latency, and simulator updates continue in the meantime. The second is separability: the size of the wind perturbation is annealed from well above $\sigma_s$ to below it. The third is the size of the mismatch region, which scales $|B|$ against the size of the grid.

Baselines

The naive reference is unconstrained sim-to-real fine-tuning: train in the simulator, then keep learning on the real system with no safety mechanism. It is run with the same real-sample budget as the other methods. Provably Safe Sim-to-Real Transfer reports 68.5% unsafe iterations for this baseline on the 5x5 windy gridworld. Any method must sit far below that number.

The second reference ignores the simulator and runs reward-free safe exploration from scratch on the real system. The same paper reports that this baseline needs 296,910 real samples. It sets the ceiling our sample count must fall well under.

The strongest competitor is Algorithm 1 of Provably Safe Sim-to-Real Transfer, which reports 0% unsafe iterations and 77,958 real samples. It is the strongest because it already solves the tabular synchronous version of our problem with a matching guarantee. We will re-implement it and reproduce its numbers before any comparison. We then run it and our learner on identical real transition streams, with the same safe policy, the same confidence parameters and the same budget $\epsilon$. Algorithm 1 is allowed a full re-solve after every real episode, which is its most favorable schedule. Our learner beats it if it matches its 0% unsafe rate while reaching the same real-policy return in no more real samples under delay. It also beats it if it keeps a lower violation rate than Algorithm 1 once separability fails. We do not expect to beat it on the undelayed, separable benchmark, and parity there is the success criterion.

For continuous-mismatch and function-approximation settings we add the reachability-shield pipeline of Sim-to-Lab-to-Real as a comparison, re-implemented on our domains with the same simulator pretraining budget. That paper reports about 86.7% safety and about 76.7% success on the Ghost Spirit quadruped. Those numbers come from a different platform, so we cite them only as context and not as a target.

Two ablations isolate our key ingredients. The first removes $\kappa_t$ and keeps only count-based confidence widths. This isolates the claim that staleness, and not just sample counts, drives transient violations. The second lets real samples update every pair rather than only pairs in $B_t$. This tests whether restricting the real stream actually matters for sample efficiency under function approximation.

Why not just…?

Why not just rerun the batch Algorithm 1 after every episode? In the tabular, separable, undelayed setting this probably works as well as our method, and we say so. The proposal does not claim a win there. The batch method stops working in three places. It needs a linear-program solve over the full occupancy space, which does not extend to function approximation. It treats real transitions that arrive after a delay as if they were synchronous, so its exploration weight is computed from a model older than the executed policy. Its certification assumes separable errors. Our delay and separability sweeps are designed to show whether these failures cause violations or extra samples in practice. If Algorithm 1 with frequent re-solves shows neither, then the incremental learner's contribution reduces to the function-approximation extension and the theory, and we will report that result.

Why not just randomize the wind in simulation and deploy zero-shot? Domain randomization over the missing physics removes the need for real-world adaptation. It succeeded for drone navigation in (CAD)$^2$RL and for quadrotor recovery in Control of a Quadrotor with Reinforcement Learning. It carries no guarantee when the real perturbation lies outside the randomized range, and wide ranges push policies away from the hazard boundary, where the task reward in our gridworld lies. We will run it with wind ranges that include and that exclude the true value. We measure real violations and the return lost to conservatism. It is a real threat when the range is chosen correctly, and the experiment shows how much that correct choice is worth.

Why not just add a large cost penalty to the reward? Penalties force a choice of weight. The file's account of Quanyi Li et al. describes the two failure modes: a weight too small tolerates violations, and a weight too large yields a stationary, safe but useless policy. We will sweep the penalty weight and report the full return-versus-violation curve. The goal is to check whether any single weight reaches both zero violations and competitive return on the windy grid.

Why not just use a reachability shield and ignore asynchrony? A shield from Sim-to-Lab-to-Real intervenes at execution time and does not care how the values were learned. Its own hardware results still leave about 13% of runs unsafe. Under latency, the state can cross the boundary before the override takes effect. Our delay sweep runs the shield under the same latencies to test whether the violations it misses are the ones $\kappa_t$ was designed to catch.

Proposed Experiments

The first check is replication. We reproduce the 5x5 windy-gridworld numbers for Algorithm 1 (0% unsafe, 77,958 samples), the unconstrained baseline (68.5%) and the from-scratch baseline (296,910). We then show that our learner reaches 0% unsafe iterations with a real-sample count within noise of Algorithm 1's. If our learner is safe but needs many more samples, the claimed convergence rate is wrong.

The decisive transient test measures violations at the moments states are added to $B_t$, with and without $\kappa_t$. The theory predicts that the ablation without $\kappa_t$ violates the constraint specifically in these windows and that the full method does not. If both violate, or neither does, the staleness mechanism is either insufficient or unnecessary, and we will say which.

The delay and separability sweeps extend the gridworld to larger sizes with more windy cells, random latencies on real transitions, and wind strengths straddling $\sigma_s$. We record violation rate, real samples to reach a target return, and wall-clock time, for our learner and for Algorithm 1 with re-solves.

For the continuous check we use the CommonPower smart-grid suite (CommonPower). There we define "real" as a held-out simulator with perturbed line and load parameters, and apply the linear-function-approximation version with the shield and randomization baselines. No reported result was found on Lacuna for CommonPower under a sim-to-real mismatch, so every number there will be new, and we will label the protocol as sim-to-sim. The main measurement is leakage: how much a real-data update inside $B$ moves value estimates in certified regions, and whether the weight-space version of $\kappa_t$ bounds that movement. If it does not, the extension beyond tables needs a different staleness measure, and that negative result is worth reporting in its own right.