Safe Sim-to-Real Transfer When the Simulator Is Wrong in Only a Few Feature Directions

A robot trained in a simulator usually fails on real hardware because the simulator gets some of the physics wrong. The fixes that work in practice, such as randomizing simulator parameters or learning a correction on the real robot, come with no guarantee about how many real trials they need or whether those trials will be safe. One recent paper does give such a guarantee: Provably Safe Sim-to-Real Transfer shows that if you locate the region where the simulator disagrees with reality, you only need to explore that region on the real system, and you can do it without violating safety constraints. Its analysis, however, only works for small discrete problems where states can be enumerated. It also assumes the simulator is either exactly right or wrong by at least a fixed margin, and that assumption does not hold for continuous physics.

This proposal carries that guarantee into continuous control by changing what "the region where the simulator is wrong" means. We model the dynamics as linear in a feature vector. The simulator then supplies a full estimate of the dynamics, and reality differs from it in only a few feature directions. Those directions might correspond to the friction terms or an actuator-lag term, while rigid-body kinematics stays correct. The real robot then only has to explore enough to pin down those few directions. It does so under a pessimistic safety check that mixes in a known safe fallback controller. The claim to test is that real-world samples and safety violations can be made to scale with the number of wrong directions rather than with the full feature dimension, and that this holds up on continuous-control tasks where the simulator's errors are physically structured.

Core Claim

Assume real transitions follow a linear model: the probability of reaching $s'$ from state $s$ under action $a$ is $\phi(s,a)^\top \mu(s')$. Here $\phi(s,a)\in\mathbb{R}^d$ is a known feature map and $\mu$ is a $d$-dimensional vector of measures over next states. The simulator gives $\mu_{\text{sim}}$ cheaply and exactly. Reality has

$$\mu_{\text{real}} = \mu_{\text{sim}} + \Delta, \qquad \Delta \text{ has at most } e \ll d \text{ nonzero rows},$$

so the simulator is wrong along at most $e$ of the $d$ feature coordinates. A per-step safety cost is also linear in $\phi$, and a baseline controller $\pi_0$ is known to be safe on the real system. Under these assumptions, we aim to design and analyze an algorithm with two properties. First, it keeps every real episode within the safety budget with high probability. Second, it reaches an $\epsilon$-optimal policy for any reward revealed after exploration, using a number of real samples that grows with $e$ and only logarithmically with $d$. For comparison, Safe Reinforcement Learning with Linear Function Approximation ignores the simulator and has regret $\tilde{\mathcal{O}}(\kappa\sqrt{d^3H^3T})$, where $d$ is the feature dimension, $H$ the episode length, $T$ the number of steps, and $\kappa$ a constant from its safety analysis. The logarithmic dependence on $d$ that we target is the usual payoff of sparse (lasso-type) estimation, and it is what Transfer Q-Learning with Composite MDP Structures obtains in its target phase, where regret is $\tilde{\mathcal{O}}(\sqrt{eH^5N})$ with $N$ the number of target trajectories.

Relative to prior work, the approach differs on three fronts. The tabular safe-transfer paper certifies a set of states; we certify a set of feature directions, which is what makes continuous spaces tractable. The composite-MDP paper learns its shared low-rank part from source samples and has no safety constraint. In our setting the "source" is a simulator we can query at will, and every real episode must be safe. Finally, the safe linear-RL paper provides the pessimistic safe-action machinery but spends its budget learning all $d$ directions. We reuse that machinery and aim it only at the uncertain rows of $\Delta$.

Research Program

The algorithm has three phases.

Simulator phase. Using only the simulator, we estimate $\mu_{\text{sim}}$ to high accuracy. We then run reward-free exploration in the style of Provably Efficient Reward-Agnostic Navigation with Linear Value Iteration to build a library of policies. Each policy excites a different feature direction, and each is screened in the simulator against the safety cost.

Real phase. Each episode runs a mixture of one library policy and the safe controller $\pi_0$, following the safe-mode mixing of the tabular paper. The mixing weight is set by a pessimistic safety value. That value is computed with $\mu_{\text{sim}}$ on the rows already certified as correct, and with a confidence set from a group-lasso estimate of $\Delta$ on the rows not yet certified. Which library policy to run is chosen to shrink the confidence set on the uncertain rows, in the spirit of the G-optimal design used by FRANCIS.

Planning phase. After exploration, any reward is optimized in the hybrid model: simulator dynamics on the certified rows, corrected dynamics on the rest.

The theoretical work has three parts. The first is a sample-complexity bound for the real phase in terms of $e$, $\log d$, $H$, and a coverage quantity. That coverage quantity is how well safe mixtures of library policies can excite the wrong directions, and it plays the role of the explorability coefficient $\nu$ in FRANCIS. The second is a high-probability safety guarantee. The third is a graceful-degradation result when $\Delta$ is only approximately sparse. That result replaces the tabular paper's $\sigma_s$-separability assumption: small entries of $\Delta$ are treated as model misspecification and absorbed into the confidence width, instead of having to be detected.

We will state the conditions on the coverage quantity plainly. On Reward-Free Reinforcement Learning with Linear Function Approximation shows that exploration becomes exponentially hard when transitions are not linear, so the linear-transition assumption is necessary rather than cosmetic.

The empirical work has to face the main weakness of the assumption, which is choosing $\phi$ for real robots. We will compare three feature choices: hand-designed physics features (terms linear in friction, damping, and mass-dependent accelerations), random Fourier features of state and action, and features learned from simulator rollouts and then frozen. For each, we measure how much misspecification error remains and whether simulator errors actually concentrate in few directions.

Baselines

The naive reference is zero-shot deployment of the simulator-optimal policy, together with the safe controller $\pi_0$ alone. Together these bound the performance range from below and show whether any real-world learning is needed.

The strongest safety-oriented competitor is the tabular algorithm of Provably Safe Sim-to-Real Transfer, run on a discretization of each continuous task and on its own windy 5x5 gridworld. On that gridworld it reports 0% unsafe iterations against 68.5% for an unconstrained baseline, and 77,958 real samples against 296,910 for a simulator-free baseline. On the gridworld, beating it means matching 0% unsafe iterations with fewer real samples than 77,958 when the wind mismatch is expressed as a sparse feature correction. On continuous tasks, beating it means achieving comparable safety with far fewer samples than its discretized version, whose cost grows with the number of grid cells.

The strongest performance-oriented competitor is Reinforced Grounded Action Transformation, which learns a neural action transformer end to end with RL. That paper reports only "near-optimal" transfer and gives no reported result found on Lacuna for RGAT as a scalar return. We will therefore rerun it on the same simulator/surrogate-real pairs, with the same budget of real transitions, the same policy class for the final controller, and a hyperparameter search of equal size over its KL limit and entropy bonus. For context on grounding methods, the original Grounded Action Transformation reports raising NAO walking speed from 19.52 to 27.97 cm/s using 65 hardware trajectories. We will also include domain randomization as benchmarked in RRLS, trained with the same simulator compute. RRLS reports no individual baseline returns, so there is no reported result found on Lacuna for its domain-randomization numbers. Beating these two means reaching equal or higher real return with fewer real transitions, and with zero or near-zero unsafe episodes where they incur some.

Two ablations isolate the key ingredient. The first removes the simulator prior and runs SLUCB-QVI from the safe linear-RL paper on the real system alone; this measures what the simulator buys. The second keeps the simulator but replaces the sparse estimate of $\Delta$ with a dense ridge estimate; this measures what the sparsity assumption buys.

Why not just…?

Why not just use domain randomization? Domain randomization needs no real data, but it optimizes for the average over a parameter range. That is known to be conservative in contact-rich tasks, and SPARR shows zero-shot assembly policies leaving a 38.4% success gap that real-world residual learning closes. Domain randomization also cannot certify safety for the particular real system. Our comparison uses the same simulator compute. If domain randomization matches our returns with zero real samples on a task, that task did not need adaptation, and we will report it as such.

Why not just fit a sparse correction to logged data from the safe controller and plan with it? This is the most serious threat, because it is simply system identification with a lasso. It fails when the safe controller never excites the directions where the simulator is wrong. In a cautious balancing controller, for example, high-velocity friction terms may never appear in the data. It succeeds whenever the safe controller's data happen to cover those directions. We will run exactly this passive-identification baseline with the same real-sample budget. We will also report the coverage quantity of $\pi_0$'s data, so the reader can see whether our gains appear only when that coverage is poor. If passive identification ties on every task, the directed-exploration component is not worth its complexity, and we will say so.

Why not just fine-tune a deep policy on the real system with off-policy RL, seeded by simulator data? This is the SPARR and RLPD route. It is sample-hungry, and it gives no protection during learning: the tabular safe-transfer study shows unconstrained transfer violating constraints in 68.5% of iterations. We run it with the same real budget and count unsafe episodes. A win for fine-tuning on return, paired with a loss on safety, is a legitimate trade-off that we will report rather than hide.

Proposed Experiments

The first experiment checks the theory where the assumptions hold exactly. We use synthetic linear MDPs with $d$ from 20 to 500 and $e$ from 1 to 20. The decisive check is that real samples needed for $\epsilon$-optimality grow roughly linearly in $e$ and roughly flat in $d$, while the no-simulator ablation grows with $d$. The second experiment is the windy gridworld from the tabular paper, recast with indicator features, to check agreement with the published 77,958-sample, 0%-unsafe result.

The main experiments use RRLS continuous tasks (code): InvertedPendulum, Hopper, and HalfCheetah. The nominal model serves as the simulator and a perturbed model as the surrogate real system. Perturbations are either structured (a change in one friction coefficient or one link mass, which is plausibly sparse in physics features) or diffuse (joint damping changed across many joints). Safety costs are joint-limit and torso-angle constraints, and $\pi_0$ is a conservative stabilizing controller. The metrics are real transitions needed to reach 90% of the surrogate-real optimal return, the fraction of real episodes violating the safety budget, and final return on three rewards revealed after exploration.

The decisive checks come in two outcomes. On structured perturbations, the method should dominate RGAT, domain randomization, and passive identification in real samples while keeping unsafe episodes near zero. On diffuse perturbations, its advantage should shrink toward the dense-ridge ablation, as the theory predicts. If the advantage persists on diffuse perturbations, sparsity is not the mechanism. If it disappears on structured perturbations, the feature map does not capture the simulator's errors. Either result would tell us more than a single benchmark win. We will also report, for each feature choice, the measured misspecification error and the fraction of the fitted $\Delta$ concentrated in its top $e$ rows, so readers can judge how realistic the sparsity assumption is.