A robot policy trained in simulation usually has to be fine-tuned on real hardware. Every real trial is expensive and risky, because the simulator is wrong in places: contact, friction and motor lag are modeled crudely. Current safe fine-tuning methods handle that risk with one global mechanism. It is either a hard switch that hands control to a backup policy whenever a learned safety estimate looks bad, or a single mixing weight between a safe policy and an exploratory one. Such a mechanism cannot tell the parts of the state space where the simulator is trustworthy, and supervision could be dropped immediately, from the parts where it is badly wrong and supervision should stay tight. We borrow from human tutoring the idea of contingent scaffolding: support that is graded, that is withdrawn as the learner proves competent, and that is re-imposed at the first sign of error. We turn it into a safety supervisor for real-world fine-tuning.
In two sentences: the supervisor corrects the policy's actions by the smallest amount needed to keep a pessimistic safety margin, and falls back to a full override only when no small correction suffices. The pessimism is set separately in each region of state space by how much the simulator has been shown to disagree with reality there, so supervision fades locally as real data arrive and snaps back after a near-miss. The claim we test is that this local, graded, reversible supervision reduces real-world constraint violations and also reaches higher return per real sample than global binary shields or globally decaying supervision.
Core Claim and What Is New
Three lines of prior work each supply one piece. Sim-to-Lab-to-Real supplies a continuous safety critic: a network trained with a Bellman equation derived from reachability analysis to predict distance to failure, rather than a binary collision flag. It uses this critic as a shield that swaps in a backup policy during lab fine-tuning. That shield is binary and its threshold is static, so it never gives the learner more room as the learner improves. Provably Safe Sim-to-Real Transfer supplies the idea of a "mismatch region": the set of state-action pairs where simulator and reality differ, which starts as everything and shrinks with statistical confidence as real transitions are observed. It is tabular, it assumes discrepancies are either zero or larger than a detectable margin, and it supervises through one scalar mixing weight. Safe Reinforcement Learning via Curriculum Induction (CISR) supplies evidence that intervention schedules should move from forgiving to strict. It learns them by Bayesian optimization over many complete student runs in simulation, which is infeasible on hardware.
Our proposal is a continuous-control analogue of the mismatch region, estimated from a residual dynamics ensemble fit to real data. This estimate drives a graded, per-region supervisor. We add two ingredients from the tutoring literature. The first is the "good student privilege" of FITS: a learner who has shown proficiency may skip steps, and fine-grained supervision is re-imposed the moment an error appears downstream. The second is the tiered hint hierarchy of CallJ, which moves from subtle cues through structural nudges to explicit correction. What is new is the combination: fading and re-imposing supervision locally, keyed to measured simulator error, inside a real-world safe learning loop. A global switch or schedule cannot do this.
Mechanism
Let $a_\pi$ be the action the learning policy proposes in state $s$. Let $\hat h(s,a)$ be the predicted safety margin (positive means safe), computed by a critic trained in simulation and refined with short rollouts of the grounded model. Let $\sigma(s,a)$ be the local uncertainty of the grounded model, defined as the disagreement across an ensemble of residual models fit to real transitions. Let $\beta_k \ge 0$ be the pessimism assigned to the region $k$ containing $s$. The supervisor executes
$$a_t = \arg\min_{a} \lVert a - a_\pi \rVert \quad \text{s.t.} \quad \hat h(s,a) - \beta_k\, \sigma(s,a) \ge 0,$$
and hands control to the backup policy only when this set is empty. The action is therefore left untouched when it is clearly safe, nudged by the minimum amount when it is marginal, and overridden only when no nearby action is certifiably safe. These are the three tiers.
The residual models follow the grounding idea of Using inaccurate models in reinforcement learning, which corrects a crude simulator with sparse real data and shows that locally correct derivatives matter more than globally accurate states. That work optimizes without any safety constraint, and our ensemble replaces its deterministic bias with an uncertainty estimate. The pessimism term plays the same role as SPiDR's variance penalty (SPiDR). SPiDR's variance comes from randomized simulators and is fixed before deployment. Ours comes from real residuals and shrinks as real data arrive.
Fading happens in two ways. The first is passive: $\sigma$ drops in regions the robot has visited. The second is privilege: once region $k$ has logged a set number of real transitions without the margin falling below a slack threshold, $\beta_k$ is lowered in steps toward zero. Any near-miss in or downstream of region $k$ resets $\beta_k$ to its initial value, which re-imposes fine-grained supervision exactly where it failed.
Regions are defined by clustering the policy's visited states in a learned embedding. Their granularity is itself a variable we study.
Pessimism alone would trap the agent. Adaptive Probabilistic Shielding proves that a shield which never permits unshielded actions can stay conservative forever. Breaking Safety Paradox with Feasible Dual Policy Iteration shows that suppressing unsafe transitions starves the safety critic of the data it needs. We therefore reserve a small per-episode budget of supervised probes. These are actions whose optimistic margin $\hat h + \beta_k\sigma$ is positive but whose pessimistic margin is not. They are allowed only where the backup policy's own margin certifies that recovery is possible. Probes are the continuous counterpart of that paper's bounded unshielded exploration, and they are the only way a region's supervision can fade before the agent is forced to act there.
Research Program
The work has three stages. The first stage builds the supervisor on top of a constrained actor-critic learner, PPO with a Lagrangian cost multiplier. It measures whether the residual-ensemble uncertainty actually tracks true simulator error: on held-out real transitions, $\sigma$ should rank-correlate with the true prediction error.
The second stage studies behavior over the course of fine-tuning in controlled sim-to-"real" settings, where "real" is a separate simulator with dynamics hidden from the learner. We track per-region intervention rates over time, how often privileges are revoked, and whether supervision fades first in regions where the hidden dynamics match the training simulator. That last pattern would be the signature that the scaffold is doing what it claims.
The third stage transfers to hardware with a Unitree Go1 or a 1/10-scale race car, the platforms SPiDR used, if access allows. The protocol is short: a fixed budget of real episodes with a tether or spotter.
Baselines
The naive reference point is the simulation policy deployed zero-shot, with no fine-tuning and no supervisor. We run it alongside an unsupervised fine-tuning run using PPO with a Lagrangian multiplier directly on the "real" system. The first shows the size of the reality gap. The second shows the violation cost of learning with no scaffold at all. On Safety Gymnasium velocity tasks, the benchmark authors report that this Lagrangian learner gives a 98% reduction in safety violations relative to standard PPO, with a 45% drop in total reward. That trade-off is exactly the conservatism we aim to avoid.
The strongest existing competitor for training-time safety is a binary value shield in the style of Sim-to-Lab-to-Real: the same margin critic, the same backup policy, and a fixed threshold. It is the closest system that fine-tunes in the real world under a shield. That paper reports lab safety violations reduced by 4% to 77% relative to the safe RL baselines SQRL and Recovery RL. On hardware it reports a 76.7% empirical success rate and an 86.7% safety rate on the Ghost Spirit quadruped. Those numbers come from vision navigation on a different platform, and no reported result was found on Lacuna for a binary shield on our perturbed Safety-Gymnasium protocol, so we reimplement it. It will share our critic, backup policy, learner, real-sample budget and hyperparameter search budget, and its threshold will be tuned over the same grid we use for $\beta$. Beating it means at least one of two outcomes at equal real samples: fewer cumulative real violations at matched final return, or higher final return at matched violations. The comparison should hold across seeds with non-overlapping confidence intervals.
The strongest zero-shot competitor is SPiDR with an ensemble of at least eight randomized simulators. The research file reports that SPiDR satisfied safety constraints in all of its real-world hardware tests; the file is cut off at that point, so we cite no further figure. SPiDR spends no real samples. We therefore report it as a horizontal line, and count our method as beating it only if we exceed its return while staying under the same cost budget.
The continuous analogue of the Provably Safe Sim-to-Real Transfer mixture policy blends the backup and exploratory policies with a single weight that decays on a global schedule. On its 5x5 windy gridworld, the original method reports 0% unsafe iterations versus 68.5% for an unconstrained transfer baseline, and 77,958 versus 296,910 real samples against a no-simulator baseline. We run our method on that gridworld as a sanity check, where it should match the 0% figure.
Four ablations isolate the key ingredients: a single global $\beta$ decayed on a schedule (removes locality), local fading with only the override tier (removes grading), no privilege revocation, and no probe budget.
Why not just…?
Why not just use the binary shield from Sim-to-Lab-to-Real with a better-tuned threshold? A fixed threshold forces one level of conservatism everywhere. In regions where the simulator is accurate it overrides good actions, and those overrides cost return and hand control to the backup policy, so the learner gathers fewer on-policy samples. In regions where the simulator is badly wrong the same threshold is too loose. The graded-only and local-only ablations settle whether grading, locality, or both drive any gain. If a well-tuned binary shield matches us, the claim fails, and we will report that.
Why not just deploy SPiDR zero-shot and skip real-world learning? If the true dynamics lie inside the randomized family, SPiDR may well be enough. That is a real threat on tasks with mild gaps. Its own limitations point the other way: strong pessimism can freeze the policy, and it assumes reality lies within the randomization support. Our protocol includes "real" systems deliberately placed outside that support, including frequent low-intensity disturbances from Robust Gymnasium, which that benchmark found more damaging than rare large shocks. If SPiDR's return on these systems is within noise of ours, the extra real samples are not worth spending.
Why not just decay supervision globally over time, as a mixing weight or a CISR-style forgiving-to-strict schedule? Global decay is the simplest competitor that also fades, and it is the most plausible one to match us. It fails in principle only when simulator error is heterogeneous across the state space, which is where our argument lives. We therefore vary how concentrated the hidden dynamics mismatch is: uniform friction scaling versus mismatch confined to one contact-rich region. We predict no advantage under uniform mismatch and a growing advantage as mismatch concentrates. Absence of that pattern would falsify the idea. CISR's actual optimizer, Bayesian optimization over many full student runs, cannot run on hardware budgets. We therefore compare against the best global schedule found by offline search in simulation, which is a generous version of it.
Proposed Experiments
The main benchmark consists of Safety-Gymnasium navigation and velocity-constrained tasks (Safety Gymnasium). Each "real" environment perturbs friction, actuator delay and mass beyond the training randomization, with Robust Gymnasium action and dynamics disruptors layered on top. The windy gridworld from Provably Safe Sim-to-Real Transfer serves as a tabular check. Metrics are cumulative real constraint violations during fine-tuning, final return, return per real sample, per-region intervention rate over time, and the fraction of initially supervised regions that are fully faded by the end. That last metric measures escape from the conservativeness trap.
Three checks decide the claim. The first is the concentration sweep above: our method's advantage over global decay should grow with mismatch concentration and vanish under uniform mismatch. The second is a calibration check: the order in which regions fade should correlate with the true local dynamics error, which only the evaluator knows. The third is a revocation check: when we inject a sudden local dynamics change mid-training, such as a low-friction patch, privileges in that region should be revoked before a violation occurs. We count how often revocation comes before failure versus after it. If all three hold on at least one hardware platform, with violations below the binary shield at matched return, we will consider local contingent scaffolding established as a mechanism for safe sim-to-real fine-tuning.
References
- Sim-to-Lab-to-Real: Safe RL with Shielding and Generalization Guarantees
- Provably Safe Sim-to-Real Transfer
- Safe Reinforcement Learning via Curriculum Induction
- FITS: A Fraction Intelligent Tutoring System
- Prototyping a call system for students of Japanese using dynamic diagram generation and interactive hints
- Using inaccurate models in reinforcement learning
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- Adaptive Probabilistic Shielding by Learning MDPs for Safe Reinforcement Learning
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning