Spending the Safety Budget Honestly Across the Reality Gap

A robot trained in simulation often has to respect a limit that applies to a whole run rather than to each moment: a total amount of wheel slip, a total number of contacts with obstacles, a total amount of energy. One clean way to learn such policies is to give the policy its remaining budget as an extra input. This lets a single network behave boldly when plenty of budget is left and cautiously when little is left. On real hardware, however, the simulator is wrong. The same maneuver spends more of the budget than the policy expects, and a run-level limit, once overspent, cannot be recovered. We propose to use the budget input itself as an adaptation knob at deployment time. The robot compares the cost it actually incurs against what its simulator-trained cost predictor expected. It then feeds the policy a deflated "effective budget" that accounts for how much reality is overcharging it. No gradient updates happen on the robot, and nothing new is learned online beyond a single scalar.

The core claim is that a budget-conditioned policy already encodes a whole family of risk levels, so correcting for a reality gap in cost can be reduced to choosing which member of that family to run. That choice can be made online from measured costs. This gives an adaptation mechanism that remains valid when the real dynamics fall outside the range the simulator was randomized over, which is the case where existing zero-shot safe transfer methods lose their guarantee.

What exists and what is missing

Three lines of work show that adding the remaining budget to the state restores the Markov property for run-level constraints. Budgeted Reinforcement Learning in Continuous State Space trains one network over the whole safety–performance trade-off curve by sampling initial budgets. Reinforcement Learning with Almost Sure Constraints proves that policies without the budget input cannot satisfy probability-one per-run constraints. Enhancing Safe Exploration Using Safety State Augmentation (Simmer) shows that budget-conditioned agents violate constraints less during training. Simmer also shows that adjusting the initial budget with a proportional-integral controller, driven by observed cost, stabilizes learning.

That controller is the closest method to ours, and we reuse its idea. The differences are in where it acts, what drives it, and whether the dynamics are assumed fixed:

  • Where it acts. Simmer adjusts the episode-start budget between training episodes. We adjust the budget passed to a frozen policy within each episode on the deployment system.
  • What drives it. Simmer tracks the gap between observed cost and a target. We track the gap between real cost and the cost the simulator-trained critic predicted.
  • Dynamics assumption. All three budget papers assume the training and deployment dynamics are the same.

The sim-to-real safety work has the opposite profile: it handles mismatch but uses a fixed safety criterion.

  • SPiDR adds a penalty on disagreement among randomized physics models to the training cost. It reports 100% real-world constraint satisfaction on a Unitree Go1 and RC cars, and 26.1x and 6.2x fewer violations than plain domain randomization. Its own limitation is the relevant one here: if reality lies outside the randomization range, the pessimism no longer covers it, and the penalty weight is tuned by hand.
  • Provably Safe Sim-to-Real Transfer estimates the regions where the simulator is wrong, but only in tabular problems.
  • Sim-to-Lab-to-Real needs an intermediate lab and shields against a distance-to-collision margin, not a cumulative budget.

None of these lets the robot change its risk level as budget is spent faster than planned.

Core Claim and Mechanism

We train a budget-conditioned policy $\pi(s,b)$ and a matching cost critic in simulation with domain randomization. Optionally, we include SPiDR's disagreement penalty. During each run on the target system, the remaining budget $b_t$ is updated from measured costs, so $b_t$ itself is exact. What is wrong is the critic's forecast of future cost.

At each step $k$, we record the critic's prediction $\hat C_k$ of cost over the next $H$ steps. Once those steps have elapsed, we record the realized cost $C_k$. The overcharge estimate is

$$\hat\rho_t = \frac{\alpha + \sum_{k \le t-H} C_k}{\alpha + \sum_{k \le t-H} \hat C_k},$$

where $\alpha$ is a pseudo-count that sets the prior $\rho = 1$ and keeps the ratio defined when predicted costs are near zero. We take an upper confidence value $\hat\rho_t^{+}$ (bootstrap over the window pairs) and give the policy

$$b^{\text{eff}}_t = b_t / \max(1, \hat\rho_t^{+}).$$

The intuition is simple. If reality charges roughly $\rho$ times what the simulator charged along the policy's own trajectories, then a real budget $b$ buys what a simulated budget $b/\rho$ bought. The single trained network already knows how to act at that tighter budget.

We frame the multiplicative model as a testable approximation, not a theorem. The main empirical question is where it breaks. The expected breaking point is when the overcharge is concentrated in rare states, so a global scalar averages it away. For that case we test a per-region variant, with $\hat\rho$ indexed by a coarse clustering of states.

This mechanism also targets the "initial burst" of violations that Simmer reports. Simulation supplies the prior, so the robot does not start from zero knowledge. The online correction handles only the residual mismatch. It does not make the robot learn costs from scratch.

Research Program

The work has three parts.

  1. Training. We train budget-conditioned agents with an off-policy constrained learner whose budget is sampled per episode, following BFTQ's practice of covering the whole trade-off curve. Constraints are per-run cumulative costs, in the spirit of Castellano et al.
  2. Surrogate realities. We build target systems in two classes. In-support shifts fall inside the randomization ranges. Out-of-support shifts reuse the physics edits from SR²: doubled gravity, friction cut to 30%, and altered limbs. We add actuator latency, which Peng et al. identify as destabilizing.
  3. Diagnosis. We measure how well the budget-conditioned critic's predictions transfer. We also measure how often a single scalar explains the realized overcharge, compared with needing per-region correction.

If resources allow, a final stage repeats the key comparison on a small RC car of the kind SPiDR used. We acknowledge that surrogate-reality evaluation is a known weakness of work like RGAT.

Baselines

The naive reference is the budget-conditioned policy deployed with its true remaining budget and no correction, which is the direct transfer of Simmer/BFTQ. If it already self-corrects well enough, because faster depletion of $b_t$ automatically makes it cautious, our mechanism adds nothing. Beating it means fewer runs that exceed the budget at matched or better return.

The strongest static competitor is the same policy with the budget scaled by a fixed factor. We choose that factor in hindsight per target system, which gives this baseline an unfair advantage. Beating it on out-of-support shifts, with one deployment rule for all shifts, is the bar that matters. We also report the best single fixed factor across all shifts, which is what a practitioner would actually ship.

The strongest existing method is SPiDR. We train it in the same randomized simulators with the same compute and environment-step budget, and give it the same hyperparameter search budget as our method, including for its penalty weight. Its published numbers are 100% real constraint satisfaction on hardware, against 26.1x (Go1) and 6.2x (RC car) more violations for plain domain randomization. We expect to match it in-support. Beating it means lower violation rates out-of-support at equal or higher return. We also run SPiDR-trained budget-conditioned policies with our correction to test whether the two are complementary.

For the expectation-constrained, non-augmented reference we use PPO-Lagrangian and FOCOPS. They are trained once per budget level, so the comparison shows the cost of having no run-time knob.

Two ablations isolate the key ingredient:

  • replacing the realized-versus-predicted signal with the SPiDR disagreement signal, which tests whether measured cost is necessary;
  • using the point estimate of $\hat\rho$ instead of its upper confidence value.

No reported result was found on Lacuna for budget-conditioned policies under any sim-to-real dynamics shift.

Why not just…?

Why not just deploy with a conservatively shrunk budget? This is the most serious threat, and it is tested directly by the hindsight-tuned fixed-scaling baseline. A fixed factor pays the safety price everywhere, including in states where the simulator is accurate. It also has to be chosen without knowing which shift will occur. If a single fixed factor matches our return–violation curve across both in-support and out-of-support shifts, the mechanism is unnecessary and we will report that.

Why not just use SPiDR? SPiDR is excellent within its randomization range. By its own analysis, however, its pessimism does not cover dynamics outside that range, and it fixes risk at training time through its penalty weight. Our out-of-support shifts are designed to separate the two approaches. If SPiDR remains safe there, the case for online correction narrows to the run-level adjustability that SPiDR lacks.

Why not just fine-tune a Lagrangian policy on the target system? Fine-tuning spends real interactions, and therefore real budget, exactly when the policy is least calibrated. The resulting oscillations of the Lagrange multiplier, documented in Safety Gymnasium, are dangerous on hardware. We include a budget-matched fine-tuning arm that gives it the same number of target-system steps, and we count every violation incurred during fine-tuning.

Proposed Experiments

We use Safety-Gymnasium PointGoal and CarGoal tasks, the RWRL Cartpole and Walker setups used by SPiDR, and MuJoCo HalfCheetah and Walker2d with SR²'s physics edits, perturbed through RRLS.

The metrics are:

  • the fraction of runs exceeding the per-run budget;
  • mean return;
  • the full return-versus-violation curve swept over initial budgets;
  • calibration error of $\hat\rho$ against the true overcharge.

There are three decisive checks:

  1. Out-of-support safety. On out-of-support shifts, the corrected policy must have a violation rate below both uncorrected deployment and SPiDR, at no worse return than the best fixed scaling.
  2. In-support cost. On in-support shifts, the correction must cost little return relative to uncorrected deployment, since $\hat\rho$ should stay near 1.
  3. Scalar versus per-region correction. On tasks where overcharge is concentrated in rare contact states, we test whether the scalar correction fails and the per-region variant fixes it. This determines whether the claim holds in its simple form or needs the more complex one.