Worst-Case Trust Regions for Safe Policy Updates Across Simulator Ensembles

A policy trained in simulation usually has to be improved again once it meets the real robot, and each of those updates is a gamble: an update that lowers expected cost in the simulator can raise it on hardware. The reason is that the real robot visits states the simulator rarely produced, such as slipping feet or delayed motor responses, and a small policy change can act very differently there. Trust-region safe reinforcement learning bounds how much cost can rise per update. Its main example is Constrained Policy Optimization, or CPO, a method that limits each update by the divergence between old and new action distributions and solves a small constrained problem using curvature from the Fisher information matrix. The Fisher matrix measures how sensitive the policy's action distribution is to each parameter direction. The catch is that CPO's bound holds only under the state distribution of the single environment in which the curvature was measured.

We propose a different rule. The update should satisfy a trust-region bound and a cost constraint in every member of a randomized simulator ensemble at once. It should also satisfy them on states logged from the real robot, which enter as one more member. The obvious way to do this multiplies second-order cost by the ensemble size. We remove that cost with a shared-plus-low-rank curvature decomposition, so a worst-case update over $n$ simulators plus real data costs close to one natural-gradient step. The bet is concrete. In the regime where the real dynamics fall outside the randomization range, this per-member curvature is what keeps hardware fine-tuning safe. Penalty-based pessimism and first-order step-size control do not.

Core Claim

Under dynamics mismatch, an update's safety depends on where it is measured, not on its average size. Suppose a step is small in Fisher distance under the ensemble-averaged state distribution. That step can still be large on states that only one simulator, or only the real robot, reaches. Those are exactly the states where the reality gap lives.

We claim three things. First, constraining divergence and linearized cost separately for each ensemble member, including a member built from real-robot states, cuts constraint violations during real-world fine-tuning substantially, compared with methods that use one pooled curvature or none. Second, the per-member curvatures share most of their structure, so the worst-case problem can be solved with a shared Kronecker-factored matrix plus rank-$k$ per-member corrections. The per-update overhead then stays close to that of a single natural-gradient step instead of growing with the ensemble size. Third, this answers an open tension in the literature. Curvature that is measured per dynamics model stabilizes transfer. Curvature pooled across models is what lets the optimizer exploit simulator flaws.

Research Program

The update. At policy parameters $\theta$, each member $i$ gives us four quantities. These are a reward gradient $g_i$, a cost gradient $b_i$, a current expected cost $J^i_C$, and a Fisher matrix $F_i$ estimated on that member's visited states. Members $i = 1,\dots,n$ are randomized simulators and member $0$ is the real robot. We solve

$$ \max_{\Delta}\; \bar g^\top \Delta \quad \text{s.t.}\quad J^i_C + b_i^\top \Delta \le \epsilon,\;\; \tfrac12 \Delta^\top F_i \Delta \le \delta \quad \text{for all } i \in \{0,\dots,n\}, $$

where $\bar g$ is a weighted reward gradient, $\epsilon$ is the cost budget, and $\delta$ is the per-member divergence radius. CPO is the special case with a single member. Applying CPO's per-step cost bound to each member gives a step whose worst-case cost increase is bounded in every member. That includes the real-data member, because its Fisher matrix is measured on real states. One property matters a great deal here: the Fisher matrix at a real state needs no extra robot interaction. Actions can be resampled from the policy offline at each logged state, so the real member's curvature costs computation, not samples.

The real member's cost gradient $b_0$ is the noisy quantity. We estimate it from the few real episodes using a critic initialized from the simulator ensemble. When the real-data estimate is too uncertain, we fall back to its upper confidence value, which makes the constraint more conservative rather than less.

Making it cheap. Running conjugate-gradient solves separately for each of $n+1$ members is the expensive path. Each solve is a loop of repeated Hessian-vector products, which Zeng et al. and Rathore et al. show make iterations several to hundreds of times slower than backpropagation. Instead we write each member's Fisher matrix as $F_i \approx F_0^{\text{shared}} + U_i U_i^\top$. Here the shared term is a Kronecker-factored approximation pooled over all members, and $U_i$ is a rank-$k$ sketch of member $i$'s score vectors, projected off the shared term.

Inverting the shared term is one Kronecker-factored solve, and each correction is handled with the Woodbury identity. The dual problem has only $2(n+1)$ multipliers and is solved directly. The shared factors are refreshed on an amortized schedule, following the result of Mueller et al. that spreading curvature updates over several steps reduces wall-clock cost. The low-rank sketches reuse the randomized approximation idea of SPAN. Sketch computation runs while the vectorized simulators step physics. This adapts the bubble-filling idea of PipeFisher, which computes curvature in otherwise idle accelerator time during pipelined training, to simulation workers waiting on synchronization.

Our target is concrete. We aim for per-update wall-clock within the 10% to 25% overhead range that Yong et al. report for efficient preconditioned optimizers in single-model settings, now with $n \ge 8$ members.

Two phases. In simulation pretraining, only the $n$ simulator members are active, so the method competes directly with zero-shot approaches. In hardware fine-tuning, member $0$ is switched on after each batch of real episodes. When real data reveals a mismatch region, such as states where friction behaves outside the randomized range, the real member's divergence constraint tightens the step exactly in the parameter directions that change behavior there.

Analysis. We will measure how much the member Fishers actually differ. We use the principal angles between the leading eigenspaces of each $F_i$ and the shared matrix. We then relate this to the cost-violation gap between pooled and per-member updates. If the deviations are tiny, per-member curvature buys nothing and the method reduces to CPO with domain randomization. That is a clean falsification route.

Relation to prior work. Three papers motivate the proposal. CPO (Constrained Policy Optimization) supplies the single-environment per-step bound we extend, and it was evaluated only in nominal MuJoCo. SPiDR adds the disagreement of next-state predictions across an ensemble to the cost, which yields zero-shot hardware safety. It names out-of-support dynamics as its failure mode and has no safe online adaptation step, and that step is what we supply. Provably Safe Sim-to-Real Transfer mixes real data on a detected mismatch region with simulator data elsewhere and guarantees zero violations, but only for tabular problems solved by linear programming. Our real member is a continuous, curvature-based analogue of its mismatch region.

Two papers constrain the design. FOCOPS shows that CPO's second-order subproblem is often infeasible and wastes updates. For that reason we keep CPO's recovery step and test against FOCOPS directly. Abbeel et al. show that precise optimizers exploit simulator artifacts, which is why we never let any single simulator's curvature define the step.

Two papers sit further away. Natural Policy Gradient Primal-Dual Method for Constrained Markov Decision Processes gives convergence theory for curvature-preconditioned constrained updates under one known dynamics model; we instead target per-update safety under several. LyEvO gets certified hardware safety with derivative-free search costing over 31 hours on up to 1,000 CPU cores. We aim for the per-update safety half of that goal with gradient methods, without its certificates.

Baselines

Naive reference. Standard domain randomization with PPO-Lagrangian is the naive reference. It is trained on the same ensemble and then fine-tuned on real or held-out dynamics with the same episode budget. Safety Gymnasium reports that PPO-Lag reduces velocity-task violations by 98% relative to PPO, at a 45% reward cost (Safety Gymnasium). This baseline tells us whether any structure beyond a Lagrange multiplier is needed.

Strongest existing method for zero-shot safety. That method is SPiDR. It reports 100% constraint satisfaction on a Unitree Go1 and an RC racecar, with standard domain randomization incurring 26.1x and 6.2x more violations respectively. We run SPiDR with its ensemble size, penalty weight tuned on the same grid budget we give our method, and the same simulator count. For the fine-tuning phase, which SPiDR does not define, we extend it in the most natural way: continued SPiDR-penalized PPO updates on real data.

Strongest existing method for update-level safety. That method is FOCOPS, which reports reward 1106.1 on Humanoid-Circle versus 671.0 for CPO within the cost limit (FOCOPS). It is run on the pooled ensemble and in fine-tuning. We also include CPPO-PID from SafePO, the PID-controlled multiplier that Safety Gymnasium presents as damping Lagrangian oscillations. No reported result was found on Lacuna for CPPO-PID on out-of-support fine-tuning.

Fairness and the bar for beating them. All methods share the same ensemble, the same real or held-out episode budget, the same network sizes, ten seeds, and equal hyperparameter-search trials. Beating SPiDR means two things together. We must match its zero-shot violation rate within seed noise. And during out-of-support fine-tuning we must cut cumulative violations, and the fraction of updates that raise real cost above budget, by at least half, at no worse final return.

Ablations. Three ablations isolate the key ingredient. The first is CPO with one pooled Fisher matrix and pooled constraints, which is CPO plus domain randomization. The second keeps per-member cost constraints but uses a shared divergence radius on pooled curvature. The third removes the real member during fine-tuning. If the first two match the full method, per-member curvature is not doing the work.

Why not just…?

Why not just use SPiDR and accept it? On in-support dynamics, SPiDR may well tie us; its hardware results are strong, and we expect parity in phase one. Its own analysis assumes the real dynamics lie within the randomization support, and it offers no update rule for when they do not. Our held-out-parameter experiments test exactly that regime. If SPiDR-penalized fine-tuning stays within budget there, our proposal loses its main justification, and we will report it that way.

Why not just fine-tune with FOCOPS or PID-Lagrangian and a small learning rate? This is the real threat, because it is cheap and FOCOPS already beats CPO in simulation. Our argument is that a small step in parameter space is not small on every member's states. A fixed learning rate or a pooled divergence projection cannot see that a step is large precisely on slippery-floor states. The pooled-curvature ablation and FOCOPS-on-real-data runs settle this. We will also sweep FOCOPS's divergence radius so that it is not handicapped by a single setting.

Why not just use a reachability shield as in Sim-to-Lab-to-Real? Sim-to-Lab-to-Real cut lab-training violations by 4% to 77% and reached 86.7% safety on a quadruped. However, its shield is trained in simulation and needs manually tuned thresholds. Under out-of-support dynamics, the shield's own value function is miscalibrated. A shield and our update are complementary, so we run a shield-plus-FOCOPS combination as an additional comparison rather than treat shielding as a substitute.

Proposed Experiments

Sanity check. We first reproduce the 5×5 windy gridworld from Provably Safe Sim-to-Real Transfer. There the tabular method has 0% unsafe iterations versus 68.5% for the unconstrained sim-to-real baseline. With a softmax policy, our method should approach 0% using a comparable real-sample count; the reported figure is 77,958 samples.

Main study. The main study uses RRLS (SuReLI/RRLS) and SPiDR's MJX PointGoal2 setup, with velocity and position costs added. We train on one interval of mass, friction, and damping and fine-tune on parameters outside it. Held-out friction is our stand-in for stick-slip; held-out action delay is our stand-in for actuator latency. No reported RRLS return figures were found on Lacuna, so all RRLS numbers will come from our reruns.

Metrics. We report cumulative cost violations during fine-tuning, the fraction of updates that push held-out-dynamics cost above budget, and final return. We also report wall-clock time per update relative to PPO-Lag for $n \in \{4, 8, 16, 32\}$ and correction rank $k \in \{4, 16, 64\}$.

Decisive checks. Three checks decide the outcome. The first is whether the per-member method beats the pooled-curvature ablation on update-level violations under out-of-support dynamics. The second is whether that advantage disappears in-support, which our account predicts. The third is whether per-update overhead stays within roughly 25% of a single natural-gradient step as $n$ grows, so that the curvature cost no longer scales with ensemble size.

If budget allows, we would finish with a 1/10-scale RC car on a floor with deliberately reduced friction, an out-of-support version of SPiDR's racecar task.