Backing Away from Newly Unsafe Actions: Cost-Side Desaturation for Sim-Trained Softmax Policies

A policy trained to convergence in simulation often becomes very confident. Whenever it chooses among discrete options (which gait, which side of an obstacle, which mode of a mixture policy), it puts probability close to 1 on one choice. On the real robot, that confident choice can turn out to be unsafe because of friction, latency, or contact effects the simulator never modeled. The policy then needs to move probability off that action quickly, using a handful of real episodes. This is exactly what a softmax policy is worst at. Its gradient with respect to the logit of an action scales with $\pi(a)(1-\pi(a))$, so the signal that should push the policy away from the costly action is close to zero precisely because the policy is so sure of it. We call this the saturation lock. The safety correction is suppressed when it is most needed. Each real episode spent waiting is another violation. Meanwhile, a Lagrange multiplier (the adaptive weight on the safety cost) keeps climbing, which sets up the overshoot into a frozen, "safe but useless" policy that safe-RL practitioners already struggle with.

We propose a small, first-order change to the policy-gradient update used during real-world fine-tuning, and we will test it. When an executed action incurs more real cost than the simulator predicted, its logit is pushed down by an amount that does not shrink with confidence. The reward side of the update is left in its usual form. In states where simulation and reality agree, the policy keeps its confident choices. In states where reality is worse than the simulator predicted, the policy can back away in a number of episodes that does not blow up with how saturated the policy was.

Core Claim

Consider a softmax policy with logits $\theta$ that takes action $A$ and observes a real cost $c$. The standard score-function contribution of the cost term to the logit update is $c\,(\mathbf{e}_A - \pi)$, where $\mathbf{e}_A$ is the one-hot vector for $A$. Its component on $\theta_A$ is $c\,(1-\pi(A))$. If the logit must fall by a fixed margin $\Delta$ before the action stops dominating, and the step size is $\eta$, then the number of corrective updates scales as

$$N_{\text{reg}} \approx \frac{\Delta}{\eta\, c\,(1-\pi(A))},$$

which diverges as the pretrained policy saturates. The same pathology underlies the exponential plateaus proven for vanilla softmax policy gradient by Li et al. and Li et al..

Our update replaces the cost term, only where reality is measurably worse than simulation, with the one-sided form

$$g = (R - b)(\mathbf{e}_A - \pi) \;-\; \lambda\, m(s,A)\,\big(c - \hat c_{\text{sim}}(s,A)\big)_+\,\mathbf{e}_A .$$

The symbols are as follows:

  • $R$ is the return and $b$ is the reward baseline.
  • $\lambda$ is the cost multiplier.
  • $\hat c_{\text{sim}}(s,A)$ is a cost critic trained in simulation.
  • $(\cdot)_+$ keeps only positive excess cost.
  • $m(s,A)\in\{0,1\}$ is a mismatch gate that switches on when the excess cost exceeds an ensemble-based confidence threshold.

For a softmax, a step along $-\mathbf{e}_A$ always lowers $\pi(A)$. Its size does not depend on $1-\pi(A)$, so $N$ becomes independent of saturation.

The removal of the $-\pi$ factor is borrowed from An Alternate Policy Gradient Estimator for Softmax Policies. That paper applies it to the whole update and relies on reward noise and an optimistic baseline to produce a random walk out of saturation. We deliberately do not do that. A random walk driven by reward is exactly the kind of high-variance exploration that causes violations on hardware. Our use of the trick is one-sided: it only ever removes probability from actions that reality has shown to be costlier than expected. It is also local, because the gate is off wherever simulation remains trustworthy. The claim we will test is that this asymmetry gives fast safety correction without unsafe exploration.

Research Program

The work has three parts.

The first part is an analysis in tabular and bandit settings. We will show that the saturation lock is real and quantitative. Concretely, we expect that the number of real violating episodes under standard CMDP policy gradient grows like $1/(1-\pi(A))$ in the pretraining confidence. We will also trace how the Lagrange multiplier evolves during the lock and show that its accumulated value predicts the size of the subsequent reward collapse. This ties a known open problem, Lagrangian instability as catalogued in Safety Gymnasium, to a specific and fixable cause in the transfer setting. For the one-sided estimator we will characterize its bias. Its expectation differs from the true cost gradient by $\mathbb{E}[c]\,\pi$ restricted to the gated set. We expect to show that it still lowers the expected cost monotonically when the gate is correct, and to bound the reward lost when the gate fires falsely.

The second part extends the update to neural policies. The saturation lock lives in the final logit layer, both in discrete-action networks and in the softmax that chooses between components of a mixture policy. Multimodal policies are specifically motivated for safety: Policy Bifurcation in Safe Reinforcement Learning proves that unimodal continuous policies must violate constraints in obstacle layouts with holes and introduces Gaussian-mixture policies (MUPO) to fix this. That paper does not consider what happens when the gate saturates on a mode that works in simulation but jams on real hardware. Our estimator applies directly to that gate: the excess cost of the executed mode pushes down that mode's gate logit. The gate $m$ is built in the style of SPiDR. SPiDR penalizes costs by the disagreement of an ensemble of randomized simulators as a static pessimism term. We reuse the ensemble only to decide when an observed real cost is outside what simulation could explain.

The third part handles where the freed probability goes. Pushing down one action raises the others in proportion to their current probabilities, and some of those may also be unsafe. During the correction phase we therefore keep a simulation-trained backup safety critic, as in the reachability shield of Sim-to-Lab-to-Real. The point is that the shield is a guard rail and not the learning mechanism. We will measure how often it has to intervene.

Baselines

The naive reference point is to deploy the simulation-trained policy and fine-tune it on real data with standard PPO-Lagrangian, using the usual score-function gradient for both reward and cost. This is the setting in which we predict the saturation lock, so every result is reported relative to it. For context, Safety Gymnasium reports that PPO-Lag achieved "a 98% reduction in safety violations compared to standard PPO, with only a 45% drop in total reward" on velocity tasks. That figure is for training from scratch in simulation, not for transfer, so it is not a direct target.

The strongest competitor in principle is natural-gradient primal-dual optimization, NPG-PD from Ding et al.. It comes with dimension-free global convergence guarantees for constrained problems. For tabular softmax it is known not to suffer vanishing updates, because the Fisher preconditioning cancels the $\pi(1-\pi)$ factor. In deep settings we run its practical surrogate, a trust-region Lagrangian method with Kronecker-factored Fisher estimates. We found no reported result on Lacuna for NPG-PD on any of the transfer benchmarks below, so we will report our own numbers. For fairness, every method uses the same pretrained checkpoint, the same real-interaction budget, and the same hyperparameter search budget (equal numbers of configurations, tuned on held-out shift seeds).

On the tabular gridworld, the strongest reported method is the LP-based approach of Provably Safe Sim-to-Real Transfer. It reports "0% unsafe iterations" against 68.5% for an unconstrained sim-to-real baseline, using 77,958 real samples against 296,910 for a method without a simulator. That method solves an LP over state-action visitation frequencies, so it has no gradient and cannot saturate. It is an upper reference for what a parameterized method can hope to match, not a peer. Beating our naive baseline while landing within a small factor of its sample count would be a strong result.

We also compare against SPiDR's static pessimism, against the full alternate estimator applied to both reward and cost, and against a shield-only approach.

Two ablations isolate the key ingredient. The first applies the one-sided cost term everywhere, with no mismatch gate. It tests whether localization matters. The second keeps the gate but uses the regular $(\mathbf{e}_A-\pi)$ cost term with a boosted step size inside the gate. It tests whether the benefit comes from removing the saturation factor or merely from larger steps.

Beating a baseline means fewer cumulative real-world violations at matched final reward, or higher final reward at matched violations. Both are measured over the same budget of real episodes, with confidence intervals over at least ten shift seeds.

Why not just…?

Why not just raise the temperature or add an entropy bonus at deployment? This desaturates every state, including those where the simulator is right and the confident choice is the safe one. That spends safety budget re-exploring actions that never needed revisiting. Beyond Softmax and Entropy also shows that Shannon entropy regularization leaves the flat regions of the softmax landscape in place on long-horizon chains. Our ungated ablation together with a temperature-reset baseline tests this directly. If uniform desaturation matches our gated update on violations, the localization claim fails, and we will report that.

Why not just use the natural policy gradient? This is the most serious threat. In the tabular experiments we expect NPG-PD to escape the lock about as fast as our update, and we will say so if it does. Our two claims against it concern the deep and mixture-gated settings. First, the Fisher matrix of a saturated softmax is nearly singular, so practical implementations add damping, and damping brings the $\pi(1-\pi)$ suppression back. Second, NPG amplifies the reward signal as well as the cost signal, which adds the variance on the reward side that our asymmetric update avoids. Should NPG-PD match us in deep settings at comparable compute, the contribution shrinks to a cheaper first-order equivalent, and we will frame it that way.

Why not just shield and let the shield handle safety? A shield replaces the unsafe proposal with a backup action. The policy's own logit for the bad action therefore never receives a cost signal, and the policy stays saturated and dependent on the shield indefinitely. The shield is also trained in simulation, so it is least reliable exactly in the mismatch region. Sim-to-Lab-to-Real addresses this with a physical intermediate lab stage and many layouts. We will measure the shield-intervention rate over time. Under shield-only it should stay flat. Under our method it should decay as the policy learns to stop proposing the bad action.

Why not just use better domain randomization so the real dynamics are covered? Randomization reduces how often the lock occurs, but it does not change how slowly the policy escapes once it occurs. SPiDR reports that plain randomization incurred 26.1× more violations than its pessimistic variant on a Unitree Go1 and 6.2× more on a RaceCar. We will pretrain all methods with randomization and evaluate on shifts outside the randomized range.

Proposed Experiments

The first experiment is a mechanism check on the 5×5 windy gridworld from Provably Safe Sim-to-Real Transfer. We pretrain tabular softmax policies in the windless simulator to controlled saturation levels, with $1-\pi(A)$ ranging from $10^{-1}$ to $10^{-4}$, and then fine-tune in the windy "real" grid. The decisive check is a plot of real violating episodes against $1/(1-\pi(A))$. We predict roughly linear growth for PPO-Lagrangian and a flat curve for our update and for NPG-PD. We will also report how the multiplier grows over the same episodes. The 3-armed-bandit and chain setups from An Alternate Policy Gradient Estimator for Softmax Policies, where the regular estimator "failed to learn in 100% of trials" from bad initializations, serve as sanity checks after adding a cost channel.

The second experiment tests neural and mixture-gated policies on Safety-Gymnasium navigation (Goal and Button tasks with Point and Car agents). We give the agents a discrete obstacle-bypass choice through a MUPO-style mixture gate, plus a discrete-action variant. The sim-to-"real" shift is created by perturbing friction, actuation latency, and mass at test time beyond the randomization range used in training, so that the simulator's preferred bypass mode becomes cost-incurring. The metrics are cumulative violations over the first $K$ real episodes, the number of episodes until per-episode cost falls back under budget, final reward, false-gate rate, and shield intervention rate. A second shift family comes from the perturbed environments in RRLS, run with discrete gait-mode selection on top of a low-level controller. This checks that the effect is not specific to navigation.

The result that would falsify the proposal is this: across saturation levels in the second experiment, the gated one-sided update fails to reduce cumulative violations relative to PPO-Lagrangian at matched final reward, or NPG-PD matches it at equal wall-clock compute. The result that would support it is a flat time-to-correction curve, together with violations matching the ungated ablation in mismatch states but much lower in matched states. Hardware validation on a small racecar platform comparable to SPiDR's is a stretch goal, contingent on the simulated results holding.