Safety-Gated Departure from Imitation Priors

An agent trained to copy demonstrations can be no better than the person who demonstrated. To beat that person it has to try actions the person never took. In safety-critical settings that is exactly where the danger lies. Current methods resolve this tension in one of two ways. Some keep the learner on a short leash near the demonstrator everywhere, which makes exploration safe but locks in the demonstrator's mediocrity. Others let a free reinforcement learner, a trial-and-error agent, roam, which improves on the demonstrator but pays for it in crashes during training.

We propose a different rule for how far the learner may stray. The allowed distance from the demonstrator should depend, state by state, on how confident the agent is that it has safety budget to spare. It should not depend on how much demonstration data exists. Where a pessimistic estimate of future safety cost leaves a wide margin, the agent may depart freely from the imitation policy and improve. Where the margin is thin, or the cost estimate is uncertain, the agent defers to the demonstrator. The bet is that this one change lets an agent climb well above a safe-but-suboptimal demonstrator while incurring far fewer training-time violations than imitation followed by standard constrained reinforcement learning. A second bet is that it turns the known late-training failure mode of safe RL into harmless caution.

Core Claim

The closest existing work is BRIDGE (Fine-tuning Behavioral Cloning Policies with Preference‑Based Reinforcement Learning). It confines online fine-tuning of a behavior-cloned policy to a Hellinger ball (a bound on how different two action distributions may be) of radius $O(1/\sqrt{n})$ around the expert, where $n$ is the number of demonstrations. The radius measures how well we know the expert. It is uniform across states, and it shrinks as data grows, so a suboptimal expert becomes a tighter cage. We replace it with a radius that measures how safe deviation is here.

Concretely, let $\pi_{\text{BC}}$ be the behavior-cloned policy, $\pi$ the improving policy, and $b$ the episode safety budget. Let $\bar{C}(s_t)$ be the cost accumulated so far in the episode, and $U_c(s_t,\pi)$ an upper confidence bound on expected future cost from an ensemble of cost critics. The policy maximizes task value subject to

$$ D_{\text{KL}}\big(\pi(\cdot|s_t)\,\|\,\pi_{\text{BC}}(\cdot|s_t)\big) \le \delta(s_t), \qquad \delta(s_t) = \kappa \cdot \max\big(0,\; b - \bar{C}(s_t) - U_c(s_t,\pi)\big), $$

where $\kappa$ converts spare budget into allowed deviation. The cost-so-far plus future-cost decomposition comes from backward value functions (Constrained Markov Decision Processes via Backward Value Functions). That work enforces a trajectory constraint locally at every step but has no imitation prior to fall back on. We reuse its decomposition, and our use is different: it gates how far the learner may leave a prior, rather than serving as a hard constraint on an unanchored policy.

The second claim concerns the "safety paradox" identified by Breaking Safety Paradox with Feasible Dual Policy Iteration. As a policy becomes safe, its replay buffer runs out of violations, the cost model degrades, and costs spike late in training. Under our rule, a degraded cost model shows up as wider ensemble disagreement, so $U_c$ rises and $\delta$ shrinks back toward the demonstrator. Starving the cost model of data should therefore produce conservatism, not spikes. FDPI addresses the same problem with a crash-seeking dual policy, which is unusable outside simulation. Our method needs no deliberate violations.

Research Program

Demonstrations and cost data. The first stage builds the prior and the cost model from offline logs. DSRL (offline safe RL datasets with reward and cost labels, used in OSIL and Online Optimization for Offline Safe Reinforcement Learning) already contains mixed trajectories. From each task's log we construct a safe-but-suboptimal demonstrator by keeping trajectories that respect the cost limit and fall in a below-median reward band. Behavior cloning on these gives $\pi_{\text{BC}}$, which by construction has a measurable ceiling.

The cost-critic ensemble is pretrained on the full log, including the violating trajectories. This is how we avoid the "expert blind spot" documented by SAFE-GIL, in which demonstrations never visit near-failure states. SAFE-GIL fills that hole with reachability computations that do not scale. We instead use failures someone else already logged.

Online phase. The second stage is an off-policy actor-critic in the matching Safety-Gymnasium task. A task critic is trained with temporal-difference learning on offline plus online data, so it can stitch partial successes, the mechanism that lets offline RL beat imitation (Offline Q-learning on Diverse Multi-Task Data Both Scales And Generalizes). The actor update is a per-state projection onto the KL ball of radius $\delta(s)$. This is a first-order projection in the style of FOCOPS (First Order Constrained Optimization in Policy Space), which projects onto a constraint-defined feasible set. Here the projection target is a state-varying neighborhood of the prior. At deployment the agent executes $\pi$ only where $\delta(s)>0$ and otherwise takes $\pi_{\text{BC}}$'s action.

Analysis. Beyond benchmark scores, we will log $\delta(s)$ over training and over state space, along with ensemble disagreement near cost boundaries. This directly tests whether the paradox mechanism behaves as predicted: falling violation counts in the buffer should coincide with shrinking $\delta$ near hazards rather than with cost spikes.

Stretch variant. Cost labels are often absent in practice. In a stretch variant we replace the labeled cost critic with OSIL's preference-learned cost model, which is trained from a small set of known-bad trajectories. OSIL itself is offline-only, so this variant would test whether learned costs suffice for online gating.

Baselines

The naive reference point is deploying $\pi_{\text{BC}}$ unchanged. It defines the ceiling: its average reward is the number we must exceed. Its cost tells us whether compounding error already makes the demonstrator unsafe when the policy runs in closed loop.

The strongest practical competitor is behavior cloning followed by PPO-Lagrangian fine-tuning, run from SafePO, the single-file implementations of 16 safe RL algorithms packaged with Safety-Gymnasium (Safety Gymnasium). We also run its PID-damped variant, CPPO-PID, because the same paper reports that plain Lagrange multipliers oscillate between reckless and overly cautious behavior. The reported reference is that PPO-Lag achieved a 98% reduction in safety violations relative to PPO on velocity tasks, at a 45% drop in total reward. Those figures concern the converged policy, not the violations incurred during training, which is the quantity we target.

For the safety side, FDPI reports near-zero violations across 14 Safety-Gymnasium environments (Breaking Safety Paradox with Feasible Dual Policy Iteration). It learns from scratch and deliberately induces violations through its dual policy, so we report its training-time cost alongside its final cost.

For the offline side, we take O3SRL (Online Optimization for Offline Safe Reinforcement Learning). On DSRL at cost limit 5 it was reported as the only method to keep normalized cost at or below 1 on all tasks, with CDT and CCAC exceeding limits by 10x to 20x. O3SRL followed by the same online budget tells us whether offline safe RL alone already closes the gap.

For prior-anchored safe fine-tuning, we implement a BRIDGE-style uniform demonstration-centered trust region using task reward rather than preferences. No reported result was found on Lacuna for BRIDGE on Safety-Gymnasium or DSRL.

For value-guided improvement over a frozen prior, we use Q-planning (Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning) with an added cost-critic veto on candidates. This resembles CIMRL's selection among prior proposals (CIMRL) and tests whether staying inside the prior's samples is enough.

All methods share the same $\pi_{\text{BC}}$ initialization, the same offline log, the same online step budget (one million environment steps per task), the same network sizes, and an equal hyperparameter search of 20 configurations each, with at least five seeds.

Three ablations isolate the key ingredient. The first sets $\delta$ to a tuned constant, which recovers KL-to-BC regularization with a Lagrangian. The second sets $\delta$ from demonstration visitation counts, a continuous analogue of the deferral rule in Decision-Point Guided Safe Policy Improvement. The third uses the mean cost estimate without the ensemble upper bound.

Beating the field means three things at once. Final reward must be significantly above $\pi_{\text{BC}}$. Cumulative training-time cost must be lower than BC plus PPO-Lag and CPPO-PID at matched or higher final reward. Converged cost must be within budget. The count-based and constant-$\delta$ ablations should each lose on at least one of these axes.

Why not just…?

Why not just fine-tune the behavior-cloned policy with PPO-Lagrangian? This is the real threat. It might match our final reward and cost. Where we expect it to lose is training: the Lagrange multiplier starts near zero, so the early policy behaves like unconstrained RL. Safety-Gymnasium documents the resulting oscillation. Our experiments report the cumulative training cost curve and the late-training cost spikes for both methods. If BC plus CPPO-PID matches us on training-time cost too, the proposal's value collapses to analysis, and we would say so.

Why not just add a fixed KL penalty to the cloned policy? A uniform anchor re-imposes the ceiling where the demonstrator is weak. APC-RL (APC-RL) reports that prior-anchored methods such as PARROT reached only 0–7% success in PointMaze with misaligned demonstrations, while its prior-free component reached 100%. Loosening the anchor uniformly instead brings back unsafe exploration near hazards. The constant-$\delta$ ablation, tuned over the same 20-configuration budget, tests directly whether state dependence matters.

Why not just roll out the cloned policy, discard unsafe or failed episodes, and retrain on the rest? Filtered fine-tuning cannot learn where boundaries are, because it throws the boundary data away. Q-planning reports filtered fine-tuning stalling at 55% and 30% on two real manipulation tasks, where value guidance reached 90% and 80%. We include filtered fine-tuning on the same rollouts as an additional reference.

Why not just veto unsafe candidates from the prior with a cost critic? That keeps every action inside what the prior proposes, which is the bottleneck in CIMRL's discrete selection. Gains are then capped wherever the prior never samples the better action. The safety-vetoed Q-planning baseline measures how much of our improvement comes from actions outside that support.

Proposed Experiments

The main suite uses the DSRL tasks with Safety-Gymnasium counterparts (PointGoal1, PointButton1, CarRun, and the Walker and Ant velocity tasks), each at cost limit 5. We report three numbers per task. The ceiling ratio is final reward divided by demonstrator reward. We also report cumulative online training cost and converged normalized cost.

Aggregate cost can hide a single long run of violations, as Revisiting Safe Exploration in Safe Reinforcement learning shows. We therefore also report its Expected Maximum Consecutive Cost metric, and we run all methods on its Circle2D levels 1–4, where the cost region and task optimum are visible. On Circle2D the gating can be plotted directly.

The decisive check for the paradox claim is a buffer-starvation experiment. After convergence, we continue training while holding out all offline violating data, then track three quantities: ensemble disagreement near hazards, $\delta$ at those states, and late-training cost. The claim is supported if disagreement rises, $\delta$ falls, and cost stays flat. Under the same intervention, FDPI's motivating baseline and BC plus PPO-Lag should show spikes.

The decisive check for the ceiling claim compares tasks where the demonstrator's weakness lies in the safe interior (for example, slow velocity tasks) with tasks where improvement requires passing near hazards (for example, Goal and Button tasks). Our method should exceed the count-based ablation most clearly on the first group. That pattern would show that gating by safety rather than by demonstrator familiarity is what lets the agent break the ceiling.