Robots and vehicles are usually taught from logs of human or scripted behavior, and those logs are both imperfect and partly unsafe. Copying them caps the learner at the demonstrator's skill. Letting the learner explore freely to do better invites the crashes we were trying to avoid. This note proposes a single policy-update rule that settles the tradeoff state by state. At each moment the agent may move away from what the demonstrations would do only as far as its remaining safety budget allows. The budget is the episode's cost allowance minus the cost already spent and minus a cautious estimate of the cost still to come. With plenty of slack, the update behaves like ordinary reinforcement learning and can exceed the data. Near the limit, the update pulls the policy back toward the safest behavior found in the logs, so the agent neither breaks the constraint nor freezes in place.
Technically, we treat every update as two steps. The first builds an ideal "target" policy in closed form. The second fits the neural network to that target. This is the operator view of Ghosh et al., which First Order Constrained Optimization in Policy Space (FOCOPS) already uses for constrained problems. Our change sits in the target-building step. The target becomes a geometric mixture of the current policy and a cost-reweighted policy fitted to the logs, and the mixing weight is a function of the remaining budget at that state. The setting is one that, according to the research file, no existing method addresses: a single mixed-quality offline dataset with no queryable expert and no separate "safe" and "reward" experts, followed by online fine-tuning that must stay within budget during training and not only at convergence.
Core Claim
Two strategies are on offer today, and each uses one fixed level of trust in the demonstrations everywhere. Optimism-driven methods such as ILDE and the reinforcement-learning phase of LOKI trust the data little and explore freely. Pessimistic offline methods such as CCAC, which inflates cost on unfamiliar actions, trust it completely. Gap 4 in the file is the tension between these two. Our claim is that this tension can be resolved locally. Let the trust weight depend on how much cost slack remains at the current state. That yields a learner that exceeds the best feasible behavior in its dataset while incurring fewer constraint violations during fine-tuning than a learner whose trust weight is fixed and tuned. If a well-tuned constant weight does just as well, the claim is false. The experiments are built so that this can be seen directly.
Mechanism
Some notation first. Let $\pi_k$ be the current policy and $\mu$ a reference policy fitted to the offline logs, reweighted toward low cost-to-go as described below. Let $A_R$ and $A_C$ be the reward and cost advantages, which measure how much better or worse an action is than the policy's average. Let $\nu \ge 0$ be the Lagrange multiplier, the adaptive price on cost, and $\tau$ a temperature. The improvement step builds the target
$$q(a\mid s) \;\propto\; \pi_k(a\mid s)^{\,1-\beta(s)}\,\mu(a\mid s)^{\,\beta(s)}\,\exp\!\Big(\tfrac{A_R(s,a)-\nu A_C(s,a)}{\tau}\Big),$$
and the projection step fits the network by minimizing the forward KL divergence from $q$ to $\pi_\theta$, exactly as FOCOPS does. When $\beta \equiv 0$ the update reduces to FOCOPS. When $\beta \equiv 1$ it is an advantage-weighted update anchored on the data policy, which is what a "behavior-regularized" baseline amounts to.
The new ingredient is the gate
$$\beta(s_t) \;=\; \sigma\!\Big(\tfrac{\hat C^{+}(s_t) - (\epsilon - C_{<t})}{\kappa}\Big).$$
Here $\epsilon$ is the episode cost budget and $C_{<t}$ is the cost already accrued in the episode. $\hat C^{+}$ is an upper quantile of an ensemble of forward cost critics, $\sigma$ is the logistic function, and $\kappa$ sets how sharp the gate is.
The decomposition of trajectory cost into a past part and a future part comes from Backward Value Functions. That work bounds total cost by cost-so-far plus expected future cost and learns the cost-so-far term through a time-reversed chain, because its policies are Markov in the state. We deliberately take a simpler route. In simulators like Safety-Gymnasium the per-step cost is observed, so we use the actual accrued cost and append the remaining budget to the policy's input. This removes the conservatism that BVF itself reports, which comes from using an expected cost-so-far in place of the realized one. Only the future term is estimated, and only that term carries pessimism. The asymmetry Gap 4 asks for therefore takes a concrete form. The reward advantage uses the ensemble mean, which is neutral-to-optimistic. The cost enters through an upper bound, and it restricts deviation only when that bound threatens the budget actually left.
The reference policy $\mu$ is built with a weighted fit. We train a behavior-cloning model on the full dataset, weighting each sample by $\exp(-A_C^{\text{data}}/\tau_c)$, where $A_C^{\text{data}}$ is the cost advantage under an offline-fitted cost critic. The result imitates the safest actions present in the data rather than its average behavior. This matters because DSRL reports that cloning all the data "achieves high rewards but fails safety constraints", while cloning only the safe trajectories is "overly conservative". SafeDICE also shows that subtracting unsafe behavior from a cloned policy can drive violations to zero, but only inside the data's support. Reweighting by cost advantage keeps the useful transitions from unsafe trajectories and lowers the weight of their dangerous actions. Value-based stitching, as described by Minghuan Liu, argues for keeping those transitions.
Research Program
The work has three parts.
The first part is theoretical. Mirror Learning guarantees monotonic improvement when the proximity term is a valid drift functional. Cheng et al. show that imitation and reinforcement are mirror descent with different gradient oracles, and Tomar et al. show that changing the reference distribution in the proximity term interpolates between update families. Our update is mirror descent with a state-dependent mixture reference. We will try to prove two bounds for a fixed gate within one iteration. The first is a per-iteration improvement bound on the Lagrangian objective. The second is a bound on the per-iteration increase in expected cost, and it should shrink as $\beta \to 1$ in the states where the agent spends its time. Whether the guarantee survives when $\beta$ itself changes between iterations is an open question. We will report honestly if we only obtain the fixed-gate result.
The second part builds the algorithm on top of the FOCOPS implementation in SafePO, which is distributed with Safety-Gymnasium. The offline stage fits $\mu$, a reward critic and a cost-critic ensemble on a DSRL dataset. The online stage alternates on-policy rollouts with the gated update.
The third part is a diagnostic study. It records where $\beta$ is low, which is where the agent is choosing to deviate. It then checks whether reward gains are concentrated in those states and cost incidents in the states where $\beta$ was high. If they are not, the gate is not doing what we claim.
Baselines
The naive reference points come straight from DSRL: behavior cloning on all data and behavior cloning on safe trajectories only, both run with the OSRL code on the same datasets. A useful method must have higher return than safe-only cloning and lower cost than all-data cloning.
The strongest purely offline method in the file is DRCORL. It regularizes a Gaussian policy with a diffusion model's score and projects apart conflicting reward and cost gradients. It reports a normalized return of 0.91 on CarGoal1, against 0.38 for the next safe baseline, while meeting the cost limit. We run its released code (JamesJunyuGuo/DRCORL) on identical datasets. Beating it means exceeding its normalized return at equal or lower normalized cost after a fixed online budget. To beat it is to answer the question of whether online interaction is worth its risk.
The strongest online competitors are three. The first is FOCOPS from scratch. It reports 1106.1 reward on Humanoid-Circle against CPO's 671.0 within the cost limit of 50, but it has no demonstration prior. The second is behavior-cloning warm-start followed by PPO-Lagrangian. The third is behavior-cloning warm-start followed by FOCOPS with a constant, tuned $\beta$, which is the main threat.
We also run a LOKI-style variant. It imitates the reweighted cloned policy for a randomized number of iterations and then switches to FOCOPS, because the hard-switching alternative in the file is known to cause performance dips.
The ablations isolate the gate. They are the same method with a constant $\beta$; a gate driven by forward cost alone, without accrued cost; a gate using the ensemble mean rather than an upper quantile; and $\mu$ fitted without cost reweighting.
Every method gets the same online environment-step budget, the same network sizes and the same number of hyperparameter configurations, chosen on a held-out seed set. The constant-$\beta$ baseline gets a sweep of the same size as our gate's $\kappa$ and quantile. Beating the online baselines means one of two things. Either we reach equal final return with significantly fewer cumulative training violations, or we reach higher return at equal violations, across a majority of tasks and over at least ten seeds.
Why not just…?
Why not just warm-start with behavior cloning and fine-tune with PPO-Lagrangian? Satija et al. argue that Lagrangian methods satisfy constraints only at convergence. The multiplier rises only after violations have accumulated. Safety-Gymnasium also documents oscillation between reckless and frozen behavior. Its headline result for velocity tasks is a 98% reduction in violations, but that comes with a 45% drop in reward. Cloning also tends to produce a sharply peaked policy, and Wang et al. link near-deterministic starting points to gradient explosion. The cumulative-violation curve during fine-tuning tests this directly. If warm-started PPO-Lagrangian matches us on that curve, the gated operator is unnecessary.
Why not just add a fixed KL penalty toward the cloned policy inside FOCOPS? This is the most serious threat, and it is the reason the gate exists rather than a plain behavior prior. A constant $\beta$ must be tuned once for the whole state space. It is either loose enough to explore in safe regions and then reckless near hazards, or tight enough near hazards and then stuck below the data's ceiling elsewhere. Our prediction is that the best constant $\beta$ shifts with the budget $\epsilon$, while the gate keeps working across budgets without retuning. We test this by sweeping $\epsilon$ over DSRL's budget levels with a single shared configuration. If the constant-$\beta$ baseline, retuned at each budget, matches the gate within noise, we will conclude that state-dependent gating adds nothing and say so.
Why not just stay offline with DRCORL or a constrained decision transformer? DRCORL itself acknowledges that offline policies are bounded by the data's support. DSRL further reports that sequence-conditioned methods can be misled by trajectories that were lucky to get high reward at low cost. The question is empirical: does a limited, budget-respecting online phase move return above the best feasible return in the dataset? We measure that directly against DRCORL's offline number. If online gains are marginal relative to the violations they incur, the offline method wins on these tasks.
Why not just penalize cost in the reward? Quanyi Li et al. show that a small penalty buys violations and a large one yields a "safe but useless" policy. We include a fixed-penalty PPO run only as a sanity point.
Proposed Experiments
The main testbed is offline-to-online fine-tuning on the DSRL datasets whose environments can be stepped online through Safety-Gymnasium: CarGoal1, PointGoal and the velocity-constrained AntVel tasks. They use mixed behavioral data and DSRL's normalized return and cost. A secondary check runs the FOCOPS velocity tasks, seeded with logs collected from a partially trained policy.
We report final normalized return and cost, the area under the cumulative-violation curve during fine-tuning, the fraction of training episodes that exceed $\epsilon$, and a ceiling-breaking ratio. That ratio is final return divided by the highest return among dataset trajectories that are within budget. I found no reported result on Lacuna for online fine-tuning from DSRL datasets, so all online numbers will be new.
Three checks decide whether the idea holds. The first is whether the gate beats constant $\beta$ on violations at matched return across a budget sweep with one configuration. The second is whether the ceiling-breaking ratio exceeds 1 while cost stays at or below the budget, which neither offline method nor safe-only cloning can do. The third is whether removing the accrued-cost input or the upper-quantile pessimism measurably increases training violations. That would show the specific asymmetry, optimistic on reward and pessimistic on remaining budget, is what carries the result.
A negative outcome on the first check would still be informative. It would show that behavior-anchored constrained updates need only a single trust level, and that would simplify the design space for safe fine-tuning from logs.
References
- An operator view of policy gradient methods
- First Order Constrained Optimization in Policy Space
- Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration
- Fast Policy Learning through Imitation and Reinforcement
- Constraint-Conditioned Actor-Critic for Offline Safe Reinforcement Learning
- Constrained Markov Decision Processes via Backward Value Functions
- Datasets and Benchmarks for Offline Safe Reinforcement Learning
- SafeDICE: Offline Safe Imitation Learning with Non-Preferred Demonstrations
- Curriculum Offline Imitating Learning
- Mirror Learning: A Unifying Framework of Policy Optimisation
- Mirror Descent Policy Optimization
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Don’t Trade Off Safety: Diffusion Regularization for Constrained Offline RL
- Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning