A robot that learns by copying demonstrations eventually makes a small mistake, lands in a situation the demonstrations never showed, and has no idea how to get back. Prior work fixes this by giving the policy a learned "pull" toward the demonstrated states. That pull is blind to danger. If a wall or hazard sits between the robot and the nearest demonstrated state, the pull points straight through it. Safe reinforcement learning has the opposite problem. It knows where danger is, but its learning signal is sparse, so a drifted agent near a hazard tends to freeze rather than recover. We propose to learn a recovery signal that is propagated only through states the agent has evidence are safe. The pull toward the demonstrations then follows routes around hazards instead of through them.
Concretely, we learn a potential function: a score per state that rises as the agent gets closer, by a safe route, to the more advanced parts of the demonstrations. We add its step-to-step change to the reward. Because it enters as a difference of potentials, it speeds up learning without changing which constrained policy is optimal. Because it is built only through certified-safe states, it never rewards entering danger. The result is a dense recovery signal that a constrained optimizer can follow without fighting its own safety constraint. The method is aimed at a common regime: a policy initialized from a few demonstrations, fine-tuned online under a cost budget, and required to stay safe during training.
Core Claim
The obvious combination is to add an imitation reward, such as a discriminator score or distance to the demonstrations, to a Lagrangian safe-RL agent. A Lagrangian agent is one that trades reward against cost with a learned penalty weight. In that combination the shaping term and the safety constraint point in opposite directions exactly where recovery matters. The agent then oscillates between hugging the hazard and freezing.
Our claim is that the conflict comes from how the shaping is built, not from the optimizer. Suppose recovery value is propagated backward from demonstration states with a pessimistic feasibility mask. States not yet shown to be safe pass no value, so the shaping signal agrees with the constraint by construction. We expect this to raise both safe task success and sample efficiency over imitation-plus-Lagrangian, shield-plus-fine-tuning, and unconstrained value flow, with training-time violations no worse than the safest baseline.
How the Potential Is Built
The mechanism adapts the "value flow" idea of Non-Adversarial Imitation Learning Provably Free of Compounding Errors. That paper shows that non-adversarial imitation methods like IQ-Learn give unvisited states a flat, uninformative value and so collapse to behavioral cloning. Its fix, Dual Q-DM, uses Bellman inequality constraints to push value backward from expert states into unvisited ones. This yields a gradient pointing home and linear rather than quadratic error growth with horizon. The paper supplies our method, but it is explicitly safety-agnostic. We change the backup in two ways.
First, the target is progress along the demonstration, not mere membership in it. Let $\rho(s)\in[0,1]$ be the normalized time index of the nearest demonstration state when $s$ lies within the demonstration support, and undefined otherwise. Pulling toward progress rather than toward the manifold borrows the incremental-progress lesson of Kreis et al. and Mehrabian et al.: rewarding change in progress avoids agents parking at high-reward states. It also avoids the zero-net-return parking that Escaping Offline Pessimism proves for purely conservative potentials. Here the potential is anchored to task advancement rather than to an uncertainty level, so its maximum lies at task completion, not at the frontier.
Second, value flows only through masked states. The potential $\Phi$ is fit on replay transitions $(s,a,s')$ by the backup
$$\Phi(s) \leftarrow \max\Big(\rho(s),\; m(s')\,\gamma \max_{a'} Q_\Phi(s',a')\Big),$$
where $\gamma<1$ discounts by the number of steps to rejoin the demonstrations, $Q_\Phi$ is the action-conditioned version of $\Phi$, and $m(s')\in\{0,1\}$ is a feasibility mask. The mask is 1 only if a learned feasibility estimate, trained from observed costs, is confident at its lower ensemble bound that $s'$ can stay within budget, and the replay data covers $s'$ densely enough. Everything else defaults to 0.
The pessimistic default speaks directly to the safety paradox identified in Breaking Safety Paradox with Feasible Dual Policy Iteration. That paper shows that a safe agent starves its buffer of unsafe samples, so feasibility estimates degrade at the boundary. In our scheme, an overconfident feasibility estimate at a rarely visited state cannot route recovery value through it unless coverage is also present. We therefore expect the paradox to cost speed, not violations.
The agent is trained on the sparse task reward plus
$$F(s,s') = \gamma\,\Phi(s') - \Phi(s),$$
subject to the usual expected-cost constraint. Along any trajectory, potential-based terms telescope to $-\Phi(s_0)$. The cost is unchanged. So the shaped and unshaped constrained problems have the same optimal policies, and the shaping cannot bias the agent toward violating the budget. Arbitrary dense bonuses lack this guarantee, as the research file notes for unshaped MDPs.
Training uses a PID-stabilized Lagrangian optimizer (CPPO-PID, from Safety-Gymnasium). For training-time safety it adds an optional fallback to the behavior-cloned prior when predicted cost exceeds budget, in the style of Safe Exploration via Policy Priors. If demonstrations themselves contain unsafe segments, they are first filtered with SafeDICE, which estimates a safe policy's visitation distribution by subtracting labeled bad demonstrations.
Research Program
The work has three parts.
The first is an analysis on a 2D point-navigation task with a hazard placed between the demonstration path and a set of perturbed start states. We plot $\Phi$ and its implied recovery direction for three cases: Dual Q-DM's unconstrained flow, our masked flow, and a barrier-shaped imitation reward. This visual test checks that masked flow routes around the hazard while unconstrained flow points through it.
The second is the main empirical study on benchmarks from the file, described below.
The third is a coverage study. We vary demonstration count and the strength of the pessimistic default to locate where masking makes recovery too conservative. The concern is that, with few demonstrations and a narrow safe corridor, little value flows and the method reduces to sparse-reward safe RL. Geometric Control of Out-of-Distribution Shift in Safe Offline RL reports exactly this kind of over-conservatism on Safe MetaDrive, where high-reward and low-cost regions barely overlap, so we expect the same risk there.
Baselines
The naive reference is behavioral cloning on the same demonstrations, deployed without fine-tuning. Its relevant reported numbers come from two papers. SafeDICE reports a 0.22 cost violation ratio for BC on RWRL Walker. RAIL reports that a vanilla Diffusion Policy on PointMaze reaches 100% success but only 15% safe success.
The strongest safety-first competitor at deployment is the BC or Diffusion Policy wrapped in RAIL's reachability filter. That filter computes the robot's swept volume in real time and substitutes a stopping maneuver on predicted collision. RAIL reports 97% success with 97% safe success on PointMaze, and this is the number to beat on that task. We also run a strengthened version: the same filter plus online fine-tuning of the policy on filter-corrected rollouts. This answers RAIL's own criticism that the policy "fights" the filter.
The strongest learning-based competitor is CPPO-PID trained on sparse task reward plus a standard imitation reward. We use two variants, a GAIL-style discriminator reward and a nearest-demonstration distance reward. Each gets the same demonstrations, environment steps, network sizes, prior fallback, and an equal hyperparameter search budget of 20 configurations per method on held-out seeds. The PPO-Lag result reported in Safety-Gymnasium, a 98% violation reduction versus PPO at a 45% reward drop on velocity tasks, calibrates what a pure Lagrangian costs in reward. We also include SOOPER with the BC policy as its safe prior, the closest existing method for safe online improvement of a demonstration prior. On RWRL Walker, SafeDICE's reported 0.00 violation ratio is the safety level we must match while improving return.
Two ablations isolate the key ingredient. The first is unconstrained value flow in the style of Dual Q-DM: the same progress target and the same optimizer, with the mask forced to 1. The second is an optimistic mask, which defaults unvisited states to feasible. The research file contains no per-task numbers for Dual Q-DM. Beating the baselines means statistically higher safe success than the strongest baseline at equal steps, with training-time cumulative violations no higher than SOOPER's, across at least five seeds.
Why not just…?
Why not just add an imitation reward to a Lagrangian safe-RL agent? This is the most natural combination and it will partly work. Our hypothesis is that it fails in one specific place. Where the nearest demonstration lies across a hazard, the imitation reward and the cost multiplier pull in opposite directions. That is the same balancing failure MetaDrive documents for crash penalties versus task reward. The test is the comparison above, run on start states deliberately placed so that the shortest path to the demonstrations crosses a hazard. If CPPO-PID with distance reward matches us there, the masking adds nothing and we will say so.
Why not just put a runtime shield on the imitation policy and fine-tune on the shielded rollouts? This is a genuine threat. Fine-tuning on filter-corrected actions could teach the policy to avoid the filter without any learned potential. RAIL itself does not update the policy, so no number in the file settles this. We run that variant directly and measure safe success and shield trigger rate. The shield needs an accurate geometric model that RAIL has for manipulators but that is not generally available, for example for hazard buttons in Safety-Gymnasium. So even a tie on PointMaze would leave our method useful where no model exists. A tie in the modeled settings would narrow our claim to model-free cases.
Why not just shape with a control barrier function on top of the demonstrations? Barrier Functions Inspired Reward Shaping adds a reward for moving away from hazard boundaries. It reports 1.56x faster convergence on Humanoid, but concedes no protection against violations, and it says nothing about where the agent should go to recover. Adding it to an imitation reward still sums two signals that can disagree. Our potential merges them into one direction. We include barrier shaping plus imitation reward as an extra baseline in the hazard-between-start-and-demonstrations test.
Proposed Experiments
The main suites are Safety-Gymnasium navigation tasks (Goal and Button, levels 1 and 2), PointMaze with walls following RAIL's protocol, RWRL Walker with SafeDICE's demonstration setup, and Safe MetaDrive as the stress test for narrow safe corridors. For each we collect 5 to 50 demonstrations from a trained safe expert. We evaluate from perturbed start states that are binned by whether a hazard separates them from the demonstration support. Metrics are task success, safe success (success with zero violations), cumulative training-time violations, cost violation ratio at evaluation, and environment steps to reach 80% safe success.
The decisive check is the hazard-separated bin. Masked value flow should beat both unconstrained value flow and the Lagrangian imitation baselines there. It should roughly tie them in the unseparated bin, where all signals agree. A gain uniform across both bins would mean we are measuring generic shaping benefit, not safety-aware recovery.
A second check is the buffer analysis from the FDPI paper. As training proceeds and unsafe samples become rare, the optimistic-mask ablation should show late-training violations, and the pessimistic mask should not.
Finally, if Safe MetaDrive shows our method stalling below SOOPER, that identifies the coverage limit of pessimistic masking, and we will report it as the method's boundary rather than tune around it.
References
- Non-Adversarial Imitation Learning Provably Free of Compounding Errors: The Value Flow Mechanism
- Interactive Shaping of Granular Media Using Reinforcement Learning
- Finding Increasingly Large Extremal Graphs with AlphaZero and Tabu Search
- Escaping Offline Pessimism: Vector-Field Reward Shaping for Safe Frontier Exploration
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Safe Exploration via Policy Priors
- SafeDICE: Offline Safe Imitation Learning with Non-Preferred Demonstrations
- Geometric Control of Out-of-Distribution Shift in Safe Offline RL
- RAIL: Reachability-Aided Imitation Learning for Safe Policy Execution
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
- Barrier Functions Inspired Reward Shaping for Reinforcement Learning