Robots that learn by copying demonstrations fail in a predictable way. Small mistakes carry them into states the demonstrator never visited, such as the edge of an obstacle or a joint near its limit. The policy has no examples of how to act there, so the mistakes grow. A known fix is to deliberately push the demonstrator toward danger while they are demonstrating, so that the dataset contains recovery maneuvers. That fix assumes the demonstrator can always recover. Real demonstrators are often novices, tired operators, or mediocre scripted controllers, and pushing them toward the boundary produces crashes and bad labels rather than useful recoveries.
We propose to collect these boundary-seeking demonstrations under a safety filter. A barrier function is learned in advance from offline data. A barrier function is a scalar score of each state that is positive when the state is safe and drops toward zero at the boundary. It is used twice during collection. Its gradient chooses the direction in which to disturb the system, pushing it toward the boundary. As a constraint inside a small quadratic program, it also minimally corrects the teacher's commands so that the push never becomes a violation. The student is trained on the actions that were actually executed. Each training sample is weighted to discount steps where the filter, not the teacher, was effectively in control. The claim is that this recovers most of the blind-spot coverage that reachability-guided disturbance gives with an optimal expert, while using a low-skill teacher and incurring essentially no violations during collection.
Core Claim
SAFE-GIL motivates this proposal. It injects adversarial disturbances computed by Hamilton-Jacobi reachability (a grid-based computation of the set of states from which collision becomes unavoidable) to steer the demonstrator toward that set. It reports that with 8 demonstrations, behavior cloning fails about 80% of the time on ground navigation while SAFE-GIL stays near 10%, and that it achieves 0% collisions on a Crazyflie quadrotor where behavior cloning fails 20–80% of the time. SAFE-GIL has two limits. It relies on an expert who survives worst-case pushes, and its reachability grid does not scale beyond low-dimensional systems.
We keep SAFE-GIL's core insight, which is that recovery data must be collected near the boundary on purpose. We change three things. The disturbance is computed from the gradient of a barrier learned from offline data, not from a reachability grid. The teacher is shielded during collection. The imitation loss is restricted so that the student copies what the teacher contributed rather than what the shield contributed.
The barrier comes from V-OCBF, which supplies the method. V-OCBF learns state-wise control barrier functions model-free from static datasets using expectile regression, and uses a learned dynamics surrogate only at test time inside the filter. It reports 98.28% safe episodes on a Dubins-car testbed versus 93.56% for BC+CCBF and 68.91% for COptiDICE. We deliberately follow V-OCBF's finding that putting a learned dynamics model inside barrier training degrades safety. The barrier is frozen before any demonstration is collected, and no model is unrolled during policy training.
One observation shapes the design and keeps us honest about differentiable safety layers. Suppose the safe set of actions at state $s$, written $C(s)$, is convex. Under the usual control-affine surrogate it is a half-space. Let $a$ be the demonstrated action. Then the safe action that minimizes expected squared imitation error is
$$\arg\min_{x \in C(s)} \mathbb{E}\,\|x - a\|^2 = \mathrm{Proj}_{C(s)}\big(\mathbb{E}[a]\big),$$
because the expectation splits into $\|x - \mathbb{E}[a]\|^2$ plus a constant. This is exactly what a post-hoc filter applied to a plain behavior-cloning policy outputs.
At the population optimum, then, training end-to-end through a projection layer gives the same result as filtering after training. This contrasts with the effects reported by GLAS, which found up to 6.25% less control effort from end-to-end training, and by Safe Policy Learning for Continuous Control. Whatever gain those papers saw must come from function approximation, finite data, or non-squared losses, not from the gradient itself.
We therefore do not rest the proposal on "train through the layer." The differentiable layer is one design choice, tested by ablation. The load-bearing ingredient is what data reaches the student. A filter cannot invent recovery maneuvers the dataset lacks.
Research Program
The barrier $h(s)$ is learned from the cost-labeled offline datasets of DSRL and the corresponding Safety Gymnasium environments, following V-OCBF. A dynamics surrogate gives the action-dependence of $\dot h$, which turns the barrier condition into a linear constraint on the action at each state.
Collection. At each step of collection, a disturbance of bounded magnitude is applied along $-\nabla h(s)$ with a schedule that controls how often the system is pushed. The teacher's command $a^{\text{cmd}}$ is projected onto $C(s)$ to give the executed action $a^{\text{exec}}$. Both are logged. The disturbance replaces SAFE-GIL's reachability computation and costs one gradient evaluation per step.
Teachers. Teachers are imperfect by construction, following the proxy-teacher protocol of AGSA. These are neural controllers at low, medium, and high skill. We additionally add reaction latency and temporally correlated noise to mimic the fatigue and jitter discussed under the expert-optimality direction.
Student training. The student is trained by weighted regression onto $a^{\text{exec}}$, the action consistent with the logged next state, with weight
$$w_t = \exp\big(\beta \hat A_t\big)\,\exp\big(-\kappa \|a^{\text{cmd}}_t - a^{\text{exec}}_t\|\big).$$
The first factor is the advantage weighting of MARWIL, used when rewards are logged. Here $\hat A_t$ is the normalized return-minus-baseline estimate and $\beta$ is its temperature. It addresses suboptimality inside the safe set, which the shield cannot see. The second factor discounts steps where the shield overrode the teacher, with $\kappa$ setting how sharply. At those steps the label is the closest safe action, which carries no task knowledge and would teach the student to slide along walls.
This weight penalizes being overridden, not being near the boundary. Boundary-Seeking Policy Gradient shows that optimal constrained policies often sit on an active constraint. A good teacher who hugs the boundary without being corrected therefore keeps full weight.
Deployment. The same projection runs at deployment, so the student's executed actions are certified to the extent the barrier is correct.
What would be new. The contribution is the demonstration, with clear ablations, that shielding plus barrier-directed disturbance turns weak teachers into a source of recovery data. It comes with an analysis of when end-to-end training through the layer adds anything beyond the pointwise equivalence above.
Baselines
Naive reference. Plain behavior cloning on unshielded, undisturbed data from the same teacher, with the same post-hoc filter added at deployment. This is what a practitioner would do by default. It measures how much of the final safety comes from the filter alone, and how much task success is lost to stalls and wall-sliding when no recovery data exists.
Strongest existing method for blind-spot coverage. SAFE-GIL, run in two forms. The first gives it an optimal teacher, which serves as an upper reference. On its ground-navigation testbed it reports about 10% failure with 8 demonstrations, against about 80% for behavior cloning. The second gives it the same low- and medium-skill teachers as our method, without a shield. Crashed episodes are counted as collection violations and either kept or discarded, both reported.
On Safety Gymnasium tasks, where a reachability grid is infeasible, SAFE-GIL is approximated by our barrier-gradient disturbance without the shield. This approximation doubles as the "disturbance only" ablation.
Strongest offline competitor. FISOR is reported as the only method satisfying constraints on all 26 DSRL tasks it evaluated, and it is also evaluated on safe offline imitation without rewards. No reported per-task return figure for FISOR was found on Lacuna. For comparison we therefore rerun it on the same DSRL datasets plus our collected demonstrations, with the same total transition count.
Interactive competitors. ThriftyDAgger reports 73% autonomous success on Robosuite peg insertion. AGSA has no reported number on Lacuna for the MuJoCo teacher-skill experiments. Both are given the same teacher-interaction budget, counted in teacher-controlled steps.
Fairness. All methods share student architecture, number of teacher steps, barrier (where one is used), and a fixed hyperparameter-search budget of equal size.
Ablations. Shield only, with no disturbance. Disturbance only, with no shield. Shield plus Gaussian disturbance of matched magnitude in place of $-\nabla h$, which isolates directed pushing. Override weight off, $\kappa = 0$. Advantage weight off, $\beta = 0$. Finally, the same data trained either end-to-end through a differentiable projection layer or with relabeled targets and a post-hoc filter.
What counts as beating SAFE-GIL. With a low-skill teacher, our method must match or beat optimal-teacher SAFE-GIL's student failure rate within its confidence interval, while recording far fewer collection violations than unshielded SAFE-GIL with the same teacher.
Why not just…?
Why not just put a safety filter on a behavior-cloning policy at deployment? It does keep the robot out of unsafe states, and V-OCBF shows such filters work. Our pointwise equivalence even says the filter is already the right answer wherever the dataset has coverage. What it cannot provide is a competent action in states the teacher never visited. There it projects a meaningless proposal onto the boundary, producing the deadlock-style behavior GLAS attributes to reactive filters like ORCA. The naive baseline measures this directly through stall rate and success, at matched safety. If it matches our method, the data-collection idea is unnecessary, and we will report that.
Why not just run SAFE-GIL with the imperfect teacher and throw away crashed episodes? Discarding crashes removes the very boundary segments the method exists to collect. In hardware, those crashes have already happened by the time they are discarded. The "disturbance only" arm quantifies both the violation count and the coverage lost. The comparison is decisive here. If unshielded collection with filtering-by-outcome reaches our student performance with few crashes, the shield is not earning its place.
Why not just train an offline safe RL method like FISOR on more data? This is a real threat when the offline data is rich and already covers near-boundary states. The DSRL datasets were generated by safe RL experts that do approach constraints. We expect the gap to open in low-data regimes and in states that only directed disturbance reaches. We test this by holding total transitions fixed and varying how many come from our shielded collection. If FISOR wins at every mix, we will conclude that the bottleneck is policy extraction rather than data.
Why not just train through a differentiable projection layer and skip the special collection? The equivalence above predicts that, with squared loss and a convex safe set, end-to-end training changes little relative to relabeling plus filtering. Any benefit would have to come from how the network shares capacity across states. The end-to-end versus post-hoc ablation on identical data tests this prediction. A null result would be informative about the GLAS and V-OCBF tension, not a failure of the proposal.
Proposed Experiments
Testbeds. We first reproduce SAFE-GIL's ground-navigation setting so its reported numbers anchor the comparison. We then move to Safety Gymnasium navigation and velocity-limited locomotion tasks, where barriers are learned from the corresponding DSRL datasets and reachability grids are unavailable. MetaDrive is the stretch environment, chosen because DSRL reports that most algorithms struggle there.
Teacher conditions. Each environment is crossed with low, medium, and high teacher skill and with three demonstration budgets.
Metrics. Collection-time violations (teacher episodes with any cost). Student cost and safe-episode rate. Normalized return or goal success. Stall rate, defined as episodes ending with near-zero velocity short of the goal. Fraction of steps where the filter is active at deployment. Control effort.
Barrier calibration. Barrier errors set the ceiling on the shield's promise. We therefore report violations that occur despite the shield, broken down by whether the state lay inside the barrier's training support. This also tests whether pushing toward the boundary exposes barrier errors, the imitation-side version of the safety paradox described in Breaking Safety Paradox with Feasible Dual Policy Iteration.
Decisive checks. Three results settle the proposal. First, with low-skill teachers, the full method must reach student failure rates near optimal-teacher SAFE-GIL while unshielded collection crashes repeatedly. Second, shield-only must underperform the full method, showing that directed disturbance rather than safe execution supplies the recovery data. Third, the override weight $\kappa$ must matter, showing that copying shield-authored actions does harm. If the first holds but the second does not, the contribution shrinks to "shield your teachers," which is still useful but much smaller. If the first fails, the idea is wrong.
References
- SAFE-GIL: SAFEty Guided Imitation Learning
- V-OCBF: Learning Safe Filters from Offline Data via Value-Guided Offline Control Barrier Functions
- GLAS: Global-to-Local Safe Autonomy Synthesis for Multi-Robot Motion Planning With End-to-End Learning
- Safe Policy Learning for Continuous Control
- Datasets and Benchmarks for Offline Safe Reinforcement Learning
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Policy Optimization under Imperfect Human Interactions with Agent-Gated Shared Autonomy
- Exponentially Weighted Imitation Learning for Batched Historical Data
- Boundary-Seeking Policy Gradient for Safe Reinforcement Learning
- Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model
- ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning
- Breaking Safety Paradox with Feasible Dual Policy Iteration