Budget-Gated Guidance: Learning Past an Unsafe, Suboptimal Teacher

When rewards are sparse, a reinforcement learning agent (one that learns by trial and error from a reward signal) often never finds the goal unless something points the way. A common fix is to let a mediocre teacher, such as an old policy, a hand-written controller, or a set of human demonstrations, steer early exploration and then fade its influence so the agent can eventually do better than the teacher. This works when the teacher is merely slow. It breaks when the teacher is also careless. A teacher that reaches the goal by cutting through hazards will drag the learner through the same hazards for as long as its guidance is active. Safe-exploration methods avoid this only by assuming the teacher is perfectly safe.

We propose to turn guidance on and off state by state using a safety budget, instead of fading it on a clock. At each state, the learner is pulled toward the teacher's action only when two estimates agree. The first is that following the teacher from here keeps the episode's total cost within budget, counting the cost already spent. The second is that the teacher still beats the learner's own reward estimate at that state. Where the teacher is unsafe, guidance shuts off immediately. Where the learner has already overtaken the teacher, guidance shuts off by itself. Weaning therefore falls out of these two estimates and needs no hand-tuned schedule. The claim we test is that this per-state gate lets an agent exceed a teacher that is suboptimal in reward and intermittently unsafe, with far fewer constraint violations during training than time-decayed guidance under a standard constrained solver.

Where this sits

The closest work splits along a clean line. Some methods wean from imperfect teachers but ignore safety. Learning Online with Guidance Offline (LOGO) alternates trust-region improvement with a step that keeps the policy near a suboptimal behavior policy, under a constraint that loosens over time. APC-RL routes control between flow-based prior actors and a free actor using task Q-values. Demonstration-Regularized RL adds a fixed KL penalty toward a behavior-cloned prior, and its sample-complexity theory assumes near-optimal demonstrations. None of these three has a cost signal, so each will faithfully copy a teacher's dangerous shortcuts. They supply our guidance mechanism and our weaning target.

Other methods enforce safety but trust the teacher completely or need it pre-sorted. SOOPER falls back on a policy prior to get zero training violations, which only works if that prior is safe everywhere. Safe CoR needs two cleanly separated demonstration sets, one aggressive and one cautious. SafeDICE works only offline and cannot leave the support of its dataset. Learning Safe Policies with Expert Guidance uses worst-case reasoning that locks the agent inside the demonstrators' support. These papers motivate the gap. They also constrain the design: the gate must be pessimistic about danger without being pessimistic about everything unvisited, or it inherits that last paper's ceiling.

From Backward Value Functions we reuse the idea of tracking cost-so-far. Adding it to expected future cost turns a constraint on the whole episode into a check at each state, which is exactly what a state-by-state gate needs. Quanyi Li et al. show that folding cost into reward as a fixed penalty either causes crashes or freezes the agent. That result rules out the cheapest design.

Core Claim

We work in a constrained Markov decision process: maximize expected return subject to expected cumulative cost staying at or below a budget $\epsilon$. The teacher $\pi_b$ is available as logged demonstrations with per-step cost labels, which Safety-Gymnasium computes from states.

From these logs and from online data we fit three quantities. The first is the teacher's cost value $Q_c^{b}(s,a)$, its expected future cost. The second is the teacher's reward value $Q_r^{b}$. The third is a backward cost estimate $B(s)$ of the cost already spent when reaching $s$. The learner keeps its own reward critic $Q_r^{\pi}$.

The gate at state $s$ is

$$g(s) = \mathbb{1}\big[\, B(s) + \overline{Q}_c^{\,b}(s,\pi_b(s)) \le \epsilon \,\big]\cdot \mathbb{1}\big[\, Q_r^{b}(s,\pi_b(s)) > \max_a Q_r^{\pi}(s,a) \,\big],$$

where $\overline{Q}_c^{\,b}$ is an upper confidence estimate taken from an ensemble. The gate also closes automatically at states far from the demonstration support, where the ensemble disagrees.

The policy update is a standard constrained update, using a Lagrangian or FOCOPS-style projection, plus the guidance term $\lambda_g\,\mathbb{E}_s[g(s)\,\mathrm{KL}(\pi_b(\cdot|s)\,\|\,\pi(\cdot|s))]$.

The gate has three properties. Guidance is never applied toward an action predicted to exhaust the budget. Guidance disappears wherever the learner has overtaken the teacher. Outside the teacher's support the agent explores under the ordinary constraint rather than under a blanket prohibition, which is how it can surpass the teacher at all.

Research Program

We will build the method on top of SafePO from Safety Gymnasium, so every baseline shares one constrained solver.

To make the teacher's flaws a controlled variable, we will construct teachers with a dial on their hazard rate. Each teacher is an unconstrained PPO checkpoint trained only to the point of partial task success, mixed with a safe but slow scripted controller. The mixing level sets the fraction of teacher transitions that pass through hazard regions to 0%, 10%, 30%, or 60%. Teachers are deliberately suboptimal in return, so success requires beating them.

The analysis has three parts. First, measure how often the gate opens over training and where it opens, to confirm that it closes near hazards and closes where the learner has overtaken the teacher. Second, measure how well the backward cost estimate and the pessimistic cost estimate predict actual violations. Third, characterize the failure mode we expect: an overconfident cost critic near hazard boundaries the teacher rarely visited.

Baselines

The naive reference is a constrained solver with no guidance: PPO-Lagrangian and FOCOPS learning from scratch. On sparse-reward tasks we expect them to stall. Safety Gymnasium reports that PPO-Lag gives a 98% reduction in safety violations relative to PPO, with a 45% drop in total reward, on velocity tasks. That figure is our reference for what pure constraint enforcement costs in return.

The most direct competitor is LOGO-Lagrangian. It runs LOGO's decaying guidance step unchanged, with the same teacher data and its loosening schedule, on top of the identical Lagrangian solver. This is exactly our method with the gate replaced by a clock, so it isolates the key ingredient. No reported result found on Lacuna for LOGO combined with a constrained solver. The LOGO paper reports only that it beat PofD and GAIL on sparse MuJoCo, without quoted numbers.

The strongest existing safe-guidance method is Safe CoR. Because it needs two experts, we will give it the teacher logs split by trajectory cost into an aggressive set and a safe set, which is the fairest construction available from mixed data. Safe CoR reports a 29.6% increase in driving score and a 61.9% reduction in violations on MetaDrive, and up to an 88% reduction in violations on the Jackal robot. It does not report absolute Safety Gym returns in the file.

SOOPER will be run with the imperfect teacher as its prior, to test what happens when its safety assumption is broken. On simulated tasks with a truly safe prior it reports zero violations during training.

Fairness rules apply to all methods: identical teacher data, environment steps, network sizes, and a hyperparameter search of equal size over each method's guidance or penalty weights, with seeds and budgets fixed in advance.

Ablations remove one part of the gate at a time. We will drop the cost condition, which leaves reward-only gating close to APC-RL's routing logic. We will drop the reward condition, which leaves guidance that never weans. We will drop cost-so-far $B(s)$, which tests whether the Markov cost value alone suffices. We will replace the per-state gate with simple filtering that removes demonstration transitions where cost is greater than zero.

Beating a baseline means four things together: equal or higher final return, a statistically lower cumulative count of training violations, a teacher's return exceeded at matched step counts, and all of this holding at hazard rates of 30% and above.

Why not just…?

Why not just penalize cost in the reward and use LOGO as-is? Quanyi Li et al. show the fixed penalty either permits crashes or freezes the agent, and Safety Gymnasium documents the violation gap between penalized PPO and Lagrangian methods. We include the penalized variant anyway, swept over its penalty weight, to show that no single weight matches the gated method across hazard rates.

Why not just add a Lagrangian to LOGO's decaying guidance? This is the real threat. Our argument is that early in training the Lagrange multiplier is small and lags behind actual violations, while the guidance pull is strongest, so the learner copies the teacher's shortcuts exactly when it is least protected. If LOGO-Lagrangian matches our training-violation curve at the 30% and 60% hazard rates, the gate adds nothing and the idea is refuted. The hazard-rate sweep is designed to settle this.

Why not just discard the teacher's unsafe transitions and imitate the rest? Cost is usually charged at the step that enters a hazard, while the decision that made the hazard unavoidable came earlier. Dropping transitions with positive cost keeps those earlier approach actions. The gate instead uses expected future cost plus cost-so-far, which scores the approach itself. The ablation that filters out positive-cost transitions tests this directly.

Why not just split the data and run Safe CoR, or run SafeDICE offline? Splitting at the trajectory level throws away safe segments inside trajectories that were unsafe overall. SafeDICE cannot exceed the support of its dataset, and our teachers are built to be suboptimal, so the dataset contains no path that is both safe and high-return. Both are run as baselines. If cost-split Safe CoR closes the gap, that would show trajectory-level labels are enough, and we would report it.

Proposed Experiments

The main suite uses the Safety-Gymnasium Goal and Button navigation tasks with rewards made sparse, and the MuJoCo Circle and Speed-Limit tasks with the sparse rewards used by LOGO. FOCOPS's Humanoid-Circle reward of 1106.1 at a cost limit of 50 serves as a dense-reward ceiling for context. MetaDrive is a secondary domain for comparison with Safe CoR's reported numbers.

The metrics are final return normalized by the teacher's return, cumulative violations over the whole of training, environment steps until the teacher's return is exceeded, and fraction of seeds that end feasible.

The decisive check is the hazard-rate sweep. When the teacher is safe (0%), all guided methods should look alike. As the hazard rate rises, LOGO-Lagrangian's training violations should grow with it, SOOPER's should appear once its prior is unsafe, and the gated method's should stay roughly flat while it still exceeds the teacher's return.

A second check is the gate trace over training. The fraction of states where guidance is active should fall as the learner's reward critic overtakes the teacher's. That decline should happen faster on easy tasks than on hard ones, which would show that weaning adapts to task difficulty without a schedule.