Borrowed Failures: Keeping Safe Agents Informed Using Demonstrators' Mistakes

A reinforcement learning agent that must stay safe has a quiet problem. It learns where danger lies from its own mistakes, and the better it gets, the fewer mistakes it makes. Its estimate of risk near the edge of safe behavior therefore goes stale. Recent work proved this "safety paradox": the model that predicts future constraint violations (a cost critic) is starved of violation data late in training, and then abruptly lets the agent into catastrophic states (Breaking Safety Paradox with Feasible Dual Policy Iteration). That paper's remedy is to run a second, deliberately reckless policy that goes looking for violations. This works in simulation. It cannot be allowed on a physical robot or a car.

We propose getting the needed failures from somewhere else. Logged demonstrations from human operators, older controllers, and crowdsourced players are rarely the perfect expert that imitation learning assumes. They contain crashes, aborted runs, and near misses. Rather than imitating those demonstrators or discarding their mistakes, we keep the mistakes in a separate store. We feed them to the cost critic throughout online training, weighted by how relevant each one is to the boundary the current agent is actually approaching. Because only a handful of failure trajectories may exist, those relevance weights are heavy-tailed and estimated from small batches. The naive way of normalizing them gives a biased critic update, so the estimator itself is part of the contribution. The claim is that relevance-weighted, bias-corrected borrowed failures keep the cost critic calibrated to the end of training with almost no online violations. That would remove the safety paradox without deliberate crashes, and it would reduce online sample needs because the critic does not have to rediscover the boundary.

Core Claim

The claim has three parts, and each can be tested. First, a cost critic trained on online data plus a small pool of logged demonstrator failures avoids the late-training cost spikes that the safety paradox predicts, and it does so with far fewer online violations than a policy that hunts for violations. Second, which failures the critic sees matters. Uniformly replaying all logged failures helps less than weighting them toward states the current policy is likely to reach near the boundary, because the demonstrators' mistakes often happen in places the learner never goes. Third, those weights have to be normalized with a running auxiliary estimate rather than a per-batch average. The per-batch version is a ratio of expectations and is biased when failure batches are small. This part is the most at risk, and we treat it as a hypothesis to test, not an assumption.

Research Program

The setting is offline-to-online safe reinforcement learning under a constrained Markov decision process, meaning a task where the agent maximizes reward while keeping expected cumulative cost below a budget. The agent starts from a logged dataset with reward and cost labels. It warm-starts its policy from that dataset and then fine-tunes online.

For the policy warm start we reuse, and do not claim, feasibility-weighted advantage regression. This is FAWAC's method of cloning logged actions in proportion to reward advantage minus a penalty times cost advantage (FAWAC), an extension of exponential advantage weighting (MARWIL). This step handles the imperfect-expert problem on the actor side. The new work is on the critic side and during the online phase.

We split the logged data into a failure pool $\mathcal{F}$, holding transitions within $k$ steps of a recorded violation, and everything else. Online, the cost critic $Q_c$ is trained on a mixture: the agent's own replay data plus failure transitions reweighted by

$$w(s,a) = \min\!\left(\frac{d_{\pi}^{\partial}(s,a)}{d_{\mathcal{F}}(s,a)},\ \frac{1}{\rho}\right).$$

Here $d_{\pi}^{\partial}$ is the distribution of state-action pairs the current policy $\pi$ visits whose predicted cost is within a margin of the budget (its "near-boundary" visits). $d_{\mathcal{F}}$ is the distribution of the failure pool. $\rho$ caps the ratio. The ratio is estimated by a classifier that separates recent near-boundary online states from failure-pool states. Its targets use the current policy's actions in the Bellman backup, so the critic stays an estimate for $\pi$ and not for the demonstrator.

The bias question enters at normalization. The weighted critic loss is $\mathbb{E}_{\mathcal{F}}[w\,\ell]/\mathbb{E}_{\mathcal{F}}[w]$, where $\ell$ is the per-sample critic loss. Plugging in per-batch means for both numerator and denominator gives the biased plug-in ratio estimator analyzed by Optimization of utility-based shortfall risk. The weights are also computed from the same batch as the loss, which is the coupling between weights and gradient noise that On the Convergence of Stochastic Multi-Objective Gradient Manipulation and Beyond showed can push stochastic updates in the wrong direction.

We borrow two method ingredients to address this. From Improved Stochastic Optimization of LogSumExp we take its device of a capped density ratio together with an auxiliary scalar that moves the expectation out of the nonlinearity; this gives the $1/\rho$ cap above. Our scalar $\alpha$ tracks $\mathbb{E}_{\mathcal{F}}[w]$ by its own stochastic updates and replaces the per-batch denominator. From CR-MOGM we take its correlation reduction: the classifier producing $w$ is updated on batches disjoint from those used for the critic step, and its outputs are smoothed with an exponential moving average.

A second adaptive quantity is the mixing fraction $\eta$, the share of each critic batch drawn from $\mathcal{F}$. It is set to hold the fraction of boundary-relevant samples roughly constant as the agent becomes safer. It is updated from a slow average rather than from the current batch, for the same decoupling reason.

On the analysis side, we will bound the bias of the per-batch estimator as a function of failure-batch size and weight variance. The standard delta-method expansion of a ratio estimator gives a bias of order $1/B$ in batch size $B$ with a constant that grows with $\mathrm{Var}(w)$. From this we will state when correction should matter: few failures, strongly mismatched demonstrators, and small $\rho$. The empirical program is organized to check that prediction.

This differs from the closest prior work in several ways. FDPI produces boundary data by violating constraints online; we never do. Marvel (Towards Fast Safe Online Reinforcement Learning via Policy Finetuning) re-evaluates its critics on the offline data once before fine-tuning, treats that data uniformly, and then handles cost spikes with a feedback controller on the Lagrange multiplier; we keep failures in the loop for the whole run and target them. SafeDICE (SafeDICE) and DualCOIL (DualCOIL) show that non-preferred demonstrations carry a useful contrast signal, but both use it only offline for imitation. We use it online, for the critic. COX-Q (Off-Policy Safe Reinforcement Learning with Cost-Constrained Optimistic Exploration) addresses cost underestimation with truncated distributional critics, but it learns from scratch and still depends on the agent's own failures. We will measure whether that dependence leaves its critic starved late in training.

Baselines

The naive reference point is SAC-Lagrangian trained online from scratch: a soft actor-critic agent with a learned cost penalty weight. It shows the full sample cost and in-training violations that a prior-free agent pays. A second reference is offline-only: FAWAC-P on the DSRL Bullet tasks, deployed without fine-tuning. That paper reports a normalized reward of 0.59 at a normalized cost of 0.17 averaged on standard datasets, against 0.41 for FISOR. Any online method must beat 0.59 reward while keeping cost below 1.0, or the online phase is not earning its keep.

The strongest offline-to-online competitor is Marvel. It is the most direct existing answer to "use suboptimal logs, then fine-tune safely," and it reports near-optimal policies on BallCircle and CarCircle with as few as 20 gradient updates. It also reports weak gains on AntRun, which is where we expect targeted failure replay to matter most. The strongest competitor on the safety-paradox question specifically is FDPI with SAC. It reports near-zero final violations across 14 Safety-Gymnasium environments and a violation share in the replay buffer roughly ten times that of SAC-FPI in the last 10% of training. It gets that share by violating on purpose, so we treat it as a simulation-only upper reference for critic data coverage and compare against it on cumulative in-training violations. COX-Q is the strongest from-scratch off-policy method and is included for sample-efficiency comparisons. No reported per-environment numbers were found on Lacuna for FDPI, COX-Q, or Marvel's cumulative online cost, so we will rerun all three from released or reimplemented code rather than quote them.

Fairness controls apply to every method. All methods get the same DSRL dataset per task, the same online step budget, the same network sizes, and the same hyperparameter search budget, with at least 20 configurations each, tuned on a held-out seed set. Online methods share the FAWAC-P warm start where applicable, so any differences come from the critic's data.

Three ablations isolate the key ingredient. Uniform failure replay keeps $\mathcal{F}$ in the buffer with no relevance weights. Weighting without debiasing uses the plug-in per-batch normalization and same-batch weights. Online-only relevance weighting removes $\mathcal{F}$ and applies the weights to the agent's own rare failures.

Beating the field means the following. Relative to Marvel, final reward and cost should match or improve, and the maximum cost over the final 20% of training should be lower by a margin outside seed variance. Relative to FDPI, cumulative online violations should be at least an order of magnitude lower at comparable final cost.

Why not just…?

Why not just keep the logged failures in the replay buffer and oversample anything with nonzero cost? This is the real threat, and it is a baseline in its own right. Oversampling by cost does not ask whether a failure happened anywhere near where the current policy goes. Demonstrators' mistakes follow their own habits, and Beliaev et al. argue that human errors are systematic, not random noise (Imitation Learning by Estimating Expertise of Demonstrators). Uniform replay can therefore spend critic capacity on irrelevant regions. We will settle this with a "mismatched failures" condition, in which failures are logged from a behavior policy visiting different regions. If uniform oversampling matches our method even there, relevance weighting is unnecessary, and we will report the simpler method as the finding.

Why not just run FDPI in a simulator and transfer? Where an accurate simulator exists, that is reasonable. The regime we target is one where the only boundary data is logged history: driving logs, operator records. FDPI keeps its violation share high by violating. Our cumulative-violation comparison measures exactly what that costs.

Why not just use larger batches to make the normalization bias disappear? Enlarging the failure batch does not create new failures. With 5 to 10 failure trajectories, as in SafeDICE's low-label regime, large batches just resample the same few points, and weight variance stays high. We will sweep failure-batch size and pool size together. If the debiased and plug-in estimators agree across the sweep, we drop the correction and say so. The bias analysis predicts where they should diverge, so this is a falsifiable check, not a hedge.

Why not just a reward penalty? Fixed penalties trade catastrophic violations against the "safe but useless" freeze, as shown by Quanyi Li et al.. DSRL also reports that cloning only safe data yields over-conservative agents (Datasets and Benchmarks for Offline Safe Reinforcement Learning). A tuned-penalty SAC is included at the same tuning budget.

Proposed Experiments

We use the DSRL datasets for the Bullet tasks BallCircle, CarCircle, and AntRun, the Safety-Gymnasium Point and Car navigation tasks, and MetaDrive, where DSRL reports that most methods struggle to be both safe and rewarding (Safety Gymnasium). For each task we build three data conditions: the full DSRL dataset; a reduced pool of only 5, 10, or 50 failure trajectories; and the mismatched-failure pool.

Metrics are DSRL-normalized final reward and cost, with cost below 1 counted as safe. We also report cumulative online cost during fine-tuning, maximum cost over the final 20% of training (the paradox signature), and online steps to reach 90% of the best final reward. Finally, we measure cost-critic calibration on near-boundary states, computed against Monte Carlo cost returns from simulator rollouts that are used only for evaluation. All results use 10 seeds with confidence intervals.

Two checks decide the outcome. The first tracks cost-critic calibration error over training. The safety paradox predicts that this error rises for SAC-Lagrangian and Marvel as their own violations dry up. Our claim predicts it stays flat with borrowed failures and flat for FDPI only at the price of violations. If our curve rises like the baselines', the core idea fails. The second is the three-way comparison of uniform replay, weighting without debiasing, and the full method across failure-pool size and mismatch. It shows which ingredient carries the effect. We expect relevance weighting to matter under mismatch and debiasing to matter only with small pools. Either result being null is reported and prunes the method.