Rehearsing Failure in Imagination: Fixing the Safety Paradox When Reward and Danger Share One Signal

A reinforcement learning agent that has learned to avoid danger stops seeing danger. Its memory of past experience fills with safe transitions, the part of the agent that estimates "how close am I to a failure" loses its examples, and late in training the agent blunders back into hazards. The fix in Breaking Safety Paradox with Feasible Dual Policy Iteration is a second "reckless" policy that deliberately commits violations near the safe policy's behavior to refresh those examples. That works in simulation and is unusable on real hardware. We propose to move the reckless policy into a learned world model, a network that predicts what happens next. Failures are then rehearsed in imagination rather than committed in reality, and each imagined probe is cut short as soon as the model's own uncertainty says the prediction can no longer be trusted.

The setting we target is harsher than the one FDPI studies, and harsher than most model-based safe RL. The agent gets one scalar signal per step, with no separate cost channel. Hitting a hazard ends the episode with a negative payoff, reaching the goal gives a positive payoff at the end, and nothing else is observed. Neither the payoff of actions not taken nor whether a nearby action would have been unsafe is ever revealed. This is bandit feedback in the single-agent sense: only the outcome of the chosen action is visible. Separate cost signals, dense rewards, and on-hardware violations are exactly the three things existing methods lean on. Our claim is that a world model can stand in for the missing counterfactuals, but only along short, uncertainty-gated branches. That gating is what keeps compounding model error from manufacturing fake failures or fake safe corridors.

Core Claim

Under single-channel, sparse, terminal-hazard feedback, the safety paradox can be broken without real violations. The method branches imagined "reckless" rollouts from real states the safe policy actually visits. It labels the imagined terminations as failure examples for the feasibility critic, the network that estimates whether a state can still be kept safe. It stops each branch at the first step where ensemble disagreement exceeds a rank-based threshold.

Short branches should be enough, and the reason comes from FDPI itself. FDPI's dual policy is held near the primal policy by a KL bound, so the boundary states it needs sit only a few steps from where the primal policy already goes. Rollouts therefore never need to be long. That is fortunate, because rollout length is where compounding error lives. The paper of Wang et al. shows that errors in long autoregressive rollouts drive sample complexity up exponentially with horizon, and our design avoids that regime by construction rather than fighting it.

What would be new is the combination. To our reading of the file, none of the prior methods gets failure data near the current policy's distribution, from one signal, without real violations. FDPI gets such data by violating for real. SafeDreamer and CERL plan safely inside world models but assume a separate, dense cost channel and do not address critic starvation. ROSARL shows that safety can come from the reward alone, through a minimum penalty that scales with how predictable the dynamics are and how far the goal is, but it is model-free and tends toward extreme caution, which is especially damaging when goal reward is sparse.

Research Program

The base agent is a DreamerV3-style latent world model, the same backbone SafeDreamer uses, so that comparisons isolate our changes. It has four parts: an ensemble of $K$ latent transition heads, a payoff head, a termination head, and a feasibility critic. Because hazard and goal share one channel, the termination head predicts terminal events and the sign of the payoff separates "failed" from "succeeded." The penalty magnitude is set with ROSARL's online estimate of the minimum penalty rather than tuned by hand. This makes ROSARL the reward-shaping component of the method, not a competitor to it.

Boundary rehearsal works as follows. At each update we sample real states from recent primal-policy data. From each state we roll out a dual policy inside the model; the dual is trained to reach predicted hazard terminations while staying within a KL ball of the primal policy, as in FDPI. Each branch is truncated with the rank-based masking of M2AC, which ranks imagined transitions in a batch by ensemble disagreement and keeps only the most confident fraction. Imagined terminations that survive the mask become negative examples for the feasibility critic, weighted by importance ratios between dual and primal policies as in FDPI. The primal policy is trained on ordinary imagined rollouts with a constraint from the feasibility critic.

For the sparse goal reward we deliberately add nothing new beyond the world model's own imagined returns. Adding shaping would confound the safety question; the open problems around shaped rewards and learned progress critics are documented in Shortest-Path Constrained RL.

The analytical component asks when imagined failures can be trusted. We measure the precision of imagined hazard terminations against simulator ground truth as a function of branch depth and of ensemble disagreement. We expect precision to hold for the first few steps and then drop, and we expect the masking threshold to track the drop. If precision falls immediately, the core premise fails, and that is itself a useful negative result.

Baselines

The naive reference point is a model-free agent such as PPO with ROSARL's self-tuned minimum penalty on the single channel. This is the strongest "do nothing structural" option because it is the only existing method in the file designed for reward-only safety. It is run on identical environments with the same number of real environment steps and an equal hyperparameter search budget. Beating it means fewer cumulative real violations at matched or higher goal success.

The strongest existing method is SafeDreamer (BSRP-Lag). We run it in two modes. The first is a privileged mode that receives the separate cost channel it was designed for. The second is a fair mode in which the cost is inferred from negative terminal payoffs. The file reports SafeDreamer reaching "a 100% arrival rate in autonomous driving tests with an average episode cost of just 2.2, compared to 14.1 for standard PPO-based reward shaping" on MetaDrive, with dense signals. On our sparse single-channel variants we found no reported result on Lacuna for SafeDreamer, so its numbers there will be produced under the same world-model size, step budget, and tuning budget as ours. Beating the fair mode on late-training cost spikes and total real violations, at equal goal success, is the main target. Matching the privileged mode would be a strong result.

FDPI with SAC is included as a violation-tolerant reference. It is allowed real reckless probing, so it shows what failure-data refresh is worth when violations are free. The file reports roughly 10x more violation data in the replay buffer in the last 10% of training compared with SAC-FPI. We expect to approach its late-training safety while committing far fewer real violations.

Three ablations isolate the key ingredient. The first removes rehearsal entirely, which is our agent with an ordinary feasibility critic. The second keeps rehearsal but removes uncertainty gating, using fixed long branches; this tests whether compounding error contaminates the critic. The third replaces the KL-bounded dual with random imagined actions; this tests whether "near the current policy" matters.

Why not just…?

Why not just keep a reservoir of early-training failures and oversample it? This is the most serious threat. Early training produces real violations anyway, and replaying them is nearly free. The argument from FDPI is that the critic degrades because those failures come from states the improved policy no longer visits, so they say little about the boundary near current behavior. That is an argument, not proof, so a failure-reservoir variant with tuned oversampling is a required baseline. If it matches boundary rehearsal on critic accuracy near current states and on late-training spikes, we will report that the simpler method suffices and that the world model adds nothing.

Why not just use a Lagrangian method with a separate inferred cost? Lagrangian methods enforce constraints only at convergence and oscillate, as documented in Safety Gymnasium. The file reports that PPO-Lag in velocity tasks achieved "a 98% reduction in safety violations compared to standard PPO, with only a 45% drop in total reward." Lagrangian methods also suffer the same buffer starvation that FDPI documents. The fair-mode SafeDreamer and PPO-Lag runs test this directly on our tasks.

Why not just shorten all rollouts, MBPO-style, and skip gating? Fixed short rollouts would give little boundary coverage when the hazard is several steps away. M2AC reports that MBPO collapses on noisy dynamics where masked 10 to 20 step rollouts stay stable. The ungated-depth ablation, run at both short and long fixed depths, shows whether gating matters beyond simply picking a good constant.

Why not just raise the single-channel penalty? ROSARL's minimum penalty is the principled version of this idea. It is a component of our method, and on its own it is the naive baseline. With sparse goal reward, larger penalties are expected to produce the "safe but useless" inaction described in Safety Gymnasium. Goal success rate in the naive baseline will show whether that happens.

Proposed Experiments

We build single-channel, sparse-reward, terminal-hazard variants of Safety-Gymnasium PointGoal, CarGoal, and PointButton, with state and with vision observations. In these variants the goal pays only at completion, a hazard terminates the episode with a negative payoff, and no cost signal is exposed. We also use ROSARL's Lava Gridworld and PointGoal1-Hard, where the minimum-penalty behavior is already characterized qualitatively. The noisy-action variants used by M2AC stress model error.

The metrics are four. Cumulative real violations over training is the primary metric. Goal success rate at the end of training measures task performance. Real violation rate in the final 10% of training measures the safety paradox. Feasibility-critic AUROC on simulator-labeled states near the current policy's visitation distribution measures whether the critic has kept its knowledge of the boundary.

Three checks decide the outcome. Rehearsal must remove late-training violation spikes relative to the no-rehearsal ablation while adding zero real violations. It must outperform the failure-reservoir baseline on near-policy critic AUROC; if it does not, the idea collapses to replay. The precision-versus-depth curve must show that gated branches keep imagined-failure precision high while ungated long branches do not. That is the direct test of whether controlling compounding error is the reason the method works.