Recovery Desirability: Steering Imitation Policies Back to the Demonstrations Without Leaving the Safe Set

A policy trained by imitation drifts. Small mistakes carry it into states the expert never visited, and there it has no idea what to do. The standard fix, DAgger, lets the learner make those mistakes and asks an expert to label the states it reaches. That is exactly what you cannot afford on a robot near a cliff edge. This note proposes giving the learner, before deployment, a cheap map of how recoverable every state is. The map is a single number per state: the probability that a slightly noisy copy of the imitation policy, run forward, reaches the region the demonstrations cover before it hits a failure state. That number is the solution of a linear equation. Reweighting the policy's next-state distribution by it gives a recovery controller that provably never steps into a known failure state, that heads back toward the expert's data, and that needs no new reward function.

The mathematical tool comes from linearly solvable control. In these problems the agent pays a penalty for deviating from a default "passive" behavior, measured by the KL divergence (a measure of how different two probability distributions are). An exponential change of variables then turns the usual nonlinear Bellman equation into a linear one (Linearly-solvable Markov decision problems). Our observation is that if we choose the passive behavior to be "behavior cloning plus noise", set the target region to the demonstrated states, and set the forbidden region to the unsafe states, the transformed value has a plain probabilistic meaning. The optimal controller is then the noisy cloned policy conditioned on succeeding. That one object does three jobs at once: it corrects distribution shift, it keeps exploration safe during data collection, and it decides when to call the expert.

Core Claim

Let $P_0(s' \mid s)$ be the transition distribution of the environment under the cloned policy with added noise. Let $D$ be the set of states the demonstrations cover, defined in practice by low disagreement of a policy ensemble, as in ThriftyDAgger's novelty test. Let $U$ be a labeled set of unsafe states. With zero running state cost, the desirability $z(s) = \exp(-V(s))$, where $V$ is the optimal cost-to-go, solves a linear first-exit problem. Its value is fixed at 1 on $D$ and 0 on $U$, and it equals the passive average of its successors everywhere else:

$$z(s) = \Pr_{P_0}\big(\text{reach } D \text{ before } U \mid s_0 = s\big), \qquad u^*(s' \mid s) = \frac{P_0(s' \mid s)\, z(s')}{\sum_{x} P_0(x \mid s)\, z(x)}.$$

Here $u^*$ is the optimal controlled transition distribution. In probability theory this construction is called a Doob h-transform: $u^*$ is exactly the passive chain conditioned on the event of returning to $D$ before failing. Three consequences follow, and together they are the claim of this proposal.

First, under $u^*$ the system reaches the demonstrated region with probability one and never enters $U$. The second half holds step by step, because $z = 0$ on $U$ removes all probability mass from unsafe successors. This part needs only the unsafe-state labeler and holds whatever the error in $z$ elsewhere.

Second, the KL cost of the correction from state $s$ is exactly $-\log z(s)$. So $z$ is a calibrated measure of how far the controller must depart from the cloned policy to recover. That makes it a principled trigger for expert queries, replacing a learned risk heuristic.

Third, $z$ is linear in its boundary values. The recovery field for a union of demonstration segments is a weighted sum of per-segment fields that share the same zero boundary on $U$. When new expert corrections enlarge $D$, the field can be updated by adding basis functions instead of re-solving from scratch, and the combined field still assigns zero to unsafe states. This follows the compositionality results of Compositionality of optimal control laws and Hierarchy Through Composition with Multitask LMDPs.

The honest boundary of the claim is this. One-step masking alone is just a safety shield. The real contribution of $z$ is foresight: it steers away from "dead-end" states from which every future eventually fails, long before the shield would be forced to act. The quality of that foresight depends on how well $z$ is estimated. Most of the empirical program tests exactly this.

Relation to Prior Work

Three papers motivate the gap. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning shows that offline cloning compounds errors quadratically in the horizon, and that letting the learner visit its own mistakes reduces this to linear. It is silent on what those visits cost. ThriftyDAgger gates human intervention with an ensemble novelty score and a learned Q-function that predicts reaching the goal. That Q-function is trained from the learner's own rollouts, so it learns about danger by approaching it. Our $z$ is computed from simulated or modeled passive rollouts and has a closed-form meaning. Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPs achieves zero training violations given a safe baseline policy. It does so by solving tabular linear programs over occupancy measures with $\Theta(|S||A|H)$ variables, and it uses no demonstrations. We borrow its idea of pessimism: unknown states default to low desirability.

Three papers supply or constrain the method. Inverse Optimal Control with Linearly-Solvable MDPs (OptV) fits a value function directly to demonstrations by convex likelihood. It treats every demonstration as optimal and says nothing about unsafe states or what happens off the data. We use the same value-space view, but the region off the data is the object of interest, not a nuisance. Learning Policy Improvements with Path Integrals (PI²) shows how to compute exponentiated-cost controls by Monte Carlo trajectory sampling in continuous spaces. That is how we estimate and use $z$ beyond grid worlds. Closing the Closed-Loop Distribution Shift in Safe Imitation Learning proves that if the expert system pulls nearby trajectories together, imitation sample complexity stops depending on the horizon. Enforcing that property requires known dynamics and nonconvex optimization. Our h-transformed controller is a constructive, linear-algebraic way to obtain a pull toward the demonstrated region, though we do not claim it satisfies their stability certificate.

Two methods are the closest competitors. SafeDICE learns safe behavior offline by subtracting non-preferred trajectories from the stationary distribution of the data. It cannot recover in states outside the data's support. End-to-End Imitation Learning with Safety Guarantees using Control Barrier Functions inherits set-invariance from a robust expert. It needs samples near the safe-set boundary and conservative Lipschitz constants. Neither attempts to return the learner to the expert's distribution.

Research Program

The work proceeds in three stages.

The first stage is exact. In tabular first-exit problems with known $P_0$, we solve the linear system for $z$ directly. We then verify four properties: zero violations under $u^*$, almost-sure return to $D$, the identity between the KL cost and $-\log z$, and incremental superposition as $D$ grows with expert labels.

The second stage is continuous control with a simulator or learned model. We estimate $z$ by Feynman–Kac Monte Carlo. Feynman–Kac is the identity that expresses the solution of the linear equation as an average over passive trajectories, so we roll the noisy cloned policy forward from sampled states and record whether each rollout hits $D$ before $U$. We then fit a network $\hat z(s) = m(s)\, g_\theta(s)$. Here $m$ is the known safety indicator, which forces exact zeros on $U$, and $g_\theta$ is a sigmoid-output regressor. Its value is shrunk toward zero away from sampled states; this is the pessimism borrowed from optimistic-pessimistic exploration. At execution time we act with a path-integral controller in the style of PI². It samples short perturbations of the cloned action, gives zero weight to sampled trajectories that touch $U$, and weights the rest by $\hat z$ at the horizon.

The third stage is the interactive loop, which we call Recovery-DAgger. Training rollouts are executed with the corrected controller rather than the raw student policy. The expert is queried only where $-\log \hat z(s)$ exceeds a budget threshold, meaning recovery would demand a large departure from the clone. Expert labels enlarge $D$, and the field is updated by superposition. The analysis we aim to prove bounds the per-episode probability of entering $U$ in terms of two quantities: the one-step model error inside the shield, and the error of $\hat z$ on states of low desirability. The bound should be independent of the student's imitation error, because the student is never executed unfiltered.

Baselines

The naive reference is plain behavior cloning on the same demonstrations, run with the same network and evaluated on the same perturbation protocol. Any method that does not clearly beat it on recovery rate and violations has no case. SafeDICE reports cloning at a 0.22 cost violation ratio on RWRL Walker (SafeDICE), which calibrates how unsafe this reference is.

The strongest interactive competitor is ThriftyDAgger. It is the most refined published answer to "when should the expert take over", and it reports a 73% autonomous success rate on simulated peg insertion, against 48% for LazyDAgger and 24% for SafeDAgger (ThriftyDAgger). We will reimplement it with a scripted expert and give both methods an identical budget of expert labels and environment steps. Its thresholds are tuned with the same grid size we use for our query threshold. Beating it means matching its autonomous success while incurring significantly fewer safety violations during training, or matching its violations with fewer expert queries.

The strongest offline competitor is SafeDICE, which reports a 0.00 cost violation ratio on RWRL Walker, against 0.17 for PPL. We give it the same mixed and non-preferred demonstration sets. Because its reported violation ratio already sits at the floor, beating it on the unperturbed task is impossible. The test is instead return and violation rate under injected perturbations that push the agent outside the data's support, where it should fail and our controller should not.

The constrained-RL competitor is a Lagrangian method initialized from cloning. We use CPPO-PID from the Safety Gymnasium suite, whose authors report that PID-controlled multipliers reduce oscillation, and whose PPO-Lagrangian variant cut violations by 98% at a 45% reward cost on velocity tasks (Safety Gymnasium). It receives the same environment-step budget. Beating it means far fewer violations accumulated during training at comparable final normalized reward.

Two ablations isolate the key ingredient. The "mask only" ablation keeps the one-step shield but sets $\hat z = 1$ off $U$, which turns the method into cloning plus a shield. The "no demonstration boundary" ablation removes $D$ from the boundary conditions, so $z$ measures only survival, not return to the expert's data. If the full method does not beat both ablations, the recovery-desirability idea adds nothing beyond a shield or a survival value.

Why not just…?

Why not just put a safety shield or control barrier filter on behavior cloning? This is the most serious alternative, and it is literally our "mask only" ablation. A shield blocks unsafe actions but gives no reason to move back toward demonstrated states. The learner can therefore sit indefinitely in an unfamiliar region, or slide into a dead end where every remaining action is unsafe and the shield has nothing left to allow. The barrier-function approach also requires boundary samples and Lipschitz margins that are hard to obtain safely (End-to-End Imitation Learning with Safety Guarantees using Control Barrier Functions). If, in our perturbation experiments, the shield matches the full method on both recovery rate and dead-end entries, we will report that the value of $z$ is confined to query gating.

Why not just use model-predictive control that tracks the nearest demonstration under hard constraints? In nearly deterministic, low-noise settings this may work as well, and we include it as a baseline with the same model and planning horizon. Our argument is that tracking a nearest neighbor is local and ignores the probability of failing on the way. In contrast, $z$ integrates over all paths back to $D$ under stochastic dynamics and supplies a calibrated query signal. The noise-scaling experiment below decides this. If tracking MPC matches us across noise levels, the exponential machinery is unnecessary.

Why not just run ThriftyDAgger with a better risk Q-function? ThriftyDAgger's risk estimate is learned from the learner's own rollouts, so it improves only after the learner has visited risky states. Our $z$ comes from passive rollouts computed in simulation before deployment, and its scale has a meaning (the log KL cost of recovery). The training-violation curves under matched label budgets settle this directly.

Why not just do constrained RL with a cloning term? Lagrangian methods are known to oscillate and to guarantee safety only at convergence (Safety Gymnasium). The cumulative violation count during training is the metric on which we expect the clearest separation.

Proposed Experiments

The first experiment is a sanity check in tabular grid worlds with cliffs and stochastic slip. We solve $z$ exactly and confirm three things: zero violations, almost-sure return to $D$, and close agreement between $-\log z$ and the empirical KL cost of the corrected policy. We also compare query efficiency against OptPess-LP given the same safe baseline policy.

The main experiment uses Safety Gymnasium PointGoal and CarGoal (Safety Gymnasium). Demonstrations come from a converged CPPO-PID expert. We inject action perturbations of increasing magnitude, and the simulator supplies the passive rollouts. Metrics are normalized reward, normalized cost, cumulative training violations, expert queries, and recovery rate within $k$ steps after a perturbation.

We then run on RWRL Walker against SafeDICE under SafeDICE's data protocol plus out-of-support perturbations, and on Robosuite peg insertion against ThriftyDAgger with matched label budgets.

The decisive checks are three. First, $\hat z$ must be calibrated: binned predictions should match the empirical success frequency of held-out passive rollouts. If it is not, the gate and the foresight claim both fail. Second, the full method must beat the "mask only" ablation on dead-end entries as the noise level grows. Third, cumulative training violations must stay flat while ThriftyDAgger's and CPPO-PID's grow. A secondary scaling check measures how the cost of the Monte Carlo estimate of $z$ grows with state dimension, from Point to Car to Walker. This tells us where the approach stops being practical.