Choosing How Hard to Return: Preference-Conditioned Recovery for Safe Imitation Without an Expert

A robot trained by imitation (copying recorded expert actions) drifts. Small mistakes carry it into situations the expert never showed, where its mistakes grow larger. Existing fixes either keep a human on call to take over, or build policies that are pulled back toward the demonstrated path no matter what lies in between. Neither works when the expert is gone and the shortest way back to familiar territory passes through a hazard.

We propose to treat recovery as a problem with three separate scores that should not be merged in advance: progress on the task, closeness to the demonstrations, and a safety cost. The agent learns one value function that predicts all three scores for every possible weighting of them. At each moment it then chooses the weighting from what the situation allows. Near a hazard it gives up task progress and even closeness to the demonstrations to stay within its safety budget. In open space it returns quickly to the demonstrated behavior. Where it is safe and well covered by data, it pushes on the task. Because one set of online experience trains every weighting at once, the approach should also need fewer environment interactions than tuning a fixed trade-off.

Core Claim

In offline-to-online safe imitation, the right balance between "return to the demonstrations" and "avoid cost" changes from state to state. A single global balance leaves the agent either too timid far from hazards or too reckless near them. It makes no difference whether that balance is a hand-set penalty weight, a Lagrange multiplier, or a fixed constraint budget. We claim that a preference-conditioned vector critic, combined with a per-state feasibility rule for choosing the preference, reduces constraint violations during online fine-tuning and raises final return compared with the best tuned fixed trade-off at equal sample budget. The benefit should be largest exactly where perturbations push the agent into states where hazards and demonstration support are close together.

The vector reward is $\mathbf{r}(s,a) = (r_{\text{task}}, r_{\text{sup}}, -c)$. Here $r_{\text{task}}$ is the environment reward, $c$ is the environment's safety cost, and $r_{\text{sup}}(s) = -\mathrm{Var}_k[\pi^{\text{BC}}_k(s)]$ is the negative disagreement of an ensemble of cloned policies $\pi^{\text{BC}}_k$. Disagreement is high far from the data and low near it. At execution, with $\mathbf{Q}(s,a,\mathbf{w}) \in \mathbb{R}^3$ the learned vector value for preference $\mathbf{w}$ on the simplex and $\pi(s,\mathbf{w})$ the preference-conditioned actor, the agent picks

$$ \mathbf{w}^*(s) = \arg\max_{\mathbf{w}} \; Q_{\text{task}}(s,\pi(s,\mathbf{w}),\mathbf{w}) \quad \text{s.t.} \quad B(s) - Q_{c}(s,\pi(s,\mathbf{w}),\mathbf{w}) \le \epsilon . $$

In this rule, $B(s)$ is the cost accumulated so far in the episode, $Q_c$ is the cost component (the negated third coordinate), and $\epsilon$ is the episode cost budget. If no preference is feasible, the agent falls back to the one that minimizes predicted cost. The choice is made over a small grid of candidate preferences, so it costs a few forward passes per step.

Relation to Prior Work

The mechanism combines pieces from three lines of work, each of which lacks one part of this setting. Envelope Q-learning trains a single network that returns vector Q-values for any linear preference and shares experience across preferences. We reuse its envelope backup as the learning rule. That work has no hard constraints and no demonstrations, and it assumes the preference is supplied from outside. In our method the agent chooses the preference itself, state by state, subject to a budget.

The budget test borrows the "cost so far plus cost to come" decomposition of Backward Value Functions. That paper turns a whole-episode budget into a per-state check, but it uses the check to project actions for a scalar-reward agent, and it is conservative. We use the check only to rule out preferences, so conservatism shows up as shifted weights rather than as a projected action.

The support term comes from the novelty signal in ThriftyDAgger. ThriftyDAgger uses ensemble disagreement to decide when to call a human. We turn the same signal into a reward to trade off against task and cost, with no human involved.

Two papers motivate the gap. SafeDAgger and ThriftyDAgger show that gating on novelty and risk controls compounding error, but only by handing control to an online expert. Contractive imitation policies (SCDS) show that a policy can be built to return to the demonstrations on its own. It reports up to 2.5× lower out-of-sample error on Robomimic, but it has no notion of cost, so its guaranteed return path can cross an obstacle.

The closest combined work constrains our design. CR-MOPO already handles several rewards plus several constraints online, but it learns from scratch and its authors report that policies sit on the edge of the constraint boundary. DRCORL anchors a constrained policy to a diffusion model of the data, but only offline and with one scalar reward. Our proposal differs from both in one respect: the weight on staying near the data is chosen per state online, not fixed as a regularizer strength or a global multiplier. Quanyi Li et al. document that fixed reward penalties produce either violations or a "safe but useless" frozen agent. That is precisely the failure we expect a single global trade-off to show when perturbation locations vary.

Research Program

We would build the method in three stages. In the first stage we pretrain on offline data: a five-member behavior-cloning ensemble, and a preference-conditioned actor and vector critic initialized with offline envelope backups on the logged transitions. In the second stage we fine-tune online under the episode cost budget. Each episode samples a preference for exploration, but actions are executed under the feasibility rule above, and every transition updates the critic for all preferences through the envelope backup. The third stage is analysis. We would log the chosen $\mathbf{w}^*(s)$ against distance to hazards and to the data, to check that the learned behavior matches the intended one: support weight high in open space, cost weight high near hazards.

A secondary question is whether exploration near the data but guided by the cost critic eases the safety paradox (Breaking Safety Paradox with Feasible Dual Policy Iteration). That paper finds that a safe agent stops producing the unsafe samples its cost critic needs. We would measure cost-critic calibration on held-out near-boundary states. We treat this as something to observe, not as a claim.

Baselines

The naive reference point is the behavior-cloning ensemble mean executed with no online learning. It shows how much drift costs and sets the floor for violations under perturbation.

The strongest existing method for this question is DRCORL pretraining followed by Lagrangian online fine-tuning. We would use the released DRCORL code for the offline stage and a PPO-Lagrangian update from the SafePO suite that ships with Safety Gymnasium for the online stage. It is the strongest competitor because it already combines a demonstration anchor with constraint handling. Offline it reports a normalized return of 0.91 against 0.38 for the next best safe baseline on CarGoal1 DRCORL. Its fixed regularizer strength and global multiplier are exactly what our method replaces.

The second strong baseline is CR-MOPO Safe and Balanced, initialized from the same behavior-cloning policy. It is given the support term as an extra reward objective and the cost as a constraint. It reports a Walker move reward above 800 against 250 for LP3, and it tests whether conflict-averse gradients alone suffice.

The key ablation is our own method with the preference frozen at the single best fixed $\mathbf{w}$. That $\mathbf{w}$ is chosen from a grid by the same validation protocol, so the only difference from the full method is per-state selection.

For fairness, all methods use the same offline data, the same online step budget (1M steps), the same network sizes, and five seeds. Each method gets the same tuning budget of twenty configurations, and we count the configurations the fixed-trade-off baselines need to find their best setting. Beating them means that, at matched final normalized cost at or below 1 in the DSRL convention, the method has fewer cumulative violations during training and higher final return, with confidence intervals that do not overlap on the perturbation suite.

Why not just…?

Why not just tune a penalty or Lagrange multiplier plus a behavior-cloning regularizer? This is the most serious threat. If hazards and data coverage are spread evenly across the state space, a well-tuned global trade-off should match per-state selection. Such a baseline can work well: PPO-Lag in Safety Gymnasium cut violations by 98% at the cost of a 45% drop in reward Safety Gymnasium. That reward drop is the kind of global conservatism we expect per-state selection to reduce. Our frozen-preference ablation is exactly this alternative given its best tuning. The perturbation suite varies whether hazards lie near or far from the demonstrations. If the ablation matches our method in both conditions, the core claim is false and we would say so.

Why not just use a contractive imitation policy with a safety shield? SCDS combined with a projection layer in the style of BVF would return to the data and cut unsafe actions. However, projection only removes the unsafe part of an action whose direction still points back to the data. When the path back to the data crosses a hazard, the agent can end up pinned against the constraint. We would run this combination on the same perturbation suite and expect stalls rather than violations. If it does not stall, it is a simpler answer and wins.

Why not just treat "stay near the data" as a second constraint? A two-constraint CMDP (constrained Markov decision process, a reinforcement learning problem with explicit cost budgets) handled by CR-MOPO is the CR-MOPO baseline above. Its support budget is one fixed number per episode, so it cannot spend closeness to the data freely in one region and save it in another. CR-MOPO's reported edge-of-boundary behavior also suggests it will leave no margin under perturbations.

Proposed Experiments

The main testbed uses DSRL offline datasets for PointGoal and CarGoal to pretrain all methods, followed by online fine-tuning in the matching Safety Gymnasium tasks. The perturbation protocol follows the out-of-sample recovery evaluation of SCDS: at random times the agent is pushed off-path, and pushes are split into two conditions depending on whether they land near hazards or away from them.

To test the setting with a true multi-objective reward, we would use the Safe Multi-Objective MuJoCo Walker task from CR-MOPO. Demonstrations would come from a policy trained at one preference, and evaluation would sweep test-time preferences. There the reward vector becomes velocity, energy, support, and cost.

Metrics are cumulative violations during training, final normalized return and cost, environment steps needed to reach the offline policy's return, and success after perturbation. We have no reported result on Lacuna for any method on these perturbation-recovery variants of Safety Gymnasium, so the baselines' numbers there will be established in this work.

The decisive check is the interaction between method and perturbation location. Per-state selection should beat the best frozen preference by a clear margin in the near-hazard condition and roughly tie in the far-from-hazard condition. The logged $\mathbf{w}^*(s)$ should shift toward cost near obstacles. A uniform gain across both conditions would suggest the benefit comes from something else, such as experience sharing across preferences. A second ablation, training separate fixed-preference critics without envelope sharing, would isolate that contribution.